Configuring Uppy to Use Multipart Uploads with Laravel/Vue - laravel

Question

I figured it out
This was the missing piece. Once I clean up my code, I'll post an answer so that hopefully the next poor soul that has to deal with this will not have to go through the same hell I went through ;)
$command = $client->getCommand('UploadPart', array(
'Bucket' => 'the-bucket-name',
'Key' => $key,
'PartNumber' => $partNumber,
'UploadId' => $uploadId,
'Body' => '',
));
$signedUrl = $client->createPresignedRequest($command, '+20 minutes');
$presignedUrl = (string)$signedUrl->getUri();
return response()->json(['url' => $presignedUrl]);
I'm trying to figure out how to configure my server to work with Uppy for uploading multipart uploads to AWS S3 by using the CompanionUrl option. https://uppy.io/docs/aws-s3-multipart/#createMultipartUpload-file.
This is where I got the idea to go this route https://github.com/transloadit/uppy/issues/1189#issuecomment-445521442.
I can't figure this out and I feel like others have been stuck as well with no answer, so I'm posting what I've come up with so far in trying to get Uppy to work with multipart uploads using Laravel/Vue.
For the Vue component I have this:
<template>
<div>
<a id="uppy-trigger" #click="isUppyOpen = !isUppyOpen">Open Uppy</a>
<dashboard-modal
:uppy="uppy"
:open="isUppyOpen"
:props="{trigger: '#uppy-trigger'}"
/>
</div>
</template>
<script>
import Uppy from '#uppy/core'
import AwsS3Multipart from '#uppy/aws-s3-multipart';
import '#uppy/core/dist/style.css';
import '#uppy/dashboard/dist/style.css';
export default {
components: {
'dashboard-modal': DashboardModal,
},
data() {
return {
isUppyOpen: false,
}
},
computed: {
// Uppy Instance
uppy: () => new Uppy({
logger: Uppy.debugLogger
}).use(AwsS3Multipart, {
limit: 4,
companionUrl: 'https://mysite.local/',
}),
},
beforeDestroy () {
this.uppy.close();
},
}
</script>
Then for the routing I've added this to my web.php file.
// AWS S3 Multipart Upload Routes
Route::name('s3.multipart.')->prefix('s3/multipart')
->group(function () {
Route::post('/', ['as' => 'createMultipartUpload', 'uses' => 'AwsS3MultipartController#createMultipartUpload']);
Route::get('{uploadId}', ['as' => 'getUploadedParts', 'uses' => 'AwsS3MultipartController#getUploadedParts']);
Route::get('{uploadId}/{partNumber}', ['as' => 'signPartUpload', 'uses' => 'AwsS3MultipartController#signPartUpload']);
Route::post('{uploadId}/complete', ['as' => 'completeMultipartUpload', 'uses' => 'AwsS3MultipartController#completeMultipartUpload']);
Route::delete('{uploadId}', ['as' => 'abortMultipartUpload', 'uses' => 'AwsS3MultipartController#abortMultipartUpload']);
});
Basically what is happening is that I've set the "companionUrl" to "https://mysite.local/", then Uppy will send five requests when uploading a multipart upload file to these routes, ie "https://mysite.local/s3/multipart/createMultipartUpload".
I then created a controller to handle the requests:
<?php
namespace App\Http\Controllers;
use Aws\S3\S3Client;
use Illuminate\Http\Request;
class AwsS3MultipartController extends Controller
{
public function createMultipartUpload(Request $request)
{
$client = new S3Client([
'version' => 'latest',
'region' => 'us-east-1',
]);
$key = $request->has('filename') ? $request->get('filename') : null;
$type = $request->has('type') ? $request->get('type') : null;
if (!is_string($key)) {
return response()->json(['error' => 's3: filename returned from "getKey" must be a string'], 500);
}
if (!is_string($type)) {
return response()->json(['error' => 's3: content type must be a string'], 400);
}
$response = $client->createMultipartUpload([
'Bucket' => 'the-bucket-name',
'Key' => $key,
'ContentType' => $type,
'Expires' => 60
]);
$mpuKey = !empty($response['Key']) ? $response['Key'] : null;
$mpuUploadId = !empty($response['UploadId']) ? $response['UploadId'] : null;
if (!$mpuKey || !$mpuUploadId) {
return response()->json(['error' => 'Unable to process upload request.'], 400);
}
return response()->json([
'key' => $mpuKey,
'uploadId' => $mpuUploadId
]);
}
public function getUploadedParts($uploadId)
{
// Haven't configured this route yet as I haven't made it this far.
return $uploadId;
}
public function signPartUpload(Request $request, $uploadId, $partNumber)
{
$client = new S3Client([
'version' => 'latest',
'region' => 'us-east-1',
]);
$key = $request->has('key') ? $request->get('key') : null;
if (!is_string($key)) {
return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
}
if (!intval($partNumber)) {
return response()->json(['error' => 's3: the part number must be a number between 1 and 10000.'], 400);
}
// Creating a presigned URL. I don't think this is correct.
$cmd = $client->getCommand('PutObject', [
'Bucket' => 'the-bucket-name',
'Key' => $key,
'UploadId' => $uploadId,
'PartNumber' => $partNumber,
]);
$response = $client->createPresignedRequest($cmd, '+20 minutes');
$presignedUrl = (string)$response->getUri();
return response()->json(['url' => $presignedUrl]);
}
public function completeMultipartUpload(Request $request, $uploadId)
{
$client = new S3Client([
'version' => 'latest',
'region' => 'us-east-1',
]);
$key = $request->has('key') ? $request->get('key') : null;
$parts = json_decode($request->getContent(), true)['parts'];
if (!is_string($key)) {
return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
}
if (!is_array($parts) || !$this->arePartsValid($parts)) {
return response()->json(['error' => 's3: "parts" must be an array of {ETag, PartNumber} objects.'], 400);
}
// The completeMultipartUpload method fails with the following error.
// "Error executing "CompleteMultipartUpload" on "https://the-bucket-name.s3.amazonaws.com/NewProject.png?uploadId=nlWLdbNgB9zgarpLBXnj17eOIGAmQM_xyBArymtwdM71fhbFvveggDmL6fz4blz.B95TLhMatDvodbMb5p2ZMKqdlLeLFoSW1qcu33aRQTlt6NbiP_dkDO90DFO.pWGH"; AWS HTTP error: Client error: `POST https://the-bucket-name.s3.amazonaws.com/NewProject.png?uploadId=nlWLdbNgB9zgarpLBXnj17eOIGAmQM_xyBArymtwdM71fhbFvveggDmL6fz4blz.B95TLhMatDvodbMb5p2ZMKqdlLeLFoSW1qcu33aRQTlt6NbiP_dkDO90DFO.pWGH` resulted in a `400 Bad Request` response:
// <Error><Code>InvalidPart</Code><Message>One or more of the specified parts could not be found. The part may not have be (truncated...)
// InvalidPart (client): One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag. - <Error><Code>InvalidPart</Code><Message>One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's en"
$result = $client->completeMultipartUpload([
'Bucket' => 'the-bucket-name',
'Key' => $key,
'UploadId' => $uploadId,
'MultipartUpload' => [
'Parts' => $parts,
],
]);
return response()->json(['location' => $result['location']]);
}
public function abortMultipartUpload($uploadId)
{
// Haven't configured this route yet as I haven't made it this far.
return $uploadId;
}
private function arePartsValid($parts)
{
// Validation for the parts will go here, but returning true for now.
return true;
}
}
I can upload a multipart file fine purely PHP/server-side. For huge files though, this isn't going to work though since I would have to wait for the upload to finish on my server, then upload it to AWS in the parts.
$s3_client = new S3Client([
'version' => 'latest',
'region' => 'us-east-1',
]);
$bucket = 'the-bucket-name';
$tmp_name = $request->file('file')->getPathname();
$folder = Carbon::now()->format('Y/m/d/');
$filename = pathinfo($request->file('file')->getClientOriginalName(), PATHINFO_FILENAME);
$extension = $extension = pathinfo($request->file('file')->getClientOriginalName(), PATHINFO_EXTENSION);
$timestamp = Carbon::now()->format('H-i-s');
$name = "{$folder}{$filename}_{$timestamp}.{$extension}";
$response = $s3_client->createMultipartUpload([
'Bucket' => $bucket,
'Key' => $name,
]);
$uploadId = $response['UploadId'];
$file = fopen($tmp_name, 'r');
$parts = [];
$partNumber = 1;
while (! feof($file)) {
$result = $s3_client->uploadPart([
'Bucket' => $bucket,
'Key' => $name,
'UploadId' => $uploadId,
'PartNumber' => $partNumber,
'Body' => fread($file, 5 * 1024 * 1024),
]);
$parts[] = [
'PartNumber' => $partNumber++,
'ETag' => $result['ETag'],
];
}
$result = $s3_client->completeMultipartUpload([
'Bucket' => $bucket,
'Key' => $name,
'UploadId' => $uploadId,
'MultipartUpload' => [
'Parts' => $parts,
],
]);
What I believe is happening is that Uppy is handling the while loop part client-side. In order to do that, I have to return a pre-signed URL Uppy can use, but the pre-signed URL I'm currently returning isn't correct.
One thing I noted is that when I step through the while loop when initiating the multipart upload purely server-side, no file is uploaded to my bucket until the completeMultipartUpload method is fired. If however, I step through the parts being uploaded via Uppy, the parts seem to be being uploaded as the final file and each part is just overwriting the previous part. I'm then left with a fragment of the file, ie the last 3.5MB of a 43.5MB file.

Answer 1

Here's how I was able to get Uppy, Vue, and Laravel to play nicely together.
The Vue Component:
<template>
<div>
<a id="uppy-trigger" #click="isUppyOpen = !isUppyOpen">Open Uppy</a>
<dashboard-modal
:uppy="uppy"
:open="isUppyOpen"
:props="{trigger: '#uppy-trigger'}"
/>
</div>
</template>
<script>
import Uppy from '#uppy/core'
import AwsS3Multipart from '#uppy/aws-s3-multipart';
import '#uppy/core/dist/style.css';
import '#uppy/dashboard/dist/style.css';
export default {
components: {
'dashboard-modal': DashboardModal,
},
data() {
return {
isUppyOpen: false,
}
},
computed: {
// Uppy Instance
uppy: () => new Uppy({
logger: Uppy.debugLogger
}).use(AwsS3Multipart, {
limit: 4,
companionUrl: 'https://mysite.local/',
}),
},
beforeDestroy () {
this.uppy.close();
},
}
</script>
The Routing:
// AWS S3 Multipart Upload Routes
Route::name('s3.multipart.')->prefix('s3/multipart')
->group(function () {
Route::post('/', ['as' => 'createMultipartUpload', 'uses' => 'AwsS3MultipartController#createMultipartUpload']);
Route::get('{uploadId}', ['as' => 'getUploadedParts', 'uses' => 'AwsS3MultipartController#getUploadedParts']);
Route::get('{uploadId}/{partNumber}', ['as' => 'signPartUpload', 'uses' => 'AwsS3MultipartController#signPartUpload']);
Route::post('{uploadId}/complete', ['as' => 'completeMultipartUpload', 'uses' => 'AwsS3MultipartController#completeMultipartUpload']);
Route::delete('{uploadId}', ['as' => 'abortMultipartUpload', 'uses' => 'AwsS3MultipartController#abortMultipartUpload']);
});
The Controller:
<?php
namespace App\Http\Controllers;
use Aws\S3\S3Client;
use Carbon\Carbon;
use Exception;
use Illuminate\Http\Request;
class AwsS3MultipartController extends Controller
{
private $bucket;
private $client;
public function __construct()
{
$this->bucket = 'the-name-of-the-bucket';
$this->client = new S3Client([
'version' => 'latest',
'region' => 'us-east-1',
]);
}
/**
* Create/initiate the multipart upload
* #param Request $request
* #return JsonResponse
*/
public function createMultipartUpload(Request $request)
{
// Get the filename and type from request
$filename = $request->has('filename') ? $request->get('filename') : null;
$type = $request->has('type') ? $request->get('type') : null;
// Check filename
if (!is_string($filename)) {
return response()->json(['error' => 's3: filename returned from "getKey" must be a string'], 500);
}
// Check type
if (!is_string($type)) {
return response()->json(['error' => 's3: content type must be a string'], 400);
}
// Set up key equal to YYYY/MM/DD/filename_H-i-s.ext
$fileBaseName = pathinfo($filename, PATHINFO_FILENAME);
$extension = pathinfo($filename, PATHINFO_EXTENSION);
$folder = Carbon::now()->format('Y/m/d/');
$timestamp = Carbon::now()->format('H-i-s');
$key = "{$folder}{$fileBaseName}_{$timestamp}.{$extension}";
// Create/initiate the multipart upload
try {
$response = $this->client->createMultipartUpload([
'Bucket' => $this->bucket,
'Key' => $key,
'ContentType' => $type,
'Expires' => 60
]);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 400);
}
// Multipart upload key and id
$mpuKey = !empty($response['Key']) ? $response['Key'] : null;
$mpuUploadId = !empty($response['UploadId']) ? $response['UploadId'] : null;
// Check multipart upload key and id
if (!$mpuKey || !$mpuUploadId) {
return response()->json(['error' => 'Unable to process upload request.'], 400);
}
return response()->json([
'key' => $mpuKey,
'uploadId' => $mpuUploadId
]);
}
/**
* Get parts that have been uploaded
* #param Request $request
* #param string $uploadId
* #return JsonResponse
*/
public function getUploadedParts(Request $request, string $uploadId)
{
$key = $request->has('key') ? $request->get('key') : null;
// Check key
if (!is_string($key)) {
return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
}
$parts = [];
$getParts = true;
$startAt = 0;
// Get parts uploaded so far
while ($getParts) {
$partsPage = $this->listPartsPage($key, $uploadId, $startAt, $parts);
if (isset($partsPage['error'])) {
return response()->json(['error' => $partsPage['error']], 400);
}
if ($partsPage['isTruncated']) {
$startAt = $partsPage['nextPartNumberMarker'];
} else {
$getParts = false;
}
}
return response()->json(
$parts,
);
}
/**
* Create a pre-signed URL for parts to be uploaded to
* #param Request $request
* #param string $uploadId
* #param int $partNumber
* #return JsonResponse
*/
public function signPartUpload(Request $request, string $uploadId, int $partNumber)
{
$key = $request->has('key') ? $request->get('key') : null;
// Check key
if (!is_string($key)) {
return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
}
// Check part number
if (!intval($partNumber)) {
return response()->json(['error' => 's3: the part number must be a number between 1 and 10000.'], 400);
}
// Create the upload part command and get the pre-signed URL
try {
$command = $this->client->getCommand('UploadPart', [
'Bucket' => $this->bucket,
'Key' => $key,
'PartNumber' => $partNumber,
'UploadId' => $uploadId,
'Body' => '',
]);
$presignedUrl = $this->client->createPresignedRequest($command, '+20 minutes');
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 400);
}
// Convert the pre-signed URL to a string
$presignedUrlString = (string)$presignedUrl->getUri();
return response()->json(['url' => $presignedUrlString]);
}
/**
* Complete the multipart upload
* #param Request $request
* #param string $uploadId
* #return JsonResponse
*/
public function completeMultipartUpload(Request $request, string $uploadId)
{
$key = $request->has('key') ? $request->get('key') : null;
$parts = json_decode($request->getContent(), true)['parts'];
// Check the key
if (!is_string($key)) {
return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
}
// Check the parts
if (!is_array($parts) || !$this->arePartsValid($parts)) {
return response()->json(['error' => 's3: "parts" must be an array of {ETag, PartNumber} objects.'], 400);
}
// Complete the multipart upload
try {
$result = $this->client->completeMultipartUpload([
'Bucket' => $this->bucket,
'Key' => $key,
'UploadId' => $uploadId,
'MultipartUpload' => [
'Parts' => $parts,
],
]);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 400);
}
// Change forwardslash entities to forwardslashes
$location = urldecode($result['Location']);
return response()->json(['location' => $location]);
}
public function abortMultipartUpload(Request $request, $uploadId)
{
$key = $request->has('key') ? $request->get('key') : null;
// Check the key
if (!is_string($key)) {
return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
}
// Cancel the multipart upload
try {
$response = $this->client->abortMultipartUpload([
'Bucket' => $this->bucket,
'Key' => $key,
'UploadId' => $uploadId,
]);
} catch (Exception $e) {
//
}
return response()->json();
}
private function listPartsPage(string $key, string $uploadId, int $startAt, array &$parts)
{
// Configure response
$response = [
'isTruncated' => false,
];
// Get list of parts uploaded
try {
$result = $this->client->listParts([
'Bucket' => $this->bucket,
'Key' => $key,
'PartNumberMarker' => $startAt,
'UploadId' => $uploadId,
]);
} catch (Exception $e) {
return ['error' => 's3: unable to continue upload. The upload may have been aborted.'];
}
// Add found parts to parts array
if ($result->hasKey('Parts')) {
array_push($parts, ...$result->get('Parts'));
}
// Check if parts are truncated
if ($result->hasKey('IsTruncated') && $result->get('IsTruncated')) {
$response['isTruncated'] = true;
$response['nextPartNumberMarker'] = $result->get('NextPartNumberMarker');
}
return $response;
}
/**
* Validate the parts for the multipart upload
* #param array $parts An associative array of parts with PartNumber and ETag
* #return bool
*/
private function arePartsValid(array $parts)
{
if (!is_array($parts)) {
return false;
}
foreach ($parts as $part) {
if (!is_int($part['PartNumber']) || !is_string($part['ETag'])) {
return false;
}
}
return true;
}
}

Answer 2

You can use this pre-built laravel package to easily achieve multipart uploading via laravel and uppy:
https://github.com/TappNetwork/laravel-uppy-s3-multipart-upload