I'm reading the official doc for Amazon S3 Multipart Upload REST API, I'm wondering whether it's possible to use this API via AJAX.
The reason I'm asking is I try to upload a very large file (>5GB) from my browser to S3 bucket, I know there's a S3 upload API for Javascript and a way of leveraging AJAX to upload file to S3 but none of these address the large file issue.
The reason for not supporting MultiPart upload API using AJAX that I can think of is browser is not able to split the local file but I want to make sure it's really the case.
Does anyone around here ever used multipart upload api in AJAX or if it's impossible doing that, how do people usually deal with large file upload from browser?
Lots of thanks in advance!
I don't think it's necessary for you to use the Rest API for this. The s3.upload() method used in the javascript example you linked does support multipart uploads for large files according to the following AWS blog post: Announcing the Amazon S3 Managed Uploader in the AWS SDK for JavaScript. A browser example is included, although it uses bucket.upload rather than s3.upload. It also includes examples of tracking progress, configuring concurrency and part size and handling failures.
It does say with respect to browser uploads that "In order to support large file uploads in the browser, you must ensure that your CORS configuration exposes the ETag header; otherwise, your multipart uploads will not succeed. See the guide for more information on how to expose this header."
Possibly the CORS configuration may also need to allow more methods than listed in the 'Configuring CORS' section of the example you linked.
Related
I have created a collection in my Fauna database. I also made an index that returns all the data in that collection. I want to know if there is a way to get JSON REST API to work with this data. What I mean by this is I want a URL that I can 'GET' using XMLHttpRequest and then parse into JSON.
You might say that Fauna is "HTTP native". All of the language drivers (JS, C#, Python, etc.) are built on top of http requests. That is, there are no special connections or connection-pools to maintain. You can get pretty far with using the JS driver in the browser, for example, without using a server.
But to answer your question more directly, there are many ways to serve a REST API, and it is going to depend on how you want to serve the API.
Examples
AWS
Here is a blog post on how to set up a REST API with AWS App Runner, for example.
https://fauna.com/blog/deploying-a-rest-api-with-aws-app-runner-and-fauna
Vercel and Netlify
There are also some example apps on github that demonstrate using a REST API with
Vercel:
https://github.com/vercel/next.js/tree/canary/examples/with-cookie-auth-fauna
Or Netlify:
https://github.com/netlify/netlify-faunadb-example
Both of which host serverless functions for you to create a REST API.
I have a three-tier app running in AWS. The midware was written in Python Flask and stored on a Linux machine.
However, they asked me to move to AWS Lambda services. There is a limit of 6 M for the returning data function. As I´m dealing with GEOJson, sometimes it´s necessary to return up to 15 M.
Despite the AWS lambda stuff is stateless, I could provide some way to return data partitioned, but it´s problematic I think it will be necessary to generate the whole map again and again until I could fulfill all data.
Is there a better way to deal with this? I´m programming in Python.
I'd handle this by sending the data to S3, and issuing a redirect or JSON response that points to the URL on S3 (with a temporary, expiring URL if the data should be secure). If the data's long-lived, you can just leave it there; if not, you could use S3's lifecycle rules to have the files automatically delete after 24 hours or so.
If you have control of the client too that receives those data, you can send a compressed result that is then uncompressed client side. So you'll be able to send that 15MB response too, which can become really small when compressed.
Or you can send a fragment of the whole response with a token or something indicating the client that the response is not complete. Than the client will make another request with that token to get the next fragment, and so on until there are no more fragments. At this point the client can join all fragments to get the full response.
Speaking of the 6MB limit, I hope that at some point we will have the ability to set what is the max payload size. since 6MB is fine for most cases, but not ALL cases
You can use presigned S3 URL to upload, using this there will be no bound by payload size.
Get HTTP GET request to API Gateway, then Lambda function get generate presigned URL and return it presigned S3 URL.
Then client directly update content to s3 using pre-signed s3 URL.
https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
The webContentLink is supposed to be a direct link to the file represented by the fileResource.
Let's first ignore the fact that the link will infact always respond with a 302 pointing to a generated downloadUrl (which has a very finite life-time).
I understand that the resource pointed to by webContentLink uses cookie-based authentication and CORS would pose a security risk, however the documentation explicitly states that no authentication is needed when the file in question is public (obviously).
So for the case where the file is publicly available I do not understand why CORS is not supported.
Is there some security risk associated with this that I fail to realise? Bug? Was it deemed too much work to check the permissions on the file before responding with or without the Access-Control-Allow-Origin.
On a side note, using a server-side proxy to fetch the downloadUrl pointed to by the 302 location produces a resource that does support CORS.
Not a solution to the issue posed by the question but using the GDrive hosting feature I was able to work around the limitation of the webContentLink without using a server side proxy.
https://googledrive.com/host/shared_folder_id/relative_path
That link will be a direct link to the file and the resource supports CORS. However this has the limitation that the file's name has to be unique inside its own folder, otherwise it does not work.
I am making a get request via http to my other application. Both apps use Paperclip to manage attachment. How can I achieve to make the target app make a correct response with the attachment? Or any suggestion ?
If you are trying to serve same asset from two different applications. that's a bad idea because it will lead to sync issus & race conditions.
Instead, you should create a image service which runs on images.example.com. Here you will expose an endpoint where images can be uploaded via HTTP POST and download via URL.
Now, get both of your applications to upload & download images from this service.
First I'm not in the web side of our world, so be nice with the backend guy.
A quick background : For a personal need I've developped a google chrome extension. They are basically a webpage loaded in a chrome windows and... yeah that's it. Everything is on the client side (scripts, styles, images, etc...) Only the data are coming from a server through ajax calls. A cron job call a php script every hours to generate two files. One, data.json contains the "latest" datas in a json format. Another one hash.json contain the hash of the data. The client chrome application use local storage. If the remote hash differ from the local one, he simply retrieve the data file from the remote server.
As I have a BizSpark account with Azure my first idea was : Azure Web Site with php for the script, a simple homepage and the generated file and the Azure Scheduler for the jobs.
I've developed everything locally and everything is running fine... but once on the azure plateform I get this error
XMLHttpRequest cannot load http://tso-mc-ws.azurewebsites.net/Core/hash.json. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:23415' is therefore not allowed access.
But what I really can't understand is that I'm able (and you'll be too) to get the file with my browser... So I just don't get it... I've also tried based on some post I've found on SO and other site to manipulate the config, add extra headers, nothing seems to be working...
Any idea ?
But what I really can't understand is that I'm able (and you'll be
too) to get the file with my browser... So I just don't get it
So when you type in http://tso-mc-ws.azurewebsites.net/Core/hash.json in your browser's address bar, it is not a cross-domain request. However when you make an AJAX request from an application which is running in a different domain (http://localhost:23415 in your case), that's a cross-domain request and because CORS is not enabled on your website, you get the error.
As far as enabling CORS is concerned, please take a look at this thread: HTTP OPTIONS request on Azure Websites fails due to CORS. I've never worked with PHP/Azure Websites so I may be wrong with this link but hopefully it should point you in the right direction.
Ok, will perhap's be little troll answer but not my point (I'm .net consultant so... nothing against MS).
I pick a linux azure virtual machine, installed apache and php, configure apache, set some rights and define the header for the CROS and configure a cron in +/- 30minutes... As my goal is to get it running the problem is solved, it's running.