I have a three-tier app running in AWS. The midware was written in Python Flask and stored on a Linux machine.
However, they asked me to move to AWS Lambda services. There is a limit of 6 M for the returning data function. As I´m dealing with GEOJson, sometimes it´s necessary to return up to 15 M.
Despite the AWS lambda stuff is stateless, I could provide some way to return data partitioned, but it´s problematic I think it will be necessary to generate the whole map again and again until I could fulfill all data.
Is there a better way to deal with this? I´m programming in Python.
I'd handle this by sending the data to S3, and issuing a redirect or JSON response that points to the URL on S3 (with a temporary, expiring URL if the data should be secure). If the data's long-lived, you can just leave it there; if not, you could use S3's lifecycle rules to have the files automatically delete after 24 hours or so.
If you have control of the client too that receives those data, you can send a compressed result that is then uncompressed client side. So you'll be able to send that 15MB response too, which can become really small when compressed.
Or you can send a fragment of the whole response with a token or something indicating the client that the response is not complete. Than the client will make another request with that token to get the next fragment, and so on until there are no more fragments. At this point the client can join all fragments to get the full response.
Speaking of the 6MB limit, I hope that at some point we will have the ability to set what is the max payload size. since 6MB is fine for most cases, but not ALL cases
You can use presigned S3 URL to upload, using this there will be no bound by payload size.
Get HTTP GET request to API Gateway, then Lambda function get generate presigned URL and return it presigned S3 URL.
Then client directly update content to s3 using pre-signed s3 URL.
https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
Related
I am trying to connect to an API that uses websockets. I need to do the following:
Connect to the websocket using a given URI
Send a login request
Send a request for the required data stream(s)
Store the returned streamed data in an array for immediate processing (the array will be continually updated while data is streamed)
When finished collecting data, send a logout request
I have a general understanding of websockets, but have never tried to connect to a websocket. I have read through the “documentation” for packages HTTP (which I have used before), WebSockets, and DandelionWebSockets. Each has left me scratching my head trying to understand how to implement the above tasks.
Would someone please help by showing me, line-by-line, how to set up the above tasks and also explain why each line or function is used? (Assume I have the correct URI, login, data, and logout request formats.)
I am trying to send binary data from my AWS Lambda function as a response to an AWS Gateway GET Method that DOES NOT use Lambda Proxy integration. I have tried all sorts of variations but still can't make it work, although I feel like I am close.
My API Gateway HTTP request is returning:
But what I want is is the actual binary data:
I did attempt using a mapping template, but was unsuccessful due to my lack of understanding the templating syntax/behavior(I tried $util.base64Decode($input.body) but that produced a server error).
But I wasn't sure if that was even necessary since I have the content handling set to Convert to binary.
I ran into this problem but I used a proxy url. Make sure to enable Binary Media Types. Also do not forget to deploy your changes, simply saving is not enough. Also make sure you have the correct content type in the header with client sending the payload.
Google Analytics uses Get Request for .gif image to server
http://www.google-analytics.com/__utm.gif?utmwv=4&utmn=769876874&utmhn=example.com&utmcs=ISO-8859-1&utmsr=1280x1024&utmsc=32-bit&utmul=en-us&utmje=...
We can observer that all parameters are sent in this Get Request and the requested image is no where found useful (Its just 1px by 1px Image)
Known Information: If requesting query string is large then Google are going for Post Request.
Now the question is why not Post Request always irrespective of the query string is large or not.
Being data sent via Get Request its leads to security issue. Since, the parameters will be stored in browser history or in web server logs in case of Get Request.
Could someone give any supportive reasons why Google Analytics is depending on both the things?
Because GET requests is what you use for retrieving information that does not alter stuff.
Please note that the use of POST has quite some downsides, the browser usually warns against reloading a resource requested via POST (to prevent double data-entry), POST requests are not cached (which is why some analytics misuse it), proxied etc.
If you want to retrieve a LOT of data using a URL (advice: rethink if there might be a better option), then it's necessary to use post, from Wikipedia:
There are times when HTTP GET is less suitable even for data retrieval. An example of this is when a great deal of data would need to be specified in the URL. Browsers and web servers can have limits on the length of the URL that they will handle without truncation or error. Percent-encoding of reserved characters in URLs and query strings can significantly increase their length, and while Apache HTTP Server can handle up to 4,000 characters in a URL, Microsoft Internet Explorer is limited to 2048 characters in any URL. Equally, HTTP GET should not be used where sensitive information, such as user names and passwords have to be submitted along with other data for the request to complete. In these cases, even if HTTPS is used to encrypt the message body, data in the URL will be passed in clear text and many servers, proxies, and browsers will log the full URL in a way where it might be visible to third parties. In these cases, HTTP POST should be used.
A POST request would require an ajax call and it wouldn't work because of http://en.wikipedia.org/wiki/Same-origin_policy. But images can easily be cross-site, so they just need to add an img tag to the DOM with the required url and the browser will load it, sending the needed information to their servers for tracking.
I'm trying to work around the fact that twitter uses a call-limit by letting the client do a call to Twitter(By parameters given by my script, like last_id, username etc) and give me the newly found tweets by posting them via an AJAX request that I store in my database after.
However, if i figure out the parameters that are being sent from my getTweets javascript function to my save_tweets.php via a JSON array it's not that hard to post random stuff with an extension like the REST console in chrome.
Obviously i want the tweets to be legit and can not be manipulated(or posted) by anyone. I understand that javascript is clientside and therefore there is not much control over the content being grabbed and put away again but is there a way to be safe that the POST data you send comes from a user/webpage that is allowed to do so?
I've tried thinking of a PHP session token or something, but that doesn't fly since you have to cross-check that token which you therefore have to send along with the JSON array to my PHP.
Thanks in advance,
p.s If you know a different way to not being obstructed by a call limit to Twitter i'd be happy with that too. But 150 calls from a server isn't that much if you get a 1000+ users an hour to your page.
Let's say we have a simple node JS backend, paired with a standard NoSQL document store such as CouchDB. Since our database is just a document key-store with no schema, anything can get inserted. And since our server is built on JSON as well, ultimately POST requests that come in from the client with JSON payloads end up getting stored directly into our data store.
This of course is very convenient and makes for a lightweight application. I've been wondering, though, short of writing code for every possible insertion endpoint to verify that each POST or PUT request is well-formed, is there anything to prevent an attacker from firing up their developer console and spoofing POST/PUT requests, allowing them to insert any kind of junk data they wish into our datastore? It would not be too difficult to wreck an application's data this way.
Clearly token-based authentication can ensure that only authenticated users can access these service endpoints, but that doesn't prevent them from spoofing these request with the same HTTP headers that valid requests have. This is all quite simple with today's browser developer tools.
In a traditional server language like Java, JSON PUTs and POSTs are marshalled to a highly-structured class-based Object. Requests whose payloads do NOT match these formats are rejected with HTTP errors.
Does anyone know of tools or paradigms for node which ensures that requests like this meet some basic structure criteria?