I'm sending an AJAX request with a few form fields (maybe 200 bytes max) and an image in base64. The image is 770KB but the content length says 1.1MB....is all of that extra data overhead?! Is there a better way to send an image?
Related
Is there any max size for send post payload ?
Actually, I have a service which send a json payload to another micro-service, In side of JSON payload, there is a filed which hold encoded String (basically i have converted file into encoded String and set it into that payload field value).
File size is not fixed, So i am curious about it, is there any max size support for Post json payload ?
I know spring-boot have following properties for set max size for file upload.
spring.http.multipart.max-file-size
spring.http.multipart.max-request-size
But as i mentioned i am not upload file, i am convert file into encoded string and send this string with json payload.
So how can i set the max size support for post request.
Also In another scenario I am hitting a Get request to micro-service and then this micro service return a large payload,is there any limitation of receive payload size?
It depends, not only in spring configuration as you've pointed, but also in server configuration.
Most servers have a post request loadout limit. For example in tomcat you can configure it with the "maxPostSize" property. Quoting from the documentation:
The maximum size in bytes of the POST which will be handled by the container FORM URL parameter parsing.
The limit can be disabled by setting this attribute to a value less than or equal to 0. If not specified, this attribute is set to 2097152 (2 megabytes).
I am sending 2 large query string in AJAX requests, which are basically, a Base64 encoding of a jpeg(s). When the Camera is not a high-resolution one, AJAX request doesn't abort.
At first, I thought its a Nginx issue, Because I was getting an error as request entity too large I resolved it, Then I made changes to my Plug as
plug Plug.Parsers,
parsers: [
:urlencoded,
{:multipart, length: 20_000_000},
:json
],
pass: ["*/*"],
query_string_length: 1_000_000,
json_decoder: Poison
After defining query_string_length, Now I am not getting any errors like above but ajax request still abort.
Base64 encoding string size is 546,591 bytes or max.
I have tried to increase the AJAX request timeout to a very large timespan as well but it still fails. And I don't have any clue where the problem is right now.
How can we receive long strings in Plug?
Some of few answers on StackOverflow about this issue where people used AJAX and PHP, suggesting to change post_max_size, How can we do that in Elixir Plug?
As you are sending AJAX request with JSON data, you should put the length config of json in the plug.
plug Plug.Parsers,
parsers: [
:urlencoded,
{:multipart, length: 20_000_000},
{:json, length: 80_000_000},
],
pass: ["*/*"],
json_decoder: Poison
I suppose you will not put the data in the query string of the post, so the query_string_length - the maximum allowed size for query strings is not needed.
---Original answer---
For plug version around 1.4.3 and have no query_string_length option.
When you post the data as string, you are using Plug.Parsers.
If you are willing to process larger requests, please give a :length
to Plug.Parsers.
You should change the code query_string_length: 1_000_000 to length: 20_000_000.
EDIT: Later on it seemed that some browsers are confusing the term "response headers" with "response message (without response body)". So that's where this question was coming wrong. The browsers were just incorrect. Meanwhile I gave answer to my own question.
In Firefox you can check the "raw headers" via "Firefox Developer Tools" > "Network".
Example of the raw response headers:
Date: Thu, 23 Nov 2017 12:43:21 GMT
Server: Apache/2.4.17 (Unix) OpenSSL/1.0.1e-fips PHP/5.6.16
Connection: Keep-Alive
Keep-Alive: timeout=1, max=100
Cache-Control: max-age=9, public
Vary: User-Agent
But I miss (for example): "HTTP/1.1 304 Not Modified". Firefox shows somewhere else "304 Not Modified", but not RAW.
So they let me think I have the response headers in raw form, but actually it's only a part of the response headers, excluding the status code. This can be really confusing for people.
In my opinion it would make much more sense to add the "status code" too at that place. Now it's not really logical.
Is this a bug or how I have to see it?
Meanwhile I can answer my own question.
At this moment, some browsers are usings terms wrong. They are confusing "reponse headers" and "response message (without message body)". So that's why I was confused when asking the question and that was what the question was about.
See: https://www.rfc-editor.org/rfc/rfc7230#page-8 (2.1. Client/Server Messaging)
A server responds to a client's request by sending one or more HTTP
response messages, each beginning with a status line that includes
the protocol version, a success or error code, and textual reason
phrase (Section 3.1.2), possibly followed by header fields containing
server information, resource metadata, and representation metadata
(Section 3.2), an empty line to indicate the end of the header
section, and finally a message body containing the payload body (if
any, Section 3.3).
So with other words.
RESPONSE MESSAGE:
Status line (ending with CRLF, so 2 extra Bytes)
Header field 1, in case of (ending with CRLF, so 2 extra Bytes)
Header field 2, in case of (ending with CRLF, so 2 extra Bytes)
Header field 3, et cetera (ending with CRLF, so 2 extra Bytes)
Empty line to indicate the end of the header section (CRLF, so 2 extra Bytes)
Message body / Response body, if any
RESPONSE HEADERS / RESPONSE HEADER FIELDS:
Header field 1 in case of (ending with CRLF, so 2 extra Bytes)
Header field 2 in case of (ending with CRLF, so 2 extra Bytes)
Header field 3 et cetera (ending with CRLF, so 2 extra Bytes)
So officially the status-line is not part of the "reponse headers", but only part of the "response message".
Firefox is for example showing the size of the "reponse headers" in: "Firefox Developer Tools" > "Network" > click on row > Headers tab.
This size also includes the size of i.a. the status-line. The size of the raw response headers must correspond this size, but that's not the case at this moment. So or they need to change the size, or they must extra include the "status-line" + "empty line" in the (raw) response headers and they must give it another name (for example: response message - message body).
Chrome is also doing this wrong. For example see: https://developers.google.com/web/tools/chrome-devtools/network-performance/reference#requests
They are saying there:
Size. The combined size of the response headers plus the response body, as delivered by the server.
But they actually mean something different (also according the value of size in practise). They actually mean this:
Size. The combined size of the response message, without the message body (instead of response headers) plus the response body, as delivered by the server.
So actually with other words:
Size. The combined size of the response message, as delivered by the server.
So that's where my origin question was coming from. Apparently it's a difficult subject for browsers at this moment, because I tested it in 2 browsers and both are making mistakes with it.
So because of that, it's not weird if people would think the status-line is part of the response headers.
I have built an audio stream for mp3 files, and each time client hits the audio it receives something like this:
But what it does is just plays 1 minute sample instead of 120 minute
What am I doing wrong here?
Not 100% sure because you didn't provide code or an example stream to test, but your handling of HTTP range requests is broken.
In your example request, the client sends Range: bytes=0-, and your server responds with a 1MiB response:
Content-Length: 1048576 (aka. 1 MiB)
Content-Range: 0-1048575/...
This is wrong, the client did not request this! It did request bytes=0-, meaning all data from position 0 to the end of the entire stream (See the http 1.1 RFC), i.e. a response equal to one without any Range. (IIRC, Firefox still sends the Range: bytes=0- to detect if the Server handles ranges in the first place).
This, combined with the Content-Length, leads the client (Firefox) to think the whole resource is just 1MiB in size, instead of the real size. I'd imagine the first 1 MiB of your test stream comes out as 1:06 of audio.
PS: The Content-Duration header (aka. RFC 3803) is something browsers don't usually implement at all and just ignore.
Just an idea. Did you tried some of the http 3xx header like:
'308 Resume Incomplete' or '503 Service Temporarily Unavailable' plus 'retry-after:2' or '413 Request Entity Too Large' plus 'retry-after:2'
agent = Mechanize.new
url = "---------------------------"
page = agent.get(url)
Now, I want to know the KB(kilobytes) of data that has been used by my internet service provider to scrape that data.
More specifically, whats the size in KB, of the variable "page"?
page.content.bytesize / 1024.0
It's really two separate things. The size of unzipped response body and the amount of bytes that were transferred. You can get the first by inspecting page.body, for the second you would need to measure response and request headers as well as account for things like gzip and redirects. Not to mention dns lookups, etc.