HTTP/1.1 - Transfer-Encoding: chunked - delay in the middle of response - chunked-encoding

I am using WINC1500 WiFi (with Arduino) to connect to a server (https) and to send some API requests.
In the header I send:
POST /api/myapi.php HTTP/1.1
Host: myserver.com
Accept: application/json, text/javascript, */*; q=0.01
Content-Type: application/x-www-form-urlencoded; charset=iso-8859-1
Connection: keep-alive
Content-Length: 10
my_content
The server response is ok, something like this:
HTTP/1.1 200 OK
Date: Tue, 03 Sep 2019 16:31:14 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json
18f
BlaBlaBla.... (399 chars = 18f in hexa)
0
The problem is that, after the first chunk, there is a delay in waiting for the terminating chunk. THe last chunk is an empty one. This is an example to understand it:
HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked
7\r\n
Mozilla\r\n
9\r\n
Developer\r\n
7\r\n
Network\r\n
0\r\n
\r\n
Ok... Let's look at the time line:
19:31:13.685 -> connecting...
19:31:14.325 -> posting...
19:31:15.745 -> end posting...
19:31:15.864 -> HTTP/1.1 200 OK
19:31:15.864 -> Date: Tue, 03 Sep 2019 16:31:14 GMT
19:31:15.864 -> Server: Apache
19:31:15.864 -> Expires: Thu, 19 Nov 1981 08:52:00 GMT
19:31:15.864 -> Cache-Control: no-store, no-cache, must-revalidate
19:31:15.864 -> Pragma: no-cache
19:31:15.904 -> Keep-Alive: timeout=5, max=100
19:31:15.904 -> Connection: Keep-Alive
19:31:15.904 -> Transfer-Encoding: chunked
19:31:15.904 -> Content-Type: application/json
19:31:15.904 ->
19:31:15.904 -> waitingChunkLength: 18f
19:31:15.904 -> 399
19:31:15.945 -> waitingChunk: BlaBlaBla.... (399 chars)
19:31:20.875 -> waitingChunkLength: 0
19:31:20.875 -> 0
19:31:20.875 -> waitingChunkTrailer:
19:31:20.875 -> Done...
Connection closed
After receiving the first and only chunk, there is a huge delay before receiving the last (terminating) chunk. Actually, the last chunk is send just before the connection to be closed by server. I believe that is happening because of the idle time. There are 5 seconds of inactivity, so the server must close the connection.
I try to send a message ("OK") to server just after receiving the first chunk. In this case, the server is responding immediately, but is closing the connection (I believe because of inadequate message):
19:45:43.818 -> connecting...
19:45:44.688 -> posting...
19:45:46.207 -> end posting...
19:45:46.322 -> HTTP/1.1 200 OK
19:45:46.322 -> Date: Tue, 03 Sep 2019 16:45:44 GMT
19:45:46.322 -> Server: Apache
19:45:46.322 -> Expires: Thu, 19 Nov 1981 08:52:00 GMT
19:45:46.322 -> Cache-Control: no-store, no-cache, must-revalidate
19:45:46.322 -> Pragma: no-cache
19:45:46.322 -> Keep-Alive: timeout=5, max=100
19:45:46.358 -> Connection: Keep-Alive
19:45:46.358 -> Transfer-Encoding: chunked
19:45:46.358 -> Content-Type: application/json
19:45:46.358 ->
19:45:46.358 ->
19:45:46.358 -> waitingChunkLength: 18f
19:45:46.358 -> 399
19:45:46.398 -> waitingChunk: BlaBlaBla.... (399 chars)
19:45:46.517 -> waitingChunkLength: 0
19:45:46.517 -> 0
19:45:46.517 -> waitingChunkTrailer:
19:45:46.517 -> Done...
Connection closed
I need to know where is the problem. Should I send a special message to server after each chunk? Because of this behaviour I cannot take advantage of "Keep-alive" session.

After changing firmware a couple of times without success, I go back to the code and try different things.
The problem seems to be last line in POST action:
client.print("Content-Length: ");
client.println(10);
client.println();
client.println("my_content");
This last line should be:
client.print("my_content");
... println appends a carriage return character (ASCII 13, or '\r') and a newline character (ASCII 10, or '\n') after text. So, this extra (unexpected) characters give me so much troubles those days. Hope this solve your problems too.

Related

netcat: hexadecimal numbers before and after response body

If I run the following:
sed 's/$/\r/g' <<EOF | netcat "www.google.com" 80
GET / HTTP/1.1
Host:www.google.com
Connection: close
EOF
I get the usual headers, followed by a hexadecimal number before the response body I haven't seen before (in this case, 5172, although another example is 51C8):
HTTP/1.1 200 OK
Date: Thu, 09 Sep 2021 20:03:53 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: ...
Accept-Ranges: none
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked
5172
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-GB"><head>...
In presumed this was content length, although the response body length did not match this by a large order.
What is this number?

Varnish is cached but Google Chrome showing not cached?

I have installed Varnish and configured it as per as the guideline.
And when I try it curl -I https://d-o-m-a-i-n.com I get following which seems its working correctly.
HTTP/1.1 200 OK
Date: Sat, 28 Mar 2020 03:17:02 GMT
Server: Apache/2.4.18 (Ubuntu)
Expires: Sun, 29 Mar 2020 03:17:03 GMT
Cache-Control: max-age=86400, public, s-maxage=86400
Pragma: cache
X-Magento-Tags: cms_b_porto_homeslider_3,store,cms_b,cms_p_91,cms_b_porto_custom_notice_new,cat_p,cat_c_p_30,cat_p_22,cat_p_1,cat_p_34,cat_p_21,cat_p_41,cat_p_11,cat_p_39,cat_p_35,cat_p_33,cms_b_porto_footer_top_1_for_5,cms_b_porto_footer_middle_1_for_5,cms_b_porto_footer_middle_2_for_5
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=UTF-8
X-Varnish: 33268 3
Age: 13957
Via: 1.1 varnish (Varnish/5.2)
X-Cache: cached
Accept-Ranges: bytes
But when I check in Google Chrome header response I get followings where it shows Varnish is not caching and age is 0?
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=86400, public, s-maxage=86400
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 20027
Content-Type: text/html; charset=UTF-8
Date: Sat, 28 Mar 2020 07:13:35 GMT
Expires: Sun, 29 Mar 2020 07:13:35 GMT
Keep-Alive: timeout=5, max=100
Pragma: cache
Server: Apache/2.4.18 (Ubuntu)
Vary: Accept-Encoding
Via: 1.1 varnish (Varnish/5.2)
X-Cache: uncached
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Magento-Tags: cms_b_porto_homeslider_3,store,cms_b,cms_p_91,cms_b_porto_custom_notice_new,cat_p,cat_c_p_30,cat_p_1,cat_p_22,cat_p_21,cat_p_11,cat_p_34,cat_p_41,cat_p_39,cat_p_35,cat_p_33,cms_b_porto_footer_top_1_for_5,cms_b_porto_footer_middle_1_for_5,cms_b_porto_footer_middle_2_for_5
X-UA-Compatible: IE=edge
X-Varnish: 1704417
X-XSS-Protection: 1; mode=block
Do you have any ideas why that is?
How are you generating the key for caching(vcl_hash)?
Check it and make sure there is no user agent involved.

How Can I Detect something from HTTP Response

I'am Developing bash script can detect web application firewall from header tags but i can find example like my idea ?
To find the request headers from bash you can simply use curl. If you're on windows you'll want the new windows bash shell or cygwin to run it.
There are dozens more tricks you can play with curl to get whatever you want in whatever format you want, lots of SO questions on it to answer any questions you come up with.
curl --head www.google.com
HTTP/1.1 200 OK
Date: Thu, 06 Apr 2017 02:07:00 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=100=IoNzfnVsz_oaEwIQE182ysgVSHoZYRVKjTqSQ5GqKrz1ewxwav2ae5GPo_bx0apr39Pnn4yvM5RfsmQnJ_QFmllVwS34ts-bNrvkzDFIfaokkDTo1BXHDDI69duBn1f9kx4sXJ_rcCK28og6; expires=Fri, 06-Oct-2017 02:07:00 GMT; path=/; domain=.google.com; HttpOnly
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding
Here's an example of getting response headers using curl:
curl -D - www.google.com
HTTP/1.1 200 OK
Date: Thu, 06 Apr 2017 02:11:26 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=100=DrUalBDiHKiZkX0yETtowdWhEfjJy7ioPU0Fe7Wch9pbbYI8MeSbg8M42dHmwu-hKZmYUlnE7VIgLhJ_Zi6byG_PYpTu5s2KYUv9XjPeH-GfSOTSq22I2GnEqXZwhJv-Bdn0aYzCUugF9FHb3Q; expires=Fri, 06-Oct-2017 02:11:26 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop=<cut the rest of the HTTP request>

Failed to decode downloaded font on Heroku

I'm seeing the following warning in Chrome Developer Tools when I access my JHipster application at http://www.jhipster-book.com. I don't see this error when running locally.
Failed to decode downloaded font: http://www.jhipster-book.com/assets/fonts/sourcesanspro/sourcesanspro.woff
I'm using these same fonts on a client's project (that runs on GAE) and it works fine. It looks like the content-type header is not being set on Heroku. Here's what I see locally:
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.9
Date: Fri, 12 Jun 2015 21:57:23 GMT
Content-type: application/x-font-woff
Content-Length: 27248
Last-Modified: Fri, 12 Sep 2014 16:22:14 GMT
And on Heroku:
HTTP/1.1 200 OK
Connection: keep-alive
Server: Apache-Coyote/1.1
X-Application-Context: application:prod,heroku:56079
Cache-Control: max-age=2678400000, public
Pragma: cache
Expires: Mon, 13 Jul 2015 21:59:11 GMT
Last-Modified: Fri, 12 Jun 2015 21:47:41 GMT
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
Content-Length: 27247
Date: Fri, 12 Jun 2015 21:59:11 GMT
Via: 1.1 vegur
Is there something I have to configure in Heroku for this to work?

"301 Moved", Google APIs with Opera

In my web app I'm making a CURL call to Google's (unofficial) weather API at:
http://www.google.com/ig/api?weather=,,, ...
This works fine in all cases except when the page involved is accessed in Opera. When accessed in Opera, instead of the weather I get the following in the xml response:
301 Moved</H1>
The document has moved
here.
How can I fix this? I've seen some suggestions around the web that clearing the cache is a solution, but that hasn't worked for me. Note I'm particularly concerned with Opera Mini.
Thanks a lot.
Update 2012-06-20: Tested with Opera 12 and Google has fixed the sniffing issue it seems.
I do not think you can fix it. It would be interesting to know why Google does server-side user agent sniffing and redirects Opera to another URI. Could you paste the full weather URI, so we can test it at Opera ourselves? If do a get of the one you have given into Opera I get
<xml_api_reply version="1">
<weather module_id="0" tab_id="0" mobile_row="0" mobile_zipped="1" row="0" section="0">
<problem_cause data=""/>
</weather>
</xml_api_reply>
I also do not get a redirection, but I guess it is because the URI is not the one you have used.
% curl -sI "http://www.google.com/ig/api?weather=,,,"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=ISO-8859-1
Date: Fri, 02 Mar 2012 12:04:44 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=fe13590e95ceb98e:TM=1330689884:LM=1330689884:S=h1eocRzhNcZ_Kwoa; expires=Sun, 02-Mar-2014 12:04:44 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
More details will help, and I will complete here.
Update 2012-03-12 First I tried with a simple curl.
→ curl -sI "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=ISO-8859-1
Date: Mon, 12 Mar 2012 13:16:43 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=9bc71bbf2edb7ebb:TM=1331558203:LM=1331558203:S=K5Ew69E5IsYhA0s8; expires=Wed, 12-Mar-2014 13:16:43 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
Then with Opera User agent.
→ curl -sI -A "Opera/9.80 (Macintosh; Intel Mac OS X 10.7.3; U; fr) Presto/2.10.229 Version/11.61" "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Mon, 12 Mar 2012 13:17:47 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=54cc62619394059e:TM=1331558267:LM=1331558267:S=JRCO-WNJMUNMMHsO; expires=Wed, 12-Mar-2014 13:17:47 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
And finally with Firefox User Agent
→ curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0" "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Mon, 12 Mar 2012 13:20:09 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=ab709995945767a8:TM=1331558409:LM=1331558409:S=bom-8pa-x9gGY5Sb; expires=Wed, 12-Mar-2014 13:20:09 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
There is a no X-Content-Type-Options: nosniff in all cases and no redirection. What is the user agent you are using. You can type about:opera in the addressbar and the user agent string will appear.

Resources