I'm trying to download a zipped file from Amazon Datafeed url and then decompress it.
This is my code:
open('public/files/amazon_ce.xml', 'w') do |local_file|
open('https://assoc-datafeeds-eu.amazon.com/datafeed/it_amazon_ce.xml.gz', :http_basic_authentication=>[USERNAME, PASSWORD]) do |remote_file|
local_file.write(Zlib::GzipReader.new(remote_file).read)
end
end
If I try with another file everything is ok, but not with this Amazon file: the error is:
OpenURI::HTTPError: 500 Internal Server Error
I logged the request when I download the same file using the browser...
GET /datafeed/getFeed?filename=it_amazon_ce.xml.gz HTTP/1.1
Host: assoc-datafeeds-eu.amazon.com:443
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: it-IT,it;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: xxxxx
Referer: https://assoc-datafeeds-eu.amazon.com/datafeed/listFeeds?format=text/html
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36
HTTP/1.1 302 Moved Temporarily
Cache-Control: no-cache
Content-Length: 0
Date: Wed, 30 Sep 2015 21:24:22 GMT
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Location: http://prod-deamazon.outputs.datafeeds.s3.amazonaws.com/it_amazon_dvd.xml.gz?Signature=xxxx&AWSAccessKeyId=xxxx&Expires=xxxx
Pragma: No-cache
Server: Apache-Coyote/1.1
Any idea?
Your browser is already authenticated and has a cookie which allows you to download the file.
You need to make sure that your Rails app has proper credentials to download the file (I can't in my browser, it asks for username/password)
Related
I am working on a web file share service, the service can decompress the .gz file content before transferring to client if user asks for it: Windows has not gzip build-in.
Following is an example: un-gzip and download long.txt.gz.
Capture from Fiddler
Request
GET https://szdc-data-explorer-test.spiral.exchange/v1/fs/file/tmp/long.txt.gz?pipelines=ungzip&file_name=long.txt&otk=da39a3ee5e6b4b0d3255bfef95601890afd80709 HTTP/1.1
Host: szdc-data-explorer-test.spiral.exchange
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Referer: https://next-test.spiral.exchange/
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-site
Response
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Disposition: attachment; filename="long.txt"
Content-Type: text/plain; charset=UTF-8
Date: Thu, 13 Jan 2022 00:44:14 GMT
Vary: Accept-Encoding
Vary: Origin
Content-Length: 293216
Firefox save the file with name long.txt as specified in Content-Disposition header, but the content is gzip-ed. It sounds like Firefox auto gzip due to request path ends with .gz: /v1/fs/file/tmp/long.txt.gz
Well, it is a known behavior of firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=610679
I've been working on this issue for two days. If I create a new project following the tutorial here, and run it through IIS Express, it works fine. As soon as I switch it to IIS, it fails.
So, when hosted in Express I send:
OPTIONS http://localhost:63274/api/test HTTP/1.1
Host: localhost:63274
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: PUT
Origin: http://localhost:4001
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
DNT: 1
Accept: */*
Referer: http://localhost:4001/admin/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-CA,en;q=0.9,fr;q=0.8
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Server: Microsoft-IIS/10.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: PUT
X-SourceFiles: =?UTF-8?B?RDpcSGVicmFcTWluZGZsYXllclxNaW5kZmxheWVyXGFwaVx0ZXN0?=
X-Powered-By: ASP.NET
Date: Wed, 24 Jul 2019 15:45:48 GMT
Content-Length: 0
And after the preflight:
PUT http://localhost:63274/api/test HTTP/1.1
Host: localhost:63274
Connection: keep-alive
Content-Length: 0
Pragma: no-cache
Cache-Control: no-cache
Origin: http://localhost:4001
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
DNT: 1
Accept: */*
Referer: http://localhost:4001/admin/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-CA,en;q=0.9,fr;q=0.8
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 17
Content-Type: text/plain; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
Access-Control-Allow-Origin: *
X-SourceFiles: =?UTF-8?B?RDpcSGVicmFcTWluZGZsYXllclxNaW5kZmxheWVyXGFwaVx0ZXN0?=
X-Powered-By: ASP.NET
Date: Wed, 24 Jul 2019 15:45:48 GMT
PUT: Test message
And when hosted from IIS I send (an identical request of):
OPTIONS http://localhost:4013/api/test HTTP/1.1
Host: localhost:4013
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: PUT
Origin: http://localhost:4001
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
DNT: 1
Accept: */*
Referer: http://localhost:4001/admin/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-CA,en;q=0.9,fr;q=0.8
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Server: Microsoft-IIS/10.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: PUT
Date: Wed, 24 Jul 2019 15:50:34 GMT
Content-Length: 0
And after the preflight:
PUT http://localhost:4013/api/test HTTP/1.1
Host: localhost:4013
Connection: keep-alive
Content-Length: 0
Pragma: no-cache
Cache-Control: no-cache
Origin: http://localhost:4001
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
DNT: 1
Accept: */*
Referer: http://localhost:4001/admin/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-CA,en;q=0.9,fr;q=0.8
HTTP/1.1 405 Method Not Allowed
Cache-Control: private
Allow: GET, HEAD, OPTIONS, TRACE
Content-Type: text/html; charset=utf-8
Server: Monsters/4.0
Date: Wed, 24 Jul 2019 15:47:58 GMT
Content-Length: 5298
... standard iis 500 page content...
The OPTIONS request and response from both is practially identical; the only change is the host (port) and timestamps. However, after the preflight, when hosted in IIS it responds with 405 Method Not Allowed and a different header Allow: GET, HEAD, OPTIONS, TRACE.
Is there something I need to add to the web.config for IIS specifically?
I'm just using the Chrome devtools to create an XMLHttpRequest for testing.
I have already removed WebDav from IIS.
The web.config is setup exactly as it would be from the linked tutorial.
I am running Windows 10 and IIS 10.
The only module I have installed is URL Rewrite, but no rules are enabled (and it is not configured in the web.config).
Thanks in advance for any hints as to what is going wrong.
I uninstalled WebDav then rebooted.
It's working.
...
Kill me.
I have a web application developed with Java 7, PrimeFaces 5 (JSF), Tomcat 7. Some time ago a user reported a defect from production environment where everything he clicked caused the page to be refreshed. I couldn't reproduce it then I forgot about it... but right now I just reproduced the issue, for the first time, running the application in my local machine. Any button I click, that triggers an ajax request, will be followed by a page refresh. I tried to restart tomcat and to clear Chrome's cache with CMD+SHIFT+R but I still can reproduce the issue.
I can reproduce on Google Chrome, but not on Firefox and Safari. I opened the network tab from Developer Tools from both Firefox and Chrome to compare the requests... but I still can't understand... does anybody have any idea what may be happening?
I will provide below the network activities for the following close event:
<p:panel id="myPanel" widgetVar="myPanelVar" closable="true">
<p:ajax event="close" listener="#{myBean.changeSomeVariables}" />
<h:outputText value="Example panel"/>
<a onclick="PF('myPanelVar').close();"><span class="ui-icon ui-icon-closethick"></span></a>
</p:panel>
Network activity from Firefox (working fine):
Request URL:http://localhost:8080/myMainPage;jsessionid=3BF25F4A5CF149326D3561C0F2002FB9
Request Method:POST
Status Code:200 OK
Remote Address:[::1]:8080
REQUEST HEADERS:
Host: localhost:8080
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: application/xml, text/xml, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Faces-Request: partial/ajax
X-Requested-With: XMLHttpRequest
Referer: http://localhost:8080/myMainPage
Content-Length: 309
Connection: keep-alive
RESPONSE HEADERS:
Cache-Control: no-cache, no-store, must-revalidate, no-cache
Content-Length: 335
Content-Type: text/xml;charset=UTF-8
Date: Sun, 21 Aug 2016 03:02:14 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Server: Apache-Coyote/1.1
FORM DATA:
javax.faces.partial.ajax=true&javax.faces.source=myPanel&javax.faces.partial.execute=myPanel&javax.faces.behavior.event=close&javax.faces.partial.event=close&myHeaderForm=myHeaderForm&myHeaderForm%3Aemail=userTest&myHeaderForm%3Apassword=passTest&javax.faces.ViewState=3276447570579140768:319272761245257759
Network activity from Chrome (not working fine, I don't understand where the GET request is coming from):
Request URL:http://localhost:8080/myMainPage
Request Method:POST
Status Code:200 OK
Remote Address:[::1]:8080
REQUEST HEADERS:
POST /myMainPage HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 325
Origin: http://localhost:8080
Faces-Request: partial/ajax
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: application/xml, text/xml, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Referer: http://localhost:8080/myMainPage
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,pt;q=0.6
Cookie: JSESSIONID=5C1AE4DB6B1B5148BD81FD0E47002EA4
RESPONSE HEADERS:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=B68E094F6CF3C5AD57D96F0D1823AEBD; Domain=.mySite.com; Path=/; HttpOnly
Cache-Control: no-cache
Content-Type: text/xml;charset=UTF-8
Content-Length: 76
Date: Sun, 21 Aug 2016 03:05:36 GMT
FORM DATA:
javax.faces.partial.ajax=true&javax.faces.source=myPanel&javax.faces.partial.execute=myPanel&javax.faces.behavior.event=close&javax.faces.partial.event=close&myHeaderForm=myHeaderForm&myHeaderForm%3Aemail=userTest&myHeaderForm%3Apassword=passTest&javax.faces.ViewState=8492705007558548241%3A8154221161100277529
__________________________________________________________
Request URL:http://localhost:8080/myMainPage
Request Method:GET
Status Code:200 OK
Remote Address:[::1]:8080
REQUEST HEADERS:
GET /myMainPage HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: http://localhost:8080/myMainPage
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,pt;q=0.6
Cookie: JSESSIONID=5C1AE4DB6B1B5148BD81FD0E47002EA4
RESPONSE HEADERS:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=D352B9B5644288F3095D1A702075AFEB; Domain=.mySite.com; Path=/; HttpOnly
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Content-Encoding: gzip
Vary: Accept-Encoding
Date: Sun, 21 Aug 2016 03:05:36 GMT
Thanks
I am using nginx and Adaptive Images to deliver dynamically sized images based on device resolution. The response headers being set by the adaptive-images.php file are shown below but every time I refresh the page the browser requests the images again. Why is the browser not caching these images? The browser is Google Chrome and it seems to be setting max-age=0 in the request headers no matter how I refresh. I've tried F5, Ctrl+F5 and entering the URL in the address bar and hitting Enter.
Request Headers:
GET /img/photos/p8.jpg HTTP/1.1
Host: example.com
Connection: keep-alive
Cache-Control: max-age=0
Accept: image/webp,*/*;q=0.8
Pragma: no-cache
If-Modified-Since: Wed, 16 Jul 2014 12:01:31 GMT
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36
Referer: http://example.com/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,en-GB;q=0.6
Response Headers:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 16 Jul 2014 12:08:55 GMT
Content-Type: image/jpeg
Content-Length: 391104
X-Powered-By: PHP/5.4.30
Cache-Control: private, max-age=604800
Expires: Wed, 23 Jul 2014 12:08:55 GMT
Last-Modified: Wed, 16 Jul 2014 12:08:55 GMT
Connection: Keep-Alive
It turns out that this seems to be a Chrome feature
See this other SO answer for why: Chrome doesn't cache images/js/css
Don't use control shift r when testing this (reload)
I've implemented a web application that takes advantage of CORS to gather JSON data from another server. The servers run on different subdomains. Everything seems implemented correctly, and it works fine with Chromium. Below is a copy of my requests, from Chromium.
My problem is that in Firefox (tested with 13.0.1), no request is ever made for my AJAX resource. No preflight request is ever sent, and no actual request is made. Instead, I get this error, from the XMLHttpRequest.send() function:
[21:40:27.546] uncaught exception: [Exception... "Access to restricted URI denied" code: "1012" nsresult: "0x805303f4 (NS_ERROR_DOM_BAD_URI)" location: "http://192.168.1.99:2502/static/mootools-core-1.4.5.js Line: 5398"]
I am using Mootools' Request.JSON object, which sets various extra headers, meaning that a preflight would indeed be required. However, it is never sent.
Unfortunately, JSONP is not an option, as the data is sensitive.
Does anyone have insight into what the problem could be?
Thanks very much.
Working example, from Chromium:
Preflight request:
OPTIONS /api/resource HTTP/1.1
Host: dev0.mydomain.com
Connection: keep-alive
Access-Control-Request-Method: GET
Origin: http://192.168.1.99:2502
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Ubuntu/12.04 Chromium/18.0.1025.151 Chrome/18.0.1025.151 Safari/535.19
Access-Control-Request-Headers: origin, x-request, x-requested-with, accept
Accept: */*
Referer: http://192.168.1.99:2502/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: [redacted]
Preflight response:
HTTP/1.0 200 OK
Server: PasteWSGIServer/0.5 Python/2.7.3
Date: Fri, 29 Jun 2012 01:43:37 GMT
Content-Length: 0
Access-Control-Allow-Headers: Cookie, Origin, X-Request, X-Requested-With, Accept
Access-Control-Max-Age: 1
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://192.168.1.99:2502
Access-Control-Allow-Methods: GET
Content-Type: text/html; charset=UTF-8
"Real" request:
GET /api/resource HTTP/1.1
Host: dev0.mydomain.com
Connection: keep-alive
Origin: http://192.168.1.99:2502
X-Request: JSON
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Ubuntu/12.04 Chromium/18.0.1025.151 Chrome/18.0.1025.151 Safari/535.19
Accept: application/json
Referer: http://192.168.1.99:2502/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: [redacted]
"Real" response:
HTTP/1.0 200 OK
Server: PasteWSGIServer/0.5 Python/2.7.3
Date: Fri, 29 Jun 2012 01:43:37 GMT
Access-Control-Allow-Origin: http://192.168.1.99:2502
Content-Type: text/html; charset=UTF-8
Content-Length: 22
Access-Control-Allow-Credentials: true
The answer is given in the comments to the question. Firefox was not sending the request due to the HTTP authentication username I had provided.