Executable downloaded from Cloudfront in IE11 (Windows 7) downloads without a file extension - windows

The long-short of it is an .exe downloaded from Cloudfront (using signed URLs) in IE11/Win7 downloads without an extension (exe_file.exe -> exe_file)
I don't think that it's the same issue as described here (among many, many other places) as the file is not renamed exe_file_exe, the extension is just dropped.
The file is being served from Cloudfront from S3 - and was uploaded via aws-cli
$ aws s3 cp exe_file.exe s3://cdn/exe_file.exe --content-type "application/x-msdownload"
as far as I'm aware the content-type argument isn't absolutely necessary as CF/S3/something, at some point, tries to do some intelligent MIME assigning (plus, before, when I was uploading without that arg, inspecting the download headers would show the correct MIME type).
Headers received when downloading the file
HTTP/1.1 200 OK
Content-Type: application/x-msdownload
Content-Length: 69538768
Connection: keep-alive
Date: Tue, 27 Dec 2016 17:36:51 GMT
Last-Modified: Thu, 22 Dec 2016 22:31:59 GMT
ETag: "c8fc68a920e198dca95e5549f8657bfb"
Accept-Ranges: bytes
Server: AmazonS3
Age: 335
X-Cache: Hit from cloudfront
This only happens in IE11 on Windows 7 - it works fine on IE11/Windows 10 (I say only but I have not tried on, for example, IE8 - you couldn't pay me enough money to put myself through that). And it does not happen with other downloads - dmg_file.dmg and linux_file.zip are both downloaded with the extension. Other browsers are also not impacted - they all download the file as-is in S3.
I have tried with and without AVs present - it does not make a difference.

You need to set the content-disposition correctly:
Forcing SaveAs using the HTTP header
In order to force the browser to show SaveAs dialog when clicking a hyperlink you have to include the following header in HTTP response of the file to be downloaded:
Content-Disposition: attachment; filename="<file name.ext>"; filename*=utf-8''<file name.ext>
Note: Those user agents that do not support the RFC 5987 encoding ignore filename* when it occurs after filename.
Where is the filename you want to appear in SaveAs dialog (like finances.xls or mortgage.pdf) - without < and > symbols.
You have to keep the following in mind:
The filename should be in US-ASCII charset and shouldn't contain special characters: < > \ " / : | ? * space.
The filename should not have any directory path information specified.
The filename should be enclosed in double quotes but most browsers will support file names without double quotes.
Ancient browsers also required the following (not needed nowadays, but for a fool proof solution might be worth doing):
Content-Type header should be before Content-Disposition.
Content-Type header should refer to an unknown MIME type (at least until the older browsers go away).
So, you should use cp with options:
--content-type (string) Specify an explicit content type for this operation. This value overrides any guessed mime types.
--content-disposition (string) Specifies presentational information for the object.
--metadata-directive REPLACE Specifies whether the metadata is copied from the source object or replaced with metadata provided when copying S3 objects.
Note that if you are using any of the following parameters: --content-type, content-language, --content-encoding, --content-disposition, --cache-control, or --expires, you will need to specify --metadata-directive REPLACE for non-multipart copies if you want the copied objects to have the specified metadata values.
try:
aws s3 cp exe_file.exe s3://cdn/exe_file.exe --content-type "application/x-msdownload" --content-disposition "attachment; filename=\"exe_file.exe\"; filename*=utf-8''exe_file.exe" --metadata-directive REPLACE

In addition to the accepted answer, I supplied my own response-content-disposition parameter to the Cloudfront Signer:
in Python, it looked like
from botocore.signers import CloudFrontSigner
def generate_presigned_url(filename, headers={}):
cf_signer = CloudFrontSigner(CF_KEY_ID, rsa_signer)
headers = '&'.join(["%s=%s" % (key, urllib.quote_plus(value)) for key, value in headers.iteritems()])
return cf_signer.generate_presigned_url(
'https://' + CF_DOMAIN + '/' + filename + ("" if len(headers) == 0 else "?%s" % (headers)),
# ... other params
)
called using
cloudfront.generate_presigned_url(file_name, {
'response-content-disposition': 'attachment; filename="exe_file.exe"; filename*=utf-8\'\'exe_file.exe'
})

Related

How can I output the full web-page specific content with Powershell wget?

When I issue the command powershell.exe wget http://IP_ADDR:8080/config/version/ I get the response:
StatusCode : 200
StatusDescription : OK
Content : {"Content of the web page does not fully show through this output… How can I use wget to just show the full content of the webpage without cutting it out?
RawContent : HTTP/1.1 200 OK
Content-Length: 307
Content-Type: application/json
Date: Wed, 26 Apr 2017 04:20:04 GMT
Server: CherryPy/3.2.2
{"Content of the web page does not fully show...
Forms : {}
Headers : {[Content-Length, 307], [Content-Type, application/json], [Date, Wed, 26 Apr 2017 04:20:04 GMT],
[Server, CherryPy/3.2.2]}
Images : {}
InputFields : {}
Links : {}
ParsedHtml : mshtml.HTMLDocumentClass
RawContentLength : 307
As above, the command shows only part of the content, and not the full output of what is returned when I go to the address normally.
My other alternative is to just use curL, but I would like a native resolution to the above other than a third-party tool.
My question is, how can I use the wget command to only show the output of the content, and the full output of it?
Per the comments, you need to access either the .Content or .RawContent property (RawContent contains the HTTP header fields where as Content does not. Note that the Headers are also available in the .Headers property):
powershell.exe (wget http://IP_ADDR:8080/config/version/).Content
or
powershell.exe (wget http://IP_ADDR:8080/config/version/).RawContent
To explain what is occurring, PowerShell returns Objects rather than plain text which to put simply means they are like mini databases with properties that can be returned/filtered etc. What you see when you make your call is the default view of a subset of properties of the object, which is not all of the properties available.
To learn more about wget look up Invoke-WebRequest which is the full cmdlet name (wget is an alias of it).
As a further aside, if your web call is returning JSON or XML you might want to consider using Invoke-RestMethod instead as that will take the JSON or XML and convert it automatically a PowerShell object (which you could then further manipulate within PowerShell).

How to get custom header in bash

I'm adding a custom header in Asp.Net app:
context.Response.Headers.Add("X-Date", DateTime.Now.ToString());
context.Response.Redirect(redirectUrl, false);
When I'm using Fiddler I can see the "X-Date" header in the response.
I need to receive it by using bash.
I tried curl -i https://my.site.com and also wget -O - -o /dev/null --save-headers https://my.site.com with no success.
In both cases I see just the regular headers like: Content-Type, Server, Date, etc...
How I can receive the "X-Date" header?
Thanks,
Lev
protocol headers are different than file-headers (like http-header and tcp-header are different). When you create a protocol header you wiil need a server to resolve it and use the associated enviroment variables. Example ...
#!/bin/bash
# Apache - CGI
echo "text/plain"
echo ""
echo "$CONTENT_TYPE"
echo "$HTTP_ACCEPT"
echo "$SERVER_PROTOCOL"
When calling this script via web, The response ony my browser was...
text/html
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP/1.1
What you looking for are enviorment variables called $HTTP_ACCEPT, $CONTENT_TYPE and maybe $SERVER_PROTOCOL too.

Why would a web server reply with 301 and the exact location that was requested?

I'm trying to retrieve pages from a web server via https, using lua with luasec. For most pages my script works as intended, but if the ressource contains special characters (like ,'é), I'm being sent into a loop with 301 responses.
let this code sniplet illustrate my dilemma (actual server details redacted to protect the innocent):
local https = require "ssl.https"
local prefix = "https://www.example.com"
local suffix = "/S%C3%A9ance"
local body,code,headers,status = https.request(prefix .. suffix)
print(status .. " - GET was for \"" .. prefix .. suffix .. "\"")
print("headers are " .. myTostring(headers))
print("body is " .. myTostring(body))
if suffix == headers.location then
print("equal")
else
print("not equal")
end
local body,code,headers,status = https.request(prefix .. headers.location)
print(status .. " - GET was for \"" .. prefix .. suffix .. "\"")
which results in the paradoxical
HTTP/1.1 301 Moved Permanently - GET was for "https://www.example.com/S%C3%A9ance"
headers are { ["content-type"]="text/html; charset=UTF-8";["set-cookie"]="PHPSESSID=e80oo5dkouh8gh0ruit7mj28t6; path=/";["content-length"]="0";["connection"]="close";["date"]="Wed, 15 Mar 2017 19:31:24 GMT";["location"]="S%C3%A9ance";}
body is ""
equal
HTTP/1.1 301 Moved Permanently - GET was for "https://www.example.com/S%C3%A9ance"
How might one be able to retrieve the elusive pages, using lua and as little additional dependencies as possible?
Obvious as it may seem, perhaps the requested url does differ from the actual location.
If you have a similar problem, do check deep within your external libraries to make sure they do what you think they do.
In this case, luasocket did urldecode and then urlencode the url and thus the final request was not what it seemed to be.

JMeter Not Sending File with HTTP Request

I'm new to JMeter and trying to put a file to our API using an HTTP Request. When I put the file via curl using the -F flag, it works no problem.
Here's my curl request:
curl -X PUT -u uname:pword https://fakehostname.com/psr-1/controllers/vertx/upload/file/big/ADJTIME3 -F "upload1=#ADJTIME" -vis
and here's the relevant part of the response from the server:
> User-Agent: curl/7.37.1 Host: myfakehost.com Accept: */*
> Content-Length: 4190 Expect: 100-continue Content-Type:
> multipart/form-data; boundary=------------------------d76566a6ebb651d3
When I do the same put via JMeter, the Content-Length is 0 which makes me think that JMeter isn't reading the file for some reason. I know the path is correct because I browsed to the file from JMeter. Litte help?
In File Upload, make your file path RELATIVE to .jmx file or place next to it and specify file name only.
Thanks to everyone who offered solutions and suggestions. It turns out that the API I was trying to load test was the issue. I can PUT a file via curl no problem, but there's something about the Jmeter PUT that the API does not like. I finally tried doing a PUT to an unrelated API and was successful.

How to properly handle a gzipped page when using curl?

I wrote a bash script that gets output from a website using curl and does a bunch of string manipulation on the html output. The problem is when I run it against a site that is returning its output gzipped. Going to the site in a browser works fine.
When I run curl by hand, I get gzipped output:
$ curl "http://example.com"
Here's the header from that particular site:
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; charset=utf-8
X-Powered-By: PHP/5.2.17
Last-Modified: Sat, 03 Dec 2011 00:07:57 GMT
ETag: "6c38e1154f32dbd9ba211db8ad189b27"
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: must-revalidate
Content-Encoding: gzip
Content-Length: 7796
Date: Sat, 03 Dec 2011 00:46:22 GMT
X-Varnish: 1509870407 1509810501
Age: 504
Via: 1.1 varnish
Connection: keep-alive
X-Cache-Svr: p2137050.pubip.peer1.net
X-Cache: HIT
X-Cache-Hits: 425
I know the returned data is gzipped, because this returns html, as expected:
$ curl "http://example.com" | gunzip
I don't want to pipe the output through gunzip, because the script works as-is on other sites, and piping through gzip would break that functionality.
What I've tried
changing the user-agent (I tried the same string my browser sends, "Mozilla/4.0", etc)
man curl
google search
searching stackoverflow
Everything came up empty
Any ideas?
curl will automatically decompress the response if you set the --compressed flag:
curl --compressed "http://example.com"
--compressed
(HTTP) Request a compressed response using one of the algorithms libcurl supports, and save the uncompressed document. If this option is used and the server sends an unsupported encoding, curl will report an error.
gzip is most likely supported, but you can check this by running curl -V and looking for libz somewhere in the "Features" line:
$ curl -V
...
Protocols: ...
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
Note that it's really the website in question that is at fault here. If curl did not pass an Accept-Encoding: gzip request header, the server should not have sent a compressed response.
In the relevant bug report Raw compressed output when not using --compressed but server returns gzip data #2836 the developers says:
The server shouldn't send content-encoding: gzip without the client having signaled that it is acceptable.
Besides, when you don't use --compressed with curl, you tell the command line tool you rather store the exact stream (compressed or not). I don't see a curl bug here...
So if the server could be sending gzipped content, use --compressed to let curl decompress it automatically.

Resources