I'm trying to retrieve pages from a web server via https, using lua with luasec. For most pages my script works as intended, but if the ressource contains special characters (like ,'é), I'm being sent into a loop with 301 responses.
let this code sniplet illustrate my dilemma (actual server details redacted to protect the innocent):
local https = require "ssl.https"
local prefix = "https://www.example.com"
local suffix = "/S%C3%A9ance"
local body,code,headers,status = https.request(prefix .. suffix)
print(status .. " - GET was for \"" .. prefix .. suffix .. "\"")
print("headers are " .. myTostring(headers))
print("body is " .. myTostring(body))
if suffix == headers.location then
print("equal")
else
print("not equal")
end
local body,code,headers,status = https.request(prefix .. headers.location)
print(status .. " - GET was for \"" .. prefix .. suffix .. "\"")
which results in the paradoxical
HTTP/1.1 301 Moved Permanently - GET was for "https://www.example.com/S%C3%A9ance"
headers are { ["content-type"]="text/html; charset=UTF-8";["set-cookie"]="PHPSESSID=e80oo5dkouh8gh0ruit7mj28t6; path=/";["content-length"]="0";["connection"]="close";["date"]="Wed, 15 Mar 2017 19:31:24 GMT";["location"]="S%C3%A9ance";}
body is ""
equal
HTTP/1.1 301 Moved Permanently - GET was for "https://www.example.com/S%C3%A9ance"
How might one be able to retrieve the elusive pages, using lua and as little additional dependencies as possible?
Obvious as it may seem, perhaps the requested url does differ from the actual location.
If you have a similar problem, do check deep within your external libraries to make sure they do what you think they do.
In this case, luasocket did urldecode and then urlencode the url and thus the final request was not what it seemed to be.
Related
I am newbie in luci and lua api. Now I have problem about remove firewall rule from luci webgui in opnewrt. I know delete the rule is using the uci command : uci delete firewall.#rule[index]
But I don't know the index exectly. So I use blow shell script name index.sh to find the index by key word.
uci show firewall | grep $1 | grep -oE '\[.*?\]' | sed 's/[][]//g' | head -n 1
Here $1 is the mac address.
My process is get the mac address from luci web and remove the firewall rule of block the
network access from the mac address I set before. To recover the network access of the mac address.
But I use the lua api of luci.sys.exec and luci.util.exec to get the output of firewall rule index that are all wrong.
My example code:
1.Get the mac address from luci
local del_mac = protocol.urldecode(luci.http.formvalue("deleteMac"))
2.Execute the the shell script to get the index
local ruleindex = ut.trim(tostring(luci.sys.exec("index.sh '" .. del_mac .."'"))
local uci_delete_rule = "uci delete firewall.#rule['" .. ruleindex .. "]'"
luci.sys.exec(uci_delete_rule)
or
local index = "index.sh %s" %{list_del_mac}
local rule_index = ut.trim(tostring(luci.util.exe(index)))
local uci_delete_rule = "uci delete firewall.#rule['" .. rule_index .. "]'"
luci.sys.exec(uci_delete_rule)
3.Reload the firewall rule
luci.sys.exec("uci commit firewall")
luci.sys.exec("fw3 reload")
Can anyone help me to reslove the problem?
Thank you for your help
This doesn't look right to me: "index.sh %s" %{list_del_mac} Unless it's short for string.format in luci, that's not proper Lua. Concatenate your command string, or actually use string.format
"index.sh " ..list_del_mac
or
string.format( "index.sh %s", list_del_mac )
also, to get the return value after execution, you need to send it to a variable
return_value = luci.sys.exec(uci_delete_rule)
print( return_value )
The long-short of it is an .exe downloaded from Cloudfront (using signed URLs) in IE11/Win7 downloads without an extension (exe_file.exe -> exe_file)
I don't think that it's the same issue as described here (among many, many other places) as the file is not renamed exe_file_exe, the extension is just dropped.
The file is being served from Cloudfront from S3 - and was uploaded via aws-cli
$ aws s3 cp exe_file.exe s3://cdn/exe_file.exe --content-type "application/x-msdownload"
as far as I'm aware the content-type argument isn't absolutely necessary as CF/S3/something, at some point, tries to do some intelligent MIME assigning (plus, before, when I was uploading without that arg, inspecting the download headers would show the correct MIME type).
Headers received when downloading the file
HTTP/1.1 200 OK
Content-Type: application/x-msdownload
Content-Length: 69538768
Connection: keep-alive
Date: Tue, 27 Dec 2016 17:36:51 GMT
Last-Modified: Thu, 22 Dec 2016 22:31:59 GMT
ETag: "c8fc68a920e198dca95e5549f8657bfb"
Accept-Ranges: bytes
Server: AmazonS3
Age: 335
X-Cache: Hit from cloudfront
This only happens in IE11 on Windows 7 - it works fine on IE11/Windows 10 (I say only but I have not tried on, for example, IE8 - you couldn't pay me enough money to put myself through that). And it does not happen with other downloads - dmg_file.dmg and linux_file.zip are both downloaded with the extension. Other browsers are also not impacted - they all download the file as-is in S3.
I have tried with and without AVs present - it does not make a difference.
You need to set the content-disposition correctly:
Forcing SaveAs using the HTTP header
In order to force the browser to show SaveAs dialog when clicking a hyperlink you have to include the following header in HTTP response of the file to be downloaded:
Content-Disposition: attachment; filename="<file name.ext>"; filename*=utf-8''<file name.ext>
Note: Those user agents that do not support the RFC 5987 encoding ignore filename* when it occurs after filename.
Where is the filename you want to appear in SaveAs dialog (like finances.xls or mortgage.pdf) - without < and > symbols.
You have to keep the following in mind:
The filename should be in US-ASCII charset and shouldn't contain special characters: < > \ " / : | ? * space.
The filename should not have any directory path information specified.
The filename should be enclosed in double quotes but most browsers will support file names without double quotes.
Ancient browsers also required the following (not needed nowadays, but for a fool proof solution might be worth doing):
Content-Type header should be before Content-Disposition.
Content-Type header should refer to an unknown MIME type (at least until the older browsers go away).
So, you should use cp with options:
--content-type (string) Specify an explicit content type for this operation. This value overrides any guessed mime types.
--content-disposition (string) Specifies presentational information for the object.
--metadata-directive REPLACE Specifies whether the metadata is copied from the source object or replaced with metadata provided when copying S3 objects.
Note that if you are using any of the following parameters: --content-type, content-language, --content-encoding, --content-disposition, --cache-control, or --expires, you will need to specify --metadata-directive REPLACE for non-multipart copies if you want the copied objects to have the specified metadata values.
try:
aws s3 cp exe_file.exe s3://cdn/exe_file.exe --content-type "application/x-msdownload" --content-disposition "attachment; filename=\"exe_file.exe\"; filename*=utf-8''exe_file.exe" --metadata-directive REPLACE
In addition to the accepted answer, I supplied my own response-content-disposition parameter to the Cloudfront Signer:
in Python, it looked like
from botocore.signers import CloudFrontSigner
def generate_presigned_url(filename, headers={}):
cf_signer = CloudFrontSigner(CF_KEY_ID, rsa_signer)
headers = '&'.join(["%s=%s" % (key, urllib.quote_plus(value)) for key, value in headers.iteritems()])
return cf_signer.generate_presigned_url(
'https://' + CF_DOMAIN + '/' + filename + ("" if len(headers) == 0 else "?%s" % (headers)),
# ... other params
)
called using
cloudfront.generate_presigned_url(file_name, {
'response-content-disposition': 'attachment; filename="exe_file.exe"; filename*=utf-8\'\'exe_file.exe'
})
Recently I needed to check for a huge list of filenames if they exist on a server. I did this by running a for loop which tried to wget each of those files. That was efficient enough, but took about 30 minutes in this case. I wonder if there is a faster way to check whether a file exists or not (since wget is for downloading files and not performing thousands of requests).
I don't know if that information is relevant, but it's an Apache server.
Curl would be the best option in a for loop and here is a straight forward simple way, run this in your forloop
curl -I --silent http://www.yoururl/linktodetect | grep -m 1 -c 404
What this simply does is check the http response header for a 404 returned on the link and if its detected as a missing file/link throwing a 404 then the command line output will display you a number 1; otherwise, if the file/link is valid and does not return a 404 then the command line output will display you a number 0.
I face funny problem: I have an FreeBSD 8.2 server, and I need to set up web server that wil answer zero (0) length answer to any request. Just '200 OK' and zero body.
Ok, I can setup nginx (and already did that) and set 404 error document to /dev/null, but I think maybe there is more optimal and elegant solution? I know there is nginx module that output 1x1 gif, may there be anything like that for zero-length file?
It's possible to return a status code from Nginx:
location /empty {
return 200;
}
NOTE Generally, HTTP status code 204 No Content is meant to say "I've completed the request, but there is no body to return". You may return it in the same fashion:
location /empty {
return 204;
}
You could use netcat like in the example here http://howtoforge.com/useful-uses-of-netcat
E.g.
while true; do echo 'HTTP/1.1 200 OK
Content-Length: 0
Connection: close
' | nc -l 80; done
I've developed an application that connects to the PayPal API with libcurl, which I use through the OCurl bindings for OCaml from a process running on a Debian server. The code always works when inside the Paypal sandbox (endpoint https://api-3t.sandbox.paypal.com/nvp) but never works when connecting to the actual Paypal servers (endpoint https://api-3t.paypal.com/nvp).
Libcurl always returns CURLE_RECV_ERROR. The general consensus is that this error happens when there is a network problem, so I have investigated that.
I ran the exact same request with the command-line curl tool from the exact same server, using the exact same process uid/gid, and it consistently works. Tracking the transfers with tcpdump does not reveal any significant difference in the structure of transactions made by the working command-line curl and the non-working application, so it all appears as if the HTTP request is successfully performed in both cases. Then again, it's HTTPS, so I cannot be certain.
Here is the OCaml code that performs the request:
let c = new Curl.handle in
let buffer = Buffer.create 1763 in
c # set_url "https://api-3t.paypal.com/nvp" ;
c # set_post true ;
c # set_postfields "SOMEDATA" ;
c # set_postfieldsize (String.length "SOMEDATA") ;
c # set_writefunction (fun x -> Buffer.add_string buffer x ; String.length x) ;
c # perform ;
c # cleanup ;
Buffer.contents buffer
Here is the equivalent curl command line:
curl -X POST https://api-3t.paypal.com/nvp -d SOMEDATA
EDIT: By increasing the libcurl verbosity, I determined that the underlying error is this:
GnuTLS recv error (-9): A TLS packet with unexpected length was received.
What could be the cause of this error? How can I investigate to find out?
EDIT 2: It appears that the difference between command-line and library use is that the command-line version is linked to OpenSSL and the library is linked to GnuTLS.
How do I link both to OpenSSL?
First, the key to further debugging those issues is to use curl's debugging facilities, namely the VERBOSE setting (and also, possibly, the DEBUGFUNCTION setting for printing the data your way).
c # set_verbose true ;
This identified the error as being a problem with GnuTLS that is also discussed here, and which is solved by setting SSLVERSION to 3 to force the use of SSLv3.
c # set_sslversion 3 ;
I always call curl#set_postfieldsize to the length of the data passed to curl#set_postfields. My code would be, then:
let make_get url =
let curl = new Curl.handle in
curl#set_writefunction String.length; (* ignore result *)
curl#set_tcpnodelay true;
curl#set_verbose false;
curl#set_post false;
curl#set_url url;
curl
let make_post url =
let curl = make_get url in
curl#set_post true;
curl#set_httpheader [
"Content-Type: text/xml; charset=\"UTF-8\"";
"SOAPAction: \"\"";
];
curl#set_postfields xml;
curl#set_postfieldsize (String.length xml);
curl
I hope this helps.