I have a problem with code (this code may to get request & download generated file):
require 'rubygems'
require 'mechanize'
require 'hpricot'
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
user = "xxx"
pass = "xxx"
auth_key = "xxx"
ip = "xxx"
agent = Mechanize.new
page = agent.get("https://#{ip}/cgi-bin/welcome.cgi")
form = page.forms.first
form.username = user
form.password = Digest::MD5.hexdigest(pass + auth_key)
page = agent.submit form
page = agent.get("https://#{ip}/cgi-bin/config.exp")
form = page.forms.first
agent.pluggable_parser.default = Mechanize::FileSaver
agent.post("https://#{ip}/cgi-bin/config.exp", {"submitstatus" => "1"})
With this I have a error:
/var/lib/gems/1.8/gems/mechanize-2.4/lib/mechanize/http/agent.rb:291:in `fetch': 400 => Net::HTTPBadRequest for https://31.223.225.133/cgi-bin/config.exp -- unhandled response (Mechanize::ResponseCodeError)
from /var/lib/gems/1.8/gems/mechanize-2.4/lib/mechanize.rb:407:in `get'
from /home/lord/Dropbox/work/ruby/ruby_backup/backup.ru:22
How to simulate this wget request for downloading file:
`wget --no-check-certificate --load-cookies cookie --post-data='submitstatus=1' \
--header='Host: 10.1.25.254' \
--header='User-Agent: Mozilla/5.0' \
--header='Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
--header='Accept-Language: ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3' \
--header='Accept-Encoding: gzip, deflate' \
--header='Connection: keep-alive' \
--header='Referer: https://10.1.25.254/sys_setting.htm' \
--header='Content-Type: application/x-www-form-urlencoded' \
--header='Content-Length: 14' \
https://$ip/cgi-bin/config.exp`
advance thanks
If you proxy your mechanize and wget requests through a debugging proxy such as fiddler or charles you can compare them side by side and thereby better understand the problem.
Have you tried Mechanize::Download?
Here is a reference
Using WWW:Mechanize to download a file to disk without loading it all in memory first
Related
In short: I'm trying to get that page source of https://www.etoro.com/app/sv-iframe using curl in Bash.
I understand this ask is quite "simple". I have read thru 10+ similar questions here. Unfortunately, none of them could solve my problem.
When you open the URL above in Chrome browser, it's blank. You can either right click -> View Page Source, or sniff network using Chrome Developer Tool. Both will give you the correct page source. The page contains javascripts, in which there is a long hex string - what I need ultimately. I tried disabling javascript and reloading the page. I still got the right page source. So javascript doesn't play trick here. It sounds getting such page source via curl should be just straight forward, right?
When I right click the request in Chrome Developer Tool -> Copy as cURL, and execute it in terminal, things turned nasty - I got a CloudFlare security check page. I reopened the page several times in Chrome Incognito mode. I swear never saw a CloudFlare security check in browser. I double checked the cURL command. It has user-agent set as well.
Here is what I tried so far:
Manually compose curl command and fill headers from Chrome Developer Tool
Sniff packages on an Android device, and use headers set on mobile browser
Post request online from Postman Web
All gave me the same CloudFlare security check page.
The CloudFlare page says "Please enable cookies". I suspect if server in this way determined I was not calling from a browser. Following some threads, I tried to set -b/-c/-j flag with curl. Also no luck.
Here's more detailed steps what I've done:
Open Chrome Incognito mode
Open Developer Tool
Use Command+Shift+P (Mac) to open command menu
Type "disable javascript" and hit enter
Switch to Network tab
Open https://www.etoro.com/app/sv-iframe
Observe the request list - there should be only 1 request (request screenshot 1 / request screenshot 2 / response body / response cookie)
Right click on the request -> Copy as cURL
Here's my curl command:
curl 'https://www.etoro.com/app/sv-iframe' \
-H 'authority: www.etoro.com' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'upgrade-insecure-requests: 1' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36' \
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'sec-fetch-site: none' \
-H 'sec-fetch-mode: navigate' \
-H 'sec-fetch-user: ?1' \
-H 'sec-fetch-dest: document' \
-H 'accept-language: en-US,en;q=0.9' \
--compressed
The request itself I don't think it requires cookie, as page was able to be opened in Incognito mode. I anyways tried to set the response cookie together with the request. It doesn't help either.
-H 'cookie: __cfduid=d2edf...; TS01047baf=01d53...; __cf_bm=a3803...; __cflb=02Di3...'
Already spent whole evening on it but couldn't get it resolved. I appreciate any suggestions or help to get me thru it. I have a feeling that the actual fix would be fairly simple. The request has no cookie. Only thing to update is header. Maybe I didn't have correct header specified? Or some extra curl flag would help?
There is some obfuscated js eval code on that page, that is basically setting cookies or sending logs, digging a bit deeper, this is what ended up with:
(function() {
var s = '9a7xxx......';
function setCookie(cname, cvalue, domain, exdays) {
var d = new Date();
d.setTime(d.getTime() + (exdays * 1000 * 60 * 60 * 24));
var expires = "expires=" + d.toUTCString();
var cookie = cname + "=" + cvalue;
if (domain) {
cookie += ";" + "domain=" + domain;
}
cookie += ";" + expires + ";path=/";
document.cookie = cookie;
}
function deleteCookie(cname, domain) {
setCookie(cname, "", domain, 0);
}
var ta = ["window.callPhantom", "window.__nightmare", "window._phantom", "window.__webdriver_script_fn", "navigator.webdriver", "document.$cdc_asdjflasutopfhvcZLmcfl_"];
var re;
try {
re = [!!window.callPhantom, !!window.__nightmare, !!window._phantom, !!window.__webdriver_script_fn, !!navigator.webdriver, !!document.$cdc_asdjflasutopfhvcZLmcfl_];
} catch (err) {}
if (re && re.indexOf(true) == -1) {
setCookie("TMIS2", s, ".etoro.com", 14);
} else {
var resultsObj = {};
for (var i = 0; i < ta.length; i++) {
resultsObj[ta[i]] = re[i];
}
var img = new Image();
img.src = 'https://etorologsapi.etoro.com/api/v2/monitoring?applicationIdentifier=JSCClient&LogEvents=' + encodeURIComponent(JSON.stringify([{
ApplicationIdentifier: 'JSCClient',
ApplicationVersion: '0.0.11',
Level: "error",
Message: "ClientSel",
Results: resultsObj,
Type: 'log'
}]));
}
})();
We are developing a WP8 app that requires push notifications.
To test it we have run the push notification POST request with CURL command line, making sure that it actually connects, authenticates with the client SSL certificate and sends the correct data. We know for a fact that this work as we are receiving pushes to the devices.
This is the CURL command we have been using for testing purposes:
curl --cert client_cert.pem -v -H "Content-Type:text/xml" -H "X-WindowsPhone-Target:Toast" -H "X-NotificationClass:2" -X POST -d "<?xml version='1.0' encoding='utf-8'?><wp:Notification xmlns:wp='WPNotification'><wp:Toast><wp:Text1>My title</wp:Text1><wp:Text2>My subtitle</wp:Text2></wp:Toast></wp:Notification>" https://db3.notify.live.net/unthrottledthirdparty/01.00/AAF9MBULkDV0Tpyj24I3bzE3AgAAAAADCQAAAAQUZm52OkE1OUZCRDkzM0MyREY1RkE
Of course our SSL cert is needed to actually use the URL, but I was hoping someone else has done this and can see what we are doing wrong.
Now, our problem is that we need to make this work with Ruby instead, something we have been unable to get to work so far.
We have tried using HTTParty with no luck, and also net/http directly without any luck.
Here is a very simple HTTParty test script I have used to test with:
require "httparty"
payload = "<?xml version='1.0' encoding='utf-8'?><wp:Notification xmlns:wp='WPNotification'><wp:Toast><wp:Text1>My title</wp:Text1><wp:Text2>My subtitle</wp:Text2></wp:Toast></wp:Notification>"
uri = "https://db3.notify.live.net/unthrottledthirdparty/01.00/AAF9MBULkDV0Tpyj24I3bzE3AgAAAAADCQAAAAQUZm52OkE1OUZCRDkzM0MyREY1RkE"
opts = {
body: payload,
headers: {
"Content-Type" => "text/xml",
"X-WindowsPhone-Target" => "Toast",
"X-NotificationClass" => "2"
},
debug_output: $stderr,
pem: File.read("/Users/kenny/Desktop/client_cert.pem"),
ca_file: File.read('/usr/local/opt/curl-ca-bundle/share/ca-bundle.crt')
}
resp = HTTParty.post uri, opts
puts resp.code
This seems to connect with SSL properly, but then the MS IIS server returns 403 to us for some reason we don't get.
Here is essentially the same thing I've tried using net/http:
require "net/http"
url = URI.parse "https://db3.notify.live.net/unthrottledthirdparty/01.00/AAF9MBULkDV0Tpyj24I3bzE3AgAAAAADCQAAAAQUZm52OkE1OUZCRDkzM0MyREY1RkE"
payload = "<?xml version='1.0' encoding='utf-8'?><wp:Notification xmlns:wp='WPNotification'><wp:Toast><wp:Text1>My title</wp:Text1><wp:Text2>My subtitle</wp:Text2></wp:Toast></wp:Notification>"
pem_path = "./client_cert.pem"
cert = File.read pem_path
http = Net::HTTP.new url.host, url.port
http.use_ssl = true
http.cert = OpenSSL::X509::Certificate.new cert
http.key = OpenSSL::PKey::RSA.new cert
http.ca_path = '/etc/ssl/certs' if File.exists?('/etc/ssl/certs') # Ubuntu
http.ca_file = '/usr/local/opt/curl-ca-bundle/share/ca-bundle.crt' if File.exists?('/usr/local/opt/curl-ca-bundle/share/ca-bundle.crt') # Mac OS X
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
r = Net::HTTP::Post.new url.path
r.body = payload
r.content_type = "text/xml"
r["X-WindowsPhone-Target"] = "toast"
r["X-NotificationClass"] = "2"
http.start do
resp = http.request r
puts resp.code, resp.body
end
Like the HTTParty version, this also returns 403..
I'm starting to get the feeling that this won't actually work with net/http, but I've also seen a few examples of code claiming to work, but I can't see any difference compared to what we have tested with here.
Does anyone know how to fix this? Is it possible? Should I use libcurl instead perhaps? Or even do a system call to curl? (I may have to do the last one as an interim solution if we can't get this to work soon).
Any input is greatly appreciated!
Thanks,
Kenny
Try using some tool like http://mitmproxy.org to compare requests from your code and curl.
For example curl in addition to specified headers does send User-Agent and Accept-headers, microsoft servers may be checking for these for some reason.
If this does not help - then it's ssl-related
I've been trying to use several libraries to make an HTTP HEAD request, but nothing seems to be working.
I've seen some examples, but nothing quite what I want.
Here's the Curl request, now I have to do it in ruby:
curl -XHEAD -H x-auth-user: myusername -H x-auth-key: mykey "url"
Also, this is an HTTPS url, if that makes a difference.
Try this:
require 'net/http'
url = 'http://...'
myusename = '...'
mykey = '...'
request = Net::HTTP.new(url, 80)
request.request_head('/', 'x-auth-user' => myusername, 'x-auth-key' => my_key)
I'm trying to do some digest authorization to a server, then parse the resulting HTML with nokogiri. I'm using the net-http-digest_auth gem (https://github.com/drbrain/net-http-digest_auth) to do the url connection. All is fine up until I start the digest_auth code (line 20); it throws an 'unknown algorithm ""MD5"" error'..
The full error message from the console:
~/.rvm/gems/ruby-1.9.3-p194#rails32/gems/net-http-digest_auth-1.2.1/lib/net/http/digest_auth.rb:105:in 'auth_header': unknown algorithm ""MD5"" (Net::HTTP::DigestAuth::Error)
from ./server_connection.rb:20:in '<main>'
Line 20 is the auth line:
auth = digest_auth.auth_header uri, res['www-authenticate'], 'GET'
Here's my complete code (almost completely verbatim from the sample code used at the github link):
#!/usr/bin/env ruby
require 'uri'
require 'net/http'
require 'net/http/digest_auth'
digest_auth = Net::HTTP::DigestAuth.new
uri = URI.parse 'http://url/controlpage?name=_internal_variables_&asList=1&useJS=True'
uri.user = 'username'
uri.password = 'password'
h = Net::HTTP.new uri.host, uri.port
req = Net::HTTP::Get.new uri.request_uri
res = h.request req
# res is a 401 response with a WWW-Authenticate header
auth = digest_auth.auth_header uri, res['www-authenticate'], 'GET'
# create a new request with the Authorization header
req = Net::HTTP::Get.new uri.request_uri
req.add_field 'Authorization', auth
# re-issue request with Authorization
res = h.request req
if res.code == "200"
page = Nokogiri::HTML(res)
isDaylight = page.css('.controlTitle:contains("isDaylight") ~ .controlValue');
puts isDaylight.content
end
Updated this question to include the request headers via Chrome's dev tools:
GET /_getupdatedcontrols?name=_internal_variables_&asList=True&folderFilter=0&changeCount=479&serverState=idle HTTP/1.1
Host: url
Connection: keep-alive
Cache-Control: no-cache
Authorization: Digest username="username", realm="Indigo Control Server", nonce="71079e9f29f7210325ae451d0f423f07", uri="/_getupdatedcontrols?name=_internal_variables_&asList=True&folderFilter=0&changeCount=479&serverState=idle", algorithm=MD5, response="bc056cc472d35f7967973cb51c5b1a65", qop=auth, nc=00005649, cnonce="18dfcf3e4a7b809d"
X-Indigo-Web-Server-Version: 1
X-Prototype-Version: 1.6.0.3
X-Requested-With: XMLHttpRequest
Pragma: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.17 Safari/536.11
Accept: text/javascript, text/html, application/xml, text/xml, */*
Referer: http://url/controlpage?name=_internal_variables_&asList=1&useJS=True
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
I ended up using the httpclient gem to accomplish the same thing.
The final code to do exactly what i was after:
#!/usr/bin/env ruby
require 'httpclient'
require 'nokogiri'
c = HTTPClient.new
c.debug_dev = STDOUT
c.set_auth("http://domain.com", "username", "password")
doc = Nokogiri::HTML(c.get_content("http://domain.com"))
isDaylight = "";
doc.css('.controlTitle:contains("isDaylight") ~ .controlValue').each do |var|
isDaylight = var.content
end
if (!isDaylight)
system("curl -X PUT --digest -u username:password -d isOn=1 http://domain.com")
else
system("curl -X PUT --digest -u username:password -d isOn=0 http://domain.com")
end
I hope this helps others that may be working with a home automation server and needing to easily do digest-based authentication.
Seth,
I ran into this same issue while working on a script in ruby. I am new to ruby but after a few google searches and some Charles Proxy showing me what was going on, I see that it is common for HTTP implementations to include quotes in the algorithm="MD5" portion of the Auth header, which is incorrect according to spec (it should be algorithm=MD5, with out quotes). Your updated header logs from Chrome devtools shows that your server response is honoring the spec, but the ruby library is NOT when it interprets that response string. This can be seen by
You server's 401 response included:
note the algorithm=MD5
Authorization: Digest username="username", realm="Indigo Control Server", nonce="71079e9f29f7210325ae451d0f423f07", uri="/_getupdatedcontrols?name=_internal_variables_&asList=True&folderFilter=0&changeCount=479&serverState=idle", algorithm=MD5, response="bc056cc472d35f7967973cb51c5b1a65", qop=auth, nc=00005649, cnonce="18dfcf3e4a7b809d"
But the console output of the initial request using this Ruby library shows:
note the algorithm=\"MD5\"
<- "GET /some/request HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Ruby\r\nConnection: close\r\nHost: 10.1.0.15\r\n\r\n"
-> "HTTP/1.1 401 Unauthorized\r\n"
-> "Content-Length: 530\r\n"
-> "Server: SomeServer/5.0\r\n"
-> "Allow: GET, HEAD, POST, PUT\r\n"
-> "Date: Sun, 27 Jan 2013 00:29:23 GMT\r\n"
-> "Content-Type: text/html;charset=utf-8\r\n"
-> "Www-Authenticate: Digest realm=\"Some Realm\", nonce=\"5a8b8b46cfb84466431baf454eb9ddb9\", algorithm=\"MD5\", qop=\"auth\"\r\n"
For the script example in the original post, I would insert the following two lines:
www_auth_response = res['www-authenticate']
www_auth_response["algorithm=\"MD5\""] = "algorithm=MD5"
And Modify the third line:
auth = digest_auth.auth_header uri, www_auth_response, 'GET'
As follows:
...
res = h.request req
# res is a 401 response with a WWW-Authenticate header
www_auth_response = res['www-authenticate']
www_auth_response["algorithm=\"MD5\""] = "algorithm=MD5"
auth = digest_auth.auth_header uri, www_auth_response, 'GET'
# create a new request with the Authorization header
req = Net::HTTP::Get.new uri.request_uri
req.add_field 'Authorization', auth
...
The important thing that is going on here is that we are modifying the www-authenticate string that is coming back from your initial unauthorized 401 request (as interpreted by this ruby library). Sending the modified header string (www_auth_response) to the digest_auth.auth_header method produces no errors. At least that worked for me in my script!
I hope that helps!
Matt
I'm doing a Post request to github at this url:
https://api.github.com/gists/2710948/comments
Theoretically, this should create a comment with the text being formed from what's in the request body. However, when I try to make that post, I get a 404 error. That leads me to believe that the gist is not being found, however, if you do a Get request at the same address it comes up just fine.
Is there an authentication thing I need to be doing? I've tried adding a username and password to my headers collection but I've got no idea if I'm using the right format. I've tried making this work via Ruby, HTTP Client, and curl, and I get the same error either way.
The curl command I'm using is this:
curl -X POST -d "This is my sample comment" https://api.github.com/gists/2710948/comments
I think that if I can get the curl command working, I'll be able to figure out the HTTP Client and then the Ruby. This will be my first attempt at consuming an API, so there's nothing too basic for me to double-check; all suggestions will be helpful.
curl -d '{ "body": "Test comment" }' -u "Username:Pass" -X POST https://api.github.com/gists/2710948/comments
Ruby code:
require 'net/http'
uri = URI("https://api.github.com/gists/2710948/comments")
req = Net::HTTP::Post.new(uri.to_s)
req.basic_auth("Username", "Pass")
req.body = '{"body": "Test message"}' # `to_json` can be used
req["content-type"] = "application/json"
Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
p response = http.request(req)
end
See also http://developer.github.com/v3/gists/comments/