cURL got different page source than what Chrome browser did - bash

In short: I'm trying to get that page source of https://www.etoro.com/app/sv-iframe using curl in Bash.
I understand this ask is quite "simple". I have read thru 10+ similar questions here. Unfortunately, none of them could solve my problem.
When you open the URL above in Chrome browser, it's blank. You can either right click -> View Page Source, or sniff network using Chrome Developer Tool. Both will give you the correct page source. The page contains javascripts, in which there is a long hex string - what I need ultimately. I tried disabling javascript and reloading the page. I still got the right page source. So javascript doesn't play trick here. It sounds getting such page source via curl should be just straight forward, right?
When I right click the request in Chrome Developer Tool -> Copy as cURL, and execute it in terminal, things turned nasty - I got a CloudFlare security check page. I reopened the page several times in Chrome Incognito mode. I swear never saw a CloudFlare security check in browser. I double checked the cURL command. It has user-agent set as well.
Here is what I tried so far:
Manually compose curl command and fill headers from Chrome Developer Tool
Sniff packages on an Android device, and use headers set on mobile browser
Post request online from Postman Web
All gave me the same CloudFlare security check page.
The CloudFlare page says "Please enable cookies". I suspect if server in this way determined I was not calling from a browser. Following some threads, I tried to set -b/-c/-j flag with curl. Also no luck.
Here's more detailed steps what I've done:
Open Chrome Incognito mode
Open Developer Tool
Use Command+Shift+P (Mac) to open command menu
Type "disable javascript" and hit enter
Switch to Network tab
Open https://www.etoro.com/app/sv-iframe
Observe the request list - there should be only 1 request (request screenshot 1 / request screenshot 2 / response body / response cookie)
Right click on the request -> Copy as cURL
Here's my curl command:
curl 'https://www.etoro.com/app/sv-iframe' \
-H 'authority: www.etoro.com' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'upgrade-insecure-requests: 1' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36' \
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'sec-fetch-site: none' \
-H 'sec-fetch-mode: navigate' \
-H 'sec-fetch-user: ?1' \
-H 'sec-fetch-dest: document' \
-H 'accept-language: en-US,en;q=0.9' \
--compressed
The request itself I don't think it requires cookie, as page was able to be opened in Incognito mode. I anyways tried to set the response cookie together with the request. It doesn't help either.
-H 'cookie: __cfduid=d2edf...; TS01047baf=01d53...; __cf_bm=a3803...; __cflb=02Di3...'
Already spent whole evening on it but couldn't get it resolved. I appreciate any suggestions or help to get me thru it. I have a feeling that the actual fix would be fairly simple. The request has no cookie. Only thing to update is header. Maybe I didn't have correct header specified? Or some extra curl flag would help?

There is some obfuscated js eval code on that page, that is basically setting cookies or sending logs, digging a bit deeper, this is what ended up with:
(function() {
var s = '9a7xxx......';
function setCookie(cname, cvalue, domain, exdays) {
var d = new Date();
d.setTime(d.getTime() + (exdays * 1000 * 60 * 60 * 24));
var expires = "expires=" + d.toUTCString();
var cookie = cname + "=" + cvalue;
if (domain) {
cookie += ";" + "domain=" + domain;
}
cookie += ";" + expires + ";path=/";
document.cookie = cookie;
}
function deleteCookie(cname, domain) {
setCookie(cname, "", domain, 0);
}
var ta = ["window.callPhantom", "window.__nightmare", "window._phantom", "window.__webdriver_script_fn", "navigator.webdriver", "document.$cdc_asdjflasutopfhvcZLmcfl_"];
var re;
try {
re = [!!window.callPhantom, !!window.__nightmare, !!window._phantom, !!window.__webdriver_script_fn, !!navigator.webdriver, !!document.$cdc_asdjflasutopfhvcZLmcfl_];
} catch (err) {}
if (re && re.indexOf(true) == -1) {
setCookie("TMIS2", s, ".etoro.com", 14);
} else {
var resultsObj = {};
for (var i = 0; i < ta.length; i++) {
resultsObj[ta[i]] = re[i];
}
var img = new Image();
img.src = 'https://etorologsapi.etoro.com/api/v2/monitoring?applicationIdentifier=JSCClient&LogEvents=' + encodeURIComponent(JSON.stringify([{
ApplicationIdentifier: 'JSCClient',
ApplicationVersion: '0.0.11',
Level: "error",
Message: "ClientSel",
Results: resultsObj,
Type: 'log'
}]));
}
})();

Related

Can curl default to using https?

I have a script which acts as a wrapper around curl: it accepts all of curl's arguments but also adds some of its own (like -H 'Content-Type: application/json'), and then it does some parsing of the output.
The problem is that curl accepts curl google.com as meaning curl http://google.com. I want to force an HTTPS connection, but I don't want to parse curl's command line to find and edit the hostname. (The user might have typed curlwrapper -H "foo: bar" -XPOST google.com -d '{"hello":"world"}')
Is there any way to tell curl "use an HTTPS connection when you're not given a URL scheme"?
It does not appear to be possible due to how libcurl determines the protocol to use when no scheme is given. An excerpt from the code:
/*
* Since there was no protocol part specified, we guess what protocol it
* is based on the first letters of the server name.
*/
/* Note: if you add a new protocol, please update the list in
* lib/version.c too! */
if(checkprefix("FTP.", conn->host.name))
protop = "ftp";
else if(checkprefix("DICT.", conn->host.name))
protop = "DICT";
else if(checkprefix("LDAP.", conn->host.name))
protop = "LDAP";
else if(checkprefix("IMAP.", conn->host.name))
protop = "IMAP";
else if(checkprefix("SMTP.", conn->host.name))
protop = "smtp";
else if(checkprefix("POP3.", conn->host.name))
protop = "pop3";
else {
protop = "http";
}
HTTPS protocol for URL with missing scheme part (and thus also bypass protocol guessing mentioned in (obsolete) answer by #FatalError) can be set with option
--proto-default https
since version 7.45.0 from October 2015. See also https://github.com/curl/curl/pull/351.
It can be put into ~/.curlrc.
Example:
$ curl -v example.org
* Trying XXXXIPv6redacted:80...
* Connected to example.org (XXXXIPv6redacted) port 80 (#0)
> GET / HTTP/1.1
...
$ curl --proto-default https -v example.org
* Trying XXXXIPv6redacted:443...
* Connected to example.org (XXXXIPv6redacted) port 443 (#0)
* ALPN: offers h2
...
(Note that it's not a magic option to assure security. It e.g. won't affect http proxy, if set, according to the manual.)

Connect to Microsoft Push Notification Service for Windows Phone 8 from Ruby

We are developing a WP8 app that requires push notifications.
To test it we have run the push notification POST request with CURL command line, making sure that it actually connects, authenticates with the client SSL certificate and sends the correct data. We know for a fact that this work as we are receiving pushes to the devices.
This is the CURL command we have been using for testing purposes:
curl --cert client_cert.pem -v -H "Content-Type:text/xml" -H "X-WindowsPhone-Target:Toast" -H "X-NotificationClass:2" -X POST -d "<?xml version='1.0' encoding='utf-8'?><wp:Notification xmlns:wp='WPNotification'><wp:Toast><wp:Text1>My title</wp:Text1><wp:Text2>My subtitle</wp:Text2></wp:Toast></wp:Notification>" https://db3.notify.live.net/unthrottledthirdparty/01.00/AAF9MBULkDV0Tpyj24I3bzE3AgAAAAADCQAAAAQUZm52OkE1OUZCRDkzM0MyREY1RkE
Of course our SSL cert is needed to actually use the URL, but I was hoping someone else has done this and can see what we are doing wrong.
Now, our problem is that we need to make this work with Ruby instead, something we have been unable to get to work so far.
We have tried using HTTParty with no luck, and also net/http directly without any luck.
Here is a very simple HTTParty test script I have used to test with:
require "httparty"
payload = "<?xml version='1.0' encoding='utf-8'?><wp:Notification xmlns:wp='WPNotification'><wp:Toast><wp:Text1>My title</wp:Text1><wp:Text2>My subtitle</wp:Text2></wp:Toast></wp:Notification>"
uri = "https://db3.notify.live.net/unthrottledthirdparty/01.00/AAF9MBULkDV0Tpyj24I3bzE3AgAAAAADCQAAAAQUZm52OkE1OUZCRDkzM0MyREY1RkE"
opts = {
body: payload,
headers: {
"Content-Type" => "text/xml",
"X-WindowsPhone-Target" => "Toast",
"X-NotificationClass" => "2"
},
debug_output: $stderr,
pem: File.read("/Users/kenny/Desktop/client_cert.pem"),
ca_file: File.read('/usr/local/opt/curl-ca-bundle/share/ca-bundle.crt')
}
resp = HTTParty.post uri, opts
puts resp.code
This seems to connect with SSL properly, but then the MS IIS server returns 403 to us for some reason we don't get.
Here is essentially the same thing I've tried using net/http:
require "net/http"
url = URI.parse "https://db3.notify.live.net/unthrottledthirdparty/01.00/AAF9MBULkDV0Tpyj24I3bzE3AgAAAAADCQAAAAQUZm52OkE1OUZCRDkzM0MyREY1RkE"
payload = "<?xml version='1.0' encoding='utf-8'?><wp:Notification xmlns:wp='WPNotification'><wp:Toast><wp:Text1>My title</wp:Text1><wp:Text2>My subtitle</wp:Text2></wp:Toast></wp:Notification>"
pem_path = "./client_cert.pem"
cert = File.read pem_path
http = Net::HTTP.new url.host, url.port
http.use_ssl = true
http.cert = OpenSSL::X509::Certificate.new cert
http.key = OpenSSL::PKey::RSA.new cert
http.ca_path = '/etc/ssl/certs' if File.exists?('/etc/ssl/certs') # Ubuntu
http.ca_file = '/usr/local/opt/curl-ca-bundle/share/ca-bundle.crt' if File.exists?('/usr/local/opt/curl-ca-bundle/share/ca-bundle.crt') # Mac OS X
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
r = Net::HTTP::Post.new url.path
r.body = payload
r.content_type = "text/xml"
r["X-WindowsPhone-Target"] = "toast"
r["X-NotificationClass"] = "2"
http.start do
resp = http.request r
puts resp.code, resp.body
end
Like the HTTParty version, this also returns 403..
I'm starting to get the feeling that this won't actually work with net/http, but I've also seen a few examples of code claiming to work, but I can't see any difference compared to what we have tested with here.
Does anyone know how to fix this? Is it possible? Should I use libcurl instead perhaps? Or even do a system call to curl? (I may have to do the last one as an interim solution if we can't get this to work soon).
Any input is greatly appreciated!
Thanks,
Kenny
Try using some tool like http://mitmproxy.org to compare requests from your code and curl.
For example curl in addition to specified headers does send User-Agent and Accept-headers, microsoft servers may be checking for these for some reason.
If this does not help - then it's ssl-related

How to make an HTTP head request with headers in ruby?

I've been trying to use several libraries to make an HTTP HEAD request, but nothing seems to be working.
I've seen some examples, but nothing quite what I want.
Here's the Curl request, now I have to do it in ruby:
curl -XHEAD -H x-auth-user: myusername -H x-auth-key: mykey "url"
Also, this is an HTTPS url, if that makes a difference.
Try this:
require 'net/http'
url = 'http://...'
myusename = '...'
mykey = '...'
request = Net::HTTP.new(url, 80)
request.request_head('/', 'x-auth-user' => myusername, 'x-auth-key' => my_key)

How do I form a Github API POST request to add a new comment to a gist?

I'm doing a Post request to github at this url:
https://api.github.com/gists/2710948/comments
Theoretically, this should create a comment with the text being formed from what's in the request body. However, when I try to make that post, I get a 404 error. That leads me to believe that the gist is not being found, however, if you do a Get request at the same address it comes up just fine.
Is there an authentication thing I need to be doing? I've tried adding a username and password to my headers collection but I've got no idea if I'm using the right format. I've tried making this work via Ruby, HTTP Client, and curl, and I get the same error either way.
The curl command I'm using is this:
curl -X POST -d "This is my sample comment" https://api.github.com/gists/2710948/comments
I think that if I can get the curl command working, I'll be able to figure out the HTTP Client and then the Ruby. This will be my first attempt at consuming an API, so there's nothing too basic for me to double-check; all suggestions will be helpful.
curl -d '{ "body": "Test comment" }' -u "Username:Pass" -X POST https://api.github.com/gists/2710948/comments
Ruby code:
require 'net/http'
uri = URI("https://api.github.com/gists/2710948/comments")
req = Net::HTTP::Post.new(uri.to_s)
req.basic_auth("Username", "Pass")
req.body = '{"body": "Test message"}' # `to_json` can be used
req["content-type"] = "application/json"
Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
p response = http.request(req)
end
See also http://developer.github.com/v3/gists/comments/

Save a file with ruby mechanize

I have a problem with code (this code may to get request & download generated file):
require 'rubygems'
require 'mechanize'
require 'hpricot'
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
user = "xxx"
pass = "xxx"
auth_key = "xxx"
ip = "xxx"
agent = Mechanize.new
page = agent.get("https://#{ip}/cgi-bin/welcome.cgi")
form = page.forms.first
form.username = user
form.password = Digest::MD5.hexdigest(pass + auth_key)
page = agent.submit form
page = agent.get("https://#{ip}/cgi-bin/config.exp")
form = page.forms.first
agent.pluggable_parser.default = Mechanize::FileSaver
agent.post("https://#{ip}/cgi-bin/config.exp", {"submitstatus" => "1"})
With this I have a error:
/var/lib/gems/1.8/gems/mechanize-2.4/lib/mechanize/http/agent.rb:291:in `fetch': 400 => Net::HTTPBadRequest for https://31.223.225.133/cgi-bin/config.exp -- unhandled response (Mechanize::ResponseCodeError)
from /var/lib/gems/1.8/gems/mechanize-2.4/lib/mechanize.rb:407:in `get'
from /home/lord/Dropbox/work/ruby/ruby_backup/backup.ru:22
How to simulate this wget request for downloading file:
`wget --no-check-certificate --load-cookies cookie --post-data='submitstatus=1' \
--header='Host: 10.1.25.254' \
--header='User-Agent: Mozilla/5.0' \
--header='Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
--header='Accept-Language: ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3' \
--header='Accept-Encoding: gzip, deflate' \
--header='Connection: keep-alive' \
--header='Referer: https://10.1.25.254/sys_setting.htm' \
--header='Content-Type: application/x-www-form-urlencoded' \
--header='Content-Length: 14' \
https://$ip/cgi-bin/config.exp`
advance thanks
If you proxy your mechanize and wget requests through a debugging proxy such as fiddler or charles you can compare them side by side and thereby better understand the problem.
Have you tried Mechanize::Download?
Here is a reference
Using WWW:Mechanize to download a file to disk without loading it all in memory first

Resources