I am trying to check if I can access a page the same way in cURL as I can access it in Chrome.
With ChromeDevTools you can copy the request as a cURL command depending upon your OS.
So I selected Windows(cmd), which gave me roughly this command:
curl "https://practice.org/tracks/DSA/?batch=339" ^
-H "authority: practice.geeksforgeeks.org" ^
-H "cache-control: max-age=0" ^
-H "sec-ch-ua: ^\^" Not;A Brand^\^";v=^\^"99^\^", ^\^"Google Chrome^\^";v=^\^"91^\^", ^\^"Chromium^\^";v=^\^"91^\^"" ^
-H "sec-ch-ua-mobile: ?0" ^
-H "upgrade-insecure-requests: 1" ^
-H "user-agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" ^
-H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" ^
-H "sec-fetch-site: none" ^
-H "sec-fetch-mode: navigate" ^
-H "sec-fetch-user: ?1" ^
-H "sec-fetch-dest: document" ^
-H "accept-language: en-US,en;q=0.9" ^
But when the 3rd header is encountered in Windows CMD the whole command is executed:
-H "sec-ch-ua: ^\^" Not;A Brand^\^";v=^\^"99^\^", ^\^"Google Chrome^\^";v=^\^"91^\^", ^\^"Chromium^\^";v=^\^"91^\^"" ^
When the above line is encountered the whole command is executed without waiting for more input.
The formatting is not done correctly.
I don't know much about Batch Programming in Windows, and have no way to correct the input.
Can anyone help ?
Related
As I made a batch file to update NirSoft tools, I had a strange experience using wget.
First I downloaded a text file with pad links:
wget http://www.nirsoft.net/pad/pad-links.txt --backups=20 --append-output=C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
After, I used fart-js to delete rows I did not need from the pad-links.txt file. Also I used that program to change the download links to https://www.nirsoft.net/utils, and change the file extensions to .zip.
fart ".\pad-links.txt" "http://www.nirsoft.net/pad" "http://www.nirsoft.net/utils" | tee --append C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
and
fart ".\pad-links.txt" ".xml" ".zip" | tee --append C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
After, to download the programs, I used:
wget --timestamping --input-file=C:\Path\UtilSuit\NirLauncher\Download\pad-links.txt --append-output=C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
Having a look at the log file I found out that not all programs are stored in this location. For example WirelessKeyView is stored in https://www.nirsoft.net/toolsdownload/wirelesskeyview.zip.
Trying to get this file with wget leads to downloaded corrupt files at size of 4kb. The same with cURL and aria2. When I download it with Mozilla, or IDM, I have no problems to get the file. So I tried out wget --auth-no-challenge or wget --header="Accept: text/html" --user-agent="Mozilla/5.0 …"
I also tried cliget, the wget/aria2/curl lines it produced while normal downloading with Mozilla.
wget --header 'Host: www.nirsoft.net' --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: de,en-US;q=0.7,en;q=0.3' --referer 'https://www.nirsoft.net/utils/wirelesskeyview.html' --header 'Upgrade-Insecure-Requests: 1' --header 'Sec-Fetch-Dest: document' --header 'Sec-Fetch-Mode: navigate' --header 'Sec-Fetch-Site: same-origin' --header 'Sec-Fetch-User: ?1' --header 'DNT: 1' --header 'Sec-GPC: 1' 'https://www.nirsoft.net/toolsdownload/wirelesskeyview.zip' --output-document 'wirelesskeyview.zip'
I googled and found this reference for powershell, (same error), but cannot reproduce the working answer in batch, (I am not familiar with powershell scripting).
So how is is possible to download the single wirelesskey.zip file with wget/curl or aria2 in a batch script?
A workaround I found out is downloading it directly from the pad Panel but I want the .zip-file, including the updated .chm-file, and also the 64-bit versions, if available.
One more note, within my anti-virus tool the nirsoft site is exempted from scanning, so that is not the answer.
Any solutions?
Aah, this one is simple. If you look at the actual page downloaded, it's called "403.html". So, let's open it. The first thing that strikes you is this:
<title>Error 403: Missing HTTP referer in the HTTP request</title>
So, the server wants a Referer header. Sure, let's give it one:
$ wget --referer foo <URL>
And it downloads the zip file correctly as expected.
Now, really, the server should not be returning a HTTP 200 response with a file called 403. It really should have sent back a HTTP 403 response. But what can you do? There's broken servers everywhere
I am trying to use CURL command to API call from AWS EC-2 server for our one application. I am using below request:
result1=`curl "https://localhost:15443/statusapi/admintool/insertOutage"
-H "cookie: JSESSIONID=5C06D8608D18AF2B87C5ADBCD4E206BF; XSRF-TOKEN=$x_csrf_token"
-H "origin: https://localhost:15443"
-H "accept-encoding: gzip, deflate, br"
-H "x-csrf-token: $x_csrf_token"
-H "authorization: Bearer $XSRF_TOKEN"
-H "content-type: application/json;charset=UTF-8"
-H "accept: application/json, text/plain, */*"
-H "authority: status.vizitechsolutions.com:15443"
-H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36"
--data-binary "[{\"serviceTypeEn\":\"dummy_service\",
\"serviceNameList\":\"VeriTrans 3G,\",
\"startTime\":\"1532684645737\",
\"startStatusEn\":\"Down\",
\"startStatusJp\":\"ダウン\",
\"startMessageEn\":\"Cannot process transaction due to outage at dummy service.\",
\"startMessageJp\":\"ダミ<81>[サ<81>[ビス<8f>瘧Qのためエラ<81>[となる事<8f>ﾛが発<90>ｶしております<81>B ダウン\",
\"locationEn\":\"dummy_location\",
\"locationJp\":\"ダミ<81>[<8f>・8f>浬",
\"rank\":\"-\",
\"createdBy\":\"lp\",
\"createdOn\":\"1532676990290\"}]"
--compressed --insecure -vvv`
English Field data is stored in DB properly, but Japanese data getting garbled in DB. When we try same through web application data get stored properly for both English and Japanese.
I would like to get the specific information from this site BTC value:
foxbit.com.br
I would just like to take the value of the BTC , but with WGET and CURL can not , because the value of the page is not static .
Do this is possible?
Here's how you can go about doing such things:
Load the page and see that the values load shorly after the page. This indicates that it's updated by a separate ajax request.
Open the Chrome developer console and look through the small handful of requested xhr resources to see which document it comes from.
Right click the request and copy as cURL.
You get some monstrosity like this:
curl 'http://www.foxbit.com.br/cdn-cgi/pe/bag2?r\[\]=https%3A%2F%2Fapi.blinktrade.com%2Fapi%2Fv1%2FBRL%2Fticker%3Fcallback%3DjQuery21109214518333319575_1448214593730%26crypto_currency%3DBTC%26_%3D1448214593731' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'PE-Token: 136c4c7c7a1e6a4bea75c61a93655067d5752459-1448214593-1800' -H 'Accept-Language: en-US,en;q=0.8,no;q=0.6,nb;q=0.4' -H 'User-Agent: Mozilla/5.0 (X11; CrOS x86_64) AppleWebKit/530.00 (KHTML, like Gecko) Chrome/45.0.0.0 Safari/530.00' -H 'Accept: */*' -H 'Referer: http://www.foxbit.com.br/' -H 'Cookie: __cfduid=d069fe8cecbd20bd0d09a5c04fbc398601448214586; _gat=1; _ga=GA1.3.1862709458.1448214588' -H 'Connection: keep-alive' --compressed
However, the URL clearly indicates that it just fetches from some other API, so we decode that URL and get:
https://api.blinktrade.com/api/v1/BRL/ticker?callback=jQuery21109214518333319575_1448214593730&crypto_currency=BTC&_=1448214593731
And we can drop the parameters that seem irrelevant and see what we get now:
$ curl 'https://api.blinktrade.com/api/v1/BRL/ticker?crypto_currency=BTC'
{"high": 1272.5, "vol": 50.77552492, "buy": 1238.51, "last": 1244.0, "low": 1238.13, "pair": "BTCBRL", "sell": 1243.99, "vol_brl": 63522.33386051}
Nice clean JSON. We can now easily make a command fetching and picking out the value we're interested in with the json tool jq:
$ curl -s 'https://api.blinktrade.com/api/v1/BRL/ticker?crypto_currency=BTC' |
jq '.last'
1244
I have a simple terminal shell command here:
curl -v -m 60 -H 'Accept-Language: en-US,en;q=0.8,nb;q=0.6' -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://imeidata.net -e http://imeidata.net/iphone/model-check http://imeidata.net/iphone/model-check?sn=C8PKTRF5DTC1
When I run it for first time I get full result (Page is about 23KB)
When I run this command for second time I get only a sample page
(about 17KB)
I am still able to visit this website, so my IP is not blocked but only cURL request are denied.
Again if I change my IP.. same will happen.
Why do my requests get blocked?
Any solution will be highly appreciated.
Thank you for helping.
I would fire it with "-k" # the end of the request.
So fire your request + " -k" like below:
curl -v -m 60 -H 'Accept-Language: en-US,en;q=0.8,nb;q=0.6' -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://imeidata.net -e http://imeidata.net/iphone/model-check http://imeidata.net/iphone/model-check?sn=C8PKTRF5DTC1 -k
I try to download something with wget using for loop in bash script:
When i'm not using variables everything work fine, when i assign it into variables i have 500 server error. This is strange for me, because this is only copy-paste.
What i'm trying to do is take number from loop i and paste it into body.
Here is my code:
#!/bin/bash
for i in {1..5}
do
STR="some_static_stuff_before"$i"some_static_suff_after"
echo $STR
wget -O ready/page$i.aspx --header="Host: www.something.com" --header="Pragma: no-cache" --header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" --header="Accept-Language: en-en" --header="Accept-Encoding: gzip, deflate" --header="Content-Type: application/x-www-form-urlencoded" --header="Origin: http://something.com" --header="Connection: keep-alive" --header="User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" --header="Referer: http://www.something.com/something.aspx" --header="Cookie: ASP.NET_SessionId=u5cmt0figi4bvs40a30gnwsa; __utma=20059042.38323768.1389369038.1389710153.1389780868.6; __utmb=20059042.2.10.1389780868; __utmc=20059042; __utmz=20059042.1389627823.2.2.utmcsr=something.com|utmccn=(referral)|utmcmd=referral|utmcct=/something.aspx" --post-data='"$STR"' http://something.com/something.aspx
done
And when i paste object directly to --post-data there is no problem with download content.
I've tried --post-data= "/"$STR/"" and --post-data='"$STR"' and still not working.
You single-quoted the variable reference (in addition to double-quoting it), which prevents substitution of the variable value.
Instead of
--post-data='"$STR"'
use
--post-data="$STR"