I have a simple terminal shell command here:
curl -v -m 60 -H 'Accept-Language: en-US,en;q=0.8,nb;q=0.6' -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://imeidata.net -e http://imeidata.net/iphone/model-check http://imeidata.net/iphone/model-check?sn=C8PKTRF5DTC1
When I run it for first time I get full result (Page is about 23KB)
When I run this command for second time I get only a sample page
(about 17KB)
I am still able to visit this website, so my IP is not blocked but only cURL request are denied.
Again if I change my IP.. same will happen.
Why do my requests get blocked?
Any solution will be highly appreciated.
Thank you for helping.
I would fire it with "-k" # the end of the request.
So fire your request + " -k" like below:
curl -v -m 60 -H 'Accept-Language: en-US,en;q=0.8,nb;q=0.6' -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://imeidata.net -e http://imeidata.net/iphone/model-check http://imeidata.net/iphone/model-check?sn=C8PKTRF5DTC1 -k
Related
I am trying to check if I can access a page the same way in cURL as I can access it in Chrome.
With ChromeDevTools you can copy the request as a cURL command depending upon your OS.
So I selected Windows(cmd), which gave me roughly this command:
curl "https://practice.org/tracks/DSA/?batch=339" ^
-H "authority: practice.geeksforgeeks.org" ^
-H "cache-control: max-age=0" ^
-H "sec-ch-ua: ^\^" Not;A Brand^\^";v=^\^"99^\^", ^\^"Google Chrome^\^";v=^\^"91^\^", ^\^"Chromium^\^";v=^\^"91^\^"" ^
-H "sec-ch-ua-mobile: ?0" ^
-H "upgrade-insecure-requests: 1" ^
-H "user-agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" ^
-H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" ^
-H "sec-fetch-site: none" ^
-H "sec-fetch-mode: navigate" ^
-H "sec-fetch-user: ?1" ^
-H "sec-fetch-dest: document" ^
-H "accept-language: en-US,en;q=0.9" ^
But when the 3rd header is encountered in Windows CMD the whole command is executed:
-H "sec-ch-ua: ^\^" Not;A Brand^\^";v=^\^"99^\^", ^\^"Google Chrome^\^";v=^\^"91^\^", ^\^"Chromium^\^";v=^\^"91^\^"" ^
When the above line is encountered the whole command is executed without waiting for more input.
The formatting is not done correctly.
I don't know much about Batch Programming in Windows, and have no way to correct the input.
Can anyone help ?
I am trying to use CURL command to API call from AWS EC-2 server for our one application. I am using below request:
result1=`curl "https://localhost:15443/statusapi/admintool/insertOutage"
-H "cookie: JSESSIONID=5C06D8608D18AF2B87C5ADBCD4E206BF; XSRF-TOKEN=$x_csrf_token"
-H "origin: https://localhost:15443"
-H "accept-encoding: gzip, deflate, br"
-H "x-csrf-token: $x_csrf_token"
-H "authorization: Bearer $XSRF_TOKEN"
-H "content-type: application/json;charset=UTF-8"
-H "accept: application/json, text/plain, */*"
-H "authority: status.vizitechsolutions.com:15443"
-H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36"
--data-binary "[{\"serviceTypeEn\":\"dummy_service\",
\"serviceNameList\":\"VeriTrans 3G,\",
\"startTime\":\"1532684645737\",
\"startStatusEn\":\"Down\",
\"startStatusJp\":\"ダウン\",
\"startMessageEn\":\"Cannot process transaction due to outage at dummy service.\",
\"startMessageJp\":\"ダミ<81>[サ<81>[ビス<8f>瘧Qのためエラ<81>[となる事<8f>ﾛが発<90>ｶしております<81>B ダウン\",
\"locationEn\":\"dummy_location\",
\"locationJp\":\"ダミ<81>[<8f>・8f>浬",
\"rank\":\"-\",
\"createdBy\":\"lp\",
\"createdOn\":\"1532676990290\"}]"
--compressed --insecure -vvv`
English Field data is stored in DB properly, but Japanese data getting garbled in DB. When we try same through web application data get stored properly for both English and Japanese.
I would like to get the specific information from this site BTC value:
foxbit.com.br
I would just like to take the value of the BTC , but with WGET and CURL can not , because the value of the page is not static .
Do this is possible?
Here's how you can go about doing such things:
Load the page and see that the values load shorly after the page. This indicates that it's updated by a separate ajax request.
Open the Chrome developer console and look through the small handful of requested xhr resources to see which document it comes from.
Right click the request and copy as cURL.
You get some monstrosity like this:
curl 'http://www.foxbit.com.br/cdn-cgi/pe/bag2?r\[\]=https%3A%2F%2Fapi.blinktrade.com%2Fapi%2Fv1%2FBRL%2Fticker%3Fcallback%3DjQuery21109214518333319575_1448214593730%26crypto_currency%3DBTC%26_%3D1448214593731' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'PE-Token: 136c4c7c7a1e6a4bea75c61a93655067d5752459-1448214593-1800' -H 'Accept-Language: en-US,en;q=0.8,no;q=0.6,nb;q=0.4' -H 'User-Agent: Mozilla/5.0 (X11; CrOS x86_64) AppleWebKit/530.00 (KHTML, like Gecko) Chrome/45.0.0.0 Safari/530.00' -H 'Accept: */*' -H 'Referer: http://www.foxbit.com.br/' -H 'Cookie: __cfduid=d069fe8cecbd20bd0d09a5c04fbc398601448214586; _gat=1; _ga=GA1.3.1862709458.1448214588' -H 'Connection: keep-alive' --compressed
However, the URL clearly indicates that it just fetches from some other API, so we decode that URL and get:
https://api.blinktrade.com/api/v1/BRL/ticker?callback=jQuery21109214518333319575_1448214593730&crypto_currency=BTC&_=1448214593731
And we can drop the parameters that seem irrelevant and see what we get now:
$ curl 'https://api.blinktrade.com/api/v1/BRL/ticker?crypto_currency=BTC'
{"high": 1272.5, "vol": 50.77552492, "buy": 1238.51, "last": 1244.0, "low": 1238.13, "pair": "BTCBRL", "sell": 1243.99, "vol_brl": 63522.33386051}
Nice clean JSON. We can now easily make a command fetching and picking out the value we're interested in with the json tool jq:
$ curl -s 'https://api.blinktrade.com/api/v1/BRL/ticker?crypto_currency=BTC' |
jq '.last'
1244
I want to get host of a web site like filehippo.com:
What I tried is:
#!/bin/bash
AGENT='Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1'
page=1
for line in $(cat /tmp/IpList.txt)
do
REQUEST="http://www.bing.com/search?q=ip%3a108.168.208.206&qs=n&pq=ip%3a108.168.208.206&sc=0-0&sp=-1&sk=&first=1&FORM=PERE"
curl $REQUEST --user-agent "$AGENT"
let page=page+10
done
What I want:
I want to search in pages and get result, In this case I have 1 page but some of my servers have more than 1 pages..
Thank you
In the test script so many times I use the command "curl". In order to optimize the code, I want options of "curl" to carry out in a global variable.
I read the terms of use of "curl", and it says that to pass a parameter that contains spaces must be to frame it in single quotes.
But it is not working.
$ curl_options="-i -L -k -S --connect-timeout 30 --user-agent 'Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14'"
$ curl $curl_options "http://google.com"
Output Result:
curl: (6) Couldn't resolve host'' Opera '
curl: (6) Couldn't resolve host '(Windows'
curl: (6) Couldn't resolve host 'NT'
curl: (6) Couldn't resolve host '6 .1; '
curl: (6) Couldn't resolve host 'WOW64)'
curl: (6) Couldn't resolve host 'Presto'
curl: (6) Couldn't resolve host 'Version'
In bash, you should use an array. This way, you don't need to worry whether a space in the string is part of an option, or separating two options:
curl_options=( ... )
curl_options+=( "--user-agent" "Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14")
curl "${curl_options[#]}" "http://google.com"
If you can't use arrays (e.g., they aren't available in the shell you are using), you'll have to fall back to using eval:
$ curl_options="-i -L -k -S --connect-timeout 30 --user-agent 'Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14'"
$ eval "curl $curl_options http://google.com"
This isn't ideal, since you need to be very careful about how you set the value of curl_options, because eval has no knowledge about what the value represents. The shell merely interpolates the value into the string passed to eval, and eval executes it. Typos can have unintended consequences.