In the test script so many times I use the command "curl". In order to optimize the code, I want options of "curl" to carry out in a global variable.
I read the terms of use of "curl", and it says that to pass a parameter that contains spaces must be to frame it in single quotes.
But it is not working.
$ curl_options="-i -L -k -S --connect-timeout 30 --user-agent 'Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14'"
$ curl $curl_options "http://google.com"
Output Result:
curl: (6) Couldn't resolve host'' Opera '
curl: (6) Couldn't resolve host '(Windows'
curl: (6) Couldn't resolve host 'NT'
curl: (6) Couldn't resolve host '6 .1; '
curl: (6) Couldn't resolve host 'WOW64)'
curl: (6) Couldn't resolve host 'Presto'
curl: (6) Couldn't resolve host 'Version'
In bash, you should use an array. This way, you don't need to worry whether a space in the string is part of an option, or separating two options:
curl_options=( ... )
curl_options+=( "--user-agent" "Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14")
curl "${curl_options[#]}" "http://google.com"
If you can't use arrays (e.g., they aren't available in the shell you are using), you'll have to fall back to using eval:
$ curl_options="-i -L -k -S --connect-timeout 30 --user-agent 'Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14'"
$ eval "curl $curl_options http://google.com"
This isn't ideal, since you need to be very careful about how you set the value of curl_options, because eval has no knowledge about what the value represents. The shell merely interpolates the value into the string passed to eval, and eval executes it. Typos can have unintended consequences.
Related
I'm learning feathersjs as per below link
http://feathersjs.com/quick-start/
I need to run below command and monitor the output at http://localhost:3000/todo
curl 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
When I tried to run below on cmd, it shows 'curl' is not recognized in cmd prompt.
If I tried to run it using git-bash.exe, bash.exe or sh.exe (in Git/bin or shell.w32-ix86), Cygwin.bat (in cygwin64), it will run fine and showing result in browser.
[{"text":"Do something","id":0}]
But if tried to run it by including into my PATH "C:\Program Files\Git\mingw64\bin, which has curl.exe", I will be getting below error, but "C:\Program Files\Git\usr\bin" will do just fine...
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
I managed to remove the error with below command
curl "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
But the resulting output will have the Json object "text" missing...
[{"text":"Do something","id":0},{"id":1}]
Question:
1) After modifying the command, the Json object is not parsed successfully. Is there a syntax problem?
2) If there is no syntax problem, does this means that curl need to be run in Unix environment as per original attempt, that it cant be run in cmd directly, but will function ok in bash, cygwin, etc2?
3) What is the difference between curl.exe in C:\Program Files\Git\usr\bin and C:\Program Files\Git\mingw64\bin, which has curl.exe?
Update:
Not OK Raw Cap output
http.content_type == "'application/json'"
OK Raw Cap output
http.content_type == "application/json"
Update2:
Removing single quote in application/json on the 2nd command... shows error
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
{"id":1}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: application/json" --data-binary "{ 'text': 'Do something' }"
SyntaxError: Unexpected token '<br> at Object.parse (native)<br> at parse (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\types\json.js:88:17)<br> at C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\read.js:116:18<br> at invokeCallback (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:262:16)<br> at done (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:251:7)<br> at IncomingMessage.onEnd (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:308:7)<br> at emitNone (events.js:67:13)<br> at IncomingMessage.emit (events.js:166:7)<br> at endReadableNT (_stream_readable.js:905:12)<br> at doNTCallback2 (node.js:441:9)
Update 3:
Tried to replace libcurl-4.dll used by curl.exe. Downloaded libcurl from "http://curl.haxx.se/download.html", MingW32 from "http://sourceforge.net/projects/mingw/files/MSYS/" and added "C:\MinGW\bin" to PATH. Then grep -rl "libcurl.dll" . | xargs sed -i 's/libcurl.dll/libcurl-4.dll/g' to create libcurl-4.dll as per suggested "http://curl.haxx.se/mail/lib-2010-11/0174.html". Then execute ./buildconfig, make, make install. Then copied the libcurl-4.dll to C:\Program Files\Git\mingw64\bin folder, but the result is the same...
Update 4
Changing origin of curl.exe but still using same command, shows error on the mingw64 version. I'm suspecting that mingw64 curl need special escaping to make it work?
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\mingw64\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\usr\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
{"text":"Do something","id":38}
Update 5
From manual... curl --manual
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP
server, in the same way that a browser does when a user has
filled in an HTML form and presses the submit button. This will
cause curl to pass the data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. --data-raw is almost the
same but does not have a special interpretation of the # charac-
ter. To post data purely binary, you should instead use the
--data-binary option. To URL-encode the value of a form field
you may use --data-urlencode.
If any of these options is used more than once on the same com-
mand line, the data pieces specified will be merged together
with a separating &-symbol. Thus, using '-d name=daniel -d
skill=lousy' would generate a post chunk that looks like
'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a
file name to read the data from, or - if you want curl to read
the data from stdin. Multiple files can also be specified. Post-
ing data from a file named 'foobar' would thus be done with
--data #foobar. When --data is told to read from a file like
that, carriage returns and newlines will be stripped out. If you
don't want the # character to have a special interpretation use
--data-raw instead.
So I tried...
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=doing --data complete=false
{"text":"doing","complete":"false","id":145}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=ding
{"text":"ding","id":146}
But I cant figure out how to make more than 1 word for the JSON object for example instead of "doing", I need "doing something". Seems that MingW64 git curl is accepting different format...
After modifying the curl command it works for you, because you need to use double quote for windows system.
After modifying the command, you have mistakenly added single quote around the application/json. That's why despite of having working curl command server was unsure what you have exactly sent to them!
"Content-Type: 'application/json'"
^ ^ notice the unwanted singles
So it should be
"Content-Type: application/json"
If you do not provide path for any binary (i.e. curl.exe, mysql.exe, php.exe, etc) then system looks for them inside the available paths provided in PATH variable and if they found multiple path there then it will only select one, and I don't know which one!
I have a simple terminal shell command here:
curl -v -m 60 -H 'Accept-Language: en-US,en;q=0.8,nb;q=0.6' -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://imeidata.net -e http://imeidata.net/iphone/model-check http://imeidata.net/iphone/model-check?sn=C8PKTRF5DTC1
When I run it for first time I get full result (Page is about 23KB)
When I run this command for second time I get only a sample page
(about 17KB)
I am still able to visit this website, so my IP is not blocked but only cURL request are denied.
Again if I change my IP.. same will happen.
Why do my requests get blocked?
Any solution will be highly appreciated.
Thank you for helping.
I would fire it with "-k" # the end of the request.
So fire your request + " -k" like below:
curl -v -m 60 -H 'Accept-Language: en-US,en;q=0.8,nb;q=0.6' -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://imeidata.net -e http://imeidata.net/iphone/model-check http://imeidata.net/iphone/model-check?sn=C8PKTRF5DTC1 -k
I am trying to get the index position using Bash 'expr index".
e.g.
$ echo `expr index "Info.out.2014-02-08:INFO|SID:sXfzRjbmKbwX7jyaW1sog7n|Browser[Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0]" Mozilla`
I am trying to get the index position of the word "Mozilla", and then get the substring using index value.
The result I got back is 4. Is it the period after Info caus the issue? How do I fix this issue?
I followed the Advanced Bash scripting guide www.tldp.org/LDP/abs/html/. See section Table B-5. String Operations
expr index "$string" $substring Numerical position in $string of first character in $substring* that matches [0 if no match, first character counts as position 1]
I tried with something simple, and it works.
I am running bash in cygwin.
$ ./bash --version
GNU bash, version 4.1.10(4)-release (i686-pc-cygwin)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Thanks.
In general, you shouldn't be using expr index unless you have a very good reason to.
For instance, let's say you want to get the browser name.
s="Info.out.2014-02-08:INFO|SID:sXfzRjbmKbwX7jyaW1sog7n|Browser[Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0]"
# strip everything up to and including the first instance of 'Browser['
browser="${s#*Browser[}"
# strip everything after the first ']', again, inclusive
browser="${browser%%]*}"
# ...and show the result...
echo "$browser"
This would return:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0
If you really do want to know how many characters precede Mozilla, well, you can do that too:
s="Info.out.2014-02-08:INFO|SID:sXfzRjbmKbwX7jyaW1sog7n|Browser[Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0]"
# strip everything after the first instance of 'Mozilla'
prefix=${s%%Mozilla*}
# count number of characters in the string
index=${#prefix}
# ...and show the result...
echo "$index"
This should return 61.
For the "why" and "how" of the above examples, see BashFAQ #73.
To split by | separators, by contrast, I'd personally choose to use read, as documented in BashFAQ #1:
s="Info.out.2014-02-08:INFO|SID:sXfzRjbmKbwX7jyaW1sog7n|Browser[Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0]"
IFS='|' read -r _ _ browser _
echo "$browser"
...which would emit...
Browser[Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0]
The expr index command searches through your first string looking the the first occurrence of any character from your second string. In this case, it is recognizing that the 'o' in the characters 'Mozilla' matches the 4th character in "Info.out..."
This using this as a test to see what happens. It will return 4 as the first match for 'd':
echo `expr index "abcdefghijklmnopqrstuvwxyz" xyzd`
This one should do what you want:
echo "Info.out.2014-02-08:INFO|SID:sXfzRjbmKbwX7jyaW1sog7n|Browser[Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0]" | grep -o -b Mozilla
The echo puts your string into stdout, so it can be piped into grep.
The -b prints the byte offset of the string shown.
The -o ensures that only the matching portion gets printed.
GNU expr does not match a substring using index; rather, it looks for the first occurrence of any character from the second string in the first. Your example returns 4 because the 4th character of the string is "o", the first character in "Mozilla" that is found in "Info.out...".
There is no built-in function of this kind in either bash or expr, but you can indirectly get the index of a given substring by first removing the substring and everything after it from the original string, then computing the remaining length.
string="Info.out..."
substring=Mozilla
tmp=${string%%$substring*}
index=${#tmp}
I try to download something with wget using for loop in bash script:
When i'm not using variables everything work fine, when i assign it into variables i have 500 server error. This is strange for me, because this is only copy-paste.
What i'm trying to do is take number from loop i and paste it into body.
Here is my code:
#!/bin/bash
for i in {1..5}
do
STR="some_static_stuff_before"$i"some_static_suff_after"
echo $STR
wget -O ready/page$i.aspx --header="Host: www.something.com" --header="Pragma: no-cache" --header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" --header="Accept-Language: en-en" --header="Accept-Encoding: gzip, deflate" --header="Content-Type: application/x-www-form-urlencoded" --header="Origin: http://something.com" --header="Connection: keep-alive" --header="User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" --header="Referer: http://www.something.com/something.aspx" --header="Cookie: ASP.NET_SessionId=u5cmt0figi4bvs40a30gnwsa; __utma=20059042.38323768.1389369038.1389710153.1389780868.6; __utmb=20059042.2.10.1389780868; __utmc=20059042; __utmz=20059042.1389627823.2.2.utmcsr=something.com|utmccn=(referral)|utmcmd=referral|utmcct=/something.aspx" --post-data='"$STR"' http://something.com/something.aspx
done
And when i paste object directly to --post-data there is no problem with download content.
I've tried --post-data= "/"$STR/"" and --post-data='"$STR"' and still not working.
You single-quoted the variable reference (in addition to double-quoting it), which prevents substitution of the variable value.
Instead of
--post-data='"$STR"'
use
--post-data="$STR"
I'm trying to use cURL to get data to the form an URL:
http://example.com/site-explorer/get_overview_text_data.php?data_type=refdomains_stats&hash=19a53c6b9aab3917d8bed5554000c7cb
which needs a cookie, so I first store it on a file:
curl -c cookie-jar http://example.com/site-explorer/overview/subdomains/example.com
Trying curl with these values:
curl -b cookie-jar -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" --referer "http://example.com/site-explorer/overview/subdomains/example.com" http://example.com/site-explorer/get_overview_text_data.php?data_type=refdomains_stats&hash=19a53c6b9aab3917d8bed5554000c7cb
There is one problem which leaps out at me: You aren't quoting the URL, which means that characters such as & and ? will be interpreted by the shell instead of getting passed to curl. If you're using a totally static URL, enclose it in single quotes, as in 'http://blah.com/blah/blah...'.