I need to use the curl command:
curl -d '{ "auth_token": "YOUR_AUTH_TOKEN", "text": "Hey, Look what I can do!" }' \http://localhost:3030/widgets/welcome
in a bash script but instead of "Hey, Look what I can do!" after the "YOUR_AUTH_TOKEN"; I need a variable $p .
This is my first time trying a bash script and read the tutorials on quotes and such but I still am not able to make it work.
The easiest thing to do is to read the data from a here document (a type of dynamically created temporary file), rather then trying to quote the entire thing as a string:
curl -d#- http://localhost:3030/widgets/welcome <<EOF
{ "auth_token": "YOUR_AUTH_TOKEN",
"text": "$p" }
EOF
If the argument to -d begins with a #, the remainder of the argument is taken as the name of a file containing the data. A file name of - indicates standard input, and the standard input to curl is supplied by the lines between the EOF tokens.
The alternative is to double-quote the string and escape all the embedded double quotes. Yuck.
... -d "{\"auth_token\": \"YOUR_AUTH_TOKEN\", \"text\": \"$p\"}" ...
An alternative to chepner's very good answer:
data=$( printf '{"auth_token":"YOUR_AUTH_TOKEN","text":"%s"}' "$p" )
curl -d "$data" http://localhost:3030/widgets/welcome
Related
Im trying to hit a rest api with token in header.
apikeyName="$(date '+%s')"
key=$(curl -k -X POST -H "Content-Type: application/json" \
-d '{"name":"'$apikeyName'", "role": "Admin"}' \
http://admin:admin#localhost:3000/api/auth/keys | jq '.key')
echo $key
# # Alerting API
curl -k -X GET 'http://localhost:3000/api/alert-notifications' -H 'Authorization: Bearer '$key'';
Terminal output
"eyJrIjoiaWJPaDFFZXZMeW1RYU90NUR4d014T3hYUmR6NDVUckoiLCJuIjoiMTY3NTM1OTc4OCIsImlkIjoxfQ=="
{"message":"invalid API key","traceID":""}
First 1 is the key printing and last one from api response. I tried to hardcode the key and it works.
Short answer: Use jq -r '.key' to extract the key from the json response without adding quotes to it.
Long answer: There is a difference between quotes on the command line and quotes embedded in a variable. Consider:
key='"abcd"'
printf '%s\n' $key "abcd"
# prints:
# "abcd"
# abcd
Quotes on the command line are bash syntax. Bash notes what is being quoted and then removes the quotes from the command line when it's done, thus printf only prints abcd in the second case above.
Quotes inside a variable are plain old data. Bash doesn't do anything with them, so they get passed through to the command like any other data and printf prints "abcd" in the first case.
In your curl case the receiver doesn't expect the key to have quotes embedded in the data. So, curl -blah "keydata" works fine because bash takes the quotes out, but curl -blah $key fails because bash does NOT remove the embedded quotes.
See also: BashParser
When write bash scripts, I want to store my whole curl command in heredoc to get a better layout. The following works fine:
#/bin/bash
read -r -d '' command1 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
MULTI_STRING_SCOPE
But when add some json data with the -d option, the command is executed weirdly. For example:
#/bin/bash
read -r -d '' command2 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
-d '{
"hello":"world"
}'
MULTI_STRING_SCOPE
response2=$(${command2})
Wrong logs from terminal:
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 1:
}'
And it seems that the curl take line }' as a seperated url, and thus the json data not sent as a unit.
How to solve the problem? Any suggestions will be highly appreciated.
I learned from this post to make heredoc work with curl command.
As the comment made by #Gordon Davisson in current post, we should not mix command with data. Since the json data set to -d option is only data and other parts is command, so I decide to use heredoc to store only the json data and remain other parts to be command itself, rather than store them in string by heredoc.
The result bash script should be something like this:
#/bin/bash
response3=$(curl -v www.stackoverflow.com \
-d #- <<- MULTI_STRING_SCOPE
{
"hello":"world"
}
MULTI_STRING_SCOPE
)
Notice: heredoc indent only works with tab, not with blanks. Be careful, especially when you are working with editors like Visual Studio Code, which may have already set indent as blanks for you.
When I stumble across an evil web site that I want blocked from corporate access, I edit my named.conf file on my bind server and then update my proxy server blacklist file. I'd like to automate this somewhat with a bash script. Say my script is called "evil-site-block.sh" and contains the following:
ssh root#192.168.0.1 'echo "#date added $(date +%m/%d/%Y)" >> /var/named/chroot/etc/named.conf; echo "zone \"$1\" { type master; file \"/etc/zone/dummy-block\"; };" >> /var/named/chroot/etc/named.conf'
It is then run as
$ evil-site-block.sh google.com
When I look at the contents of named.conf on the remote machine I see:
#date added 09/16/2014
zone "" { type master; file "/etc/zone/dummy-block"; };
What I can't figure out is how to pass "google.com" as $1.
First off, you don't want this to be two separately redirected echo statements -- doing that is both inefficient and means that the lines could end up not next to each other if something else is appending at the same time.
Second, and much more importantly, you don't want the remote command that's run to be something that could escape its quotes and run arbitrary commands on your server (think of if $1 is '$(rm -rf /)'.spammer.com).
Instead, consider:
#!/bin/bash
# ^ above is mandatory, since we use features not found in #!/bin/sh
printf -v new_contents \
'# date added %s\nzone "%s" { type master; file "/etc/zone/dummy-block"; };\n' \
"$(date +%m/%d/%Y)" \
"$1"
printf -v remote_command \
'echo %q >>/var/named/chroot/etc/named.conf' \
"$new_contents"
ssh root#192.168.0.1 bash <<<"$remote_command"
printf %q escapes data such that an evaluation pass in another bash shell will evaluate that content back to itself. Thus, the remote shell will be guaranteed (so long as it's bash) to interpret the content correctly, even if the content attempts to escape its surrounding quotes.
Your problem: Your entire command is put into single quotes – obviously so that bash expressions are expanded on the server and not locally.
But this also applies to your $1.
Simple solution: “Interupt” the quotation by wrapping your local variable into single quotes.
ssh root#192.168.0.1 'echo "#date added $(date +%m/%d/%Y)" >> /var/named/chroot/etc/named.conf; echo "zone \"'$1'\" { type master; file \"/etc/zone/dummy-block\"; };" >> /var/named/chroot/etc/named.conf'
NB: \"$1\" → \"'$1'\".
NOTE: This solution is a simple fix for the one-liner as posted in the question above. If there's the slightest chance that this script is executed by other people, or it could process external output of any kind, please have a look at Charles Duffy's solution.
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
curl 'https://shoesworkshop.net/libraries/ajax/ajax.invoice.php?act=viewallinvoice&invoiceid="${line}"&sEcho=1&iColumns=8&iDisplayStart=0&iDisplayLength=20&bRegex=false&bRegex_0=false&bSearchable_0=true&bRegex_1=false&bSearchable_1=true&bRegex_2=false&bSearchable_2=true&bRegex_3=false&bSearchable_3=true&bRegex_4=false&bSearchable_4=true&bRegex_5=false&bSearchable_5=true&bRegex_6=false&bSearchable_6=true&bRegex_7=false&bSearchable_7=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true&bSortable_4=true&bSortable_5=true&bSortable_6=true&bSortable_7=true' -H 'Host: shoesworkshop.net'| sed 's/^[^[[]*:/:/'
done < "$1"
inside $line there is a value like this
AAAAA
SSSSS
DDDDD
and i want to pass $line into curl command
can someone help me how?
i tried "'${line}'" and '${line}' and it still not working
i want to make a repeat call using curl get request from the url using variable from $line
For simple URLs, one way is to just use double quotes for the complete URL, including your variable expansion, ${line}, like this:
curl "https://shoe...&invoiceid=${line}&sEcho=1&iCo...table_7=true"
(Under single quotes, your shell variable line is not expanded.)
If your URL contains shell-special characters like $, it's best to combine both single and double quotes (and concatenate several strings, like explained here). For Example:
curl 'https://shoe...&invoiceid='"$line"'&sEcho=1&iCo...table_7=true'
# ^------ fixed part ------^ ^var^ ^------- fixed part ------^
However, if your variable contains characters that have to be URL-encoded (like space, &, ?, etc.) it's best to let curl handle that with --data-urlencode option. When called with this option, curl will default to POST method, but you can override this with -G, in which case your parameters will be appended to URL query. For example:
line="1&2?3 4"
curl "http://httpbin.org/get?x=1&y=2" --data-urlencode z="$line" -G
produces the right URL:
http://httpbin.org/get?x=1&y=2&z=1%262%3F3%204
Your script, fixed:
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
curl --data-urlencode invoiceid="$line" -G 'https://shoesworkshop.net/libraries/ajax/ajax.invoice.php?act=viewallinvoice&sEcho=1&iColumns=8&iDisplayStart=0&iDisplayLength=20&bRegex=false&bRegex_0=false&bSearchable_0=true&bRegex_1=false&bSearchable_1=true&bRegex_2=false&bSearchable_2=true&bRegex_3=false&bSearchable_3=true&bRegex_4=false&bSearchable_4=true&bRegex_5=false&bSearchable_5=true&bRegex_6=false&bSearchable_6=true&bRegex_7=false&bSearchable_7=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true&bSortable_4=true&bSortable_5=true&bSortable_6=true&bSortable_7=true' -H 'Host: shoesworkshop.net' | sed 's/^[^[[]*:/:/'
done < "$1"
I have a bash script that creates and executes an expect script by stitching together dozens of different files containing pieces of expect code. Those files contain environment variables that need to be expanded. Example:
expect.piece:
send "command\r"
sleep $timeout
send "command argument\r"
script.sh:
#let's try it like this
eval echo $(cat expect.piece)
#or maybe like this
eval "echo \"$(cat expect.piece)\""
output:
send command\r sleep 1 send command argument\r
send commandr
sleep 1
send command argumentr
Desired otput:
send "command\r"
sleep 1
send "command argument\r"
I need a solution without sed string substitution (there is a lot of environment variables) and without modifying original expect script files. I guess it could be done line by line, but is there a more elegant solution?
Default field separator in bash is a space so set input file separator as a new line like IFS=$(echo -e '\n') before executing eval echo $(cat expect.piece) .
Final script would be :
#Storing original field separator in variable OFS
OFS=$IFS
#Setting IFS as new line using echo -e. Unfortunately IFS="\n" does not work in bash
IFS=$(echo -e '\n')
eval echo $(cat expect.piece)
#Resetting the field separator as space
IFS=$OFS
There is one another way which you can put your expect code with -c flag in the shell script as shown below.
script.sh
#Calling the expect.piece file code here
expect -c expect.piece
You can make use of the optional command line values as ,
expect -c "set count 1" myscript.exp
where the variable count will be used in the expect script file myscript.exp.
You can directly give the whole code as
expect -c "
send \"command\r\"
sleep $timeout
send \"command argument\r\"
"
Notice the use of backslash to escape the double quotes wherever needed. Single quotes can also be used. But, if you use double quotes, then only shell substitution can happen.
Please refer here to know more about the -c flag in expect.
It's not clear from your question in what context you want this output. If it's okay to embed the Expect script as a here document, what you want is trivial.
#!/bin/sh
timeout=1
cat <<____HERE
send "command\r"
sleep $timeout
send "command argument\r"
____HERE
(Maybe you can even replace the cat with expect but I'm not familiar enough with Expect to make any recommendations.)
If you need to take the input from a file, and only have a limited set of variables you want expanded, you could do that with sed.
sed "s/\$timeout/$timeout/g" file
If you need a more general solution, you might want to swich to Perl:
perl -pe 's/\$(\w+)/$ENV{$1} || "\$$1" /ge' file
but this requires you to export (or otherwise expose to Perl) the shell environment variables you want exported. (This will just not substitute any undefined variables; change the value after || if you want to change that aspect of the behavior.)
I invented the solution for this problem, it is a kludge, but it works.
expect.piece:
sleep $timeout
send "foo bar\r"
send "$(date)\r"
script.sh:
timeout=1
eval echo $(sed " \
s/\\\/\\\\\\\/g; \
s/\"/\\\\\"/g; \
s/\"/\\\\\`/\"/\\\\\`/g; \
" "expect.piece" | tr '\n' '+') | tr '+' '\n'
output:
sleep 1
send "foo bar\r"
send "Wed Feb 18 03:19:24 2015\r"
First, we need to escape all the backslashes, backticks and quotes in the file, because they will be removed during the evaluation. Then, we need to replace all the newline characters with pluses, in order to make it in a single line. After that, we evaluate that line (the evaluation will "expand" all the environment variables and command substitutions) and replace pluses back to newlines.