bash - assign variable to curl get request - bash

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
curl 'https://shoesworkshop.net/libraries/ajax/ajax.invoice.php?act=viewallinvoice&invoiceid="${line}"&sEcho=1&iColumns=8&iDisplayStart=0&iDisplayLength=20&bRegex=false&bRegex_0=false&bSearchable_0=true&bRegex_1=false&bSearchable_1=true&bRegex_2=false&bSearchable_2=true&bRegex_3=false&bSearchable_3=true&bRegex_4=false&bSearchable_4=true&bRegex_5=false&bSearchable_5=true&bRegex_6=false&bSearchable_6=true&bRegex_7=false&bSearchable_7=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true&bSortable_4=true&bSortable_5=true&bSortable_6=true&bSortable_7=true' -H 'Host: shoesworkshop.net'| sed 's/^[^[[]*:/:/'
done < "$1"
inside $line there is a value like this
AAAAA
SSSSS
DDDDD
and i want to pass $line into curl command
can someone help me how?
i tried "'${line}'" and '${line}' and it still not working
i want to make a repeat call using curl get request from the url using variable from $line

For simple URLs, one way is to just use double quotes for the complete URL, including your variable expansion, ${line}, like this:
curl "https://shoe...&invoiceid=${line}&sEcho=1&iCo...table_7=true"
(Under single quotes, your shell variable line is not expanded.)
If your URL contains shell-special characters like $, it's best to combine both single and double quotes (and concatenate several strings, like explained here). For Example:
curl 'https://shoe...&invoiceid='"$line"'&sEcho=1&iCo...table_7=true'
# ^------ fixed part ------^ ^var^ ^------- fixed part ------^
However, if your variable contains characters that have to be URL-encoded (like space, &, ?, etc.) it's best to let curl handle that with --data-urlencode option. When called with this option, curl will default to POST method, but you can override this with -G, in which case your parameters will be appended to URL query. For example:
line="1&2?3 4"
curl "http://httpbin.org/get?x=1&y=2" --data-urlencode z="$line" -G
produces the right URL:
http://httpbin.org/get?x=1&y=2&z=1%262%3F3%204
Your script, fixed:
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
curl --data-urlencode invoiceid="$line" -G 'https://shoesworkshop.net/libraries/ajax/ajax.invoice.php?act=viewallinvoice&sEcho=1&iColumns=8&iDisplayStart=0&iDisplayLength=20&bRegex=false&bRegex_0=false&bSearchable_0=true&bRegex_1=false&bSearchable_1=true&bRegex_2=false&bSearchable_2=true&bRegex_3=false&bSearchable_3=true&bRegex_4=false&bSearchable_4=true&bRegex_5=false&bSearchable_5=true&bRegex_6=false&bSearchable_6=true&bRegex_7=false&bSearchable_7=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true&bSortable_4=true&bSortable_5=true&bSortable_6=true&bSortable_7=true' -H 'Host: shoesworkshop.net' | sed 's/^[^[[]*:/:/'
done < "$1"

Related

curl set dynamic header token throws invalid key

Im trying to hit a rest api with token in header.
apikeyName="$(date '+%s')"
key=$(curl -k -X POST -H "Content-Type: application/json" \
-d '{"name":"'$apikeyName'", "role": "Admin"}' \
http://admin:admin#localhost:3000/api/auth/keys | jq '.key')
echo $key
# # Alerting API
curl -k -X GET 'http://localhost:3000/api/alert-notifications' -H 'Authorization: Bearer '$key'';
Terminal output
"eyJrIjoiaWJPaDFFZXZMeW1RYU90NUR4d014T3hYUmR6NDVUckoiLCJuIjoiMTY3NTM1OTc4OCIsImlkIjoxfQ=="
{"message":"invalid API key","traceID":""}
First 1 is the key printing and last one from api response. I tried to hardcode the key and it works.
Short answer: Use jq -r '.key' to extract the key from the json response without adding quotes to it.
Long answer: There is a difference between quotes on the command line and quotes embedded in a variable. Consider:
key='"abcd"'
printf '%s\n' $key "abcd"
# prints:
# "abcd"
# abcd
Quotes on the command line are bash syntax. Bash notes what is being quoted and then removes the quotes from the command line when it's done, thus printf only prints abcd in the second case above.
Quotes inside a variable are plain old data. Bash doesn't do anything with them, so they get passed through to the command like any other data and printf prints "abcd" in the first case.
In your curl case the receiver doesn't expect the key to have quotes embedded in the data. So, curl -blah "keydata" works fine because bash takes the quotes out, but curl -blah $key fails because bash does NOT remove the embedded quotes.
See also: BashParser

How to execute curl command stored in heredoc in bash script?

When write bash scripts, I want to store my whole curl command in heredoc to get a better layout. The following works fine:
#/bin/bash
read -r -d '' command1 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
MULTI_STRING_SCOPE
But when add some json data with the -d option, the command is executed weirdly. For example:
#/bin/bash
read -r -d '' command2 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
-d '{
"hello":"world"
}'
MULTI_STRING_SCOPE
response2=$(${command2})
Wrong logs from terminal:
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 1:
}'
And it seems that the curl take line }' as a seperated url, and thus the json data not sent as a unit.
How to solve the problem? Any suggestions will be highly appreciated.
I learned from this post to make heredoc work with curl command.
As the comment made by #Gordon Davisson in current post, we should not mix command with data. Since the json data set to -d option is only data and other parts is command, so I decide to use heredoc to store only the json data and remain other parts to be command itself, rather than store them in string by heredoc.
The result bash script should be something like this:
#/bin/bash
response3=$(curl -v www.stackoverflow.com \
-d #- <<- MULTI_STRING_SCOPE
{
"hello":"world"
}
MULTI_STRING_SCOPE
)
Notice: heredoc indent only works with tab, not with blanks. Be careful, especially when you are working with editors like Visual Studio Code, which may have already set indent as blanks for you.

Curl upload with spaces in filenames

I am trying to use cURL to upload files with spaces in their filenames onto a dedicated server. I am using bash. In a previous project, I just gave up, removing all spaces from filenames. This is not feasible for this project.
Running cURL in verbose mode suggests that it stops when it reads my local file path:
curl -X PUT -u $USER:$PASS --data-binary #"$LOCAL_FILE" "$SERVER/remote.php/dav/files/$USER/$REMOTE_DIR/$REMOTE_FILE"
where $LOCAL_FILE is a path to a file on my local machine (with spaces), and $REMOTE_FILE also has spaces.
This gives:
Warning: Couldn't read data from file "/Users/my_account/somepath/with
Warning: spaces
which implies the command is taking "/Users/my_account/somepath/with" and "spaces" as two separate paths.
How can I solve this?
My full code:
#!/bin/bash -ex
# Local dir - note space in path
IMAGES_DIR="/Users/my_account/somepath/with spaces"
# Remote server, and credentials
SERVER="http://myserver"
REMOTE_DIR="mydir"
USER="myname"
PASS="mypass"
FILE="$1"
LOCAL_FILE="$IMAGES_DIR/$FILE"
REMOTE_FILE=urlencode $FILE
# Move file to the server
curl -v -X PUT -u $USER:$PASS --data-binary #"$LOCAL_FILE" "$SERVER/remote.php/dav/files/$USER/$REMOTE_DIR/$REMOTE_FILE"
# Check that file has made it
echo 'Waiting for file to be on server'
until [ $result > 0 ]
do
result=$(curl -I "$SERVER/remote.php/dav/files/$USER/$REMOTE_DIR/$REMOTE_FILE" -u $USER:$PASS 2>/dev/null | grep Content-Length | awk -F ': ' '{print $2}')
echo "."
sleep 2
done
echo "File $FILE is now on server."
urlencode() {
# urlencode <string>
old_lc_collate=$LC_COLLATE
LC_COLLATE=C
local length="${#1}"
for (( i = 0; i < length; i++ )); do
local c="${1:i:1}"
case $c in
[a-zA-Z0-9.~_-]) printf "$c" ;;
*) printf '%%%02X' "'$c" ;;
esac
done
LC_COLLATE=$old_lc_collate
}
In the Windows version of curl 7.55.0 (should be in higher versions also) the below command will work:
local Windows path: path enclosed in double or single quotes
server URL: path enclosed in double or single quotes and each space replaced with '%20' an example to upload tes st.txt is given below
curl -username:password -T "./Test/te st.txt" "http://server-url/folder1/folder2/te%20st.txt"
Did you tried changing the value of IFS in your script? Normally its value is set to $' \t\n', which includes space. Set it to nothing like so: IFS= and no field splitting is performed at all. You can get the default value back by executing unset IFS.
Example:
var="a b c d"
echo $var
a b c d
IFS=
echo $var
a b c d
TL;DR : the problem still occurs on windows with filenames containing illegal characters, like | > < :
As #Jerin points out, curl on windows (at least in versions > 7.55.0) will accept spaces in filenames.
However, with curl 7.57.0 (x86_64-w64-mingw32), I'm observing that
curl -X POST http://localhost:8080/ -H "Content-Type: application/json" --data-binary "#my_dir/2019-11-04>.json"
gives me
Warning: Couldn't read data from file "my_dir/2019-11-04>.json",
Warning: this makes an empty POST.
And doesn't pass the file as POST data.

Bash script with a command with lots of single and double quotes

I need to use the curl command:
curl -d '{ "auth_token": "YOUR_AUTH_TOKEN", "text": "Hey, Look what I can do!" }' \http://localhost:3030/widgets/welcome
in a bash script but instead of "Hey, Look what I can do!" after the "YOUR_AUTH_TOKEN"; I need a variable $p .
This is my first time trying a bash script and read the tutorials on quotes and such but I still am not able to make it work.
The easiest thing to do is to read the data from a here document (a type of dynamically created temporary file), rather then trying to quote the entire thing as a string:
curl -d#- http://localhost:3030/widgets/welcome <<EOF
{ "auth_token": "YOUR_AUTH_TOKEN",
"text": "$p" }
EOF
If the argument to -d begins with a #, the remainder of the argument is taken as the name of a file containing the data. A file name of - indicates standard input, and the standard input to curl is supplied by the lines between the EOF tokens.
The alternative is to double-quote the string and escape all the embedded double quotes. Yuck.
... -d "{\"auth_token\": \"YOUR_AUTH_TOKEN\", \"text\": \"$p\"}" ...
An alternative to chepner's very good answer:
data=$( printf '{"auth_token":"YOUR_AUTH_TOKEN","text":"%s"}' "$p" )
curl -d "$data" http://localhost:3030/widgets/welcome

Substituting lines of a text file into bash commands

I want to run a bash command once for each line of a text file.
If I do:
while read l; do
echo $l;
done < myfile.txt
I get each line echoed.
But when I use my actual command:
while read l; do
curl -X POST http://somedomain -H "Content-Type: application/json" -d $l
done < myfile.txt
I get errors. I think it may have something to do with escaping characters in the text file, but nothing I have tried works. What is the proper way to run a curl expression once for each line of a text file?
To use quotes.
foocmd ... "$l"

Resources