I want to run a bash command once for each line of a text file.
If I do:
while read l; do
echo $l;
done < myfile.txt
I get each line echoed.
But when I use my actual command:
while read l; do
curl -X POST http://somedomain -H "Content-Type: application/json" -d $l
done < myfile.txt
I get errors. I think it may have something to do with escaping characters in the text file, but nothing I have tried works. What is the proper way to run a curl expression once for each line of a text file?
To use quotes.
foocmd ... "$l"
Related
When write bash scripts, I want to store my whole curl command in heredoc to get a better layout. The following works fine:
#/bin/bash
read -r -d '' command1 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
MULTI_STRING_SCOPE
But when add some json data with the -d option, the command is executed weirdly. For example:
#/bin/bash
read -r -d '' command2 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
-d '{
"hello":"world"
}'
MULTI_STRING_SCOPE
response2=$(${command2})
Wrong logs from terminal:
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 1:
}'
And it seems that the curl take line }' as a seperated url, and thus the json data not sent as a unit.
How to solve the problem? Any suggestions will be highly appreciated.
I learned from this post to make heredoc work with curl command.
As the comment made by #Gordon Davisson in current post, we should not mix command with data. Since the json data set to -d option is only data and other parts is command, so I decide to use heredoc to store only the json data and remain other parts to be command itself, rather than store them in string by heredoc.
The result bash script should be something like this:
#/bin/bash
response3=$(curl -v www.stackoverflow.com \
-d #- <<- MULTI_STRING_SCOPE
{
"hello":"world"
}
MULTI_STRING_SCOPE
)
Notice: heredoc indent only works with tab, not with blanks. Be careful, especially when you are working with editors like Visual Studio Code, which may have already set indent as blanks for you.
I am having some trouble figuring out the expansion of a variable in my shell script. If I replace the variable with the desired string it works.
#!/bin/zsh
KEY="$(curl -Ivs -X GET "http://admin:admin#192.168.1.1" &> >(awk '/^> Authorization/{ print $3 " " $4 }'))"
# The string returned by the curl and awk command is Basic "YWRtaW46YWRtaW4=" without double quotes.
curl -H "Authorization: $KEY" "http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=1"
# This doesn't work
curl -H "Authorization: Basic YWRtaW46YWRtaW4=" "http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=1"
# This works
The only thing thats different in the above two lines is.
-H "Authorization: $KEY"
-H "Authorization: Basic YWRtaW46YWRtaW4="
HTTP generally uses DOS style CR+LF line terminators. Whenever you parse data out of curl, you have to account for this.
To check if this is the problem, run your script with bash -x yourscript or zsh -x yourscript to see trace output that shows otherwise invisible carriage returns:
var=$'value\C-M' # zsh
var=$'value\r' # bash
(Dash and ash/busybox unfortunately doesn't highlight this problem, so try with one of the above shells)
To strip them, pipe your data through tr -d '\r'.
I am trying to use cURL to upload files with spaces in their filenames onto a dedicated server. I am using bash. In a previous project, I just gave up, removing all spaces from filenames. This is not feasible for this project.
Running cURL in verbose mode suggests that it stops when it reads my local file path:
curl -X PUT -u $USER:$PASS --data-binary #"$LOCAL_FILE" "$SERVER/remote.php/dav/files/$USER/$REMOTE_DIR/$REMOTE_FILE"
where $LOCAL_FILE is a path to a file on my local machine (with spaces), and $REMOTE_FILE also has spaces.
This gives:
Warning: Couldn't read data from file "/Users/my_account/somepath/with
Warning: spaces
which implies the command is taking "/Users/my_account/somepath/with" and "spaces" as two separate paths.
How can I solve this?
My full code:
#!/bin/bash -ex
# Local dir - note space in path
IMAGES_DIR="/Users/my_account/somepath/with spaces"
# Remote server, and credentials
SERVER="http://myserver"
REMOTE_DIR="mydir"
USER="myname"
PASS="mypass"
FILE="$1"
LOCAL_FILE="$IMAGES_DIR/$FILE"
REMOTE_FILE=urlencode $FILE
# Move file to the server
curl -v -X PUT -u $USER:$PASS --data-binary #"$LOCAL_FILE" "$SERVER/remote.php/dav/files/$USER/$REMOTE_DIR/$REMOTE_FILE"
# Check that file has made it
echo 'Waiting for file to be on server'
until [ $result > 0 ]
do
result=$(curl -I "$SERVER/remote.php/dav/files/$USER/$REMOTE_DIR/$REMOTE_FILE" -u $USER:$PASS 2>/dev/null | grep Content-Length | awk -F ': ' '{print $2}')
echo "."
sleep 2
done
echo "File $FILE is now on server."
urlencode() {
# urlencode <string>
old_lc_collate=$LC_COLLATE
LC_COLLATE=C
local length="${#1}"
for (( i = 0; i < length; i++ )); do
local c="${1:i:1}"
case $c in
[a-zA-Z0-9.~_-]) printf "$c" ;;
*) printf '%%%02X' "'$c" ;;
esac
done
LC_COLLATE=$old_lc_collate
}
In the Windows version of curl 7.55.0 (should be in higher versions also) the below command will work:
local Windows path: path enclosed in double or single quotes
server URL: path enclosed in double or single quotes and each space replaced with '%20' an example to upload tes st.txt is given below
curl -username:password -T "./Test/te st.txt" "http://server-url/folder1/folder2/te%20st.txt"
Did you tried changing the value of IFS in your script? Normally its value is set to $' \t\n', which includes space. Set it to nothing like so: IFS= and no field splitting is performed at all. You can get the default value back by executing unset IFS.
Example:
var="a b c d"
echo $var
a b c d
IFS=
echo $var
a b c d
TL;DR : the problem still occurs on windows with filenames containing illegal characters, like | > < :
As #Jerin points out, curl on windows (at least in versions > 7.55.0) will accept spaces in filenames.
However, with curl 7.57.0 (x86_64-w64-mingw32), I'm observing that
curl -X POST http://localhost:8080/ -H "Content-Type: application/json" --data-binary "#my_dir/2019-11-04>.json"
gives me
Warning: Couldn't read data from file "my_dir/2019-11-04>.json",
Warning: this makes an empty POST.
And doesn't pass the file as POST data.
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
curl 'https://shoesworkshop.net/libraries/ajax/ajax.invoice.php?act=viewallinvoice&invoiceid="${line}"&sEcho=1&iColumns=8&iDisplayStart=0&iDisplayLength=20&bRegex=false&bRegex_0=false&bSearchable_0=true&bRegex_1=false&bSearchable_1=true&bRegex_2=false&bSearchable_2=true&bRegex_3=false&bSearchable_3=true&bRegex_4=false&bSearchable_4=true&bRegex_5=false&bSearchable_5=true&bRegex_6=false&bSearchable_6=true&bRegex_7=false&bSearchable_7=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true&bSortable_4=true&bSortable_5=true&bSortable_6=true&bSortable_7=true' -H 'Host: shoesworkshop.net'| sed 's/^[^[[]*:/:/'
done < "$1"
inside $line there is a value like this
AAAAA
SSSSS
DDDDD
and i want to pass $line into curl command
can someone help me how?
i tried "'${line}'" and '${line}' and it still not working
i want to make a repeat call using curl get request from the url using variable from $line
For simple URLs, one way is to just use double quotes for the complete URL, including your variable expansion, ${line}, like this:
curl "https://shoe...&invoiceid=${line}&sEcho=1&iCo...table_7=true"
(Under single quotes, your shell variable line is not expanded.)
If your URL contains shell-special characters like $, it's best to combine both single and double quotes (and concatenate several strings, like explained here). For Example:
curl 'https://shoe...&invoiceid='"$line"'&sEcho=1&iCo...table_7=true'
# ^------ fixed part ------^ ^var^ ^------- fixed part ------^
However, if your variable contains characters that have to be URL-encoded (like space, &, ?, etc.) it's best to let curl handle that with --data-urlencode option. When called with this option, curl will default to POST method, but you can override this with -G, in which case your parameters will be appended to URL query. For example:
line="1&2?3 4"
curl "http://httpbin.org/get?x=1&y=2" --data-urlencode z="$line" -G
produces the right URL:
http://httpbin.org/get?x=1&y=2&z=1%262%3F3%204
Your script, fixed:
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
curl --data-urlencode invoiceid="$line" -G 'https://shoesworkshop.net/libraries/ajax/ajax.invoice.php?act=viewallinvoice&sEcho=1&iColumns=8&iDisplayStart=0&iDisplayLength=20&bRegex=false&bRegex_0=false&bSearchable_0=true&bRegex_1=false&bSearchable_1=true&bRegex_2=false&bSearchable_2=true&bRegex_3=false&bSearchable_3=true&bRegex_4=false&bSearchable_4=true&bRegex_5=false&bSearchable_5=true&bRegex_6=false&bSearchable_6=true&bRegex_7=false&bSearchable_7=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true&bSortable_4=true&bSortable_5=true&bSortable_6=true&bSortable_7=true' -H 'Host: shoesworkshop.net' | sed 's/^[^[[]*:/:/'
done < "$1"
I need to use the curl command:
curl -d '{ "auth_token": "YOUR_AUTH_TOKEN", "text": "Hey, Look what I can do!" }' \http://localhost:3030/widgets/welcome
in a bash script but instead of "Hey, Look what I can do!" after the "YOUR_AUTH_TOKEN"; I need a variable $p .
This is my first time trying a bash script and read the tutorials on quotes and such but I still am not able to make it work.
The easiest thing to do is to read the data from a here document (a type of dynamically created temporary file), rather then trying to quote the entire thing as a string:
curl -d#- http://localhost:3030/widgets/welcome <<EOF
{ "auth_token": "YOUR_AUTH_TOKEN",
"text": "$p" }
EOF
If the argument to -d begins with a #, the remainder of the argument is taken as the name of a file containing the data. A file name of - indicates standard input, and the standard input to curl is supplied by the lines between the EOF tokens.
The alternative is to double-quote the string and escape all the embedded double quotes. Yuck.
... -d "{\"auth_token\": \"YOUR_AUTH_TOKEN\", \"text\": \"$p\"}" ...
An alternative to chepner's very good answer:
data=$( printf '{"auth_token":"YOUR_AUTH_TOKEN","text":"%s"}' "$p" )
curl -d "$data" http://localhost:3030/widgets/welcome