In the following shell script I am unable to set a user-agent with spaces in it. I am getting word splitting. The bit after the first space (i.e. "(Macintosh;") is being interpreted by curl as a url.
If I type it in into the console it work fine but not when I use substitution.
PARAMS="-v"
PARAMS="${PARAMS} --user-agent \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko)\"" #does not work
#PARAMS="${PARAMS} --user-agent \"Mozilla/5.0\"" #works
curl ${PARAMS} $1 > results.txt
Can someone please explain why?
The problem is explained in the Bash FAQ
The solution is a slightly different syntax.
PARAMS=(-v)
PARAMS+=( "-A Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko)")
curl "${PARAMS[#]}" $1 > results.txt
From here: http://wiki.bash-hackers.org/syntax/quoting
These quote characters (", double quote and ', single quote) are a syntax element that influences parsing. It is not related to eventual quote characters that are passed as text to the commandline! The syntax-quotes are removed before the command is called!
So there is a fundamental difference between cmd "my args" and myargs="\"my args\""; cmd $myargs.
Try replacing the spaces with %20
You can do this in the script if you want like:
str_replace ( ' ', '%20', 'what you need here' );
Hope this helps.
Related
I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code:
useragent='--user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0"'
cookies_file="/tmp/wget-cookies.txt"
save_cookies_cmd="--save-cookies $cookies_file --keep-session-cookies"
load_cookies_cmd="--load-cookies $cookies_file --keep-session-cookies"
function mywget {
log "#!!!!!!!!!# WGET #!!!!!!!!!# wget $quiet $useragent $load_cookies_cmd $#"
wget $useragent $load_cookies_cmd "$#"
}
Saddly isn't working. Somehow I'm missing the right way to store parameters on variables $useragent, $save_cookies_cmd, $load_cookies_cmd and caling wget passing these vars as parameters.
I want the result commandline as this:
wget --user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" --load-cookies /tmp/wget-cookies.txt --keep-session-cookies http://mysite.local/myfile.php
Drop the inner quotes when setting $useragent, but retain the double quotes when you use it:
useragent='--user-agent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0'
...
wget "$useragent" $load_cookies_cmd "$#"
To understand why this works, notice that wget --user-agent="string with spaces" is entirely equivalent to wget "--user-agent=string with spaces". Wget receives (and must requires) the --user-agent=... option as a single argument, regardless of the positioning of the quotes.
The quotes serve to prevent the shell from splitting the string, which is why wget "$useragent" is necessary. On the other hand, the definition of user-agent needs quotes for the assignment to work, but doesn't need a second level of quotes, because those would be seen by Wget and become part of the user-agent header sent over the wire, which you don't want.
I've trouble with finding a list of all symbols that need to be treated specially in Windows cmd.exe command line.
For example,
C:\Users\boda>echo ">"
Outputs:
">"
But
C:\Users\boda>echo '>'
Doesn't output anything.
Similarly none of these work:
C:\Users\boda>echo \'>\'
C:\Users\boda>echo \'\>\'
C:\Users\boda>echo '\>'
There are similar issues with characters | and ^ and probably more.
Does anyone know how to use raw strings in cmd.exe?
These links might help you out!
http://ss64.com/nt/syntax-esc.html
http://www.robvanderwoude.com/escapechars.php
I have written a bash script for Mac OS X that gathers a lot of computer information and writes it to a MySQL database. Everything works fine except one thing. One of the pieces of information I gather is the Computer Name. Probably around 90% of the computers have an apostrophe in their names. The problem is that when I try to write the variable to MySQL it errors out due to the apostrophe.
Is there a way to find if a variable contains an apostrophe and if it does rewrite the variable with the escape character? I am sure I need to use sed but I am still pretty new to this.
For example if I have the following variable:
COMPUTER_NAME="Fred Flintstone's MacBook Air"
How can I change the variable to:
COMPUTER_NAME="Fred Flintstone\'s MacBook Air"
I can pass this off to perl of I need to but I would rather keep it in BASH.
I also don't want to rename the computers because there are over 300 of them.
This will escape apostrophes that are not already escaped:
${COMPUTER_NAME/[^\\]\'/\\\'}
Try doing this to escape what you want :
string="Hello, c'est bientôt l'été ?"
printf '%q\n' "$string"
Output
$'Hello, c\'est bient\303\264t l\'\303\251t\303\251 ?'
You can use Parameter expansion with substitution:
COMPUTER_NAME="${COMPUTER_NAME//\'/\'}"
I have a script in a file:
#../Python32/Test.py
y = input("Please input:")
print("'" + y + "'")
from command line, I run the script, and the result is different between XP and Win7:
On XP:
> Python Text.py
Please input:h
'h'
On Win7, the result is:
> Python Text.py
Please input:h
'h
On Win7, the second single quote is missed in the result.
But if the script is like following:
#../Python32/Test2.py
y = input("Please input:")
print("'" + y.rstrip() + "'")
The results are the same on both win7 and XP.
> Python Text.py
Please input:h
'h'
Comparing the results, there should be special character following the read string from command line on win7. I searched it on the internet and tried all the ways which I know, but cannot get the answer.
What's the special character followed the read string via command line on win7? Or is there a way that I can print the special character in my script?
Thanks advance for your response.
You have a carriage return in the string, from the CRLF line ending on Windows. This moves the cursor to the beginning of the line, overwriting the first quote. input() is supposed to remove that. I thought that bug was fixed in 3.2.1. Are you sure you're not using 3.2?
I would need to read certain data using curl. I'm basically reading keywords from file
while read line
do
curl 'https://gdata.youtube.com/feeds/api/users/'"${line}"'/subscriptions?v=2&alt=json' \
> '/home/user/archive/'"$line"
done < textfile.txt
Anyway I haven't found a way to form the url to curl so it would work. I've tried like every possible single and double quoted versions. I've tried basically:
'...'"$line"'...'
"..."${line}"..."
'...'$line'...'
and so on.. Just name it and I'm pretty sure that I've tried it.
When I'm printing out the URL in the best case it will be formed as:
/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE
or something similar. If you know what could be the cause of this I would appreciate the information. Thanks!
It's not a quoting issue. The problem is that your keyword file is in DOS format -- that is, each line ends with carriage return & linefeed (\r\n) rather than just linefeed (\n). The carriage return is getting read into the line variable, and included in the URL. The giveaway is that when you echo it, it appears to print:
/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE"
but it's really printing:
https://gdata.youtube.com/feeds/api/users/KEYWORD FROM FILE
/subscriptions?v=2&alt=json
...with just a carriage return between them, so the second overwrites the first.
So what can you do about it? Here's a fairly easy way to trim the cr at the end of the line:
cr=$'\r'
while read line
do
line="${line%$cr}"
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
> "/home/user/archive/$line"
done < textfile.txt
Your current version should work, I think. More elegant is to use a single pair of double quotes around the whole URL with the variable in ${}:
"https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"
Just use it like this, should be sufficient enough:
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" > "/home/user/archive/${line}"
If your shell gives you issues with & just put \&, but it works fine for me without it.
If the data from the file can contain spaces and you have no objection to spaces in the file name in the /home/user/archive directory, then what you've got should be OK.
Given the contents of the rest of the URL, you could even just write:
while read line
do
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
> "/home/user/archive/${line}"
done < textfile.txt
where strictly the ${line} could be just $line in both places. This works because the strings are fixed and don't contain shell metacharacters.
Since you're code is close to this, but you claim that you're seeing the keywords from the file in the wrong place, maybe a little rewriting for ease of debugging is in order:
while read line
do
url="https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"
file="/home/user/archive/${line}"
curl "$url" > "$file"
done < textfile.txt
Since the strings may end up containing spaces, it seems (do you need to expand spaces to + in the URL?), the quotes around the variables are strongly recommended. You can now run the script with sh -x (or add a line set -x to the script) and see what the shell thinks it is doing as it is doing it.