Bash script for downloading files with Curl - bash

I'm trying to knock together a little batch script for downloading files, that takes a URL as its first parameter and a local filename as its second parameter. In testing I've learned that its tripping up on spaces in the output filename, so I've tried using sed to escape them, but its not working.
#!/bin/bash
clear
echo Downloading $1
echo
filename=`sed -e "s/ /\\\ /g" $2`
echo $filename
echo eval curl -# -C - -o $filename $1
but I get the message
sed: outfile.txt: No such file or directory
which suggests its trying to load the output file as input to sed instead of treating the output filename as a string literal.
What would be the correct syntax here?

quoting the arguments correctly, rather than transforming them might be a better approach
It's quite normal to expect to have to quote spaces in arguments to shell scripts
e.g.
#!/bin/bash
clear
echo Downloading $1
echo `curl -# -C - -o "${2}" "${1}"`
called like so
./myscript http://www.foo.com "my file"
alternatively, escape the spaces with a '\' as you call them
./myscript http://www.example.com my\ other\ filename\ with\ spaces

I agree with cms. Quoting the input arguments correctly is much better style - what will you do with the next problem character?
The following is much better.
curl -# -C - -o "$2" $1
However, I hate people not answering the asked question, so here's an answer :-)
#!/bin/bash
clear
echo Downloading $1
echo
filename=`echo $2 | sed -e "s/ /\\\ /g"`
echo $filename
echo eval curl -# -C - -o $filename $1

curl -# -C - -o "$2" $1

if $2 is a text input then try
echo $2 | sed 's: :\\\ :g '
I generally avoid backslashes in sed delimiters are they are quite confusing.

Related

error because eval command seems to remove backslashes in stored command

I want to be able to add newline characters before every occurences of some tokens appearing in some .tex files that I possess, some of those tokens are '\itemQ', '\pagebreakQ'. I created a procedure that ends up creating a command for sed stored in $sedInpt:
~$ echo "$sedInpt"
-e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/
I want to use "$sedInpt" as a command for sed:
echo "$inputText" | eval "sed ${sedInpt}"
but if I do the following as a test:
echo 'hello\itemQ' | eval "sed ${sedInpt}"
hello\itemQ
you can see there ain't any newline that has been added before \itemQ.
So I've tried debugging this way of doing thing by calling bash -x to see what's happened in detail:
~$ bash -x
~$ echo "hello\itemQ" | eval "sed ${sedInpt}"
+ echo 'hello\itemQ'
+ eval 'sed -e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/'
++ sed -e 's/(\itemQ)/n1/' -e 's/(\pagebreakQ)/n1/'
hello\itemQ
you can see that the backslashes of \n and \1 and even the ones before ( and ) that I had placed in "$sedInpt" seem to have disappeared when parsed by eval.
So I am bit lost on what to do next to do what I want.. any ideas?
You could also just combine them into a single command, which in my opinion is more straightforward:
$ cat /tmp/sed.sh
sedInpt='s/\(\\itemQ\)/\n\1/; s/\(\\pagebreakQ\)/\n\1/'
echo "hello\itemQ" | sed "$sedInpt"
$ /tmp/sed.sh
hello
\itemQ
Edit: As #123 rightly points out, storing commands in variables is dangerous and should be avoided if possible. If you have complete control over what is stored, it should be safe, but if it comes from any sort of user input, it is a "Command Injection" vulnerability.
Following #Inian advice I managed to achieve what I wanted to do in this way:
~$ sedInpt=( -e 's/\(\\itemQ\)/\n\1/' -e 's/\(\\pagebreakQ\)/\n\1/' )
~$ echo "hello\itemQ" | sed "${sedInpt[#]}"
hello
\itemQ

Making bash script with command already containing '$1'

Somewhere I found this command that sorts lines in an input file by number of characters(1st order) and alphabetically (2nd order):
while read -r l; do echo "${#l} $l"; done < input.txt | sort -n | cut -d " " -f 2- > output.txt
It works fine but I would like to use the command in a bash script where the name of the file to be sorted is an argument:
& cat numbersort.sh
#!/bin/sh
while read -r l; do echo "${#l} $l"; done < $1 | sort -n | cut -d " " -f 2- > sorted-$1
Entering numbersort.sh input-txt doesn't give the desired result, probably because $1 is already in using as an argument for something else.
How do I make the command work in a shell script?
There's nothing wrong with your original script when used with simple arguments that don't involve quoting issues. That said, there are a few bugs addressed in the below version:
#!/bin/bash
while IFS= read -r line; do
printf '%d %s\n' "${#line}" "$line"
done <"$1" | sort -n | cut -d " " -f 2- >"sorted-$1"
Use #!/bin/bash if your goal is to write a bash script; #!/bin/sh is the shebang for POSIX sh scripts, not bash.
Clear IFS to avoid pruning leading and trailing whitespace from input and output lines
Use printf rather than echo to avoid ambiguities in the POSIX standard (see http://pubs.opengroup.org/onlinepubs/009604599/utilities/echo.html, particularly APPLICATION USAGE and RATIONALE sections).
Quote expansions ("$1" rather than $1) to prevent them from being word-split or glob-expanded
Note also that this creates a new file rather than operating in-place. If you want something that operates in-place, tack a && mv -- "sorted-$1" "$1" on the end.

How to use grep with quotes inside a shell script?

I am trying to use the following pattern inside a shell script eg. I want to use the following:
grep ^"#" $fn | grep -v NM$ > $op
but inside bash script. The problem is normally bash considers everything after the "#" as a comment. If i use
grep ^"\#" $fn
I think it changes the meaning. I am a newbie.
Any help will be appreciated.
There are multiple possibilities:
grep '^#' $fn
or
grep "^#" $fn
or
grep ^\# $fn
Here is a little example which I saved as foo.sh
echo "#"
echo '#' # this is a comment
echo '\#'
when run as sh foo.sh this is what is printed out
#
#
\#

overwrite a file then append

I have a loop in my script that will append a list of email address's to a file "$CRN". If this script is executed again, it will append to this old list. I want it to overwrite with the new list rather then appending to the old list. I can submit my whole script if needed. I know I could test if "$CRN" exists then remove file, but I'm interested in some other suggestions? Thanks.
for arg in "$#"; do
if ls /students | grep -q "$arg"; then
echo "${arg}#mail.ccsf.edu">>$CRN
((students++))
elif ls /users | grep -q "$arg$"; then
echo "${arg}#ccsf.edu">>$CRN
((faculty++))
fi
Better do this :
CRN="/path/to/file"
:> "$CRN"
for arg; do
if printf '%s\n' /students/* | grep -q "$arg"; then
echo "${arg}#mail.ccsf.edu" >> "$CRN"
((students++))
elif printf '%s\n'/users/* | grep -q "${arg}$"; then
echo "${arg}#ccsf.edu" >> "$CRN"
((faculty++))
fi
done
don't parse ls output ! use bash glob instead. ls is a tool for interactively looking at file information. Its output is formatted for humans and will cause bugs in scripts. Use globs or find instead. Understand why: http://mywiki.wooledge.org/ParsingLs
"Double quote" every expansion, and anything that could contain a special character, eg. "$var", "$#", "${array[#]}", "$(command)". See http://mywiki.wooledge.org/Quotes http://mywiki.wooledge.org/Arguments and http://wiki.bash-hackers.org/syntax/words
take care to false positives like arg=foo and glob : foobar, that will match. You need grep -qw then if you want word boundaries. UP2U

How can I use bash to parse out only a section of a variable with different delimiters?

I have a loop in a bash file to show me all of the files in a directory, each as its own variable. I need to take that variable (filename) and parse out only a section of it.
Example:
92378478234978ehbWHATIWANT#98712398712398723
Now, assuming "ehb" and the pound symbol never change, how can I just capture WHATIWANT into its own variable?
So far I have:
#!/bin/bash
for FILENAME in `dir -d *` ; do
done
You can use sed to edit out the parts you don't want.
want=$(echo "$FILENAME" | sed -e 's/.*ehb\(.*\)#.*/\1/')
Or you can use Bash's parameter expansion to strip out the tail and head.
want=${FILENAME%#*}; want=${want#*ehb}
One possibility:
for i in '92378478234978ehbWHATIWANT#98712398712398723' ; do
j=$(echo $i | sed -e 's/^.*ehb//' -e 's/#.*$//')
echo $j
done
produces:
WHATIWANT
using only the bash shell, no need external tools
$ string=92378478234978ehbWHATIWANT#98712398712398723
$ echo ${string#*ehb}
WHATIWANT#98712398712398723
$ string=${string#*ehb}
$ echo ${string%#*}
WHATIWANT

Resources