Cannot pass variable into while loop - bash

I've tried this a few ways but trying to pass a variable into this code when it's a text file just doesn't work. What's weird though is if the check sees it's just a url, it works perfectly.
I've tried -i in wget, quotes around $line, {} around line, putting $directory a few ways into wget. Nothing. It either reads it as blank or as the file name, not the urls in the file.
On top of this mess, $savefile.log in the first part of the loop always returns directory.txt.log. Tried $line.log to fix that and nada. I do need it stripped as : and \ are not valid in a file name.
#!/bin/bash
read -p "Enter directory or .txt file: `echo $'\n> '`" directory
savefile=$(echo "${directory//"http://"}" | cut -d '/' -f1)
if [[ $directory == *.txt ]]
echo
echo "Spidering $directory"
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "$line"
wget -np --spider -e robots=off --no-check-certificate $line 2>> $savefile.log
echo
echo "Spider saved to $savefile.log"
done < $directory
else
echo
echo "Spidering $directory"
wget -r -np --spider -e robots=off --no-check-certificate $directory 2>> $savefile.log
echo
echo "Spider saved to $savefile.log"
fi

EDITx2??:
Removing my old answer because it was wrong, finally got a chance to sit down and run the code, and based off what I THINK you were looking for this should do it:
#!/bin/bash
read -p "Enter directory or .txt file: `echo $'\n> '`" directory
if [[ $directory == *.txt ]]; then
echo
echo "Spidering $directory"
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "$line"
savefile="$(basename "${directory//.txt}")"
wget -np --spider -e robots=off --no-check-certificate $line 2>> $savefile.log
echo
echo "Spider saved to $savefile.log"
done < $directory
else
echo
echo "Spidering $directory"
savefile=$(echo "${directory//"http://"}" | cut -d '/' -f1)
wget -r -np --spider -e robots=off --no-check-certificate $directory 2>> $savefile.log
echo
echo "Spider saved to $savefile.log"
fi
One big issue was you forgot the ; then at the end of the if [[ ... ]] line.
With savefile, I am guessing you were aiming to strip off extra after the ".com" on a url and give just the file name of a file? Works perfect for a url as written, but it nukes file path. Moved it into the else as written. For a file, basename removes the directories and just leaves the file name and the variable expansion strips the troublesome '.txt'.
Tried not to deviate from your code by much, but I would recommend quoting the variables - shouldn't be spaces in the URL, but ~could be~ if input wrong, but mainly because filenames could have un-escaped spaces.
Could also make a tad more compact by using 'echo -e "\nSpidering $directory"' instead of the double echoes. Don't suppose anything wrong with them but they bug my ocd. :P
Finally, I would recommend using the newer syntax for the command substitution, "$(echo "cmd")" instead of the backticks. Again not technically wrong, but since the back-ticks are deprecated they may eventually stop working. Also, and more so I'd say, it makes the code allot more readable, I have to squint at times to tell if it is a single quote or a back-tick.

Related

Bash Read Input - Tab the Prompt

I've got a script that I'm reading the input from users. Here's my code:
if [ -z $volreadexists ]; then
echo -e "\tThis will overwrite the entire volume (/dev/vg01/$myhost)...are you sure?"
read REPLY
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
echo -e "\t\tContinuing"
syncvolume
else
echo "Fine...skipping"
fi
fi
I'm having to use read REPLY because read by itself doesn't insert tabs. What I'm looking for is something similar to:
read -p "\tDoes this look OK? (n for No)" -n 1 -r
Where \t would tab over the read prompt.
How can I add tabs to a read prompt?
UPDATE: Thanks for the great answer from #gniourf!:
read -p $'\tDoes this look OK? (n for No)' -n 1 -r
However, I found an issue. When I attempt to use a variable there it doesn't translate it:
read -p $'\tThis will overwrite the entire volume (/dev/vg01/$myhost)...are you sure? ' -n 1 -r
becomes
This will overwrite the entire volume (/dev/vg01/$myhost)...are you sure?
where I want:
This will overwrite the entire volume (/dev/vg01/server1)...are you sure?
Using doublequotes doesn't work either :(
Any ideas?
Just use ANSI-C quoting:
read -p $'\tDoes this look OK? (n for No)' -n 1 -r
Now, if you want to use variable expansions too, you can mix different quotes like so:
read -p $'\t'"This will overwrite the entire volume (/dev/vg01/$myhost)...are you sure? " -n 1 -r
Here I only used ANSI-C quoting for the tab character. Make sure you don't leave any spaces between $'\t' and "This will....".
I ended up referencing this answer:
Read a variable in bash with a default value
and created a workaround. It's not perfect, but it works:
myhost="server1"
if [ -z $volreadexists ]; then
read -e -i "$myhost" -p $'\tJust checking if it\'s OK to overwrite volume at /dev/vg01/'
echo
if [[ $REPLY =~ ^$myhost[Yy]$ ]]; then
echo -e "\t\tContinuing"
else
echo "Fine...skipping"
fi
fi

Ampersand in bash script not working

Using the Ampersand (&) to place it in the background. But in this script for some reason it doesnt work. My programming skills are not great, so please remember im a noob trying to get stuff working.
#!/bin/bash
# Date in format used by filenaming
date=$(date '+%Y%m%d')
# Location where the patch files should be downloaded
patches=~/lists/patches
# Location of the full list
blacklist=~/lists/list
while :
do
# Fetching last download date from downloaded patches
ldd=$(cd $patches && printf '%s\n' * | sed "s/[^0-9]*//g"); echo $ldd
if [ "$ldd" = "" ]
then
break
else
if [ "$ldd" = "$date" ]
then
break
else
ndd=$(date +%Y%m%d -d "${ldd}+1 days")
# Cant have multiple patches in $patches directory, otherwise script wont work
rm -rf $patches/*
sleep 1
file=$patches/changes-$ndd.diff.gz
curl -s -o "$file" "http://url.com/directory/name-$ndd.diff.gz" &
sleep 1
done=$(jobs -l | grep curl | wc -l)
until [ "$done" == 1 ]
do
echo "still here"
done
gunzip "$file"
# Apply patch directory to list's file directories
cat $(echo "$file" | sed "s/.gz//g") | sed 's/.\/yesterday//' | sed 's/.\/today//' > $patches/$ndd.diff
rm $(echo $file | sed "s/.gz//g")
cd $blacklist
patch -p1 --batch -r /root/fail.patch < $patches/$ndd.diff
rm /root/fail.patch
fi
fi
done
What i want to do is let the script wait for each command until the one before is finished. As you can see i used 'sleep' sometimes but i know that isnt a solution. I also read about the wait command, but then you have to place a command in the background using the Ampersand. And thats the problem. For some reason this script doesnt recognize the ampersand at the end of my curl command. I also tried wget, same results. Who can point me in the right direction?
It would never change done after first check. So you need to check every iteration, that's why you should test for command, not for variable
And while will be better, because you need to check before entering
while [ "$(jobs -l | grep curl | wc -l)" -ne 0 ]; do
echo "Still there"
sleep 1
done
I've added sleep because otherwise it wold just flood your console.

Having trouble with multiple variables using read in a while loop

Still learning and I'm getting lost with IFS=
#!/bin/bash
inotifywait -m -r --excludei '(.txt$|.old$|.swp$|.swx$|.swpx$|.zip$)' /path/to/watch -e create |
while read path action file; do
cd $path
USER="$(stat -c %U $file)"
echo "To: User <user#domain.com>
CC: Support <user#domain.com>
From: $USER <user#domain.com>
Subject: $USER has Uploaded a new File
The user '$USER' uploaded the file '$file' to '$path'" > /mnt/remote/ian/temp2.txt
cat /path/to/temp/file.txt | ssmtp list#domain.com
rm /path/to/temp/file.txt
done
This was my first ever script and it works great as long as there are no spaces in the the file names that get uploaded. I've read some scripts that use IFS= 'whatever' to define the field separators but I don't want to mess around with this while it's in production; it works, but it annoys me when I can't get the username of the user who uploaded the file. Please give me a hint.
This can be broken by an attacker who knows your implementation and wants to spoof arbitrary data (by creating filenames with newlines), but it's a rough first draft:
while IFS= read -r -d $'\n' path && IFS= read -r -d $'\n' file; do
user=$(stat -c %U "$file")
printf 'File %q created by %q in %q\n' "$file" "$user" "$path"
done < <(inotifywait --format $'%w\n%f\n' ~/tmp -r -e create)
I'd strongly suggest filing a ticket with upstream inotifytools requesting the ability use \0 in format strings.
Incidentally, this has already been brought up on the inotify-tools mailing list, where Stephane Chazelas offered a workaround:
nl="
"
inotifywait --syslog -e close_write -mr /tmp --format '%f///' |
while IFS= read -r file; do
while :; do
case $file in
(*///) file=${file%///}; break
esac
IFS= read -r extra || exit
file=$file$nl$extra
done
something with "$file"
done

bash call script with variable

What I want to achieve is the following :
I want the subtitles for my TV Show downloaded automatically.
The script "getSubtitle.sh" is ran as soon as the show is downloaded, but it can happen that no subtitle are released yet.
So what I am doing to counter this :
Creating a file each time "getSubtitle.sh" is ran. It contain the location of the script with its arguments, for example :
/Users/theo/logSubtitle/getSubtitle.sh "The Walking Dead - 5x10 - Them.mp4" "The.Walking.Dead.S05E10.480p.HDTV.H264.mp4" "/Volumes/Window HD/Série/The Walking Dead"
If a subtitle has been found, this file will contain only this line, if no subtitle has been found, this file will have 2 lines (the first one being "no subtitle downloaded", and the second one being the path to the script as explained above)
Now, once I get this, I'm planning to run a cron everyday that will do the following :
Remove all file that have only 1 line (Subtitle found), and execute the script again for the remaining file. Here is the full script :
cd ~/logSubtitle/waiting/
for f in *
do nbligne=$(wc -l $f | cut -c 8)
if [ "$nbligne" = "1" ]
then
rm $f
else
command=$(sed -n "2 p" $f)
sh $command 3>&1 1>&2 2>&3 | grep down > $f ; echo $command >> $f
fi
done
This is unfortunately not working, I have the feeling that the script is not called.
When I replace $command by the line in the text file, it is working.
I am sure that $command match the line because of the "echo $command >> $f" at the end of my script.
So I really don't get what I am missing here, any ideas ?
Thanks.
I'm not sure what you're trying to achieve with the cut -c 8 part in wc -l $f | cut -c 8. cut -c 8 will select the 8th character of the output of wc -l.
A suggestion: to check whether your file contains 1 or two lines (and since you'll need the content of the second line, if any, anyway), use mapfile. This will slurp the file in an array, one line per field. You can use the option -n 2 to read at most 2 lines. This will be much more efficient, safe and nice than your solution:
mapfile -t -n 2 ary < file
Then:
if ((${#ary[#]}==1)); then
printf 'File contains one line only: %s\n' "${ary[0]}"
elif ((${#ary[#]==2)); then
printf 'File contains (at least) two lines:\n'
printf ' %s\n' "${ary[#]}"
else
printf >&2 'Error, no lines found in file\n'
fi
Another suggestion: use more quotes!
With this, a better way to write your script:
#!/bin/bash
dir=$HOME/logSubtitle/waiting/
shopt -s nullglob
for f in "$dir"/*; do
mapfile -t -n 2 ary < "$f"
if ((${#ary[#]}==1)); then
rm -- "$f" || printf >&2 "Error, can't remove file %s\n" "$f"
elif ((${#ary[#]}==2)); then
{ sh -c "${ary[1]}" 3>&1 1>&2 2>&3 | grep down; echo "${ary[1]}"; } > "$f"
else
printf >&2 'Error, file %s contains no lines\n' "$f"
fi
done
After the done keyword you can even add the redirection 2>> logfile to a log file if you wish. Make sure the cron job is run with your user: check crontab -l and, if needed, edit it with crontab -e.
Use eval instead of sh. The reason it works with eval and not sh is due to the number of passes to evaluate variables. sh will treat the sed command as its command to execute while eval will evaluate the sed command first and then execute the result.
Briefly explained.

How to change parameter in a file, only if the file exists and the parameter is not already set?

#!/bin/bash
# See if registry is set to expire updates
filename=hostnames
> test.log
PARAMETER=Updates
FILE=/etc/.properties
CODE=sudo if [ ! -f $FILE] && grep $PARAMETER $FILE; then echo "File found, parameter not found."
#CODE=grep $PARAMETER $FILE || sudo tee -a /etc/.properties <<< $PARAMETER
while read -r -a line
do
hostname=${line//\"}
echo $hostname":" >> test.log
#ssh -n -t -t $hostname "$CODE" >> test.log
echo $CODE;
done < "$filename"
exit
I want to set "Updates 30" in /etc/.properties on about 50 servers if:
The file exists (not all servers have the software installed)
The parameter "Updates" is not already set in the file (e.g. in case of multiple runs)
I am a little puzzled so far how, because I am not sure if this can be done in 1 line of bash code. The rest of the script works fine.
Ok, here's what i think would be a solution for you. Like explained in this article http://www.unix.com/shell-programming-scripting/181221-bash-script-execute-command-remote-servers-using-ssh.html
invoke the script which contains the commands that you want to be executed at the remote server
Code script 1:
while read -r -a line
do
ssh ${line} "bash -s" < script2
done < "$filename"
To replace a line in a text file, you can use sed (http://www.cyberciti.biz/faq/unix-linux-replace-string-words-in-many-files/)
Code script 2:
PARAMETER=Updates
FILE=/etc/.properties
NEWPARAMETER=Updates ###(What you want to write there)
if [ ! -f $FILE] && grep $PARAMETER $FILE; then exit
sed -i 's/$PARAMETER/$NEWPARAMETER/g' $FILE
So, I'm not certain this covers all your use case, I hope this helps you out if there is anything feel free to ask!

Resources