I have a bash-script that moves backup-files to the remote location. On few occasions the temporary HDDs on the remote server had no space left, so I added a md5 check to compare local and remote files.
The remote ssh breaks however the while-loop (i.e. it runs only for first item listed in dir_list file).
# populate /tmp/dir_list
(while read dirName
do
# create archive files for sub-directories
# populate listA variable with archive-file names
...
for fileName in $listA; do
scp /PoolZ/__Prepared/${dirName}/$fileName me#server:/archiv/${dirName}/
md5_local=`md5sum /PoolZ/__Prepared/${dirName}/${fileName} | awk '{ print $1 }'`
tmpRemoteName=`printf "%q\n" "$fileName"` # some file-names have strange characters
md5_remote=`ssh me#server 'md5sum /archiv/'${dirName}'/'$tmpRemoteName | awk '{ print $1 }'`
if [[ $md5_local == $md5_remote ]]; then
echo "Checksum of ${fileName}: on local ${md5_local}, on remote ${md5_remote}."
mv -f /PoolZ/__Prepared/${dirName}/$fileName /PoolZ/__Backuped/${dirName}/
else
echo "Checksum of ${fileName}: on local ${md5_local}, on remote ${md5_remote}."
# write eMail
fi
done
done) < /tmp/dir_list
When started the script gives the same md5-sums for the first directory listed in dir_list. The files are also copied both local and remote to expected directories and then script quits.
If I remove the line:
md5_remote=`ssh me#server 'md5sum /archiv/'${dirName}'/'$tmpRemoteName | awk '{ print $1 }'`
then apparently the md5-comaprison is not working but the whole script goes through whole list from dir_list.
I also tried to use double-quotes:
md5_remote=`ssh me#server "md5sum /archiv/${dirName}/${tmpRemoteName}" | awk '{ print $1 }'`
but there was no difference (broken dirName-loop).
I went so far, that I replaced the md5_remote... line with a remote ls-command without any shell-variables, and eventually I even tried a line without setting value to the md5_remote variable, i.e.:
ssh me#server "ls /dir/dir/dir/ | head -n 1"
Every solution that has a ssh-command breaks the while-loop. I have no idea why ssh should break bash-loop. Any suggestion are welcomed.
I'm plainly stupid. I found just answer on — what a surprise — stackoverflow.com.
ssh breaks out of while-loop in bash
As suggested I added a pipe to /dev/null and it works now:
md5_remote=`ssh me#server 'md5sum /archiv/'${dirName}'/'$tmpRemoteName < /dev/null | awk '{ print $1 }'`
Related
I'm reasonably new to shell scripting, so I've had difficulty applying the answers to similar questions to my problem.
I am trying to ssh to a remote server, perform multiple awk commands, and return the value of each to a local .txt file. I am also trying to ssh to other servers, perform similar commands, and return them to the same text file.
If i manually ssh into the remote server and run df -h | awk '$6 == "/" {print $5; exit} I get the % used value for the root directory from the df -h command.
Thus far, I have the following, but while it enters and exits the remote server it doesn't save the value.
> testfile.txt
ssh {$CURRENT_ENV} << EOF
VAL=\$(df -h | awk '\$6 == "/" {print \$5; exit}')
echo "\$VAL > testfile.txt"
exit
EOF
I've looked at single line awk returns, but as I have multiple commands to run it doesn't seem optimal. I would appreciate any suggestions!
First of all, your ssh command. Using a here-document is a good idea. You can improve it in two ways by:
indenting with TABs owing to the <<- syntax. This is purely cosmetic and makes your code more readable.
avoiding the escaping of special characters like $ by quoting EOF. This is not only cosmetic but makes your code less error-prone.
This gives:
ssh {$CURRENT_ENV} <<- 'EOF'
VAL=$(df -h | awk '$6 == "/" {print $5; exit}')
echo "$VAL > testfile.txt"
exit
EOF
(we could even put a tabulation before EOF)
Now your code :
You don't tell us what CURRENT_ENV is. I assume this is something like user#server. To use that variable, write "$CURRENT_ENV", not '{CURRENT_ENV}'. Unless you know what you are doing, when using a variable, always enclose it in double-quotes to avoid any undesirable side-effect.
You put the result of df into variable VAL and write its content to textfile.txt:
As a universal convention, use lower cases for you variable names (unless they are exported to the environment which is not the case here); i.e. this should be val, not VAL.
echo "$val > testfile.txt" won't write anything into textfile.txt because your redirection is inside double-quotes, and thus belongs to the text that is echo-ed. Proper command would be echo "$val" > testfile.txt
Now, think about it: all this, including this echo is executed on the remote server, therefore this will create the file testfile.txt there, not on your machine. This is not what you want, so let's remove that echo line. Let's also remove val= since val is not needed any longer.
The exit command is of no need. Once the last command will be read and executed, the ssh session ends anyway.
We are left with this;
ssh "$CURRENT_ENV" <<- 'EOF'
df -h | awk '$6 == "/" {print $5; exit}'
EOF
(remember this is a tabulation before df but single spaces wouldn't harm in this case)
As it is now, your code outputs everything to your terminal. Let's now redirect this to your local file testfile.txt :
ssh {$CURRENT_ENV} <<- 'EOF' > testfile.txt
df -h | awk '$6 == "/" {print $5; exit}'
EOF
OK, this works for one server. You told us there were actually several ones. You don't show us your code, so I will assume there is a loop somewhere:
for ssh_target in u1#server1 u2#server2 ...; do
ssh "$ssh_target" <<- 'EOF' > testfile.txt
df -h | awk '$6 == "/" {print $5; exit}'
EOF
done
Almost there! The problem with this command is that each loop overwrites the content of testfile.txt. The solution is to redirect the for loop, NOT the ssh command inside it:
for ssh_target in u1#server1 u2#server2 ...; do
ssh "$ssh_target" <<- 'EOF'
df -h | awk '$6 == "/" {print $5; exit}'
EOF
done > testfile.txt
(the redirection must be put after done)
Here it is!
I think you can do this:
# Write all of this script's stdout to testfile.txt
> testfile.txt
# Run df -h on remote host, and pipe the stdout of df -h to awk
ssh "$host" df -h | awk '$6 == "/" {print $5; exit}'
Note that if you specify a filepath with df then you can get information for just that filepath. In your case you are only interested in /, so you can specify that.
If you are only interested in the percent used, you can use the --ouput=pcent option, and then use tail to get only the percentage part (leaving out the Use% header):
df --output=pcent / | tail -n 1
This will produce output with one or two leading spaces if the percent used is less than 100%. The leading spaces can be deleted with tr:
df --output=pcent / | tail -n 1 | tr -d ' '
If for some reason you want to avoid having two pipes, but you still want to remove leading spaces, you can use awk:
df --output=pcent / | awk 'NR == 2 {print $1}'
However using awk might be slower, I'm not sure.
If you want to print anything to local file, just do redirect immediately after EOF
here is example how to write VAL to remotefile and to localfile. Also you can use df --output pcent to show only percentage column. Unfortunately there is no option to hide header, so tail -n1
ssh {$CURRENT_ENV} << EOF >>localfile.txt
VAL=\$(df --output=pcent /|tail -n1)
echo "$VAL" > remotefile.txt
echo "$VAL" # for localfile
exit
EOF
There are a few answers on this topic already, but pretty much all of them say that it's bad to parse the output of ls -l, and therefore suggest other methods.
However, I'm using ncftpls -l, and so I can't use things like shell globs or find – I think I have a genuine need to actually parse the ls -l output. Don't worry if you're not familiar with ncftpls, the output returns in exactly the same format as if you were just using ls -l.
There is a list of files at a public remote ftp directory, and I don't want to burden the remote server by re-downloading each of the desired files every time my cronjob fires. I want to check, for each one of a subset of files within the ftp directory, whether the file exists locally; if not, download it.
That's easy enough, I just use
tdy=`date -u '+%Y%m%d'`_
# Today's files
for i in $(ncftpls 'ftp://theftpserver/path/to/files' | grep ${tdy}); do
if [ ! -f $i ]; then
ncftpget "ftp://theftpserver/path/to/files/${i}"
fi
done
But I came upon the issue that sometimes the cron job will download a file that hasn't finished uploading, and so when it fires next, it skips the partially downloaded file.
So I wanted to add a check to make sure that for each file that I already have, the local file size matches the size of the same file on the remote server.
I was thinking along the lines of parsing the output of ncftpls -l and using awk, something like
for i in $(ncftpls -l 'ftp://theftpserver/path/to/files' | awk '{print $9, $5}'); do
...
x=filesize # somehow get the file size and the filename
y=filename # from $i on each iteration and store in variables
...
done
but I can't seem to get both the filename and the filesize from the server into local variables on the same iteration of the loop; $i alternates between $9 and $5 in the awk string with each iteration.
If I could manage to get the filename and filesize into separate variables with each iteration, I could simply use stat -c "%s" $i to get the local size and compare it with the remote size. Then its a simple ncftpget on each remote file that I don't already have. I tinkered with syncing programs like lftp too, but didn't have much luck and would rather do it this way.
Any help is appreciated!
for loop splits when it sees any whitespace like space, tab, or newline. So, IFS is needed before loop, (there are a lot of questions about ...)
IFS=$'\n' && for i in $(ncftpls -l 'ftp://theftpserver/path/to/files' | awk '{print $9, $5}'); do
echo $i | awk '{print $NF}' # filesize
echo $i | awk '{NF--; print}' # filename
# you may have spaces in filenames, so is better to use last column for awk
done
The better way I think is to use while not for, so
ls -l | while read i
do
echo $i | awk '{print $9, $5}'
#split them if you want
x=echo $i | awk '{print $5}'
y=echo $i | awk '{print $9}'
done
I have written the following bash script which is designed to create a file if the UFW rules get changed on a server. The existence of this file will then be checked with Zabbix.
#!/bin/bash
file="/tmp/ports.txt"
file_open="/tmp/open_ports.txt"
md5_file=$(md5sum /tmp/ports.txt | awk '{ print $1 }')
md5_file_open=$(md5sum /tmp/open_ports.txt | awk '{ print $1 }')
file_diff="/tmp/ports_diff"
if [[ ! -f $file ]]; then
touch $file && sudo ufw status|grep ALLOW > $file
fi
if [ -f $file_diff ];then
rm $file_diff
fi
sudo ufw status|grep ALLOW > $file_open
if [ $md5_file != $md5_file_open ];then
touch $file_diff
fi
What i'm finding is that sometimes the file doesnt get created or deleted when it should but if I run the command the 2nd or 3rd time without anything further changing, it does.
Please help.
Thanks
During the first call the files "/tmp/ports.txt" and "/tmp/open_ports.txt" don't exist. After the first run both files should be created and the diff is empty.
Afer this, the ufw status changes.
In the next run, the statements
md5_file=$(md5sum /tmp/ports.txt | awk '{ print $1 }')
md5_file_open=$(md5sum /tmp/open_ports.txt | awk '{ print $1 }')
will process the original files (without the changed status), so they will be operating on the files that are equal. No diff will be found.
During this run $file_open will be filled with the new value, but the md5sum function is not called after this change. The diff wile will nogt be made.
The next run will start with the changed $file_open, and the difference will be found.
When the status is changed back to the original value, the first run the md5 on the old files, claining to see a difference and not deleting the diff file. This run will also write the $file_open with the new values, that will be detected the next run.
Solution:
Move the 2 md5sum lines until after the last sudo.
I am archiving and using split to produce several parts while also printing the output files (from split on STDERR, which I am redirecting to STDOUT). However the loop over the output data doesn't happen until after the command returns.
Is there anyway to actively split over the STDOUT output of a command before it returns?
The following is what I currently have, but it only prints the list of filenames after the command returns:
export IFS=$'\n'
for line in `data_producing_command | split -d -b $CHUNK_SIZE --verbose - $ARCHIVE_PREFIX 2>&1`; do
FILENAME=`echo $line | awk '{ print $3 }'`
echo " - $FILENAME"
done
Try this:
data_producing_command | split -d -b $CHUNK_SIZE --verbose - $ARCHIVE_PREFIX 2>&1 | while read -r line
do
FILENAME=`echo $line | awk '{ print $3 }'`
echo " - $FILENAME"
done
Note however that any variables set in the while loop will not preserve their values after the loop (the while loop runs in a subshell).
There's no reason for the for loop or the read or the echo. Just pipe the stream to awk:
... | split -d -b $CHUNK_SIZE --verbose - test 2>&1 |
awk '{printf " - %s\n", $3 }'
You are going to see some delay from buffering, but unless your system is very slow or you are very perceptive, you're not likely to notice it.
The command substitution needs1 to run before the for loop can start.
for item in $(command which produces items); do ...
whereas a while read -r can start consuming output as soon as the first line is produced (or, more realistically, as soon as the output buffer is full):
command which produces items |
while read -r item; do ...
1 Well, it doesn't absolutely need to, from a design point of view, I suppose, but that's how it currently works.
As William Pursell already noted, there is no particular reason to run Awk inside a while read loop, because that's something Awk does quite well on its own, actually.
command which produces items |
awk '{ print " - " $3 }'
Of course, with a reasonably recent GNU Coreutils split, you could simply do
split --filter='printf " - %s\n" "$FILE"'; cat >"$FILE" ... options
I have a text file holding numerous records formatted as follows:
Ford:Mondeo:1997:Blue:5
There are around 100 of them that I'm trying to sort via a bash script, and I'm looking to extract all cars made between 1994 and 1999. Here's what I have so far:
awk -F: '$3=="1994"' | awk -F: '$3<="1999"' $CARFILE > output/1994-1999.txt
The output file is containing all of the correct information, no duplicates etc but it freezes and doesn't echo out the confirmation afterwards. I have to ctrl + D my way out of the script.
Here's the full file for reference:
#CS101 Assignment BASH script
#--help option
#case $1 in
# --help | carslist.txt)
# cat <<-____HALP
# Script name: ${0##*/} [ --help | carslist.txt ]
# This script will organise the given text file and save various #sections to new files.
# No continuation checks are used but when each part is finished, a #confirmation message will print before the script continues.
#____HALP
# exit 0;;
#esac
CARFILE=$1
while [ ! -f "$CARFILE" ]
do
echo "We cannot detect a car file to load, please enter the new filename and press [ENTER]"
read CARFILE
done
echo "We have detected that you're using $CARFILE as your cars file, please continue."
if [ -f output ]
then
echo "Sorry, a file called 'output' exists in the working directory. The script will now exist."
elif [ -d output ]
then
echo "The directory 'output' has been detected, instead of creating a new one we'll be working in there instead."
else
mkdir output
echo "We couldn't find an existing file or directory named 'output' so we've made one for you. Aren't we generous?"
fi
grep 'Vauxhall' $CARFILE > output/Vauxhall_Cars.txt
echo "We've saved all Vauxhall information in the 'Vauxhall_Cars.txt' file. The script will now continue."
grep '2001' $CARFILE > output/Manufactured_2001.txt
echo "We've saved all cared manufactured in 2001 in the 'Manufactured_2001.txt' file. The script will now continue."
awk -F: '$1=="Volkswagen" && $4=="Blue"' $CARFILE > output/Blue_Volkswagen.txt
echo "We've saved all Blue Volkswagens cars in Blue_Volkswagen.txt. The script will now continue"
grep 'V' $CARFILE > output/Makes_V.txt
echo "All cars with the make starting with 'V' have been saved in Makes_V.txt. The script will now continue."
awk -F: '$3=="1994"' | awk -F: '$3<="1999"' $CARFILE > output/1994-1999.txt
echo "Cars made between 1994 and 1999 have been saved in 1994-1999.txt. The script will now continue."
With the run command being bash myScript.sh carslist.txt
Can anyone tell me why it's freezing after outputting correctly?
Just noticed that a record of 1993 has slipped through the cracks, is there a way of formatting the dates in the line above so it's only between 1994-1999?
Thanks in advance.
The expression:
awk -F: '$3=="1994"' | awk -F: '$3<="1999"' $CARFILE > output/1994-1999.txt
Means: run awk on "something" and then pipe to another awk. But you are not providing any "something", so awk is waiting for it.
This is like saying:
awk '{print}' | awk 'BEGIN {print 1}'
It indeed prints the 1 but waits for some kind of input to come.
Just join the conditions:
awk -F: '$3=="1994" && $3<="1999"' $CARFILE > output/1994-1999.txt
Regarding the rest of the script: note you are not using many double quotes. They are a good practise to prevent problems when you have names with spaces, etc. So for example you can say grep 'Vauxhall' "$CARFILE" and allow $CARFILE to contains things like "my car".
You can find out these kind of errors by pasting your script in ShellCheck.