I wrote a shell script which reads some files and copy all the data from these files to other consolidated file.
This happens multiple times and previously copied data will not be copied again. For this i am saving the last copied line in a separate file and comparing it from the second time.
Flow of logic :
For the first time, read individual files and copy the whole data except last line into the consolidated file. Saving the last copied line into a temporary file.
sed '$d' $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
From second time, read last line from the temporary file and search for that line in individual file. When found, take all the lines from the next line and append to consolidated file.
Full script:
#!/bin/bash
consolidatedCSVFile=$1
endTime=$2
curdate=$(date +%s)
CSVFiles=${#:3}
CSVFiles=${CSVFiles[#]}
lastLines=_LASTLINES
lastLine=_LASTLINE
newMerge=false
# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile ]
if [ $? == 1 ]
then
mkdir $consolidatedCSVFile$lastLines
newMerge=true
fi
testInProcess=false
if [ $endTime -ge $curdate ]
then
testInProcess=true
fi
for file in $CSVFiles
do
if $newMerge ;
then
touch $consolidatedCSVFile$1astLines5(basename $file)$lastLine
sed $d $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
availableLastLine="`cat $consolidatedCSVFile$1astLines/$(basename $file)$lastLine`"
if $testInProcess
then
sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
fi
fi
done
if ! $testInProcess ;
then
rm -rf $consolidatedCSVFile$lastLines
fi
Sometimes, i get error as sed: -e expression #1, char 26: unknown command: 'X'
My guess
I am trying to combine the files generated by JMeter.
Sometimes, the files are generating data like 1439459065780,5,SOAP/XML-RPC Request,200,OK,ws test 1-3,text,true,267,3,3,5
I think the problem is while sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile at SOAP/XML. Slash(/) could be problem here.
Without having gone through all your code, your intuition about that line seems spot on to me. The variable is going to be expanded before the sed command is executed, so if you have / in the expansion it will terminate the pattern in sed. You can use other characters than / as your separators, so if, say _ is not going to show up in your data you could use that like
sed '1,_^'"$availableLastLine"'$_d' $file | sed '$d' >> $consolidatedCSVFile
or you could skip the second sed with
sed -e '1,_^'"$availableLastLine"'$_d' -e '$d' $file >> $consolidatedCSVFile
worked after modifying availableLastLine
to availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine).
Working Script :
#!/bin/bash
consolidatedCSVFile=$1
endTime=$2
curdate=$(date +%s)
CSVFiles=${#:3}
CSVFiles=${CSVFiles[#]}
lastLines=_LASTLINES
lastLine=_LASTLINE
newMerge=false
# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile ]
if [ $? == 1 ]
then
mkdir $consolidatedCSVFile$lastLines
newMerge=true
fi
testInProcess=false
if [ $endTime -ge $curdate ]
then
testInProcess=true
fi
for file in $CSVFiles
do
if $newMerge ;
then
touch $consolidatedCSVFile$1astLines5(basename $file)$lastLine
sed $d $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
## Replaced below line
#availableLastLine="`cat $consolidatedCSVFile$1astLines/$(basename $file)$lastLine`"
availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine)
if $testInProcess
then
sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
fi
fi
done
if ! $testInProcess ;
then
rm -rf $consolidatedCSVFile$lastLines
fi
Possible error cases handled while merging of multiple files
#!/bin/bash
echo -e "\n******* $(date) *******"
echo "Arguments :: $#"
consolidatedCSVFile=$1
echo "consolidatedCSVFile :: $1"
endTime=$2
curdate=$(date +%s)
echo -e "endTime :: $2\n"
CSVFiles=${#:3}
CSVFiles=${CSVFiles[#]}
echo -e "Individual Files : ${CSVFiles[#]}\n"
lastLines=_LASTLINES
lastLine=_LASTLINE
newMerge=false
# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile ]
if [ $? == 1 ]
then
newMerge=true
fi
echo "newMerge? :: $newMerge"
if $newMerge ;
then
if [ ! -d $consolidatedCSVFile$lastLines ]
then
echo "creating LASTLINES directory"
mkdir $consolidatedCSVFile$lastLines
fi
fi
testInProcess=false
if [ $endTime -ge $curdate ]
then
testInProcess=true
fi
echo -e "testInProcess? :: $testInProcess\n"
for file in $CSVFiles
do
echo -e "\nCurrent file in loop :: $(basename $file)"
if $newMerge ;
then
# Returns 1 if file is empty
[ -s $file ]
if [ $? == 1 ]
then
echo "File empty !!"
continue
fi
echo "creating LASTLINE file"
touch $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
#sed '$d' $file >> $consolidatedCSVFile
sed '1d' $file | sed '$d' >> $consolidatedCSVFile
echo "Last line to save :: $(tail -1 $consolidatedCSVFile)"
tail -1 $consolidatedCSVFile > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
else
# Returns 1 if file is empty
[ -s $file ]
if [ $? == 1 ]
then
echo "File empty !!"
continue
fi
if [ ! -f $consolidatedCSVFile$lastLines/$(basename $file)$lastLine ]
then
echo "creating LASTLINE file"
touch $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
#sed '$d' $file >> $consolidatedCSVFile
sed '1d' $file | sed '$d' >> $consolidatedCSVFile
echo "Last line to save :: $(tail -1 $consolidatedCSVFile)"
tail -1 $consolidatedCSVFile > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
continue
else
availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine)
echo "Last line from file :: $availableLastLine"
fi
if $testInProcess
then
tempVar=`sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d'`
#echo -e "\ntempVar ::\n$tempVar\n"
[ -z "$tempVar" ]
if [ $? == 1 ]
then
#sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
echo -e "$tempVar" >> $consolidatedCSVFile
echo "Last line to save :: $(tail -1 $consolidatedCSVFile)"
tail -1 $consolidatedCSVFile > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
else
echo "No last line to save"
fi
else
sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
fi
fi
done
if ! $testInProcess ;
then
echo -e "\nRemoving LASTLINES folder"
rm -rf $consolidatedCSVFile$lastLines
fi
echo -e "\n*************"
Related
I'm trying to perform certain operation on each file in a directory but there is a problem with order it's going through. It should do one file at the time. The long line (unzipping, grepping, zipping) works fine on a single file without a script, so there is a problem with a loop. Any ideas?
Script should grep through through each zipped file and look for word1 or word2. If at least one of them exist then:
unzip file
grep word1 and word2 and save it to file_done
remove unzipped file
zip file_done to /donefiles/ with original name
remove file_done from original directory
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c 'word1\|word2' $file)
if [[ $counter -gt 0 ]]; then
echo $counter
for file in *.gz; do
filenoext=${file::-3}
filedone=${filenoext}_done
echo $file
echo $filenoext
echo $filedone
gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip -f -c $filedone > /donefiles/$file | rm -f $filedone
done
else
echo "nothing to do here"
fi
done
The code snipped you've provided has a few problems, e.g. unneeded nested for cycle and erroneous pipeline
(the whole line gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip...).
Note also your code will work correctly only if *.gz files don't have spaces (or special characters) in names.
Also zgrep -c 'word1\|word2' will also match strings like line_starts_withword1_orword2_.
Here is the working version of the script:
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c -E 'word1|word2' $file) # now counter is the number of word1/word2 occurences in $file
if [[ $counter -gt 0 ]]; then
name=$(basename $file .gz)
zcat $file | grep -E 'word1|word2' > ${name}_done
gzip -f -c ${name}_done > /donefiles/$file
rm -f ${name}_done
else
echo 'nothing to do here'
fi
done
What we can improve here is:
since we unzipping the file anyway to check for word1|word2 presence, we may do this to temp file and avoid double-unzipping
we don't need to count how many word1 or word2 is inside the file, we may just check for their presence
${name}_done can be a temp file cleaned up automatically
we can use while cycle to handle file names with spaces
#!/bin/bash
tmp=`mktemp /tmp/gzip_demo.XXXXXX` # create temp file for us
trap "rm -f \"$tmp\"" EXIT INT TERM QUIT HUP # clean $tmp upon exit or termination
find . -maxdepth 1 -mindepth 1 -type f -name '*.gz' | while read f; do
# quotes around $f are now required in case of spaces in it
s=$(basename "$f") # short name w/o dir
gunzip -f -c "$f" | grep -P '\b(word1|word2)\b' > "$tmp"
[ -s "$tmp" ] && gzip -f -c "$tmp" > "/donefiles/$s" # create archive if anything is found
done
It looks like you have an inner loop inside the outer one :
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c 'word1\|word2' $file)
if [[ $counter -gt 0 ]]; then
echo $counter
for file in *.gz; do #<<< HERE
filenoext=${file::-3}
filedone=${filenoext}_done
echo $file
echo $filenoext
echo $filedone
gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip -f -c $filedone > /donefiles/$file | rm -f $filedone
done
else
echo "nothing to do here"
fi
done
The inner loop goes through all the files in the directory if one of them contains file1 or file2. You probably want this :
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c 'word1\|word2' $file)
if [[ $counter -gt 0 ]]; then
echo $counter
filenoext=${file::-3}
filedone=${filenoext}_done
echo $file
echo $filenoext
echo $filedone
gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip -f -c $filedone > /donefiles/$file | rm -f $filedone
else
echo "nothing to do here"
fi
done
Guys when i run the below script I get error grep: Unmatched ( or (
Also Iam unable to understand the grep performed in the shell script for fetching the definition of a word from web.
#!/bin/sh
# define - given a word, return its definition from dictionary.com
url="http://www.cogsci.princeton.edu/cgi-bin/webwn2.0?stage=1&word="
if [ $# -ne 1 ] ; then
echo "Usage: $0 word" >&2
exit 1
fi
lynx -source "$url$1" | \
grep -E '(^[[:digit:]]+\.| has [[:digit:]]+$)' | \
sed 's/<[^>]*>//g' |
( while read line
do
if [ "${line:0:3}" = "The" ] ; then
part="$(echo $line | awk '{print $2}')"
echo ""
echo "The $part $1:"
else
echo "$line" | fmt | sed 's/^/ /g'
fi
done
)
exit 0
I have the following code
#!/bin/bash
output= cat $1 | sed s/$2/$3/
if [ -f "$1" ]
then
echo $output > "$1"
echo "Done"
fi
Arguments:
1 is file
2 old word
3 new word to replace
File Permission is 777 and for some reason the code will replace the current file with a newline and that's it. Any possible reason for this issue?
Try:
#!/bin/bash
output=`cat $1 | sed s/$2/$3/`
if [ -f "$1" ]
then
echo $output > "$1"
echo "Done"
fi
I am trying to create a bash dictionary script that accepts first argument and creates file named after that, then script accepts next arguments (which are files inside same folder) and outputs their content into file (first argument). It also sorts, deletes symbols etc., but main problem is, that sometimes ouptut file is empty (I am passing one non empty file and one non existing file), after deleting and running script few more times it is sometimes empty sometimes not.
#!/bin/bash
numberoffileargs=$(( $# - 1 ))
exitstat=0
counterexit=0
acceptingstdin=0;
> "$1";
#check if we have given input files given
if [ "$#" -gt 1 ]; then
#for cycle going through input files
for i in "${#:2}"
do
#check whether input file is readable
if [ -r "${i}" ]; then
cat "${i}" >> "$1"
#else redirect to standard output
else
exitstat=2
counterexit=$((counterexit + 1))
echo "file does not exist" 1>&2
fi
done
else
echo "stdin code to be done"
acceptingstdin=1
#stdin input to output file
#stdin=$(cat)
fi
#one word for each line, alphabetical sort, alphabet only, remove duplicates
#all lowercase
#sort -u >> "$1"
if [ "$counterexit" -eq "$numberoffileargs" ] && [ "$acceptingstdin" -eq 0 ]; then
exitstat=3
fi
cat "$1" | sed -r 's/[^a-zA-Z\-]+/ /g' | tr A-Z a-z | tr ' ' '\n' | sort -u | sed '/^$/d' > "$1"
echo "$numberoffileargs"
echo "$counterexit"
echo "$exitstat"
exit $exitstat
Here is your script with some syntax improvement. Your trouble came from the fact that the dictionary was both on input and output on your pipeline; I added a temp file to fix it.
#!/bin/bash
(($# >= 1)) || { echo "Usage: $0 dictionary file ..." >&2 ; exit 1;}
dict="$1"
shift
echo "Creating $dict ..."
>| "$dict" || { echo "Failed." >&2 ; exit 1;}
numberoffileargs=$#
exitstat=0
counterexit=0
acceptingstdin=0
if (($# > 0)); then
for i ; do
#check whether input file is readable
if [ -r "${i}" ]; then
cat "${i}" >> "$dict"
else
exitstat=2
let counterexit++
echo "file does not exist" >&2
fi
done
else
echo "stdin code to be done"
acceptingstdin=1
fi
if ((counterexit == numberoffileargs && acceptingstdin == 0)); then
exitstat=3
fi
sed -r 's/[^a-zA-Z\-]+/ /g' < "$dict" | tr '[:upper:]' '[:lower:]' | tr ' ' '\n' |
sort -u | sed '/^$/d' >| tmp$$
mv -f tmp$$ "$dict"
echo "$numberoffileargs"
echo "$counterexit"
echo "$exitstat"
exit $exitstat
The pipeline might be improved.
The following script is generating the desired out put but not redirecting the result to /home/myuser/slavedelay.html
#!/bin/bash
host=<ip>
echo $host
user=usr1
password=mypass
threshold=300
statusok=OK
statuscritical=CRITICAL
for i in ert7 ert9
do
echo "<html>" > /home/myuser/slavedelay.html
if [ "$i" == "ert7" ]; then
slvdelay=`mysql -u$user -p$password -h<ip> -S /backup/mysql/mysql.sock -e 'show slave status\G' | grep Seconds_Behind_Master | sed -e 's/ *Seconds_Behind_Master: //'`
if [ $slvdelay -ge $threshold ]; then
echo "<tr><td>$i</td><td>CRITICAL</td>" >> /home/myuser/slavedelay.html
echo "<tr><td>$i</td><td>CRITICAL</td>"
else
echo "<tr><td>$i</td><td>OK</td>" >> /home/myuser/slavedelay.html
echo "<tr><td>$i</td><td>OK</td>"
fi
fi
done
echo "</html>" >> /home/myuser/slavedelay.html
If I cat the output file /home/myuser/slavedelay.html it gives.
<html>
</html>
Execution result :
sh slave_delay.sh
<tr><td>sdb7</td><td>OK</td>
Each time through the loop you're emptying the output file because of the command
echo "<html>" > /home/myuser/slavedelay.html
So the first iteration writes the <tr> row to the file, then the next iteration overwrites the file and doesn't write those lines because $i isn't ert7.
Change it to:
for i in ert7 ert9
do
if [ "$i" == "ert7" ]; then
echo "<html>" > /home/myuser/slavedelay.html
slvdelay=`mysql -u$user -p$password -h<ip> -S /backup/mysql/mysql.sock -e 'show slave status\G' | grep Seconds_Behind_Master | sed -e 's/ *Seconds_Behind_Master: //'`
if [ $slvdelay -ge $threshold ]; then
echo "<tr><td>$i</td><td>CRITICAL</td>"
else
echo "<tr><td>$i</td><td>OK</td>"
fi | tee -a /home/myuser/slavedelay.html
echo "</html>" >> /home/myuser/slavedelay.html
fi
done
Replace :
if [ "$i" == "ert7" ];
with:
if [ "$i" = "ert7" ];
You use = operator in test also.