Redirection based on variables assigned within while loop setup - shell

I want to access while loop variable out side the loop
while read line
do
...
...
...
done < $file > /home/Logs/Sample_$line_$(date +"%Y%m%d_%H%M%S").log
In the above example whatever the log file is getting generated that doesn't have the value for the line variable. i.e. $line is not working here.
Please let me know how this can be written to make it work.
#!/bin/sh
exec 1> /home/Logs/`basename $0 | cut -d"." -f1 | sed 's/\.sh//g'`_$(date +"%Y%m%d_%H%M%S").log 2>&1
echo "Execution Started : `date` \n"
SQL_DIR=/home/Sql
INFILE=in_file
TEMPFILE=temp_file
RETURN_CODE=0
ls -ltr $SQL_DIR|grep ".sql"|awk -F" " '{print $9}'|awk -F"." '{print $1}' > $INFILE
sed '/^$/d' $INFILE > $TEMPFILE; mv $TEMPFILE $INFILE
while read line
do
{
START_TIME=`date +%s`
printf "\n SQL File Executed Is : $line.sql"
SQL_FILE_NM=$line.sql
SQL_FILE=$SQL_DIR/$SQL_FILE_NM
nzsql -db netezza_db -Atqf $SQL_FILE > /dev/null
RETURN_CODE=$?
if [ $RETURN_CODE -eq 0 ]; then
echo "Time taken to execute sqlfile $SQL_FILE=$TT_HRS:$TT_MINS:$TT_REM_SECS HH:MM:SS" > $TEMPFILE
printf "\n Success: Sql completed successfully at `date` \n"
cat $TEMPFILE|mailx -s "Time taken to execute sqlfile $SQL_FILE=$TT_HRS:$TT_MINS:$TT_REM_SECS HH:MM:SS" 'koushik.chandra#a.com'
else
printf "\n Error: Failed in sql execution at `date`"
exit $RETURN_CODE
fi
END_TIME=`date +%s`
TT_SECS=$(( END_TIME - START_TIME))
TT_HRS=$(( TT_SECS / 3600 ))
TT_REM_MS=$(( TT_SECS % 3600 ))
TT_MINS=$(( TT_REM_MS / 60 ))
TT_REM_SECS=$(( TT_REM_MS % 60 ))
printf "\n"
printf "Total time taken to execute the sql $line="$TT_HRS:$TT_MINS:$TT_REM_SECS HH:MM:SS
printf "\n"
} > /home/Logs/sql_query_time_$line_$(date +"%Y%m%d_%H%M%S").log
done < $INFILE
rm -f $INFILE $TEMPFILE
exit $RETURN_CODE

You actually need redirection inside the while loop:
while read -r line; do
{ cmd1; cmd2; cmd3; } > "/home/Logs/Sample_${line}_$(date +"%Y%m%d_%H%M%S").log"
done < "$file"
When you have > outfile after done then output is redirected to one file only.

Related

Bash - Extract Matching String from GZIP Files Is Running Very Slow

Complete novice in Bash. Trying to iterate thru 1000 gzip files, may be GNU parallel is the solution??
#!/bin/bash
ctr=0
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
for f in "$dir"/*.gz; do
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr" >> $1
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym
else
ctr=$((ctr+1))
fi
done
done
Any help to speed the process will be greatly appreciated !!!
#!/bin/bash
ctr=0
export ctr
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
export dir
doit() {
f="$1"
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr"
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym >&2
else
ctr=$((ctr+1))
fi
done
}
export -f doit
parallel doit ::: *gz 2>&1 > $1
The Bash while read loop is probably your main bottleneck here. Calling multiple external processes for simple field splitting will exacerbate the problem. Briefly,
while IFS="|" read -r first second third rest; do ...
leverages the shell's built-in field splitting functionality, but you probably want to convert the whole thing to a simple Awk script anyway.
echo "file_name,symbol,record_count" > "$1"
for f in "/data/myfolder"/*.gz; do
gunzip -c "$f" |
awk -F "\|" -v f="$f" -v OFS="," '
/H/ { if(ctr) print f, sym, ctr
ctr=0; sym=$3;
print sym >"/dev/stderr"
next }
{ ++ctr }'
done >>"$1"
This vaguely assumes that printing the lone sym is just for diagnostics. It should hopefully not be hard to see how this can be refactored if this is an incorrect assumption.

using tail to follow a log and execute a command instantly? Only seems to work by starting a new line

I am trying to figure out a command which will enable me to read a log file in real time and execute a command when the string matches? I am using logkeys and trying to make it when I type a word it immediately triggers a command. This script works, but only when I press enter (start a newline) does it execute, and it seems anything I have found online also requires the press of the enter key to work. Is there a way to get around this somehow?
#/bin/bash
echo Waiting...
string='test'
tail /path/to/logfile -n0 -f | while read line; do
if [[ $line =~ $string ]]; then
echo "hello"
fi
done
I've played with buffering settings to no avail, so my conclusion is that read waits for a newline before it finishes. If you instead did read -n1, read would read exactly one character, which isn't quite what we want either, because then $line would always be just that one char.
Unfortunately, grep appears to have the same behavior (even with buffering options changed), even with grep -o:
$ tail logfile -f -n0 | grep -o test &
[1] 25524
$ echo -n test >> logfile
$ echo -n test >> logfile
$ echo test >> logfile
test
test
test
I think the general solution would be to roll our own "ring buffer grep" search tool that reads character per character into a ring buffer.
Here's my perl version of that, hope it helps. (Save as: ringgrep.pl)
#!/usr/bin/perl -w
use strict;
if (!$ARGV[0]) {
print "Usage: $0 needle\n";
exit 1;
}
my $needle = $ARGV[0];
my $buffer_len = length($needle);
my #buffer = (0) x $buffer_len;
my $i = 0;
my $input;
while(sysread(STDIN, $input, 1)) {
$buffer[$i] = $input;
my $string = join("", #buffer);
$string = (($i+1)%$buffer_len == 0 ? "" : substr($string, $i-$buffer_len+1)) . substr($string, 0, $i+1);
# print "string is: $string\n";
if ($string =~ /$needle/) {
print "got test!\n";
#buffer = (0) x $buffer_len;
$i = 0
} else {
$i = (++$i) % $buffer_len
}
}
Usage:
$ chmod +x ringgrep.pl
$ tail logfile -n0 -f | ./ringgrep.pl "this is a test" &
[1] 25756
$ echo -n "test" >> logfile
$ echo -n "test" >> logfile
$ echo -n "test" >> logfile
$ echo -n "test" >> logfile
$ echo -n "this is a test" >> logfile
got test!
$ echo -n "this is a test" >> logfile
got test!
$ echo -n "this is a test" >> logfile
got test!
$ (echo -n t; echo -n h; echo -n i; echo -n s; echo -n ' '; echo -n i; echo -n s; echo -n ' '; echo -n a; echo -n ' '; echo -n t; echo -n e; echo -n s; echo -n t) >> logfile
got test!

collect output in bash before writing

I've got the following issue:
i am parsing a file in a bash script and preparing an output format like this:
echo "Evaluation of data from $date" > $outPut
printf "\n%s\t\t\t%s\t\t%s\n" "column1" "column2" "column3" >> $outPut
printf '%*s\n' "${COLUMNS:-$(tput cols)}" '' | tr ' ' - >> $outPut
cat $file | \
while read i;
do
...
printf "..."
done >> $outPut
It works fine if $outPut is a file. But depending on a program parameter $output can be /dev/stdout.
If I pipe the output to less
bash someprogram.bash --tostdout | less
less starts immediately with zero output. After a while I see everything, but if I do :G less just stops working and I can only stop it with CTRL-C. If I don't pipe it, the output works fine.
What I want is: the function to write everything at once after collecting the output + the pipe to wait for my program to finish.
I found a workaround by first writing to a tempfile and cat it to stdout afterwards
tmpFile=$(mktemp -p /tmp)
echo "Evaluation of data from $date" > $tmpFile
printf "\n%s\t\t\t%s\t\t%s\n" "column1" "column2" "column3" >> $tmpFile
printf '%*s\n' "${COLUMNS:-$(tput cols)}" '' | tr ' ' - >> $tmpFile
cat $file | \
while read i;
do
...
printf "..."
done >> $tmpFile
cat $tmpFile
rm $tmpFile
The advantage is that less gets the full output of the script and not chunk after chunk.

Bash - sometimes creates only empty output

I am trying to create a bash dictionary script that accepts first argument and creates file named after that, then script accepts next arguments (which are files inside same folder) and outputs their content into file (first argument). It also sorts, deletes symbols etc., but main problem is, that sometimes ouptut file is empty (I am passing one non empty file and one non existing file), after deleting and running script few more times it is sometimes empty sometimes not.
#!/bin/bash
numberoffileargs=$(( $# - 1 ))
exitstat=0
counterexit=0
acceptingstdin=0;
> "$1";
#check if we have given input files given
if [ "$#" -gt 1 ]; then
#for cycle going through input files
for i in "${#:2}"
do
#check whether input file is readable
if [ -r "${i}" ]; then
cat "${i}" >> "$1"
#else redirect to standard output
else
exitstat=2
counterexit=$((counterexit + 1))
echo "file does not exist" 1>&2
fi
done
else
echo "stdin code to be done"
acceptingstdin=1
#stdin input to output file
#stdin=$(cat)
fi
#one word for each line, alphabetical sort, alphabet only, remove duplicates
#all lowercase
#sort -u >> "$1"
if [ "$counterexit" -eq "$numberoffileargs" ] && [ "$acceptingstdin" -eq 0 ]; then
exitstat=3
fi
cat "$1" | sed -r 's/[^a-zA-Z\-]+/ /g' | tr A-Z a-z | tr ' ' '\n' | sort -u | sed '/^$/d' > "$1"
echo "$numberoffileargs"
echo "$counterexit"
echo "$exitstat"
exit $exitstat
Here is your script with some syntax improvement. Your trouble came from the fact that the dictionary was both on input and output on your pipeline; I added a temp file to fix it.
#!/bin/bash
(($# >= 1)) || { echo "Usage: $0 dictionary file ..." >&2 ; exit 1;}
dict="$1"
shift
echo "Creating $dict ..."
>| "$dict" || { echo "Failed." >&2 ; exit 1;}
numberoffileargs=$#
exitstat=0
counterexit=0
acceptingstdin=0
if (($# > 0)); then
for i ; do
#check whether input file is readable
if [ -r "${i}" ]; then
cat "${i}" >> "$dict"
else
exitstat=2
let counterexit++
echo "file does not exist" >&2
fi
done
else
echo "stdin code to be done"
acceptingstdin=1
fi
if ((counterexit == numberoffileargs && acceptingstdin == 0)); then
exitstat=3
fi
sed -r 's/[^a-zA-Z\-]+/ /g' < "$dict" | tr '[:upper:]' '[:lower:]' | tr ' ' '\n' |
sort -u | sed '/^$/d' >| tmp$$
mv -f tmp$$ "$dict"
echo "$numberoffileargs"
echo "$counterexit"
echo "$exitstat"
exit $exitstat
The pipeline might be improved.

Errors with a shell-script

i found some freaky error. I want to increment a counter, but the variable isnt visible outside the while do.
The script as follows:
## $1 - The file which should be examined
## $2 - The time passed between the checks. If $2 is 5 then all lines from the last 5 minutes are taken
## $3 - The Errormessage to search for
outputOK="OK - nothing happened"
output_logcheck=0;
errlines="";
cat $1 | grep "$3" | while read line
do
linedate=`date -d "$(echo $line | cut -d " " -f 2)" '+%s'`
nowdate=`date '+%s'`
if [ $(( $nowdate - (60 * $2) )) -le $linedate ]
then
$output_logcheck=$[$output_logcheck+1]
$errlines="${errlines} -- ${line}"
fi
done;
if [ $output_logcheck -eq 0 ]
then
echo $outputOK
else
echo "CRITICAL - There are -= ${output_logcheck} =- $3 -- Lines: $errlines"
fi
So i dont know what else to try.
Thanks in advance.
The problem is that pipe create a SubShell.
change
cat $1 | grep "$3" | while read line
do
...
done
to
while read line
do
...
done <(cat $1 | grep "$3")
As noted, the Bash shell, creates a subshell whenever a pipe is opened to a loop. In that case, variables within the loop are local to the loop.
One kludge is to substitute (if possible) a Korn ('ksh') shell for the Bash one.
Try something like:
## $1 - The file which should be examined
## $2 - The time passed between the checks. If $2 is 5 then all lines from the last 5 minutes are taken
## $3 - The Errormessage to search for
outputOK="OK - nothing happened"
outfile="/tmp/result.$$"
trap { rm $outfile } 0 1 2 3
cat $1 | grep "$3" | (output_logcheck=0; errlines=""; while read line
do
linedate=`date -d "$(echo $line | cut -d " " -f 2)" '+%s'`
nowdate=`date '+%s'`
if [ $(( $nowdate - (60 * $2) )) -le $linedate ]
then
$output_logcheck=$[$output_logcheck+1]
$errlines="${errlines} -- ${line}"
fi
done; echo $output_logcheck ":" $errlines > $outfile)
output_logcheck=`cat $outfile| cut -f1 -d:`
errlines=`cat $outfile|cut -f2 -d:`
if [ $output_logcheck -eq 0 ]
then
echo $outputOK
else
echo "CRITICAL - There are -= ${output_logcheck} =- $3 -- Lines: $errlines"
fi
while is executed in a separate process. Variables that are changed in the context of that process still hold their unchanged valus in the parent process.

Resources