Use awk to print largest alphanumeric grep result and create variable next in sequence - bash

I have a source file with the following information in it.
WABEL8499IPM101
WABEL8499IPM102
WABEL8499IPM103
WABEL8499IPM104
WABEL8499IPM105
WABEL8499IPM106
WABEL8499IPM107
WABEL8499IPM108
I need to be able to find the largest name in the sequence and then create a new variable with the next logical name in the sequence. I need to be able to create multiple if necessary. For example:
Use grep to search the file for WABEL8499IPM which shows all of the above results. I need to find WABEL8499IPM108 because it's the largest in the sequence and then create a new variable (how many depends on what the user inputs) with the value WABEL8499IPM109. If user inputs a quantity of 2 then I need both 109 and 110. My goal is to build a bash script to input the base name (without the last 3 digits), find the largest in the sequence and then output to a log file the next names in the sequence however many times the user needs.
I'm not really sure where to start. I can find all using grep but having difficulty finding only the largest value/sequence. The user will only input the base name because they won't know the last 3 digits. Currently I don't have any code that works.
SRCFILE="~/Desktop/deviceinfo.csv"
LOGDIR="~/Desktop/"
LOGFILE="$LOGDIR/DeviceNames.csv"
echo -e "\n"
echo "What is the base device name?"
read deviceName
echo "How many device names do you need?"
read quantityName
lines=$(grep -c "$deviceName" $SRCFILE)
echo -e "\n"
echo "There are $lines results."
deviceResults=$(grep -F "$deviceName" $SRCFILE)
echo -e "\n"
echo Device Name\'s Currently Enrolled:
echo "$deviceResults"
echo -e "\n"
echo "Your output file has been created."
CODE FOR CREATING OUTPUT FILE HERE
echo "$deviceName1" >> "$LOGFILE"
echo "$deviceName2" >> "$LOGFILE"
echo "$deviceName3" >> "$LOGFILE"
Would there be a way with this method to use a reference file for the input? For example if I had to research and create multiple names with different quantities could we use an input reference file for that so we don't have to type them each individually and run the script multiple times?

SRCFILE="~/Desktop/deviceinfo.csv"
LOGDIR="~/Desktop/"
LOGFILE="$LOGDIR/DeviceNames.csv"
# base name, such as "WABEL8499IPM"
device_name=$1
# quantity, such as "2"
quantityNum=$2
# the largest in sequence, such as "WABEL8499IPM108"
max_sequence_name=$(cat $SRCFILE | grep -o -e "$device_name[0-9]*" | sort --reverse | head -n 1)
# extract the last 3digit number (such as "108") from max_sequence_name
max_sequence_num=$(echo $max_sequence_name | rev | cut -c 1-3 | rev)
# creat a sequence of files starting from "WABEL8499IPM101" if there is not any "WABEL8499IPM".
if [ -z "$max_sequence_name" ];
then
max_sequence_name=device_name
max_sequence_num=100
fi
# create new sequence_name
# such as ["WABEL8499IPM109", "WABEL8499IPM110"]
array_new_sequence_name=()
for i in $(seq 1 $quantityNum);
do
cnum=$((max_sequence_num + i))
array_new_sequence_name+=($(echo $device_name$cnum))
done
#CODE FOR CREATING OUTPUT FILE HERE
#for fn in ${array_new_sequence_name[#]}; do touch $fn; done;
# write log
for sqn in ${array_new_sequence_name[#]};
do
echo $sqn >> $LOGFILE
done
Usage:
bash test.sh WABEL8499IPM 2
Result in the log file:
WABEL8499IPM109
WABEL8499IPM110
EDITED
The input reference file (input.txt) :
WABEL8499IPM,2
WABEL8555IPM,6
WABEL8444IPM,5
The driver shell script :
INPFIL="./input.txt"
PSRC="./test.sh"
cat $INPFIL | while read line;
do
device_name=`echo $line | cut -d "," -f 1`
quantity_num=`echo $line | cut -d "," -f 2`
bash $PSRC $device_name $quantity_num
done;

You can try
logdir="~/Desktop/"
srcfile="$logdir/deviceinfo.csv"
logfile="$logdir/DeviceNames.csv"
echo
read -p "What is the base device name? " deviceName
echo
read -p "How many device names do you need? " quantityName
echo
awk -v name="$deviceName" \
-v q="$quantityName" \
-v lelog="$logfile" '
$0 ~ "^"name {
sub(name,"")
a=a>$0?a:$0
}
END {
if ( a )
for ( i = 1 ; i <= q ; i++ )
print name ( a + i ) >> lelog
}
' "$srcfile"

Related

Concatenate the output of 2 commands in the same line in Unix

I have a command like below
md5sum test1.txt | cut -f 1 -d " " >> test.txt
I want output of the above result prefixed with File_CheckSum:
Expected output: File_CheckSum: <checksumvalue>
I tried as follows
echo 'File_Checksum:' >> test.txt | md5sum test.txt | cut -f 1 -d " " >> test.txt
but getting result as
File_Checksum:
adbch345wjlfjsafhals
I want the entire output in 1 line
File_Checksum: adbch345wjlfjsafhals
echo writes a newline after it finishes writing its arguments. Some versions of echo allow a -n option to suppress this, but it's better to use printf instead.
You can use a command group to concatenate the the standard output of your two commands:
{ printf 'File_Checksum: '; md5sum test.txt | cut -f 1 -d " "; } >> test.txt
Note that there is a race condition here: you can theoretically write to test.txt before md5sum is done reading from it, causing you to checksum more data than you intended. (Your original command mentions test1.txt and test.txt as separate files, so it's not clear if you are really reading from and writing to the same file.)
You can use command grouping to have a list of commands executed as a unit and redirect the output of the group at once:
{ printf 'File_Checksum: '; md5sum test1.txt | cut -f 1 -d " " } >> test.txt
printf "%s: %s\n" "File_Checksum:" "$(md5sum < test1.txt | cut ...)" > test.txt
Note that if you are trying to compute the hash of test.txt(the same file you are trying to write to), this changes things significantly.
Another option is:
{
printf "File_Checksum: "
md5sum ...
} > test.txt
Or:
exec > test.txt
printf "File_Checksum: "
md5sum ...
but be aware that all subsequent commands will also write their output to test.txt. The typical way to restore stdout is:
exec 3>&1
exec > test.txt # Redirect all subsequent commands to `test.txt`
printf "File_Checksum: "
md5sum ...
exec >&3 # Restore original stdout
Operator &&
e.g. mkdir example && cd example

How to read multiple lines in while statement in ksh

I am creating a script to help me through my daily work and automate it. I have encountered my problem when trying to input multiple lines in my while loop. I usually do it in my for loop but I execute it via command.
Sample:
for i in `cat listoffiles.txt`
do
echo $i
find <path> -name *$i* | awk -F "." {'print $4'} #to display a specific value
done
Now I am trying to automate it with a while loop. Having problems to read multiple input lines in it.
For example:
i want to search for these inputs:
For
Example
only
here is my script for it:
#!/bin/ksh
echo Please enter file #:
read Var1
while true
do
VarSession=`find $OT_DIR/archive*/ -name *$Var1* | awk -F "." {'print $4'}`
if [ "$VarSession" = "" ]
then
echo No match for File# $Var1 on this leg or is out of retention.
else
echo File# $Var1 is under Session# $VarSession
fi
done
VarSession=`find $OT_DIR/archive*/ -name *$Var1* | awk -F "." {'print $4'}`
Assuming that you provide 1 2 3 as input, The line above translates to this
VarSession=`find $OT_DIR/archive*/ -name "1 2 3" | awk -F "." {'print $4'}`
But you want to search all those values separately so you need another loop. for loop serves the purpose if traversing white-space separated entries.
Also, based upon the original script that you showed, I assume you want the script to search file by file, rather than scanning entire directories. However, the statement above will put all output in the variable without traversing it. To traverse line by line, while loop does the job.
#!/bin/ksh
# -n switch suppresses printing a newline
echo -n 'Please enter file #: '
read Var1
# Traverse over all entered values in Var1 (separated by white space)
for i in $Var1
do
#Set a flag to zero, logic explained later
Flag=0
find $OT_DIR/archive*/ -name *$i* | while read FileName
do
#Set the Flag to 1 if find command finds something
Flag=1
VarSession=`echo $FileName | awk -F "." {'print $4'}`
if [ "$VarSession" = "" ]
then
#If find found a file but VarSession has nothing then file name is not correct
echo "Some conventions went wrong in file name: $FileName"
else
echo "File# $Var1 is under Session# $VarSession"
fi
done
#If find found nothing, there was no match
if [ $Flag -eq 0 ]
then
echo No match for File# $Var1 on this leg or is out of retention.
fi
done

if variable exists in col 1 of a file, set a separate variable equal to the corresponding value in col 2

I'm writing a script to run a grep for an input, create a file list and grep that file list in a separate directory and print the results.
echo "enter term"
read term
grep -rc "$term" /prod/directory1 | grep -v ":0" | sed -s 's%:[0-9]*%%' > file_list.txt
grep -rl --file=file_list.txt /tmp/directory2 > results.txt
but i would like to add an if statement that will identify if the $term is equal to a value in the first column of another file, and then set a new variable to the corresponding value in column 2 of that file.
echo "enter term"
read term
for i in products.txt; do
if [[ $term = $i ]]; then
var2 = $2
echo "product code set to: "$var2
else
var2 = 0
echo "product code set to 0."
fi
done
grep -rc "$term" /prod/directory1 | grep -v ":0" | sed -s 's%:[0-9]*%%' > file_list.txt
if [[ ! $var2 = 0 ]] ; then
grep -rc "var2" /prod/directory1 | grep -v ":0" | sed -s 's%:[0-9]*%%' >> file_list.txt
sed -s 's%.properties%%' file_list.txt | sort -u > file_list.txt
grep -rl --file=file_list.txt /tmp/directory2 > results.txt
the new grep will return results with .properties at the end, so I remove them and also any duplicate files from the list. Below is a sample for products.txt
Product_1 productCode_1
Product_2 productCode_2
Product_3 productCode_3
I would like the script to identify whether $term is in column one of that file, and then set $var2 equal to the corresponding productCode in column two. I believe once that variable is set everything else will work out nicely but right now it fails at:
for i in products.txt; do
if [[$term = $i ]]; then
var2 = $2
Thanks in advance for the assistance, I'm relatively new to bash scripting so I apologize in advance for my ineptitude.
Try:
var2=$(awk -v term="Product_2" '$1==term{print $2}' products.txt)
Change Product_2 to read from another variable, if required.

ksh: shell script to search for a string in all files present in a directory at a regular interval

I have a directory (output) in unix (SUN). There are two types of files created with timestamp prefix to the file name. These file are created on a regular interval of 10 minutes.
e. g:
1. 20140129_170343_fail.csv (some lines are there)
2. 20140129_170343_success.csv (some lines are there)
Now I have to search for a particular string in all the files present in the output directory and if the string is found in fail and success files, I have to count the number of lines present in those files and save the output to the cnt_succ and cnt_fail variables. If the string is not found I will search again in the same directory after a sleep timer of 20 seconds.
here is my code
#!/usr/bin/ksh
for i in 1 2
do
grep -l 0140127_123933_part_hg_log_status.csv /osp/local/var/log/tool2/final_logs/* >log_t.txt; ### log_t.txt will contain all the matching file list
while read line ### reading the log_t.txt
do
echo "$line has following count"
CNT=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT=`expr $CNT - 1`
echo $CNT
done <log_t.txt
if [ $CNT > 0 ]
then
exit
fi
echo "waiitng"
sleep 20
done
The problem I'm facing is, I'm not able to get the _success and _fail in file in line and and check their count
I'm not sure about ksh, but while ... do; ... done is notorious for running off with whatever variables you're using in bash. ksh might be similar.
If I've understand your question right, SunOS has grep, uniq and sort AFAIK, so a possible alternative might be...
First of all:
$ cat fail.txt
W34523TERG
ADFLKJ
W34523TERG
WER
ASDTQ34T
DBVSER6
W34523TERG
ASDTQ34T
DBVSER6
$ cat success.txt
abcde
defgh
234523452
vxczvzxc
jkl
vxczvzxc
asdf
234523452
vxczvzxc
dlkjhgl
jkl
wer
234523452
vxczvzxc
And now:
egrep "W34523TERG|ASDTQ34T" fail.txt | sort | uniq -c
2 ASDTQ34T
3 W34523TERG
egrep "234523452|vxczvzxc|jkl" success.txt | sort | uniq -c
3 234523452
2 jkl
4 vxczvzxc
Depending on the input data, you may want to see what options sort has on your system. Examining uniq's options may prove useful too (it can do more than just count duplicates).
Think you want something like this (will work in both bash and ksh)
#!/bin/ksh
while read -r file; do
lines=$(wc -l < "$file")
((sum+=$lines))
done < <(grep -Rl --include="[1|2]*_fail.csv" "somestring")
echo "$sum"
Note this will match files starting with 1 or 2 and ending in _fail.csv, not exactly clear if that's what you want or not.
e.g. Let's say I have two files, one starting with 1 (containing 4 lines) and one starting with 2 (containing 3 lines), both ending in `_fail.csv somewhere under my current working directory
> abovescript
7
Important to understand grep options here
-R, --dereference-recursive
Read all files under each directory, recursively. Follow all
symbolic links, unlike -r.
and
-l, --files-with-matches
Suppress normal output; instead print the name of each input
file from which output would normally have been printed. The
scanning will stop on the first match. (-l is specified by
POSIX.)
Finaly I'm able to find the solution. Here is the complete code:
#!/usr/bin/ksh
file_name="0140127_123933.csv"
for i in 1 2
do
grep -l $file_name /osp/local/var/log/tool2/final_logs/* >log_t.txt;
while read line
do
if [ $(echo "$line" |awk '/success/') ] ## will check the success file
then
CNT_SUCC=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT_SUCC=`expr $CNT_SUCC - 1`
fi
if [ $(echo "$line" |awk '/fail/') ] ## will check the fail file
then
CNT_FAIL=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT_FAIL=`expr $CNT_FAIL - 1`
fi
done <log_t.txt
if [ $CNT_SUCC > 0 ] && [ $CNT_FAIL > 0 ]
then
echo " Fail count = $CNT_FAIL"
echo " Success count = $CNT_SUCC"
exit
fi
echo "waitng for next search..."
sleep 10
done
Thanks everyone for your help.
I don't think I'm getting it right, but You can't diffrinciate the files?
maybe try:
#...
CNT=`expr $CNT - 1`
if [ $(echo $line | grep -o "fail") ]
then
#do something with fail count
else
#do something with success count
fi

Count mutiple occurences of a word on the same line using grep

Here I made a small script that take input from user searching some pattern from a file and displays required no of lines from that file where the pattern is found. Although this code is searching the pattern line wise due to standard grep practice. I mean if the pattern occurs twice on the same line, i want the output to print twice. Hope I make some sense.
#!/bin/sh
cat /dev/null>copy.txt
echo "Please enter the sentence you want to search:"
read "inputVar"
echo "Please enter the name of the file in which you want to search:"
read "inputFileName"
echo "Please enter the number of lines you want to copy:"
read "inputLineNumber"
[[-z "$inputLineNumber"]] || inputLineNumber=20
cat /dev/null > copy.txt
for N in `grep -n $inputVar $inputFileName | cut -d ":" -f1`
do
LIMIT=`expr $N + $inputLineNumber`
sed -n $N,${LIMIT}p $inputFileName >> copy.txt
echo "-----------------------" >> copy.txt
done
cat copy.txt
As I understood, the task is to count number of pattern occurrences in line. It can be done like so:
count=$((`echo "$line" | sed -e "s|$pattern|\n|g" | wc -l` - 1))
Suppose you have one file to read. Then, code will be following:
#!/bin/bash
file=$1
pattern="an."
#reading file line by line
cat -n $file | while read input
do
#storing line to $tmp
tmp=`echo $input | grep "$pattern"`
#counting occurrences count
count=$((`echo "$tmp" | sed -e "s|$pattern|\n|g" | wc -l` - 1))
#printing $tmp line $count times
for i in `seq 1 $count`
do
echo $tmp
done
done
I checked this for pattern "an." and input:
I pass here an example of many 'an' letters
an
ananas
an-an-as
Output is:
$ ./test.sh input
1 I pass here an example of many 'an' letters
1 I pass here an example of many 'an' letters
1 I pass here an example of many 'an' letters
3 ananas
4 an-an-as
4 an-an-as
Adapt this to your needs.
How about using awk?
Assume the pattern you are searching for is in variable $pattern and the file you are checking is $file
The
count=`awk 'BEGIN{n=0}{n+=split($0,a,"'$pattern'")-1}END {print n}' $file`
or for a line
count=`echo $line | awk '{n=split($0,a,"'$pattern'")-1;print n}`

Resources