Bash grep variable from multiple variables on a single line - bash

I am using GNU bash, version 4.2.20(1)-release (x86_64-pc-linux-gnu). I have a music file list I dumped into a variable: $pltemp.
Example:
/Music/New/2010s/2011;Ziggy Marley;Reggae In My Head
I wish to grep the 3rd field above, in the Master-Music-List.txt, then continue another grep for the 2nd field. If both matched, print else echo "Not Matched".
So the above will search for the Song Title (Reggae In My Head), then will make sure it has the artist "Shaggy" on the same line, for a success.
So far, success for a non-variable grep;
$ grep -i -w -E 'shaggy.*angel' Master-Music-MM-Playlist.m3u
$ if ! grep Shaggy Master-Music-MM-Playlist.m3u ; then echo "Not Found"; fi
$ grep -i -w Angel Master-Music-MM-Playlist.m3u | grep -i -w shaggy
I'm not sure how to best construct the 'entire' list to process.
I want to do this on a single line.
I used this to dump the list into the variable $pltemp...
Original: \Music\New\2010s\2011\Ziggy Marley - Reggae In My Head.mp3
$ pltemp="$(cat Reggae.m3u | sed -e 's/\(.*\)\\/\1;/' -e 's/\(.*\)\ -\ /\1;/' -e 's/\\/\//g' -e 's/\\/\//g' -e 's/.mp3//')"

If you realy want to "grep this, then grep that", you need something more complex than grep by itself. How about awk?
awk -F';' '$3~/title/ && $2~/artist/ {print;n=1;exit;} END {if(n=0)print "Not matched";}'
If you want to make this search accessible as a script, the same thing simply changes form. For example:
#!/bin/sh
awk -F';' -vartist="$1" -vtitle="$2" '$3~title && $2~artist {print;n=1;exit;} END {if(n=0)print "Not matched";}'
Write this to a file, make it executable, and pipe stuff to it, with the artist substring/regex you're looking for as the first command line option, and the title substring/regex as the second.
On the other hand, what you're looking for might just be a slightly more complex regular expression. Let's wrap it in bash for you:
if ! echo "$pltemp" | egrep '^[^;]+;[^;]*artist[^;]*;.*title'; then
echo "Not matched"
fi
You can compress this to a single line if you like. Or make it a stand-along shell script, or make it a function in your .bashrc file.

awk -F ';' -v title="$title" -v artist="$artist" '$3 ~ title && $2 ~ artist'

Well, none of the above worked, so I came up with this...
for i in *.m3u; do
cat "$i" | sed 's/.*\\//' | while read z; do
grep --color=never -i -w -m 1 "$z" Master-Music-Playlist.m3u \
| echo "#NotFound;"$z" "
done > "$i"-MM-Final.txt;
done
Each line is read (\Music\Lady Gaga - Paparazzi.mp3), the path is stripped, the song is searched in the Master Music List, if not found, it echos "Not Found", saved into a new playlist.
Works {Solved}
Thanks anyway.

Related

User input into variables and grep a file for pattern

H!
So I am trying to run a script which looks for a string pattern.
For example, from a file I want to find 2 words, located separately
"I like toast, toast is amazing. Bread is just toast before it was toasted."
I want to invoke it from the command line using something like this:
./myscript.sh myfile.txt "toast bread"
My code so far:
text_file=$1
keyword_first=$2
keyword_second=$3
find_keyword=$(cat $text_file | grep -w "$keyword_first""$keyword_second" )
echo $find_keyword
i have tried a few different ways. Directly from the command line I can make it run using:
cat myfile.txt | grep -E 'toast|bread'
I'm trying to put the user input into variables and use the variables to grep the file
You seem to be looking simply for
grep -E "$2|$3" "$1"
What works on the command line will also work in a script, though you will need to switch to double quotes for the shell to replace variables inside the quotes.
In this case, the -E option can be replaced with multiple -e options, too.
grep -e "$2" -e "$3" "$1"
You can pipe to grep twice:
find_keyword=$(cat $text_file | grep -w "$keyword_first" | grep -w "$keyword_second")
Note that your search word "bread" is not found because the string contains the uppercase "Bread". If you want to find the words regardless of this, you should use the case-insensitive option -i for grep:
find_keyword=$(cat $text_file | grep -w -i "$keyword_first" | grep -w -i "$keyword_second")
In a full script:
#!/bin/bash
#
# usage: ./myscript.sh myfile.txt "toast" "bread"
text_file=$1
keyword_first=$2
keyword_second=$3
find_keyword=$(cat $text_file | grep -w -i "$keyword_first" | grep -w -i "$keyword_second")
echo $find_keyword

grep from two variables

I am trying to eliminate the duplicate lines of a list like this.
LINES='opa
opa
eita
eita
argh'
DUPLICATE='opa
eita'
The output I am looking for is argh.
Till now, this is what I tried:
echo -e "$DUPLICATE" | grep --invert-match -Ff- <(echo -e "$LINES")
And:
grep --invert-match -Ff- <(echo -e "$DUPLICATE") <(echo -e "$LINES")
But unsuccessfuly.
I know that I can achieve this if I put the content of $LINES into a file:
echo -e "$DUPLICATE" | grep --invert-match -Ff- FILE
But I'd like to know if this is possible only with variables.
Passing a dash as the file name to -f means "read from stdin". Get rid of it so the file name given to -f is the process substitution.
There's no need for echo -e, and -v is shorter and more common than --invert-match.
echo "$LINES" | grep -vFf <(echo "$DUPLICATE")
Equivalently, using a herestring:
grep -vFf <(echo "$DUPLICATE") <<< "$LINES"
another approach which doesn't require to create a duplicate list separately,
$ awk '{a[$0]++} END{for(k in a) if(a[k]==1) print k}' <<< "$LINES"
count occurrence of each line, print only if it's not duplicated (count==1).

xargs sed and command substitution in bash

I'm trying to pass a xargs string replace into a sed replacement inside of a substitution, here's the non-working code.
CALCINT=$CALCINT$(seq $CALCLINES | xargs -Iz echo $CALCINT' -F "invoiceid'z'="'$(sed -n '/invoiceid'z'/s/.*name="invoiceid'z'"\s\+value="\([^"]\+\).*/\1/p' output.txt))
Everything works up until the sed inside the second substitution. the 'z' should be a number 1-20 based on the $CALCLINES variable. I know it has something to do with not escaping properly for sed but I'm having trouble wrapping my head around how sed wants things escaped in this situation.
Here's the surrounding lines of code:
curl -b mycookiefile -c mycookiefile http://localhost/calcint.php > output.txt
CALCLINES=`grep -o 'class="addinterest"' output.txt | wc -l`
CALCINT=$CALCINT$(seq $CALCLINES | xargs -Iz echo $CALCINT' -F "invoiceid'z'="'$(sed -n '/invoiceid17/s/.*name="invoiceid17"\s\+value="\([^"]\+\).*/\1/p' output.txt))
echo $CALCINT
Output: (What I get now)
-F "invoiceid1=" -F "invoiceid2=" -F "invoiceid3=" -F "invoiceid4=" -F "invoiceid5=" -F "invoiceid6=" -F "invoiceid7=" -F "invoiceid8=" -F "invoiceid9=" -F "invoiceid10=" -F "invoiceid11=" -F "invoiceid12=" -F "invoiceid13=" -F "invoiceid14=" -F "invoiceid15=" -F "invoiceid16=" -F "invoiceid17=" -F "invoiceid18=" -F "invoiceid19=" -F "invoiceid20="
What I'm hoping to see as output is something like this
-F "invoiceid1=2342" -F "invoiceid2=456456" -F "invoiceid3=78987" ...etc etc
-------------------------EDIT-----------------------
FWIW...here's the output.txt and other things I've tried.
for i in $(seq -f "%02g" ${CALCLINES});do
sed -n "/interest$i/s/.*name=\"interest$i\"\s\+value=\"\([^\"]\+\).*/\1/p" output.txt > output2.txt
done
output2.txt contains nothing
Thanks to #janos response for clearing things up but taking a step back makes it clear to me that the root of the issue here is that I'm struggling to get the invoice ids out. It's dynamically generated HTML "....name="invoiceid7" value="556"..." so there isn't anything consistent in those particular tags that I can grep on, which is why I was counting another tag that IS consistent then trying to use a variable sed to basically deduce the tag name then extract the value.
Annd..output.txt https://pastebin.com/ewUaddVi
------UPDATE-----
Working solution
Stuff sed into a loop. Note how I had to use ' to use variables in the sed string. That is well documented elsewhere on here. :)
for i in $(seq ${CALCLINES});do
e="interest"$i`
CALCINT=$CALCINT' -F "'$e'='
CALCINT=$CALCINT$(sed -n '/'$e'/s/.*name="'$e'"\s\+value="\([^"]\+\).*/\1/p' output.txt)'"'
done
Please read through the comments on the solution below, there is a cleaner way of doing this.
Your current approach cannot work, specifically this part:
... | xargs -Iz echo -F "invoiceid'z'="$(sed ...)"
The problem is that the $(sed ...) will not be evaluated for each line in the input during the execution of xargs.
The shell will evaluate this once, before it actually runs xargs.
And you need there dynamic values from your input.
You can make this work by taking a different approach:
Extract the invoice ids. For example, write a grep or sed pipeline that produces as output simply the list of invoice ids
Transform the invoice list to the -F "invoiceidNUM=..." form that you need
For the second step, Awk could be practical. The script could be something like this:
curl -b mycookiefile -c mycookiefile http://localhost/calcint.php > output.txt
args=$(sed ... output.txt | awk '{ print "-F \"invoice" NR "=" $0 "\"" }')
echo $args
For example if the sed step produces 2342, 456456, 78987, then the output will be:
-F "invoice1=2342" -F "invoice2=456456" -F "invoice3=78987"

grep: compare string from file with another string

I have a list of files paths that I need to compare with a string:
git_root_path=$(git rev-parse --show-toplevel)
list_of_files=.git/ForGeneratingSBConfigAlert.txt
cd $git_root_path
echo "These files needs new OSB config:"
while read -r line
do
modfied="$line"
echo "File for compare: $modfied"
if grep -qf $list_of_files" $modfied"; then
echo "Found: $modfied"
fi
done < <(git status -s | grep -v " M" | awk '{if ($1 == "M") print $2}')
$modified - is a string variable that stores path to file
Pattern file example:
SVCS/resources/
SVCS/bus/projects/busCallout/
SVCS/bus/projects/busconverter/
SVCS/bus/projects/Resources/ (ignore .jar)
SVCS/bus/projects/Teema/
SVCS/common/
SVCS/domain/
SVCS/techutil/src/
SVCS/tech/mds/src/java/fi/vr/h/service/tech/mds/exception/
SVCS/tech/mds/src/java/fi/vr/h/service/tech/mds/interfaces/
SVCS/app/cashmgmt/src/java/fi/vr/h/service/app/cashmgmt/exception/
SVCS/app/cashmgmt/src/java/fi/vr/h/service/app/cashmgmt/interfaces/
SVCS/app/customer/src/java/fi/vr/h/service/app/customer/exception/
SVCS/app/customer/src/java/fi/vr/h/service/app/customer/interfaces/
SVCS/app/etravel/src/java/fi/vr/h/service/app/etravel/exception/
SVCS/app/etravel/src/java/fi/vr/h/service/app/etravel/interfaces/
SVCS/app/hermes/src/java/fi/vr/h/service/app/hermes/exception/
SVCS/app/hermes/src/java/fi/vr/h/service/app/hermes/interfaces/
SVCS/app/journey/src/java/fi/vr/h/service/app/journey/exception/
SVCS/app/journey/src/java/fi/vr/h/service/app/journey/interfaces/
SVCS/app/offline/src/java/fi/vr/h/service/app/offline/exception/
SVCS/app/offline/src/java/fi/vr/h/service/app/offline/interfaces/
SVCS/app/order/src/java/fi/vr/h/service/app/order/exception/
SVCS/app/order/src/java/fi/vr/h/service/app/order/interfaces/
SVCS/app/payment/src/java/fi/vr/h/service/app/payment/exception/
SVCS/app/payment/src/java/fi/vr/h/service/app/payment/interfaces/
SVCS/app/price/src/java/fi/vr/h/service/app/price/exception/
SVCS/app/price/src/java/fi/vr/h/service/app/price/interfaces/
SVCS/app/product/src/java/fi/vr/h/service/app/product/exception/
SVCS/app/product/src/java/fi/vr/h/service/app/product/interfaces/
SVCS/app/railcar/src/java/fi/vr/h/service/app/railcar/exception/
SVCS/app/railcar/src/java/fi/vr/h/service/app/railcar/interfaces/
SVCS/app/reservation/src/java/fi/vr/h/service/app/reservation/exception/
SVCS/app/reservation/src/java/fi/vr/h/service/app/reservation/interfaces/
kraken_test.txt
namaker_test.txt
shmaker_test.txt
I need to compare file search pattern with a string, is it possible using grep?
I'm not sure I understand the overall logic, but a few immediate suggestions come to mind.
You can avoid grep | awk in the vast majority of cases.
A while loop with a grep on a line at a time inside the loop is an antipattern. You probably just want to run one grep on the whole input.
Your question would still benefit from an explanation of what you are actually trying to accomplish.
cd "$(git rev-parse --show-toplevel)"
git status -s | awk '!/ M/ && $1 == "M" { print $2 }' |
grep -Fxf .git/ForGeneratingSBConfigAlert.txt
I was trying to think of a way to add back your human-readable babble, but on second thought, this program is probably better without it.
The -x option to grep might be wrong, depending on what you are really hoping to accomplish.
This should work:
git status -s | grep -v " M" | awk '{if ($1 == "M") print $2}' | \
grep --file=.git/ForGeneratingSBConfigAlert.txt --fixed-strings --line-regexp
Piping the awk output directly to grep avoids the while loop entirely. In most cases you'll find you don't really need to print debug messages and the like in it.
--file takes a file with one pattern to match per line.
--fixed-strings avoids treating any characters in the patterns as special.
--line-regexp anchors the patterns so that they only match if a full line of input matches one of the patterns.
All that said, could you clarify what you are actually trying to accomplish?

shell script to read contain from file and grep on other file

I am working on shell, I want to write one liner which will read the file contents of file A and execute grep command on file B.
for example, suppose there are two file
dataFile.log which have following value
abc
xyz
... and so on
now read abc and grep on searchFile.log like grep abc searchFile.log
I have shell script for the same but want one liner for it
for i in "cat dataFile.log" do grep $i searchFile.log done;
try this:
grep -f dataFile.log searchFile.log
Note that if you want to grep as fixed string, you need -F, if you want to match the text in dataFile.log as regex, use -E or -P
How about the following: it even ignores blank lines and # comments:
while read FILE; do if [[ "$FILE" != [/a-zA-Z0-9]* ]]; do continue; fi; grep -h pattern "$FILE"; done;
Beware: have not compiled this.
You can use grep -f option:
cat dataFile.log | grep -f searchFile.log
Edit
OK, now I understand the problem. You want to use every line from dataFile.log to grep in searchFile.log. I also see you have value1|value2|..., so instead of grep you need egrep.
Try with this:
for i in `cat dataFile.log`
do
egrep "$i" searchFile.log
done
Edit 2
Following chepner suggestion:
egrep -f dataFile.log searchFile.log

Resources