I am using grep tool inside a Bash loop to count the number of found unique values inside the population but I would also like to output the found line.
Currently I am running the following command inside my loop:
n=$(grep -o -i ${uniqueVal} ${population} | wc -l)
total+=n
This however only outputs the number of occurrences found it does not output the matched line. Is there a way to count the number of occurrences while outputting only the matches to the screen?
uniqueVal
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
population
192.168.1.4
192.168.1.1
192.168.1.88
192.168.1.77
192.168.1.72
192.168.1.66
192.168.1.55
192.168.1.32
192.168.1.22
192.168.1.24
192.168.1.98
192.168.1.99
Output Total Unique Found: 2
I want the Desired Output to display the line in population that was found and then the total number found.
Found: 192.168.1.4
Found: 192.168.1.1
Total Unique Found: 2
Assuming uniqueVal and population are files:
join uniqueVal population | sed 's/.*/Found: &/' | tee /dev/stderr | wc -l
or f they are variables:
join <(echo $uniqueVal) <(echo $population) | sed 's/.*/Found: &/' | tee /dev/stderr | wc -l
Add -ne flag to get the line number, along with tee /dev/stderr to print the resulting output
grep -o -i ${uniqueVal} ${population} -ne | tee /dev/stderr | wc -l
If you want to print the lines with ${population} inside ${population} file, along with the number of lines founds, then I would use awk, you can use the following:
awk '/${population}/' ${uniqueVal} | tee /dev/stderr | wc -l
Related
I'm trying to edit a grep/sed output - currently it is as follows;
grep -Pzo '(?s)INTO .?db.? VALUES[^(]\K[^;]*' aniko.sql | grep -Pao '\(([^,]*,){2}\K[^,]*'
'yscr_bbYcqN'
'yscr_bbS4kf'
'yscr_bbhrSZ'
'yscr_bbBl0C'
'yscr_bbrsKX'
I am then wanting to add test_ prefix to them all, however, I can only get it to amend one of them - so;
#!/bin/sh
read -p "enter the cPanel username: " cpuser
cd "/home/$cpuser/public_html"
dbusers=($(grep -Pzo '(?s)INTO .?db.? VALUES[^(]\K[^;]*' aniko.sql | grep -Pao '\(([^,]*,){2}\K[^,]*' | grep -Pzo [^\']));
for dbuser in $dbusers; do sed -i "s/$dbuser/${cpuser}\_${dbuser}/g" aniko.sql; done;
The result of this only updates one of the lines (all should show sensationahosti_)
sensationalhosti_yscr_bbYcqN
yscr_bbS4kf
yscr_bbhrSZ
yscr_bbBl0C
yscr_bbrsKX
The format of the file I am changing is as follows;
INSERT INTO `db` VALUES ('localhost','blog','sensationalhosti_yscr_bbYcqN','Y','Y','Y','Y','Y','N','N','N','N','N'),('localhost','blog1','yscr_bbS4kf','Y','Y','Y','Y','Y','N','N','N','N','N'),('localhost','blog','yscr_bbhrSZ','Y','Y','Y','Y','Y','N','N','N','N','N'),('localhost','blog','yscr_bbBl0C','Y','Y','Y','Y','Y','N','N','N','N','N'),('localhost','blog','yscr_bbrsKX','Y','Y','Y','Y','Y','N','N','N','N','N');
I need a total of lines in terminal output
Like cat lista_de_compras.txt:
arroz
feijão
leite
I want to get 3
Use grep:
grep -c ^ lista_de_compras.txt
The output will be 3
or use the cat with the wc
cat lista_de_compras.txt | wc -l
The output will be 3
Or Use only the wc:
wc -l lista_de_compras.txt (suggested)
The output will be 3
A way to do it is as follows:
cat lista_de_compras.txt | wc -l
See: https://ss64.com/bash/wc.html
Situation
I have many URLs in a file, and I need to find out how many unique URLs exist.
I would like to run either a bash script or a command.
myfile.log
/home/myfiles/www/wp-content/als/xm-sf0ab5df9c1262f2130a9b313192deca4-f0ab5df9c1262f2130a9b313192deca4-c23c5fbca96e8d641d148bac41017635|https://public.rgfl.org/HS/PowerPoint%20Presentations/Health%20and%20Safety%20Law.ppt,18,17
/home/myfiles/www/wp-content/als/xm-s4bf050d47df5bfaf0486a50a8528cb16-4bf050d47df5bfaf0486a50a8528cb16-c23c5fbca96e8d641d148bac41017635|https://public.rgfl.org/HS/PowerPoint%20Presentations/Health%20and%20Safety%20Law.ppt,15,14
/home/myfiles/www/wp-content/als/xm-sad122bf22152ba4823a520cc2fe59f40-ad122bf22152ba4823a520cc2fe59f40-c23c5fbca96e8d641d148bac41017635|https://public.rgfl.org/HS/PowerPoint%20Presentations/Health%20and%20Safety%20Law.ppt,17,16
/home/myfiles/www/wp-content/als/xm-s3c0f031eebceb0fd5c4334ecef15292d-3c0f031eebceb0fd5c4334ecef15292d-c23c5fbca96e8d641d148bac41017635|https://public.rgfl.org/HS/PowerPoint%20Presentations/Health%20and%20Safety%20Law.ppt,12,11
/home/myfiles/www/wp-content/als/xm-sff661e8c3b4f94957926d5434d0ad549-ff661e8c3b4f94957926d5434d0ad549-c23c5fbca96e8d641d148bac41017635|https://quality.gha.org/Portals/2/documents/HEN/Meetings/nursesinstitute/062013/nursesroleineliminatingharm_moddydunning.pptx,17,16
/home/myfiles/www/wp-content/als/xm-s32c41ec2a5440ad220008b9abfe9add2-32c41ec2a5440ad220008b9abfe9add2-c23c5fbca96e8d641d148bac41017635|https://quality.gha.org/Portals/2/documents/HEN/Meetings/nursesinstitute/062013/nursesroleineliminatingharm_moddydunning.pptx,19,18
/home/myfiles/www/wp-content/als/xm-s28787ca2f4372ddb3616d3fd53c161ab-28787ca2f4372ddb3616d3fd53c161ab-c23c5fbca96e8d641d148bac41017635|https://quality.gha.org/Portals/2/documents/HEN/Meetings/nursesinstitute/062013/nursesroleineliminatingharm_moddydunning.pptx,22,21
/home/myfiles/www/wp-content/als/xm-s89a7b68158e38391da9f0de1e636c0d5-89a7b68158e38391da9f0de1e636c0d5-c23c5fbca96e8d641d148bac41017635|https://quality.gha.org/Portals/2/documents/HEN/Meetings/nursesinstitute/062013/nursesroleineliminatingharm_moddydunning.pptx,13,12
/home/myfiles/www/wp-content/als/xm-sc4b14e10f6151995f21334061ff1d139-c4b14e10f6151995f21334061ff1d139-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hy-wire-car-2.pptx,13,12
/home/myfiles/www/wp-content/als/xm-se589d47d163e43fa0c0d68e824e2c286-e589d47d163e43fa0c0d68e824e2c286-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hy-wire-car-2.pptx,19,18
/home/myfiles/www/wp-content/als/xm-s52f897a623c539d09bfb988bfb153888-52f897a623c539d09bfb988bfb153888-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hy-wire-car-2.pptx,14,13
/home/myfiles/www/wp-content/als/xm-sccf27a904c5b88e96a3522b2e1180fed-ccf27a904c5b88e96a3522b2e1180fed-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hy-wire-car-2.pptx,18,17
/home/myfiles/www/wp-content/als/xm-s6874bf9d589708764dab754e5af06ddf-6874bf9d589708764dab754e5af06ddf-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hy-wire-car-2.pptx,17,16
/home/myfiles/www/wp-content/als/xm-s46c55ec8387dbdedd7a83b3ad541cdc1-46c55ec8387dbdedd7a83b3ad541cdc1-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hy-wire-car-2.pptx,19,18
/home/myfiles/www/wp-content/als/xm-s08cfdc15f5935b947bbaa93c7193d496-08cfdc15f5935b947bbaa93c7193d496-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hydro-power-plant.ppt,9,8
/home/myfiles/www/wp-content/als/xm-s86e267bd359c12de262c0279cee0c941-86e267bd359c12de262c0279cee0c941-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hydro-power-plant.ppt,15,14
/home/myfiles/www/wp-content/als/xm-s5aa60354d134b87842918d760ec8bc30-5aa60354d134b87842918d760ec8bc30-c23c5fbca96e8d641d148bac41017635|https://royalmechanical.files.wordpress.com/2011/06/hydro-power-plant.ppt,14,13
Desired Result:
Unique Urls: 4
cut -d "|" -f 2 file | cut -d "," -f 1 | sort -u | wc -l
Output:
4
See: man cut, man sort
An awk solution would be
awk '{sub(/^[^|]*\|/,"");gsub(/,[^,]*/,"");i+=a[$0]++?0:1}END{print i}' file
4
If you happen to use GNU awk then below would also give you the same result
awk '{i+=a[gensub(/.*(http[^,]*).*/,"\\1",1)]++?0:1}END{print i}' file
4
Or even short as pointed out in this cracker comment by #cyrus
awk -F '[|,]' '{i+=!a[$2]++} END{print i}' file
4
which uses awk multiple field separator functionality with more idiomatic awk.
Note: See the [ awk manual ] for more info.
Parse with sed, and since file appears to be already sorted,
(with respect to URLs), just run uniq, and count it:
echo Unique URLs: $(sed 's/^.*|\([^,]*\),.*$/\1/' file | uniq | wc -l)
Use GNU grep to extract URLs:
echo Unique URLs: $(grep -o 'ht[^|,]*' file | uniq | wc -l)
Output (either method):
Unique URLs: 4
tr , '|' < myfile.log | sort -u -t '|' -k 2,2 | wc -l
tr , '|' < myfile.log translates all commas into pipe characters
sort -u -t '|' -k 2,2 sorts unique (-u), pipe delimited (-t '|'), in the second field only (-k 2,2)
wc -l counts the unique lines
I want to detect which one of my files is corrupt, and by corrupt it means that the file does not have 102 lines in it. I want the for loop that I'm writing to output a error message giving me the file name of the corrupt files. I have files named ethane1.log ethane2.log ethane3.log ... ethane10201.log .
for j in {1..10201}
do
if [ ! (grep 'C 2- C 5' ethane$j.log | cut -c 22- | tail -n +2 | awk '{for (i=1;i<=NF;i++) print $i}'; done | wc -l) == 102]
then echo "Ethane$j.log is corrupt."
fi
done
When the file is not corrupt, the input:
grep 'C 2- C 5' ethane$j.log | cut -c 22- | tail -n +2 | awk '{for (i=1;i<=NF;i++) print $i}'; done | wc -l
returns:
102
Or else it is another number.
Only thing is, I'm not sure the syntax for the if construct (How to create a variable from the 102 output of wc -l, and then how to check if it is equal to or not equal to 102.)
A sample output would be:
Ethane100.log is corrupt.
Ethane2010.log is corrupt.
Ethane10201.log is corrupt.
To count lines, use wc -l:
wc -l ethane*.log | grep -v '^ *102 ' | head -n-1
grep -v removes matching lines
^ matches the start of a line
space* matches any number of spaces (0 or more)
head removes some trailing lines
-n-1 removes the last line (the total)
Using gawk
awk 'ENDFILE{if(NR!=102)print NR,FILENAME}' ethane*.log
At the end of each file, checks the number of lines isn't 102 and prints the number of lines and the filename.
Scenario:
I have a bunch of VIPs. While doing an NSLOOKUP, the output generally returns an output with one public IP. In cases where the loadbalancer fails, the NSLOOKUP returns two public IPs. for such scenarios, I want to write a script.
Logic:
for i vip1 vip2 vip3; do nslookup $i | grep -v "<private IP> | grep 'Address:' | wc -l ; done
in an ideal scenario, the output will look like
1
1
1
If I could sum the output, it would say 3. If something goes wrong, the output will show a sum > 3. I was unable to sum in the above case. Please advice
echo vip1 vip2 vip3 | xargs -n 1 nslookup | \
awk '/Address/ && !/<private-ip>/ {s++} END{print s}'
Instead of summing, just count the matches in the entire loop. And use grep -c to do the match and count in one step.
for i in vip1 vip2 vip3
do
nslookup "$i"
done | grep -v "<private IP>" | grep -c 'Address'
Make it with awk:
user#host:~# cat blub | awk '{ SUM += $1} END { print SUM }'
4
blub is a file with contents:
user#host:~# cat blub
1
2
1
To get a sum you can substitute the command into arithmetic evaluation construct. If your pipeline produces an integer, then try using following loop:
for i in vip1 vip2 vip3; do
((sum += $(nslookup $i | .. rest of pipeline .. | wc -l)))
done
# .. do something with $sum ..
maybe not the most elegant look, but shall work