Parsing list in bash - bash

I have a text file which looks like this:
...
unique_trigger = item1
item2
item3
...
itemN
unique_end_trigger
...
Is there a quick (maybe on-liner if possible) bash script I could use to parse the file and pipe item1 item2 item3... into another command? The number of items is not determined. I looked at other bash scripts to read variables out of a file but they either source it, parse each item manually (predetermined list length) or assign a environment variable to each item based on its name (which is not my case). I am looking for something like this:
parse_command file.txt | other_command

perl:
perl -0777 -pE 's/.*unique_trigger\s*=\s*(.*)unique_end_trigger.*/$1/s; s/^\s+//gm' file.txt
item1
item2
item3
...
itemN

One-liner
cat file.txt | tr -s "[:space:]" " " | \
sed -En "s/(.*)(unique_trigger = )(.*)(unique_end_trigger)/\3/p" | \
other_command

awk '/unique_end_trigger/&&--f||f||/unique_trigger/&&f++ {printf $1 " "}' <(sed 's/unique_trigger =/ & \n /' input_file)
item1 item2 item3 ... itemN

Not quite a one-liner, but it should do the trick as long as your triggers don't contain spaces.
flag=0
while read tr eq it ; do
if [ "$tr" = "unique_trigger" ] ; then
echo "$it"
flag=1
elif [ $flag = 1 ] ; then
if [ "$tr" = "unique_end_trigger" ] ; then
flag=0
else
echo "$tr"
fi
fi
done

Two solutions, using the same concept. When the start trigger is found, add the item (3rd item in the line) to a string. Until the end trigger is found, the item is the only work in the line, so add it to the string. Output the string at the end.
Bash parsing
#!/bin/bash
file="file.txt"
start_trigger="unique_trigger"
end_trigger="unique_end_trigger"
items=''
between_trigger="no"
while IFS= read -r line; do
#echo "-----$line-----"
# Start trigger lines
if [[ "$line" =~ "$start_trigger =" ]]
then
items="$items $(echo "$line" | awk '{print $3}')"
between_trigger="yes"
continue
fi
# End trigger lines
if [[ "$line" =~ "$end_trigger" ]]
then
between_trigger="no"
continue
fi
# Lines between start and end trigger
if [[ "$between_trigger" == "yes" ]]
then
items="$items $line"
continue
fi
done < "$file"
echo ">>$items<<"
Using it: script.bash | xargs echo
Replace echo by any command.
Awk version
BEGIN {
output = ""
between = "no"
}
/unique_end_trigger/ {
between = "no";
}
/.*/ {
if (between == "yes") output = output " " $1
}
/unique_trigger/ {
between = "yes";
output = output " " $3;
}
END { print output }
Using it: awk -f script.awk file.txt | xargs echo
Replace echo with whatever command you want.

Related

How to test if an attribute exists in a file with Bash?

I'm trying to get this tested, but I'm not sure if the if is right
if [ $(lsattr /mnt/backup/*.* | grep i) ] ;
then
echo "file $_ has i attribute";
else
echo "file $_ does not have i attribute"
fi
This is the lsattr on that directory:
----i----------------- /mnt/backup/Backup-Full_02-04-2022.7z
---------------------- /mnt/backup/test.7z
Thank you
With grep i you will also match file names containing i. Moreover, $_ is not set so its value is probably just the empty string. If you really want to use an if statement you also need a loop. And grep is not needed any more if you use the bash conditional expressions:
$ lsattr /mnt/backup/*.* | while read -r attr name; do
if [[ "$attr" == "*i*" ]]; then
echo "file $name has i attribute"
else
echo "file $name does not have i attribute"
fi
done
file /mnt/backup/Backup-Full_02-04-2022.7z has i attribute
file /mnt/backup/test.7z does not have i attribute
If you can use awk instead of grep you can easily limit the search to the first word:
awk '$1 ~ "i"'
And you don't need any bash if or while loop any more, all this can be embedded in the awk script:
$ lsattr /mnt/backup/*.* | awk -vs1=" has " -vs2=" does not have " \
'{print $2 ($1 ~ "i" ? s1 : s2) "i attribute"}'
file /mnt/backup/Backup-Full_02-04-2022.7z has i attribute
file /mnt/backup/test.7z does not have i attribute

Parsing .ini file in bash

I have a below properties file and would like to parse it as mentioned below. Please help in doing this.
.ini file which I created :
[Machine1]
app=version1
[Machine2]
app=version1
app=version2
[Machine3]
app=version1
app=version3
I am looking for a solution in which ini file should be parsed like
[Machine1]app = version1
[Machine2]app = version1
[Machine2]app = version2
[Machine3]app = version1
[Machine3]app = version3
Thanks.
Try:
$ awk '/\[/{prefix=$0; next} $1{print prefix $0}' file.ini
[Machine1]app=version1
[Machine2]app=version1
[Machine2]app=version2
[Machine3]app=version1
[Machine3]app=version3
How it works
/\[/{prefix=$0; next}
If any line begins with [, we save the line in the variable prefix and then we skip the rest of the commands and jump to the next line.
$1{print prefix $0}
If the current line is not empty, we print the prefix followed by the current line.
Adding spaces
To add spaces around any occurrence of =:
$ awk -F= '/\[/{prefix=$0; next} $1{$1=$1; print prefix $0}' OFS=' = ' file.ini
[Machine1]app = version1
[Machine2]app = version1
[Machine2]app = version2
[Machine3]app = version1
[Machine3]app = version3
This works by using = as the field separator on input and = as the field separator on output.
I love John1024's answer. I was looking for exactly that. I have created a bash function that allows me to lookup sections or specific keys based on his idea:
function iniget() {
if [[ $# -lt 2 || ! -f $1 ]]; then
echo "usage: iniget <file> [--list|<section> [key]]"
return 1
fi
local inifile=$1
if [ "$2" == "--list" ]; then
for section in $(cat $inifile | grep "\[" | sed -e "s#\[##g" | sed -e "s#\]##g"); do
echo $section
done
return 0
fi
local section=$2
local key
[ $# -eq 3 ] && key=$3
# https://stackoverflow.com/questions/49399984/parsing-ini-file-in-bash
# This awk line turns ini sections => [section-name]key=value
local lines=$(awk '/\[/{prefix=$0; next} $1{print prefix $0}' $inifile)
for line in $lines; do
if [[ "$line" = \[$section\]* ]]; then
local keyval=$(echo $line | sed -e "s/^\[$section\]//")
if [[ -z "$key" ]]; then
echo $keyval
else
if [[ "$keyval" = $key=* ]]; then
echo $(echo $keyval | sed -e "s/^$key=//")
fi
fi
fi
done
}
So given this as file.ini
[Machine1]
app=version1
[Machine2]
app=version1
app=version2
[Machine3]
app=version1
app=version3
then the following results are produced
$ iniget file.ini --list
Machine1
Machine2
Machine3
$ iniget file.ini Machine3
app=version1
app=version3
$ iniget file.ini Machine1 app
version1
$ iniget file.ini Machine2 app
version2
version3
Again, thanks to #John1024 for his answer, I was pulling my hair out trying to create a simple bash ini parser that supported sections.
Tested on Mac using GNU bash, version 5.0.0(1)-release (x86_64-apple-darwin18.2.0)
You can try using awk:
awk '/\[[^]]*\]/{ # Match pattern like [...]
a=$1;next # store the pattern in a
}
NF{ # Match non empty line
gsub("=", " = ") # Add space around the = character
print a $0 # print the line
}' file
Excellent answers here. I made some modifications to #davfive's function to fit it better to my use case. This version is largely the same except it allows for whitespace before and after = characters, and allows values to have spaces in them.
# Get values from a .ini file
function iniget() {
if [[ $# -lt 2 || ! -f $1 ]]; then
echo "usage: iniget <file> [--list|<section> [key]]"
return 1
fi
local inifile=$1
if [ "$2" == "--list" ]; then
for section in $(cat $inifile | grep "^\\s*\[" | sed -e "s#\[##g" | sed -e "s#\]##g"); do
echo $section
done
return 0
fi
local section=$2
local key
[ $# -eq 3 ] && key=$3
# This awk line turns ini sections => [section-name]key=value
local lines=$(awk '/\[/{prefix=$0; next} $1{print prefix $0}' $inifile)
lines=$(echo "$lines" | sed -e 's/[[:blank:]]*=[[:blank:]]*/=/g')
while read -r line ; do
if [[ "$line" = \[$section\]* ]]; then
local keyval=$(echo "$line" | sed -e "s/^\[$section\]//")
if [[ -z "$key" ]]; then
echo $keyval
else
if [[ "$keyval" = $key=* ]]; then
echo $(echo $keyval | sed -e "s/^$key=//")
fi
fi
fi
done <<<"$lines"
}
For taking disparate sectional and tacking the section name (including 'no-section'/Default together) to each of its related keyword (along with = and its keyvalue), this one-liner AWK will do the trick coupled with a few clean-up regex.
ini_buffer="$(echo "$raw_buffer" | awk '/^\[.*\]$/{obj=$0}/=/{print obj $0}')"
Will take your lines and output them like you wanted:
+++ awk '/^\[.*\]$/{obj=$0}/=/{print obj $0}'
++ ini_buffer='[Machine1]app=version1
[Machine2]app=version1
[Machine2]app=version2
[Machine3]app=version1
[Machine3]app=version3'
A complete solution to the INI-format File
As Clonato, INI-format expert said that for the latest INI version 1.4 (2009-10-23), there are several other tricky aspects to the INI file:
character set constraint for section name
character set constraint for keyword
And lastly is for the keyvalue to be able to handle pretty much anthing that is not used in the section and keyword name; that includes nesting of quotes inside a pair of same single/double-quote.
Except for the nesting of quotes, a INI-format Github complete solution to parsing INI-format file with default section:
# syntax: ini_file_read <raw_buffer>
# outputs: formatted bracket-nested "[section]keyword=keyvalue"
ini_file_read()
{
local ini_buffer raw_buffer hidden_default
raw_buffer="$1"
# somebody has to remove the 'inline' comment
# there is a most complex SED solution to nested
# quotes inline comment coming ... TBA
raw_buffer="$(echo "$raw_buffer" | sed '
s|[[:blank:]]*//.*||; # remove //comments
s|[[:blank:]]*#.*||; # remove #comments
t prune
b
:prune
/./!d; # remove empty lines, but only those that
# become empty as a result of comment stripping'
)"
# awk does the removal of leading and trailing spaces
ini_buffer="$(echo "$raw_buffer" | awk '/^\[.*\]$/{obj=$0}/=/{print obj $0}')" # original
ini_buffer="$(echo "$ini_buffer" | sed 's/^\s*\[\s*/\[/')"
ini_buffer="$(echo "$ini_buffer" | sed 's/\s*\]\s*/\]/')"
# finds all 'no-section' and inserts '[Default]'
hidden_default="$(echo "$ini_buffer" \
| egrep '^[-0-9A-Za-z_\$\.]+=' | sed 's/^/[Default]/')"
if [ -n "$hidden_default" ]; then
echo "$hidden_default"
fi
# finds sectional and outputs as-is
echo "$(echo "$ini_buffer" | egrep '^\[\s*[-0-9A-Za-z_\$\.]+\s*\]')"
}
The unit test for this StackOverflow post is included in this file:
https://github.com/egberts/bash-ini-file
Source:
https://github.com/egberts/easy-admin/blob/main/test/section-regex.sh
https://cloanto.com/specs/ini/#escapesequences

bash, adding string after a line

I'm trying to put together a bash script that will search a bunch of files and if it finds a particular string in a file, it will add a new line on the line after that string and then move on to the next file.
#! /bin/bash
echo "Creating variables"
SEARCHDIR=testfile
LINENUM=1
find $SEARCHDIR* -type f -name *.xml | while read i; do
echo "Checking $i"
ISBE=`cat $i | grep STRING_TO_SEARCH_FOR`
if [[ $ISBE =~ "STRING_TO_SEARCH_FOR" ]] ; then
echo "found $i"
cat $i | while read LINE; do
((LINENUM=LINENUM+1))
if [[ $LINE == "<STRING_TO_SEARCH_FOR>" ]] ; then
echo "editing $i"
awk -v "n=$LINENUM" -v "s=new line to insert" '(NR==n) { print s } 1' $i
fi
done
fi
LINENUM=1
done
the bit I'm having trouble with is
awk -v "n=$LINENUM" -v "s=new line to insert" '(NR==n) { print s } 1' $i
if I just use $i at the end, it will output the content to the screen, if I use $i > $i then it will just erase the file and if I use $i >> $i it will get stuck in a loop until the disk fills up.
any suggestions?
Unfortunately awk dosen't have an in-place replacement option, similar to sed's -i, so you can create a temp file and then remove it:
awk '{commands}' file > tmpfile && mv tmpfile file
or if you have GNU awk 4.1.0 or newer, the -i inplace is added, so you can do:
awk -i inplace '{commands}' file
to modify the original
#cat $i | while read LINE; do
# ((LINENUM=LINENUM+1))
# if [[ $LINE == "<STRING_TO_SEARCH_FOR>" ]] ; then
# echo "editing $i"
# awk -v "n=$LINENUM" -v "s=new line to insert" '(NR==n) { print s } 1' $i
# fi
# done
# replaced by
sed -i 's/STRING_TO_SEARCH_FOR/&\n/g' ${i}
or use awk in place of sed
also
# ISBE=`cat $i | grep STRING_TO_SEARCH_FOR`
# if [[ $ISBE =~ "STRING_TO_SEARCH_FOR" ]] ; then
#by
if [ $( grep -c 'STRING_TO_SEARCH_FOR' ${i} ) -gt 0 ]; then
# if file are huge, if not directly used sed on it, it will be faster (but no echo about finding the file)
If you can, maybe use a temporary file?
~$ awk ... $i > tmpfile
~$ mv tmpfile $i
Or simply awk ... $i > tmpfile && mv tmpfile $i
Note that, you can use mktemp to create this temporary file.
Otherwise, with sed you can insert a line right after a match:
~$ cat f
auie
nrst
abcd
efgh
1234
~$ sed '/abcd/{a\
new_line
}' f
auie
nrst
abcd
new_line
efgh
1234
The command search if the line matches /abcd/, if so, it will append (a\) the line new_line.
And since sed as the -i to replace inline, you can do:
if [[ $ISBE =~ "STRING_TO_SEARCH_FOR" ]] ; then
echo "found $i"
echo "editing $i"
sed -i "/STRING_TO_SEARCH_FOR/{a
\new line to insert
}" $i
fi

count words in a file without using wc

Working in a shell script here, trying to count the number of words/characters/lines in a file without using the wc command. I can get the file broken into lines and count those easy enough, but I'm struggling here to get the words and the characters.
#define word_count function
count_stuff(){
c=0
w=0
l=0
local f="$1"
while read Line
do
l=`expr $line + 1`
# now that I have a line I want to break it into words and characters???
done < "$f"
echo "Number characters: $chars"
echo "Number words: $words"
echo "Number lines: $line"
}
As for characters, try this (adjust echo "test" to where you get your output from):
expr `echo "test" | sed "s/./ + 1/g;s/^/0/"`
As for lines, try this:
expr `echo -e "test\ntest\ntest" | sed "s/^.*$/./" | tr -d "\n" | sed "s/./ + 1/g;s/^/0/"`
===
As for your code, you want something like this to count words (if you want to go at it completely raw):
while read line ; do
set $line ;
while true ; do
[ -z $1 ] && break
l=`expr $l + 1`
shift ;
done ;
done
You can do this with the following Bash shell script:
count=0
for var in `cat $1`
do
count=`echo $count+1 | bc`
done
echo $count

how to map one csv file content to second csv file and write it another csv using unix

After writing some unix scripts I am able to manage to get data from different xml files to csv format and now I got stuck with the following problem
file1.csv : contains
1,5,6,7,8
2,3,4,5,9
1,6,10,11,12
1,5,11,12
file2.csv : contains
1,Mango,Tuna,Webby,Through,Franky,Sam,Sumo
2,Franky
3,Sam
4,Sumo
5,Mango,Tuna,Webby
6,Tuna,Webby,Through
7,Through,Sam,Sumo
8,Nothing
9,Sam,Sumo
10,Sumo,Mango,Tuna
11,Mango,Tuna,Webby,Through
12,Mango,Tuna,Webby,Through,Franky
output I want is
1,5,6,7,8
Mango,Tuna,Webby,Through,Franky,Sam,Sumo
Mango,Tuna,Webby
Tuna,Webby,Through
Through,Sam,Sumo
Nothing
Common word:None
2,3,4,5,9
Franky
Sam
Sumo
Mango,Tuna,Webby
Sam, Sumo
Common Word:None
1,6,10,11,12
Mango,Tuna,Webby,Through,Franky,Sam,Sumo
Tuna,Webby,Through
Sumo,Mango,Tuna
Mango,Tuna,Webby,Through
Mango,Tuna,Webby,Through,Franky
Common word: Tuna
1,5,11,12
Mango,Tuna,Webby,Through,Franky,Sam,Sumo
Mango,Tuna,Webby
Mango,Tuna,Webby,Through
Mango,Tuna,Webby,Through,Franky
Common word: Mango,Tuna,Webby
I apprecaite any help.
Thanks
I got some solution but not complete
##!/bin/bash
count=1
count_2=1
for i in `cat file1.csv`
do
echo $i > $count.txt
cat $count.txt | tr "," "\n" > $count_2.txt
count=`expr $count + 1`
count_2=`expr $count_2 + 1`
done;
#this code will create separte files for each line in file1.csv,
bash file3_search.sh
##########################
file3_search.sh
================
##!/bin/bash
cat file2.csv | sed '/^$/d' | sed 's/[ ]*$//' > trim.txt
dos2unix -q 1.txt 1.txt
dos2unix 2.txt 2.txt
dos2unix 3.txt 3.txt
echo "1st Combination results"
for i in `cat 1.txt`
do
cat trim.txt | egrep -w $i
done > Combination1.txt;
echo "2nd Combination results"
for i in `cat 2.txt`
do
cat trim.txt | egrep -w $i
done > Combination2.txt;
echo "3rd Combination results"
for i in `cat 3.txt`
do
cat trim.txt | egrep -w $i
done > Combination3.txt;
Guys I am not good at programming (I am software tester) please someone can re-factor my code and also please tell me how to get the common word in those Combination.txt file
IMHO it works:
for line in $(cat 1.csv) ; do
echo $line ;
grepline=`echo $line | sed 's/ \+//g;s/,/,|/g;s/^\(.*\)$/^(\1,)/'`;
egrep $grepline 2.csv
egrep $grepline 2.csv | \
awk -F "," '
{ for (i=2;i<=NF;i++)
{s[$i]+=1}
}
END { for (key in s)
{if (s[key]==NR) { tp+=key "," }
}
if (tp!="") {print "Common word(s): " gensub(/,$/,"","g",tp)}
else {print "Common word: None"}}'
echo
done
HTH
Here's an answer for you. It depends on associative array capabilities of bash version 4:
IFS=,
declare -a words
# read and store the words in file2
while read line; do
set -- $line
n=$1
shift
words[$n]="$*"
done < file2.csv
# read file1 and process
while read line; do
echo "$line"
set -- $line
indexes=( "$#" )
NF=${#indexes[#]}
declare -A common
for (( i=0; i<$NF; i++)); do
echo "${words[${indexes[$i]}]}"
set -- ${words[${indexes[$i]}]}
for word; do
common[$word]=$(( ${common[$word]} + 1))
done
done
printf "Common words: "
n=0
for word in "${!common[#]}"; do
if [[ ${common[$word]} -eq $NF ]]; then
printf "%s " $word
(( n++ ))
fi
done
[[ $n -eq 0 ]] && printf "None"
unset common
printf "\n\n"
done < file1.csv

Resources