The following scenario is for pattern search using UNIX Shell where the pattern between two strings need to happen and then a new column with sequence need to happen
Input Data
1|AB|1|2
2|BC|1|2
ID CLOSED
3|AB|1|2
4|BC|1|2
ID CLOSED
Query
As per the data above, we need to add SEQ column after UN and it should add
seq 1 as the first value and sequence 2 to the second part and so on till End.
Expected Output
1|AB|1|2|1
2|BC|1|2|1
3|AB|1|2|2
4|BC|1|2|2
Tried solution as first part but isn't giving correct output
sed -n '/^ID/,/^ID CLOSED/{p;/^pattern2/q}'
Any particular reason you want to use sed for this? It seems like a better fit for awk:
awk -v{,O}FS='|' '
BEGIN { seq = 1 }
/CLOSED/ { seq++ }
!/^ID/ { $5=seq; print }'
Output:
1|AB|1|2|1
2|BC|1|2|1
3|AB|1|2|2
4|BC|1|2|2
Maybe something like this:
(
seq=1
echo "ID NAME ID1 ID2 ID3 UN SEQ"
while read id name id1 id2 id3 un; do
[ "$id $name" = "ID NAME" ] && continue
[ "$id $name" = "ID CLOSED" ] && { let "seq+=1"; continue; }
echo "$id $name $id1 $id2 $id3 $un $seq"
done < /path/to/the/datafile
echo "ID CLOSED"
) | column -t -s' '
Doing this with just a sed instruction is not impossible I think, but a way much harder ;)
Related
An possible way to use while read is:
while read server application date; do ..
So now i could print only the applications, i understand that. So here comes my question:
with my example i know exactly how many "arrays" there are but how would i do it if i dont know how many "arrays" exist per line?
Example file:
Server : ID1 ; ID2 ; ID3
Server : ID1
Server : ID1 ; ID2
Server : ID1 ; ID2 ; ID3 ; ID4
It doesnt have to be with read but how could i read them so i could for example
echo "$Server $ID3"
p.s sorry for the bad english
so what i am doing so far is this:
#!/bin/bash
file=$1
csv=$2
echo Server : Applikation : AO : BV : SO > endlist.txt
while read server aid; do
grep $aid $csv | while IFS=";" read id aid2 name alia status typ beschreibung gesch gesch2 finanzierung internet service servicemodell AO BV SO it betrieb hersteller; do
if [[ $aid == $aid2 ]]
then
echo $server : $name : $AO : $BV : $SO >> endlist.txt
fi
done
done < $file
the Problem is that the first while read is for now only SERVER and AID but i want to edit this file so more than one AID is possible
It doesnt have to be with read but how could i read them so i could for example
echo "$Server $ID3"
First split the input on :, then read the array on ;. Use bash arrays and read -a to save input to an array.
# split the input on `:` and spaces
while IFS=' :' read -r server temp_ids; do
# split the ids on `;` and spaces into an array
IFS=' ;' read -r -a id <<<"$temp_ids"
# check if there are at least 3 elements
if ((${#id[#]} >= 3)); then
# array numbering starts from 0
echo "$server ${id[2]}"
else
echo There is no 3rd element...
fi
done <<EOF
Server : ID1 ; ID2 ; ID3
Server : ID1
Server : ID1 ; ID2
Server : ID1 ; ID2 ; ID3 ; ID4
EOF
will output:
Server ID3
There is no 3rd element...
There is no 3rd element...
Server ID3
112345D000000000000129
123456D000000000000129
112345C000000000000129
123456C000000000000129
123456C000000000000126
position 2-6 is the account number
position 7-22 is the debit or credit value based on D or C in 7th position
want to sum the credit and debit value on the per account basis
tried
awk '{array[substr($0,7,1)]+=substr($0,8,15)+0} END{for(i in array){print array[i]}}')"
but since the file is huge its taking more time is there a way we can find out this one more faster
MVCE
fileA contains the account number + other info
fileB contains the exampe above with debit credit
typeset -i stbal2
typeset -i endbal2
DONE=false
until $DONE; do
read s || DONE=true
accountnumber=${s:1:10} //account number
endbal=${s:26:1} //contain + or - sign
endbal1=${s:11:15} //balance
endbal2=$endbal1 //strip of leading zeros
endbal3=$endbal$endbal2 //concatenate the sign with balance
//similar process as above to get the start balance
stbal=${s:42:1}
stbal1=${s:27:15}
stbal2=$stbal1
stbal3=$stbal$stbal2
creditdebit="$(grep "${bban}" ${fileB} | awk '{array[substr($0,7,1)]+=substr($0,8,15)+0} END{for(i in array){print array[i]}}')"
set -- $creditdebit
... further logic
done < ${fileA}
Without a complete MCVE its a guess but this might be what you're looking for, using GNU awk for true 2D arrays:
$ awk '
{ tots[substr($0,7,1)][substr($0,2,5)] += substr($0,8) }
END {
for (type in tots) {
for (id in tots[type]) {
print type, id, tots[type][id]+0
}
}
}
' file
C 12345 129
C 23456 255
D 12345 129
D 23456 129
in ksh you might do something like that : (myfile.txt being your information file)
#!/bin/ksh
typeset -A Ledger
typeset -i amount
typeset -L10 Col1
typeset -L10 Col2
while read line
do
account=${line:1:5}
action=${line:6:1}
amount=${line:7:21}
if [[ $action == "C" ]]; then
Ledger[$account]=$(( ${Ledger[$account]} + $amount ))
elif [[ $action == "D" ]]; then
Ledger[$account]=$(( ${Ledger[$account]} - $amount ))
fi
done < myfile.txt
Col1="Account"
Col2="Balance"
print "$Col1$Col2\n"
for i in ${!Ledger[#]}; do
Col1=$i
Col2=${Ledger[$i]}
print "$Col1$Col2"
done
With your example my output is :
Account Balance
12345 0
23456 126
Hope it could help
Goal: I'm attempting to create an interactive version of docker ps. Basically, have each line be a "menu" such that a user can: start, stop, ssh, etc.
Example:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1. bf4a9c7de6bf app_1 "docker-php-entryp..." 7 days ago Up About an hour 443/tcp, 0.0.0.0:80->80/tcp, 9000/tcp app_1
2. 26195f0764ce app_2 "sh /var/www/html/..." 10 days ago Up About an hour 443/tcp, 127.0.0.1:8000->80/tcp app_2
Upon choosing (1/2, etc) there will be an options menu to perform various actions on the selected container.
Problem: I can't seem to figure out how to parse out each line of the docker ps command such that i'll have the Container ID and other values as array elements.
The code so far:
list=`docker ps`
IFS=$'\n' array=($list)
for index in ${!array[#]}
do
declare -a 'a=('"${array[index]}"')'
printf "%s\n" "${a[#]}"
done
The result:
CONTAINER
ID
IMAGE
COMMAND
CREATED
STATUS
PORTS
NAMES
/usr/bin/dockersh: array assign: line 9: syntax error near unexpected token `>'
/usr/bin/dockersh: array assign: line 9: `bf4a9c7de6bf app_1 "docker-php-entryp..." 7 days ago Up About an hour 443/tcp, 0.0.0.0:80->80/tcp, 9000/tcp app_1'
It looks like you've got a few issues with the quoting, maybe try:
list=$(docker ps)
IFS=$'\n' array=($list)
for index in "${!array[#]}"
do
declare -a a=("${array[index]}")
printf "%s\n" "${a[#]}"
done
Without proper quoting your string will be likely by re-split; consider checking your shell scripts # shell-check.net, as it usually will give you some good hints regarding bad syntax.
If you want to have an associative array that features a matrix with all your docker ps field accessible in row/column, you can use awk to insert separator | between fields. Then export the result in a single associative array and build the matrix according to the number of column you expect (eg 7) :
#!/bin/bash
IFS=$'|'
data=$(docker ps -a | awk '
function rtrim(s) { sub(/[ \t\r\n]+$/, "", s); return s }
{
if (NR == 1) {
head[1] = index($0,"CONTAINER ID")
head[2] = image=index($0,"IMAGE")
head[3] = command=index($0,"COMMAND")
head[4] = created=index($0,"CREATED")
head[5] = status=index($0,"STATUS")
head[6] = ports=index($0,"PORTS")
head[7] = names=index($0,"NAMES")
}
else{
for (i = 1;i < 8;i++) {
if (i!=7){
printf "%s",rtrim(substr($0, head[i], head[i+1] - 1 - head[i])) "|"
}
else{
printf "%s",rtrim(substr($0, head[i], 100)) "|"
}
}
print ""
}
}')
arr=($data)
max_column=7
row=0
column=0
declare -A matrix
for index in "${!arr[#]}"
do
matrix[$row,$column]=$(echo "${arr[index]}" | tr -d '\n')
column=$((column+1))
if [ $((column%max_column)) == 0 ]; then
row=$((row+1))
column=0
fi
done
echo "first container ID is : ${matrix[0,0]}"
echo "second container ID is : ${matrix[1,0]}"
echo "third container NAME is : ${matrix[2,6]}"
In the awk part, the aim is to insert a | character between each field for the data to be injected into an associative array with the | delimiter
As field content is aligned with field title, we store the index of each field names in head array and extract each field trimming according to the next field position
Then the matrix is build according to the max column count (7). Then each row/column can be accessed easily with ${matrix[row,column]}
Usual story ... don't read data with a for loop unless you know exactly the format and how to control it:
while IFS="\n" read -r line
do
array+=("$line")
done< <(docker ps)
Personally I would try and remove the numbers from the start of the lines (1., 2., etc) because then you can throw it into a select and it will give you numbers which can then be used to reference the relevant items.
I have a simple search script, where based on user's options it will search in certain column of a file.
The file looks similar to passwd
openvpn:x:990:986:OpenVPN:/etc/openvpn:/sbin/nologin
chrony:x:989:984::/var/lib/chrony:/sbin/nologin
rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
radvd:x:75:75:radvd user:/:/sbin/nologin
now the function based on user's option will search in different columns of the file. For example
-1 "searchedword" -2 "secondword"
will search in the first column for "searchedword" and in the second column for "secondword"
The function looks like this:
while [ $# -gt 0 ]; do
case "$1" in
-1|--one)
c=1
;;
-2|--two)
c=2
;;
-3|--three)
c=3
;;
...
esac
In the c variable is the number of the column where I want to search.
cat data | if [ "$( cut -f $c -d ':' )" == "$2" ]; then cut -d: -f 1-7 >> result; fi
Now I have something like this, where I try to select the right column and compare it to the second option, which is in this case "searchedword" and then copy the whole column into the result file. But it doesn't work. It doesn't copy anything into the result file.
Does anyone know where is the problem?
Thanks for answers
(At the end of the script I use:
shift
shift
to get the next two options)
I suggest using awk for this task as awk is better tool for processing delimited columns and rows.
Consider this awk command where we pass search column numbers their corresponding search values in 2 different strings cols and vals to awk command:
awk -v cols='1:3' -v vals='rpcuser:29' 'BEGIN {
FS=OFS=":" # set input/output field separator as :
nc = split(cols, c, /:/) # split column # by :
split(vals, v, /:/) # split values by :
}
{
p=1 # initialize p as 1
for(i=1; i<=nc; i++) # iterate the search cols/vals and set p=0
if ($c[i] !~ v[i]) { # if any match fails
p=0
break
} # finally value of p decides if a row is printing or not
} p' file
Output:
rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
I need advice on how to achieve this output:
myoutputfile.txt
Tom Hagen 1892
State: Canada
Hank Moody 1555
State: Cuba
J.Lo 156
State: France
output of mycommand:
/usr/bin/mycommand
Tom Hagen
1892
Canada
Hank Moody
1555
Cuba
J.Lo
156
France
Im trying to achieve with this shell script:
IFS=$'\r\n' GLOBIGNORE='*' :; names=( $(/usr/bin/mycommand) )
for name in ${names[#]}
do
#echo $name
echo ${name[0]}
#echo ${name:0}
done
Thanks
Assuming you can always rely on the command to output groups of 3 lines, one option might be
/usr/bin/mycommand |
while read name;
read year;
read state; do
echo "$name $year"
echo "State: $state"
done
An array isn't really necessary here.
One improvement could be to exit the loop if you don't get all three required lines:
while read name && read year && read state; do
# Guaranteed that name, year, and state are all set
...
done
An easy one-liner (not tuned for performance):
/usr/bin/mycommand | xargs -d '\n' -L3 printf "%s %s\nState: %s\n"
It reads 3 lines at a time from the pipe and then passes them to a new instance of printf which is used to format the output.
If you have whitespace at the beginning (it looks like that in your example output), you may need to use something like this:
/usr/bin/mycommand | sed -e 's/^\s*//g' | xargs -d '\n' -L3 printf "%s %s\nState: %s\n"
#!/bin/bash
COUNTER=0
/usr/bin/mycommand | while read LINE
do
if [ $COUNTER = 0 ]; then
NAME="$LINE"
COUNTER=$(($COUNTER + 1))
elif [ $COUNTER = 1 ]; then
YEAR="$LINE"
COUNTER=$(($COUNTER + 1))
elif [ $COUNTER = 2 ]; then
STATE="$LINE"
COUNTER=0
echo "$NAME $YEAR"
echo "State: $STATE"
fi
done
chepner's pure bash solution is simple and elegant, but slow with large input files (loops in bash are slow).
Michael Jaros' solution is even simpler, if you have GNU xargs (verify with xargs --version), but also does not perform well with large input files (external utility printf is called once for every 3 input lines).
If performance matters, try the following awk solution:
/usr/bin/mycommand | awk '
{ ORS = (NR % 3 == 1 ? " " : "\n")
gsub("^[[:blank:]]+|[[:blank:]]*\r?$", "") }
{ print (NR % 3 == 0 ? "State: " : "") $0 }
' > myoutputfile.txt
NR % 3 returns the 0-based index of each input line within its respective group of consecutive 3 lines; returns 1 for the 1st line, 2 for the 2nd, and 0(!) for the 3rd.
{ ORS = (NR % 3 == 1 ? " " : "\n") determines ORS, the output-record separator, based on that index: a space for line 1, and a newline for lines 2 and 3; the space ensures that line 2 is appended to line 1 with a space when using print.
gsub("^[[:blank:]]+|[[:blank:]]*\r?$", "") strips leading and trailing whitespace from the line - including, if present, a trailing \r, which your input seems to have.
{ print (NR % 3 == 0 ? "State: " : "") $0 } prints the trimmed input line, prefixed by "State: " only for every 3rd input line, and implicitly followed by ORS (due to use of print).