Can we use begin and end with condition? - bash
I have a CSV file which has multiple columns. What I am trying to achieve here is to print only the details if column 5<=25. If not then only one single line with string "No certificate will expire in 25 days".
Expectation is to print the column 3 and 6 along with Error_begin and Error_end string if condition matches.
If not then only need to print date - info - "somestring" in one line only.
Output should be in same log file for both.
Command I am using:
awk -F ',' -v date="$(date +'%Y-%m-%d')" 'BEGIN {if ($5<=25) {print date,"ERROR----"}else { print date,"INFO-- No certificate will expire in 25 days" }} {if ($
5<=25) {print $3,$6}} END {if ($5<=25) {print "ERROR_END"}}' /tmp/cert_details.csv
BEGIN is executed before processing lines, so accesing any field is equivalent of accessing unset variable, therefore it same as using 0 in numeric comparison, so this piece
BEGIN {if ($5<=25) {print date,"ERROR----"}else { print date,"INFO-- No certificate will expire in 25 days" }}
will behave like
BEGIN {if (0<=25) {print date,"ERROR----"}else { print date,"INFO-- No certificate will expire in 25 days" }}
which I suppose is not what you desire.
END is executed after all lines and here accessing field pertains to last line that is
END {if ($5<=25) {print "ERROR_END"}}
is dependant solely on last line of your file, which I suppose again is not what you desire.
Note that if you want to make first line of output depend on something which need processing all lines of file, you need to hold print until your line processing do commence, consider simple example, let say you want to print number of lines which have 5 or more character then print said lines and file.txt content is
Able
Baker
Charlie
then possible solution is
awk 'length>=5{cnt+=1;lines=lines "\n" $0}END{printf "Found %d lines:%s", cnt, lines}' file.txt
output
Found 2 lines:
Baker
Charlie
Observe that you must not print line as you go, as until reach end you do not know number to be put in first line of output, thus I do store lines of output to be printed, sheared by \n (newlines) and do printf inside END.
tested in gawk 4.2.1
Related
awk to get first column if the a specific number in the line is greater than a digit
I have a data file (file.txt) contains the below lines:
123 pro=tegs, ETA=12:00, team=xyz,user1=tom,dom=dby.com
345 pro=rbs, team=abc,user1=chan,dom=sbc.int,ETA=23:00
456 team=efg, pro=bvy,ETA=22:00,dom=sss.co.uk,user2=lis
I'm expecting to get the first column ($1) only if the ETA= number is greater than 15, like here I will have 2nd and 3rd line first column only is expected.
345
456
I tried like cat file.txt | awk -F [,TPF=]' '{print $1}' but its print whole line which has ETA at the end.
Using awk
$ awk -F"[=, ]" '{for (i=1;i<NF;i++) if ($i=="ETA") if ($(i+1) > 15) print $1}' input_file
345
456
With your shown samples please try following GNU awk code. Using match function of GNU awk where I am using regex (^[0-9]+).*ETA=([0-9]+):[0-9]+ which creates 2 capturing groups and saves its values into array arr. Then checking condition if 2nd element of arr is greater than 15 then print 1st value of arr array as per requirement.
awk '
match($0,/(^[0-9]+).*\<ETA=([0-9]+):[0-9]+/,arr) && arr[2]+0>15{
print arr[1]
}
' Input_file
I would harness GNU AWK for this task following way, let file.txt content be
123 pro=tegs, ETA=12:00, team=xyz,user1=tom,dom=dby.com
345 pro=rbs, team=abc,user1=chan,dom=sbc.int,ETA=23:00
456 team=efg, pro=bvy,ETA=02:00,dom=sss.co.uk,user2=lis
then
awk 'substr($0,index($0,"ETA=")+4,2)+0>15{print $1}' file.txt
gives output
345
Explanation: I use String functions, index to find where is ETA= then substr to get 2 characters after ETA=, 4 is used as ETA= is 4 characters long and index gives start position, I use +0 to convert to integer then compare it with 15. Disclaimer: this solution assumes every row has ETA= followed by exactly 2 digits.
(tested in GNU Awk 5.0.1)
Whenever input contains tag=value pairs as yours does, it's best to first create an array of those mappings (v[]) below and then you can just access the values by their tags (names):
$ cat tst.awk
BEGIN {
FS = "[, =]+"
OFS = ","
}
{
delete v
for ( i=2; i<NF; i+=2 ) {
v[$i] = $(i+1)
}
}
v["ETA"]+0 > 15 {
print $1
}
$ awk -f tst.awk file
345
456
With that approach you can trivially enhance the script in future to access whatever values you like by their names, test them in whatever combinations you like, output them in whatever order you like, etc. For example:
$ cat tst.awk
BEGIN {
FS = "[, =]+"
OFS = ","
}
{
delete v
for ( i=2; i<NF; i+=2 ) {
v[$i] = $(i+1)
}
}
(v["pro"] ~ /b/) && (v["ETA"]+0 > 15) {
print $1, v["team"], v["dom"]
}
$ awk -f tst.awk file
345,abc,sbc.int
456,efg,sss.co.uk
Think about how you'd enhance any other solution to do the above or anything remotely similar.
It's unclear why you think your attempt would do anything of the sort. Your attempt uses a completely different field separator and does not compare anything against the number 15.
You'll also want to get rid of the useless use of cat.
When you specify a column separator with -F that changes what the first column $1 actually means; it is then everything before the first occurrence of the separator. Probably separately split the line to obtain the first column, space-separated.
awk -F 'ETA=' '$2 > 15 { split($0, n, /[ \t]+/); print n[1] }' file.txt
The value in $2 will be the data after the first separator (and up until the next one) but using it in a numeric comparison simply ignores any non-numeric text after the number at the beginning of the field. So for example, on the first line, we are actually literally checking if 12:00, team=xyz,user1=tom,dom=dby.com is larger than 15 but it effectively checks if 12 is larger than 15 (which is obviously false).
When the condition is true, we split the original line $0 into the array n on sequences of whitespace, and then print the first element of this array.
Using awk you could match ETA= followed by 1 or more digits. Then get the match without the ETA= part and check if the number is greater than 15 and print the first field.
awk '/^[0-9]/ && match($0, /ETA=[0-9]+/) {
if(substr($0, RSTART+4, RLENGTH-4)+0 > 15) print $1
}' file
Output
345
456
If the first field should start with a number:
awk '/^[0-9]/ && match($0, /ETA=[0-9]+/) {
if(substr($0, RSTART+4, RLENGTH-4) > 15)+0 print $1
}' file
How to select two specific lines with awk?
/!\ The question is basically solved, see my own answer below for more details and a subsidiary question /!\ I'm trying to add two lines based on specific word, but all I could find is adding everything after some pattern: How to select lines between two marker patterns which may occur multiple times with awk/sed Which is not what I'm looking after. Consider the following output: aji 1 bsk 2 cmq 3 doh 4 enr 5 fwp 6 gzx 7 What I'm trying to get is something like cmq + fwp, which output should be: 9 I do know how to add values, but I'm missing the select line containing cmq, then select line containing fwp part. So, is there a way awk could strictly select two specific lines independently (then add them) ? Edit: As far as I know, matching words is awk '/cmq/', but I need to do that for let's say "fwp" too so I can add them.
$ awk '$1 ~ /^(cmq|fwp)$/{sum+=$2} END { print sum}' infile
Explanation:
awk '$1 ~ /^(cmq|fwp)$/{ # look for the match in first field
sum+=$2 # sum up 2nd field ($2) value,where sum is variable
}
END{ # at the end
print sum # print variable sum
}' infile
Test Results:
$ cat infile
aji 1
bsk 2
cmq 3
doh 4
enr 5
fwp 6
gzx 7
$ awk '$1 ~ /^(cmq|fwp)$/{sum+=$2} END { print sum}' infile
9
Now, for a more generic way this time -which even works for subtracting-:
awk '/cmq/{x=$2} /fwp/{y=$2} END {print x+y}'
Where:
awk ' # Invoking awk and its instructions
/cmq/{x=$2} # Select line with "cmq", then set its value to x. Both must be tied
/fwp/{y=$2} # Select line with "fwp", then set its value to y. Both must be tied
END # Ends pattern matching/creation
{print x+y} # Print the calculated result
' # Ending awk's instructions
Unfortuanately, two variables are used (x and y).
So, I'm still interested on finding how to do it without any variable, or only one at the very most.
I do have a single-variable way for summing:
awk '/cmq|fwp/ {x+=$2} END {print x}'
But doing this for subtracting:
awk '/cmq|fwp/ {x-=$2} END {print x}'
doesn't work.
As an subsidiary question, anyone knows to achieve such subtracting without or with only one variable ?
Eliminate useless repeats of values from CSV for line charting
Given a CSV file with contents similar to this: 2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:01,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:01,GRP3,GRP3,TOTAL,0 2017-05-01,00:00:11,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:11,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:11,GRP3,GRP3,TOTAL,0 2017-05-01,00:00:21,GRP1,GRP1,TOTAL,3 What is the best way using bash or awk scripting to tidy it up and remove all useless zeros. By useless I mean: this data will be used for line charts in web pages. However reading the entire CSV file in the web browser via JavaScript/jQuery etc is very slow. It would be more efficient to eliminate the useless zeros prior to uploading the file. If I remove all the zeros, the lines all more or less show peak to peak to peak instead of real lines from zero to some larger value back to zero, followed by a space until the next value greater than zero. As you see there are 3 groups in the list of data. Any time there are 3 in a row for example for GRP1, I'd like to remove the middle or 2nd 0 in that list. In reality, this could work for values greater than zero also...if the same values were found every 10 seconds for say 10 in a row... it would be good to leave both ends in place and remove items 2 through 9. The line chart would look the same, but the data would be much smaller to deal with. Ideally I could do this with a shell script on disk prior to reading the input file. So (just looking at GRP1) instead of: 2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:11,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:21,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:31,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:41,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:51,GRP1,GRP1,TOTAL,3 2017-05-01,00:01:01,GRP1,GRP1,TOTAL,2 The script would eliminate all useless 3 values...and leave only: 2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:51,GRP1,GRP1,TOTAL,3 2017-05-01,00:01:01,GRP1,GRP1,TOTAL,2 Or... Another Expected Result using 0 this time...instead of 3 as the common consecutive value for GRP2... 2017-05-01,00:00:01,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:11,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:21,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:31,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:41,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:51,GRP2,GRP2,TOTAL,0 2017-05-01,00:01:01,GRP2,GRP2,TOTAL,2 The script would eliminate all useless 0 values...and leave only: 2017-05-01,00:00:01,GRP2,GRP2,TOTAL,0 2017-05-01,00:00:51,GRP2,GRP2,TOTAL,0 2017-05-01,00:01:01,GRP2,GRP2,TOTAL,2 #karakfa answer gets me close but still end up with portions similar to this after applying awk to one unique group and then eliminating some duplicates that also showed up for some reason: I like it but it still ends up with this: 2017-05-02,00:05:51,DTE,DTE,TOTAL,2 2017-05-02,00:06:01,DTE,DTE,TOTAL,1 2017-05-02,00:06:51,DTE,DTE,TOTAL,1 2017-05-02,00:07:01,DTE,DTE,TOTAL,1 2017-05-02,00:07:51,DTE,DTE,TOTAL,1 2017-05-02,00:08:01,DTE,DTE,TOTAL,1 2017-05-02,00:08:51,DTE,DTE,TOTAL,1 2017-05-02,00:09:01,DTE,DTE,TOTAL,1 2017-05-02,00:09:51,DTE,DTE,TOTAL,1 2017-05-02,00:10:01,DTE,DTE,TOTAL,1 2017-05-02,00:10:51,DTE,DTE,TOTAL,1 2017-05-02,00:11:01,DTE,DTE,TOTAL,1 2017-05-02,00:11:51,DTE,DTE,TOTAL,1 2017-05-02,00:12:01,DTE,DTE,TOTAL,1 2017-05-02,00:12:51,DTE,DTE,TOTAL,1 2017-05-02,00:13:01,DTE,DTE,TOTAL,1 2017-05-02,00:13:51,DTE,DTE,TOTAL,1 2017-05-02,00:14:01,DTE,DTE,TOTAL,1 2017-05-02,00:14:51,DTE,DTE,TOTAL,1 2017-05-02,00:15:01,DTE,DTE,TOTAL,1 2017-05-02,00:15:11,DTE,DTE,TOTAL,1 2017-05-02,00:15:21,DTE,DTE,TOTAL,9 Would be wonderful to get to this instead: 2017-05-02,00:05:51,DTE,DTE,TOTAL,2 2017-05-02,00:06:01,DTE,DTE,TOTAL,1 2017-05-02,00:15:11,DTE,DTE,TOTAL,1 2017-05-02,00:15:21,DTE,DTE,TOTAL,9
That's one ill-placed question but I'll take a crack at the title, if you don't mind:
$ awk -F, ' {
if($3 OFS $4 OFS $6 in first)
last[$3 OFS $4 OFS $6]=$0
else
first[$3 OFS $4 OFS $6]=$0 }
END {
for(i in first) {
print first[i]
if(i in last)
print last[i] }
}' file
2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3
2017-05-01,00:00:21,GRP1,GRP1,TOTAL,3
2017-05-01,00:00:01,GRP2,GRP2,TOTAL,0
2017-05-01,00:00:11,GRP2,GRP2,TOTAL,0
2017-05-01,00:00:01,GRP3,GRP3,TOTAL,0
2017-05-01,00:00:11,GRP3,GRP3,TOTAL,0
Basically it keeps the first and last (if exists) occurrence of each unique combination of 3rd, 4th and 6th field.
Edit: In the new light of the word consecutive, how about this awful hack:
$ awk -F, '
(p!=$3 OFS $4 OFS $6) {
if(NR>1 && lp<(NR-1))
print q
print $0
lp=NR }
{
p=$3 OFS $4 OFS $6
q=$0 }
' file
2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3
2017-05-01,00:00:01,GRP2,GRP2,TOTAL,0
2017-05-01,00:00:01,GRP3,GRP3,TOTAL,0
2017-05-01,00:00:11,GRP1,GRP1,TOTAL,3
2017-05-01,00:00:11,GRP2,GRP2,TOTAL,0
2017-05-01,00:00:11,GRP3,GRP3,TOTAL,0
2017-05-01,00:00:21,GRP1,GRP1,TOTAL,3
and output for the second data:
2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3
2017-05-01,00:00:51,GRP1,GRP1,TOTAL,3
2017-05-01,00:01:01,GRP1,GRP1,TOTAL,2
and the third:
2017-05-01,00:00:01,GRP2,GRP2,TOTAL,0
2017-05-01,00:00:51,GRP2,GRP2,TOTAL,0
2017-05-01,00:01:01,GRP2,GRP2,TOTAL,2
Simple awk approach: awk -F, '$NF!=0' inputfile The output: 2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:11,GRP1,GRP1,TOTAL,3 2017-05-01,00:00:21,GRP1,GRP1,TOTAL,3 $NF!=0 - takes into account only those lines which don't have 0 as their last field value
awk to the rescue!
$ awk -F'[,:]' '$4==pt+10 && $NF==p {pt=$4; pl=$0; next}
pl {print pl}
{pt=$4;p=$NF}1' file
2017-05-01,00:00:01,GRP1,GRP1,TOTAL,3
2017-05-01,00:00:51,GRP1,GRP1,TOTAL,3
2017-05-01,00:01:01,GRP1,GRP1,TOTAL,2
How to set FS to eof?
I want to read whole file not per lines. How to change field separator to eof-symbol? I do: awk "^[0-9]+∆DD$1[PS].*$" $(ls -tr) $1 - param (some integer), .* - message that I want to print. There is a problem: message can contains \n. In that way this code prints only first line of file. How can I scan whole file not per lines? Can I do this using awk, sed, grep? Script must have length <= 60 characters (include spaces).
Assuming you mean record separator, not field separator, with GNU awk you'd do:
gawk -v RS='^$' '{ print "<" $0 ">" }' file
Replace the print with whatever you really want to do and update your question with some sample input and expected output if you want help with that part too.
The portable way to do this, by the way, is to build up the record line by line and then process it in the END section:
awk '{rec = rec (NR>1?RS:"") $0} END{ print "<" rec ">" }' file
using nf = split(rec,flds) to create fields if necessary.
Extracting field from last row of given table using sed
I would like to write a bash script to extract a field in the last row of a table. I will illustrate by example. I have a text file containing tables with space delimited fields like ... Table 1 (foobar) num flag name comments 1 ON Frank this guy is frank 2 OFF Sarah she is tall 3 ON Ahmed who knows him Table 2 (foobar) num flag name comments 1 ON Mike he is short 2 OFF Ahmed his name is listed twice I want to extract the first field in the last row of Table1, which is 3. Ideally I would like to be able to use any given table's title to do this. There are guaranteed carriage returns between each table. What would be the best way to accomplish this, preferably using sed and grep?
Awk is perfect for this, print the first field in the last row for each record:
$ awk '!$1{print a}{a=$1}END{print a}' file
3
2
Just from the first record:
$ awk '!$1{print a;exit}{a=$1}' file
3
Edit:
For a given table title:
$ awk -v t="Table 1" '$0~t{f=1}!$1&&f{print a;f=0}{a=$1}END{if (f) print a}' file
3
$ awk -v t="Table 2" '$0~t{f=1}!$1&&f{print a;f=0}{a=$1}END{if (f) print a}' file
2
This sed line seems to work for your sample.
table='Table 2'
sed -n "/$table"'/{n;n;:next;h;n;/^$/b last;$b last;b next;:last;g;s/^\s*\(\S*\).*/\1/p;}' file
Explanation: When we find a line matching the table name in $table, we skip that line, and the next (the field labels). Starting at :next we push the current line into the hold space, get the next line and see if it is blank or the end of the file, if not we go back to :next, push the current line into hold and get another. If it is blank or EOF, we skip to :last, pull the hold space (the last line of the table) into pattern space, chop out all but the first field and print it.
Just read each block as a record with each line as a field and then print the first sub-field of the last field of whichever record you care about:
$ awk -v RS= -F'\n' '/^Table 1/{split($NF,a," "); print a[1]}' file
3
$ awk -v RS= -F'\n' '/^Table 2/{split($NF,a," "); print a[1]}' file
2
Better tool to that is awk!
Here is a kind legible code:
awk '{
if(NR==1) {
row=$0;
next;
}
if($0=="") {
$0=row;
print $1;
} else {
row=$0;
}
} END {
if(row!="") {
$0=row;
print $1;
}
}' input.txt