I just started learning about awk programming and am still getting used to it in the bash terminal. If i were to write an expression to match strings that start with de and end with ed, i was wondering how does it go about?
Was thinking of something like:
echo -e "deed\ndeath\ndone\ndeindustrialized" |awk '/^de.ed$/'
where i match the start and match the end in the awk command but it doesn't print out anything. I'll appreciate some help.
It should produce:
deed
deindustrialized
I just started today and would like to know.
The awk part should be:
... | awk '/^de.*ed$/'
deed
deindustrialized
. matches any character and * means that the preceding item will be matched zero or more times.
try with awk:
echo -e "deed\ndeath\ndone\ndeindustrialized" | awk 'NR==1;END{print}'
Following is the explanation too on same.
awk '
NR==1; ###Checking the NR(Number of line) value is 1, if yes then print the current line(awk works on method of pattern/action, if a condition is TRUE then do actions, in case of NO action do default action which is print of current line).
END{print}' ###In END section now, so it will print the last line of Input_file.
Related
I have a string:
Log for: squid.log.2017.11.13
I need to cut out squid.log. so that I see:
Log for: 2017.11.13
I tried to cut
echo "Log for: squid.log.2017.11.13" | cut -d'.' -f3-5
But I ended up with:
2017.11.13
How can I get the result I want?
You can use sed to cut the unwanted part:
echo "Log for: squid.log.2017.11.13" | sed 's/squid\.log\.//'
Use sed to remove the part you don't want:
echo "Log for: squid.log.2017.11.13" | sed 's/squid\.log\.//'
awk to the rescue! a non-standard approach to break the monotony...
define the to be removed text as field separator and parse and print the input line.
$ echo Log for: squid.log.2017.11.13 | awk -F' squid\\.log\\.' '{$1=$1}1'
Log for: 2017.11.13
This solution is a bit more reusable than the previous ones offered:
awk '/^Log/{ split($3,x,"."); print $1" "$2" "x[length(x)-2]"."x[length(x)-1]"."x[length(x)] };'
This looks for all lines starting with Log, then grabs the 3rd column which contains squid.log.2017.11.13 and utilizes the the split built-in to break up the string in to array x using the . as the delimiter. Once we have our array x, we know that the last 3 values will always be the date, and this will work regardless of the rest of the string, (even if squid.log was something different) - we can use the length built-in to make sure we only get the last three elements.
Then we just print our reformatted string print $1" "$2" "x[length(x)-2]"."x[length(x)-1]"."x[length(x)] - reinserting the .'s in the appropriate places since they were stripped by using them as the split delimiter.
Output:
Log for: 2017.11.13
I have a tab deliminated file which I want to add "$" end of each variable, Can I do that with awk,sed or anything else?
Example
input:
a seq1 anot1
b seq2 anot2
c seq3 anot3
d seq4 anot4
I neet to have this:
output:
a$ seq1$ anot1$
b$ seq2$ anot2$
c$ seq3$ anot3$
d$ seq4$ anot4$
Any answer will be appreciated,
Thanks
In bash alone:
while read line; do echo "${line//$'\t'/\$$'\t'}\$"; done < file
This hackish solution relies on two "special" things -- parameter expansion to do the replacement, and format expansion to allow the tabs to be parsed.
In awk, you can process fields much more safely:
awk -F'\t' 'BEGIN{OFS=FS} {for(n=1;n<=NF;n++){$n=$n "$"}} 1' file
This works by stepping through each line of input and replacing each field with itself plus the dollar sign. The BEGIN block insures that your output will use the same field separators as your input. The 1 at the end is awk short-hand for "print the current line".
late to the party...
another awk solution. Prefix field and record separators with "$"
$ awk -F'\t' 'BEGIN{OFS="$"FS; ORS="$"RS} {$1=$1}1' file
With sed:
sed 's/[^ ]*/&$/g' filename
which replaces any non-space words with the word (&) followed by a $.
Oops! You said tabs. You can replace the above space with "\t" to use tab delimited.
sed 's/[^\t]*/&$/g' filename
Actually, even better, for tabs OR spaces:
sed 's/[^[:blank:]]*/&$/g' filename
awk is your friend :
awk '{for(i=1;i<=NF;i++)sub(/$/,"$",$i);print}' file
or
awk '{for(i=1;i<=NF;i++)sub(/$/,"$",$i);}1' file
Sample Output
a$ seq1$ anot1$
b$ seq2$ anot2$
c$ seq3$ anot3$
d$ seq4$ anot4$
What is happening here?
Using a for-loop we iterate thru all the fields in a record.
We use the awk sub function to replace the end ie (/$/) with a $ ie ("$") for each record ($i).
Use print explicitly to print the record. Numeric 1 also represents the default action that is to print the record.
awk '{gsub(/ /,"$ ")}{print $0 "$\r"}' file
a$ seq1$ anot1$
b$ seq2$ anot2$
c$ seq3$ anot3$
d$ seq4$ anot4$
What happens?
First replace spaces with dollar sign and new space.
Last insert dollar sign before the carriage return.
Good day to everyone.
Could you, please, help me with some of my file preparation problem:
I have a file:
2:1 3:1 4:2 5:1 7:2 34:1 37:3 ...
4:2 6:1 8:1 23:1 25:2 30:1 ...
I would like to get:
20002:1 20003:1 20004:2 20005:1 20007:2 20034:1 20037:3 ...
20004:2 20006:1 20008:1 20023:1 20025:2 20030:1 ...
I tried:
awk '{FS=":"; RS=" "; OFS=":"; ORS=" "}{$1=$1+20000; print $0}'
But it works only partially: it doesnt work with the first line, giving 20002:1:3:1:4:2.., and doesn't work with the first element of each line, giving 4:2 20006:1 20008:1 ...
You can use this (GNU awk only for RT)
awk 'BEGIN{FS=OFS=":";RS="[[:space:]]"}{ORS=RT;$1=$1+20000; print $0}' file
20002:1 20003:1 20004:2 20005:1 20007:2 20034:1 20037:3
20004:2 20006:1 20008:1 20023:1 20025:2 20030:1
Explanation
BEGIN{
#Only run at start of script
FS=OFS=":"
#Set input and output field separator to :
RS="[[:space:]]"
#Set the record separator to any space character e.g `\n` `\t` or ` `
}
{ORS=RT
#Set the output record separator to whatever was captured by the input one, i.e keep newline space or tab in the right places
$1+=20000; print
#Do your math and print, note that `+=` is shorthand for adding to the current value,
#and also that print can be used on it's own as by default it prints $0(you can also use 1
#at the end of the script as this evaluates to true and the default action if no block
#is defined is to print the current line)
}'
In case of not having GNU awk as required by #123's more elegant solution:
$ awk -F"[: ]+" '{for(i=1;i<NF;i+=2){$i+=20000; printf "%s:%s ",$i,$(i+1)} print ""}' cs.txt
20002:1 20003:1 20004:2 20005:1 20007:2 20034:1 20037:3
20004:2 20006:1 20008:1 20023:1 20025:2 20030:1
I am having this example of ini file. I need to extract the names between 2 patterns Name_Z1 and OBJ=Name_Z1 and put them each on a line.
The problem is that there are more than one occurences with Name_Z1 and OBJ=Name_Z1 and i only need first occurence.
[Name_Z5]
random;text
Names;Jesus;Tom;Miguel
random;text
OBJ=Name_Z5
[Name_Z1]
random;text
Names;Jhon;Alex;Smith
random;text
OBJ=Name_Z1
[Name_Z2]
random;text
Names;Chris;Mara;Iordana
random;text
OBJ=Name_Z2
[Name_Z1_Phone]
random;text
Names;Bill;Stan;Mike
random;text
OBJ=Name_Z1_Phone
My desired output would be:
Jhon
Alex
Smith
I am currently writing a more ample script in bash and i am stuck on this. I prefer awk to do the job.
My greatly appreciation for who can help me. Thank you!
For Wintermute solution: The [Name_Z1] part looks like this:
[CAB_Z1]
READ_ONLY=false
FilterAttr=CeaseTime;blank|ObjectOfReference;contains;511047;512044;513008;593026;598326;CL5518;CL5521;CL5538;CL5612;CL5620|PerceivedSeverity;=;Critical;Major;Minor|ProbableCause;!=;HOUSE ALARM;IO DEVICE|ProblemText;contains;AIRE;ALIMENTA;BATER;CONVERTIDOR;DISTRIBUCION;FUEGO;HURTO;MAINS;MALLO;MAYOR;MENOR;PANEL;TEMP
NAME=CAB_Z1
And the [Name_Z1_Phone] part looks like this:
[CAB_Z1_FUEGO]
READ_ONLY=false
FilterAttr=CeaseTime;blank|ObjectOfReference;contains;511047;512044;513008;593026;598326;CL5518;CL5521;CL5538;CL5612;CL5620|PerceivedSeverity;=;Critical;Major;Minor|ProbableCause;!=;HOUSE ALARM;IO DEVICE|ProblemText;contains;FUEGO
NAME=CAB_Z1_FUEGO
The fix should be somewhere around the "|PerceivedSeverity"
Expected Output:
511047
512044
513008
593026
598326
CL5518
CL5521
CL5538
CL5612
CL5620
This should work:
sed -n '/^\[Name_Z1/,/^OBJ=Name_Z1/ { /^Names/ { s/^Names;//; s/;/\n/g; p; q } }' foo.txt
Explanation: Written readably, the code is
/^\[Name_Z1/,/^OBJ=Name_Z1/ {
/^Names/ {
s/^Names;//
s/;/\n/g
p
q
}
}
This means: In the pattern range /^\[Name_Z1/,/^OBJ=Name_Z1/, for all lines that match the pattern /^Names/, remove the Names; in the beginning, then replace all remaining ; with newlines, print the whole thing, and then quit. Since it immediately quits, it will only handle the first such line in the first such pattern range.
EDIT: The update made things a bit more complicated. I suggest
sed -n '/^\[CAB_Z1/,/^NAME=CAB_Z1/ { /^FilterAttr=/ { s/^.*contains;\(.*\)|PerceivedSeverity.*$/\1/; s/;/\n/g; p; q } }' foo.txt
The main difference is that instead of removing ^Names from a line, the substitution
s/^.*contains;\(.*\)|PerceivedSeverity.*$/\1/;
is applied. This isolates the part between contains; and |PerceivedSeverity before continuing as before. It assumes that there is only one such part in the line. If the match is ambiguous, it will pick the one that appears last in the line.
An (g)awk way that doesn't need a set number of fields(although i have assumed that contains; will always be on the line you need the names from.
(g)awk '(x+=/Z1/)&&match($0,/contains;([^|]+)/,a)&&gsub(";","\n",a[1]){print a[1];exit}' f
Explanation
(x+=/Z1/) - Increments x when Z1 is found. Also part of a
condition so x must exist to continue.
match($0,/contains;([^|]+)/,a) - Matches contains; and then captures everything after
up to the |. Stores the capture in a. Again a
condition so must succeed to continue.
gsub(";","\n",a[1]) - Substitutes all the ; for newlines in the capture
group a[1].
{print a[1];exit}' - If all conditions are met then print a[1] and exit.
This way should work in (m)awk
awk '(x+=/Z1/)&&/contains/{split($0,a,"|");y=split(a[2],b,";");for(i=3;i<=y;i++)
print b[i];exit}' file
sed -n '/\[Name_Z1\]/,/OBJ=Name_Z1$/ s/Names;//p' file.txt | tr ';' '\n'
That is sed -n to avoid printing anything not explicitly requested. Start from Name_Z1 and finish at OBJ=Name_Z1. Remove Names; and print the rest of the line where it occurs. Finally, replace semicolons with newlines.
Awk solution would be
$ awk -F";" '/Name_Z1/{f=1} f && /Names/{print $2,$3,$4} /OBJ=Name_Z1/{exit}' OFS="\n" input
Jhon
Alex
Smith
OR
$ awk -F";" '/Name_Z1/{f++} f==1 && /Names/{print $2,$3,$4}' OFS="\n" input
Jhon
Alex
Smith
-F";" sets the field seperator as ;
/Name_Z1/{f++} matches the line with pattern /Name_Z1/ If matched increment {f++}
f==1 && /Names/{print $2,$3,$4} is same as if f == 1 and maches pattern Name with line if true, then print the the columns 2 3 and 4 (delimted by ;)
OFS="\n" sets the output filed seperator as \n new line
EDIT
$ awk -F"[;|]" '/Z1/{f++} f==1 && NF>1{for (i=5; i<15; i++)print $i}' input
511047
512044
513008
593026
598326
CL5518
CL5521
CL5538
CL5612
CL5620
Here is a more generic solution for data in group of blocks.
This awk does not need the end tag, just the start.
awk -vRS= -F"\n" '/^\[Name_Z1\]/ {n=split($3,a,";");for (i=2;i<=n;i++) print a[i];exit}' file
Jhon
Alex
Smith
How it works:
awk -vRS= -F"\n" ' # By setting RS to nothing, one record equals one block. Then FS is set to one line as a field
/^\[Name_Z1\]/ { # Search for block with [Name_Z1]
n=split($3,a,";") # Split field 3, the names and store number of fields in variable n
for (i=2;i<=n;i++) # Loop from second to last field
print a[i] # Print the fields
exit # Exits after first find
' file
With updated data
cat file
data
[CAB_Z1_FUEGO]
READ_ONLY=false
FilterAttr=CeaseTime;blank|ObjectOfReference;contains;511047;512044;513008;593026;598326;CL5518;CL5521;CL5538;CL5612;CL5620|PerceivedSeverity;=;Critical;Major;Minor|ProbableCause;!=;HOUSE ALARM;IO DEVICE|ProblemText;contains;FUEGO
NAME=CAB_Z1_FUEGO
data
awk -vRS= -F"\n" '/^\[CAB_Z1_FUEGO\]/ {split($3,a,"|");n=split(a[2],b,";");for (i=3;i<=n;i++) print b[i]}' file
511047
512044
513008
593026
598326
CL5518
CL5521
CL5538
CL5612
CL5620
The following awk script will do what you want:
awk 's==1&&/^Names/{gsub("Names;","",$0);gsub(";","\n",$0);print}/^\[Name_Z1\]$/||/^OBJ=Name_Z1$/{s++}' inputFileName
In more detail:
s==1 && /^Names;/ {
gsub ("Names;","",$0);
gsub(";","\n",$0);
print
}
/^\[Name_Z1\]$/ || /^OBJ=Name_Z1$/ {
s++
}
The state s starts with a value of zero and is incremented whenever you find one of the two lines:
[Name_Z1]
OBJ=Name_Z1
That means, between the first set of those lines, s will be equal to one. That's where the other condition comes in. When s is one and you find a line starting with Names;, you do two substitutions.
The first is to get rid of the Names; at the front, the second is to replace all ; semi-colon characters with a newline. Then you print it out.
The output for your given test data is, as expected:
Jhon
Alex
Smith
Hello: I have tab separated data of the form
customer-item description-purchase price-category
e.g. a.out contains:
1\t400 Bananas\t3.00\tfruit
2\t60 Oranges\t0.00\tfruit
3\tNULL\t3.0\tfruit
4\tCarrots\tNULL\tfruit
5\tNULL\tNULL\tfruit
I'm attempting to get rid of all the NULL fields. I can't rely on the simple replacement of the string "NULL" as it may be a substring; so I am attempting
sed -i 's:\tNULL\t:\t\t:g' a.out
when I do this, I end up with
1\t400 Bananas\t3.00\tfruit
2\t60 Oranges\t0.00\tfruit
3\t\t3.0\tfruit
4\tCarrots\t\tfruit
5.\t\tNULL\tfruit
what's wrong here is that #5 has only suffered a replacement of the first instance of the search string on each line.
If I run my sed command twice, I end up with the result I want:
1\t400 Bananas\t3.00\tfruit
2\t60 Oranges\t0.00\tfruit
3\t\t3.0\tfruit
4\tCarrots\t\tfruit
5.\t\t\tfruit
where you can see that line 5 has both of the NULLs removed
But I don't understand why I'm suffering this?
awk -F'\t' -v OFS='\t' '{
for (i = 1; i <= NF; ++i) {
if ($i == "NULL") {
$i = "";
}
}
print
}' test.txt
The straightforward solution is to use \t as a field separator and then loop over all of the fields looking for an exact match of "NULL". No substringing.
Here's the same thing as a one liner:
awk -F'\t' -v OFS='\t' '{for(i=1;i<=NF;++i) if($i=="NULL") $i=""} 1' test.txt
Since tabs can't be inside strings in your case since that would imply a new field you might be able to do what you want simply by doing this;
sed ':start ; s/\tNULL\(\t\|$\)/\t\1/ ; t start' a.out
First the inner part s/\tNULL\(\t\|$\)/\t\1/ searches for tab NULL followed by a tab or end of line $ and replace with a tab followed by the character that did appear after NULL (this last part is done using \1). We'll call that expression
We now have:
sed ':start ; expression ; t start' a.out
This is effectively a loop (like goto). :start is a label. ; acts as a statement delimiter. I have described what expression does above. t start says that IF the expression did any substitution that a jump will be made to label start. The buffer will contain the substituted text. This loop occurs until no substitution can be done on the line and then processing continues.
Information on sed flow control and other useful tidbits can be found here
awk makes it simpler:
awk -F '\tNULL\\>' -v OFS='\t' '{$1=$1}1' file
1\t400 Bananas\t3.00\tfruit
2\t60 Oranges\t0.00\tfruit
3\t\t3.0\tfruit
4\tCarrots\t\tfruit
5\t\t\tfruit
From grep(1) on a recent Linux:
The Backslash Character and Special Expressions
The symbols \< and > respectively match the empty string at the
beginning and end of a word. The symbol \b matches the empty string at
the edge of a word [...]
--
So, how about:
sed -i 's:\<NULL\>::g' a.out