I want to extract a substring where certain pattern exist from pipe separated file, thus I used below command,
awk -F ":" '/REWARD REQ. SERVER HEADERS/{print $1, $2, $3, $4}' sample_profile.txt
Here, 'REWARD REQ. SERVER HEADERS' is a pattern which is to be searched in the file, and print its first 4 parts on a colon separated line.
Now, I want to send bash variable to act as a pattern. thus I used below command, but it's not working.
awk -v pat="$pattern" -F ":" '/pat/{print $1, $2 , $3, $4 } sample_profile.txt
How can I use -v and -F in a single awk command?
If you want to provide the pattern through a variable, you need to use ~ to match against it:
awk -v pat="$pattern" '$0 ~ pat'
In your case, the problem does not have to do with -F.
The problem is the usage of /pat/ when you want pat to be a variable. If you say /pat/, awk understands it as a literal "pat", so it will try to match those lines containing the string "pat".
All together, your code should be:
awk -v pat="$pattern" -F ":" '$0~pat{print $1, $2, $3, $4 }' file
# ^^^^^^
See an example:
Given this file:
$ cat file
hello
this is a var
hello bye
Let's look for lines containing "hello":
$ awk '/hello/' file
hello
hello bye
Let's now try looking for "pat", contained in a variable, the way you were doing it:
$ awk -v pat="hello" '/pat/' file
$ # NO MATCHES!
Let's now use the $0 ~ pat expression:
$ awk -v pat="hello" '$0~pat' file
hello # WE MATCH!
hello bye
Of course, you can use such expressions to match just one field and say awk -v pat="$pattern" '$2 ~ pat' file and so on.
From GNU Awk User's Guide → 3.1 How to Use Regular Expressions:
When a regexp is enclosed in slashes, such as /foo/, we call it a regexp constant, much like 5.27 is a numeric constant and "foo" is a string constant.
And GNU Awk User's Guide → 3.6 Using Dynamic Regexps:
The righthand side of a ‘~’ or ‘!~’ operator need not be a regexp
constant (i.e., a string of characters between slashes). It may be any
expression. The expression is evaluated and converted to a string if
necessary; the contents of the string are then used as the regexp. A
regexp computed in this way is called a dynamic regexp or a computed
regexp:
BEGIN { digits_regexp = "[[:digit:]]+" }
$0 ~ digits_regexp { print }
This sets digits_regexp to a regexp that describes one or more digits,
and tests whether the input record matches this regexp.
awk -v pat="$pattern" -F":" '$0 ~ pat { print $1, $2, $3, $4 }' sample_profile.txt
You can't use the variable inside the regex // notation (there's no way to distinguish it from searching for pat); you have to specify that the variable is a regex with the ~ (matching) operator.
This is kind of a hack but it makes things a little simpler for me.
cmd="awk '/$pattern/'"
eval $cmd
making it a string first lets you manipulate it past the boundaries of awk
Related
I have a text file called 'file.txt' with the content like,
test:one
test_test:two
test_test_test:three
If the pattern is test, then the expected output should be one and similarly for the other two lines.
This is what I have tried.
pattern=test && awk '{split($0,i,":"); if (i[1] ~ /'"$pattern"'$/) print i[2]}'
This command gives the output like,
one
two
three
and pattern=test_test && awk '{split($0,i,":"); if (i[1] ~ /'"$pattern"'$/) print i[2]}'
two
three
How can I match the unique pattern being "test" for "test" and not for "test_test" and so on.
How can I match the unique pattern being test for test and not for test_test and so on.
Don't use a regex for comparing the value, just use equality:
awk -F: -v pat='test' '$1 == pat {print $2}' file
one
awk -F: -v pat='test_test' '$1 == pat {print $2}' file
two
If you really want to use regex, then use it like this with anchors:
awk -F: -v pat='test' '$1 ~ "^" pat "$" {print $2}' file
one
If you want to use a regex, you can create it dynamically with pattern and optionally repeating _ followed by pattern until matching a :
If it matches the start of the string, then you can print the second field.
awk -v pattern='test' -F: '
$0 ~ "^"pattern"(_"pattern")*:" {
print $2
}
' file
Output
one
two
three
Or if only matching the part before the first underscore is also ok, then splitting field 1 on _ and printing field 2:
awk -v pattern='test' -F: ' {
split($1, a, "_")
if(a[1] == pattern) print $2
}' file
Using GNU sed with word boundaries
$ sed -n '/\<test\>/s/[^:]*://p' input_file
one
I want to extract a substring where certain pattern exist from pipe separated file, thus I used below command,
awk -F ":" '/REWARD REQ. SERVER HEADERS/{print $1, $2, $3, $4}' sample_profile.txt
Here, 'REWARD REQ. SERVER HEADERS' is a pattern which is to be searched in the file, and print its first 4 parts on a colon separated line.
Now, I want to send bash variable to act as a pattern. thus I used below command, but it's not working.
awk -v pat="$pattern" -F ":" '/pat/{print $1, $2 , $3, $4 } sample_profile.txt
How can I use -v and -F in a single awk command?
If you want to provide the pattern through a variable, you need to use ~ to match against it:
awk -v pat="$pattern" '$0 ~ pat'
In your case, the problem does not have to do with -F.
The problem is the usage of /pat/ when you want pat to be a variable. If you say /pat/, awk understands it as a literal "pat", so it will try to match those lines containing the string "pat".
All together, your code should be:
awk -v pat="$pattern" -F ":" '$0~pat{print $1, $2, $3, $4 }' file
# ^^^^^^
See an example:
Given this file:
$ cat file
hello
this is a var
hello bye
Let's look for lines containing "hello":
$ awk '/hello/' file
hello
hello bye
Let's now try looking for "pat", contained in a variable, the way you were doing it:
$ awk -v pat="hello" '/pat/' file
$ # NO MATCHES!
Let's now use the $0 ~ pat expression:
$ awk -v pat="hello" '$0~pat' file
hello # WE MATCH!
hello bye
Of course, you can use such expressions to match just one field and say awk -v pat="$pattern" '$2 ~ pat' file and so on.
From GNU Awk User's Guide → 3.1 How to Use Regular Expressions:
When a regexp is enclosed in slashes, such as /foo/, we call it a regexp constant, much like 5.27 is a numeric constant and "foo" is a string constant.
And GNU Awk User's Guide → 3.6 Using Dynamic Regexps:
The righthand side of a ‘~’ or ‘!~’ operator need not be a regexp
constant (i.e., a string of characters between slashes). It may be any
expression. The expression is evaluated and converted to a string if
necessary; the contents of the string are then used as the regexp. A
regexp computed in this way is called a dynamic regexp or a computed
regexp:
BEGIN { digits_regexp = "[[:digit:]]+" }
$0 ~ digits_regexp { print }
This sets digits_regexp to a regexp that describes one or more digits,
and tests whether the input record matches this regexp.
awk -v pat="$pattern" -F":" '$0 ~ pat { print $1, $2, $3, $4 }' sample_profile.txt
You can't use the variable inside the regex // notation (there's no way to distinguish it from searching for pat); you have to specify that the variable is a regex with the ~ (matching) operator.
This is kind of a hack but it makes things a little simpler for me.
cmd="awk '/$pattern/'"
eval $cmd
making it a string first lets you manipulate it past the boundaries of awk
How can I split a string with awk but printing the match too?
Full random string:
aaa sasawf wewfTotemeswdwqewqwqtotemwewedew
I need to get "wewftotemeswdwqewqwqtotemwewedew" where the substring is random, the only constant is a space and the word totem in it. As you notice the random string might contain more than one totem word, I need awk to get the substring starting from the first match. To be clear, I need "wewftotemeswdwqewqwqtotemwewedew" not "totemwewedew". I also need it to be case insensitive
I can use awk -F ' .*totem' '{print$2}' to print eswdwqewqwqtotemwewedew but how can I print the match too?
With GNU awk for the third arg to match():
$ echo 'aaa sasawf wewftotemeswdwqewqwq' |
awk 'match($0,/[^ ]*totem[^ ]*/,a) { print a[0] }'
wewftotemeswdwqewqwq
and with any awk:
$ echo 'aaa sasawf wewftotemeswdwqewqwq' |
awk 'match($0,/[^ ]*totem[^ ]*/) { print substr($0,RSTART,RLENGTH) }'
wewftotemeswdwqewqwq
For case-insensitive matching with GNU awk:
awk -v IGNORECASE=1 'match($0,/[^ ]*totem[^ ]*/...
and with any awk:
awk 'match(tolower($0),/[^ ]*totem[^ ]*/...
Running
ip -o -f inet addr show | grep $INTERNAL |awk '/scope global/ {print $4}'
Want to replace the / in my output to _ so rather than reading
10.168.122.59/16
it reads as
10.168.122.59_16
.
|sed s///_/
didnt help
Any suggestions?
All the postprocessing requested can be done internal to awk. Expanding a one-liner provided in a comment by #123 for better readability, this can look like the following:
ip -o -f inet addr show | \
awk -v i="$INTERNAL" '
$0 ~ i && /scope global/ {
sub(/\//, "_", $4);
print $4;
}'
Breaking down how this works:
awk -v i="$INTERNAL" defines an awk variable based on a shell variable. (As an aside, all-caps shell variable names are bad form; per POSIX convention, lowercase names are reserved for application use, whereas all-caps names can have meaning to the OS and surrounding tools).
$0 ~ i filters for the entire line ($0) matching the awk variable i.
/scope global/ by default is applied as a regex against $0 as well (it's equivalent to $0 ~ /scope global/).
sub(/\//, "_", $4) substitutes /s with _s in the fourth field.
print $4 prints that field.
You need to scape the / or use a different separator as below:
echo 10.168.122.59/16 | sed s:/:_:
echo 10.168.122.59/16| awk '{sub(/\//,"_")}1'
10.168.122.59_16
I am trying to pass a for loop index i into awk but keep getting unexpected token awk errors.
First I tried using the -v option within awk:
for i in "${myarray}"
awk -v var=$i '/var/{print}' myfile.dat
done
I also tried calling the variable directly using single quotes:
for i in "${myarray}"
awk '/'"$i"'/{print}' myfile.dat
done
My end goal is to learn how to pass a for loop index variable through awk as the search pattern. I'd like the above code to search through myfile.dat and print lines which contain the strings in myarray.
There are 2 problems:
Array traversing should be like this for i in "${myarray[#]}"
awk treats text between /.../ as regex literal, to use a variable use $0 ~ var.
Your code should be:
for i in "${myarray[#]}"; do
awk -v var="$i" '$0 ~ var' myfile.dat
done
{print} is default action in awk that you can omit as shown above.
you can do the same loop free as well, e.g.,
echo "${myarray[#]}" | tr ' ' '|' | awk 'NR==FNR{pat=$0; next} $0 ~ pat' - file