I would like to extract text from file using awk what basicly it works correctly but I would like to make it dymamical using variable for looking for pattern.
HOW IT SHOULD WORKS:
File test_input contains (btw: extract from HP DP omnimm -show_locked_devs)
Type: Device
Name/Id: Drive1
Pid: 28405
Host: Host1
Type: Cartridge
Name/Id: Lib1
Pid: 28405
Host: Host1
Location: 47
...
get "Pid" number for Drive1 => command find pattern (Drive1) and display next line from file test_input (28405)
cat test_input | awk 'c&&!--c;/Drive1/{c=1}'| awk '{print $2}'
28405
get "Location" number => find all "Pid" numberes and display next 2 line(record) for each match then use grep for filter "Location" from output and display 2nd field (47)
cat test_input | awk 'c&&!--c;/28405/{c=2; print $0}'| grep Location | awk '{print $2}'
47
I have noticed that double quotes in AWK can handle SHELL variables but when I use SAME command in script then I have got error message "awk: The statement cannot be correctly parsed."
DRIVE=Drive1;cat test_input | awk "c&&!--c;/$DRIVE/{c=1}" | awk '{print $2}'
28405
If you have some hints how get work variables from SHELL please let me know.
also I know that my commands and redirections are probably complicated but yeah I am not script master :)
If You need just use environment variables then you can use the ENVIRON awk built-in hash. If You want to pass arguments to awk, you can use the -v option.
An example for both:
cat >inputfile <<EOT
aaa
bbbxxx
xxxccc
ddd
EOT
VAR=xxx
awk -vVAR="$VAR" '$0~VAR {print ENVIRON["USER"]":"$0}' inputfile
I added the creation of the sample inputfile.
As I know in some awk version a white space is needed between -v and VAR.
If I may suggest you would use ' instead of " around the whole script line. It makes the life a bit easier if you use a lot of awk.
Output:
myuser:bbbxxx
myuser:xxxccc
If I understood well, You need to collect the names of all devices and all locations in non "Device" blocks. I assume clock starting with the tag Type and the tag order is always the same. If not, pls. inform me. Based on these assumptions my code looks like:
awk '$1=="Type:"{dev=$2=="Device"}
dev && $1=="Name/Id:"{name=$2}
dev && $1=="Pid:"{pids[name]=$2}
!dev && $1=="Pid:"{pid=$2}
!dev && $1=="Location:"{locs[pid]=$2}
END {
for(i in pids) {
pid = pids[i];
print i"\t"(pid in locs ? locs[pid] : "None");
}
}
' inputfile
It fills up the pids and and locs hash, then it prints all device names found in pids hash and the location belongs to this pid (if found).
Output:
Drive1 47
Of course if the location is always after the device block, the line could be printed immediately when the location found. So the END part could be dropped.
It's not clear what you want but maybe this:
$ cat tst.awk
BEGIN{ RS=""; FS="[:[:space:]]+" }
{
for (i=1;i<=NF;i+=2)
name2val[$i] = $(i+1)
}
(name2val[key] == val) && (tgt in name2val) {
print name2val[tgt]
}
$
$ awk -v key="Name/Id" -v val="Drive1" -v tgt="Pid" -f tst.awk file
28405
$
$ awk -v key="Pid" -v val="28405" -v tgt="Location" -f tst.awk file
47
Related
I have a property file meant for Java like:
oracle {
username = "bla"
password = "blabla"
driver = "driver1"
}
postgres {
username = "pg"
password = "pg"
driver = "pg-driver"
}
when read into java I can extract the oracle.driver property which returns driver1.
Now I want to extract the same string in a bash script.
I have tried something like:
grep -A5 oracle application.conf | grep -Po 'driver = ".*?"' | grep -Po '".*"'
returning "driver1" (including the quotes). I also tried using sed substitute but that also did not yield the driver1 string.
How can I retriever only driver1?
Whenever you have name -> value mappings in your data, first creating an array to store those mappings (f[] below) and then accessing the data by it's name provides the simplest, clearest and easiest to enhance solution:
$ awk -v RS= '$1=="oracle"{ for (i=3;i<=NF;i+=3) f[$i]=$(i+2); print f["username"]}' file
"bla"
$ awk -v RS= '$1=="oracle"{ for (i=3;i<=NF;i+=3) f[$i]=$(i+2); print f["password"]}' file
"blabla"
$ awk -v RS= '$1=="oracle"{ for (i=3;i<=NF;i+=3) f[$i]=$(i+2); print f["driver"]}' file
"driver1"
$ awk -v name="driver" -v RS= '$1=="oracle"{ for (i=3;i<=NF;i+=3) f[$i]=$(i+2); print f[name]}' file
"driver1"
With single awk command - will work in ANY awk implementation:
awk '/oracle/{ f=1 }f && $1=="driver"{ gsub(/"/,""); print $3; exit }' file
/oracle/{ f=1 } - on encountering line matching the pattern oracle - set active flag f
f && $1=="driver" - if it's "active" processed section ("oracle") and the 1st field $1 is equal to driver:
gsub(/"/,"") - remove double quotes from the line
print $3 - print the 3rd field which is the driver value
exit - exit the script execution immediately avoiding redundant processing
The output:
driver1
Using awk you can do this using an empty record separator:
awk -v RS= '/^[[:blank:]]*oracle/{
gsub(/.*driver[[:blank:]]*=[[:blank:]]*|\n.*$|"/, ""); print}' application.conf
driver1
Empty RS makes all the continuous non-empty lines a single record.
You can try with sed too
database='oracle'
search='driver'
sed -n '
/'"$database"'/!d
:A
n
/'"$search"'/!bA
s/[^"]*"\([^"]*\)"/\1/
p
q
' application.conf
Sometimes I want a bash script that's mostly a help file. There are probably better ways to do things, but sometimes I want to just have a file called "awk_help" that I run, and it dumps my awk notes to the terminal.
How can I do this easily?
Another idea, use #!/bin/cat -- this will literally answer the title of your question since the shebang line will be displayed as well.
Turns out it can be done as pretty much a one liner, thanks to #CharlesDuffy for the suggestions!
Just put the following at the top of the file, and you're done
cat "$BASH_SOURCE" | grep -v EZREMOVEHEADER
So for my awk_help example, it'd be:
cat "$BASH_SOURCE" | grep -v EZREMOVEHEADER
# Basic form of all awk commands
awk search pattern { program actions }
# advanced awk
awk 'BEGIN {init} search1 {actions} search2 {actions} END { final actions }' file
# awk boolean example for matching "(me OR you) OR (john AND ! doe)"
awk '( /me|you/ ) || (/john/ && ! /doe/ )' /path/to/file
# awk - print # of lines in file
awk 'END {print NR,"coins"}' coins.txt
# Sum up gold ounces in column 2, and find out value at $425/ounce
awk '/gold/ {ounces += $2} END {print "value = $" 425*ounces}' coins.txt
# Print the last column of each line in a file, using a comma (instead of space) as a field separator:
awk -F ',' '{print $NF}' filename
# Sum the values in the first column and pretty-print the values and then the total:
awk '{s+=$1; print $1} END {print "--------"; print s}' filename
# functions available
length($0) > 72, toupper,tolower
# count the # of times the word PASSED shows up in the file /tmp/out
cat /tmp/out | awk 'BEGIN {X=0} /PASSED/{X+=1; print $1 X}'
# awk regex operators
https://www.gnu.org/software/gawk/manual/html_node/Regexp-Operators.html
I found another solution that works on Mac/Linux and works exactly as one would hope.
Just use the following as your "shebang" line, and it'll output everything from line 2 on down:
test.sh
#!/usr/bin/tail -n+2
hi there
how are you
Running this gives you what you'd expect:
$ ./test.sh
hi there
how are you
and another possible solution - just use less, and that way your file will open in searchable gui
#!/usr/bin/less
and this way you can grep if for something too, e.g.
$ ./test.sh | grep something
I'm having some trouble passing bash script variables into awk command-line.
Here is pseudocode:
for FILE in $INPUT_DIR/*.txt; do
filename=`echo $FILE | sed -n 's/^.*\(chr[0-9A-Z]*\).*.vcf$/\1/p'`
OUTPUT_FILE=$OUTPUT_DIR/$filename.snps.txt
egrep -v "^#" $FILE | awk '{print $2,$4,$5}' > $OUTPUT_FILE
done
The final line where I awk the columns, I would like it to be flexible or user input. For example, the user could want columns 6,7,and 8 as well, or column 133 and 138, or column 245 through 248. So how do I custom this so I can have that 'print $2 .... $5' be a user input thing? For example the user would run this script like : bash script.sh input_dir output_dir [user inputs whatever string of columns], and then I would get those columns in the output. I tried passing it in, but I guess I'm not getting the syntax right.
With awk, you should declare the variable before use it. This is better than the escape method (awk '{print $'$var'}'):
awk -v var1="$col1" -v var2="$col2" 'BEGIN {print var1,var2 }'
Where $col1 and $col2 would be the input variables.
Maybe you can try an input variable as string with "$2,$4,$5" and print this variable to get the values (I am not sure if this works)
The following test works for me:
A="\$3" ; ls -l | awk "{ print $A }"
I want to record the RSSI at a certain point with the distance that point is from a router. The distance will be user input and so will the output file name so the user will type something like:
sh rssi.sh output.csv 20
where output.csv is the csv I want to append the results to and 20 is the distance
at the moment rssi.sh looks like:
#!/bin/bash
RSSI_CSV=$1
DISTANCE=$2
RSSI=$(iwconfig wlan0 | awk -F'[ =]+' '/Signal level/ {print $7}\')
awk '{print $DISTANCE, $RSSI}' > $RSSI_CSV
This creates RSSI_CSV as per user input but doesn't print the required values in it and I'm not sure why.
I imagine it's
awk '{print $DISTANCE, $RSSI}' > $RSSI_CSV
that isn't working as echo RSSI or echo DISTANCE both output the values to the screen. I'm using awk as I want to have columns so i can output a csv file, perhaps though there is a better way?
There are a couple of issues with your awk need to pass the variables using the -v option and use the BEGIN block as no input is given. Also note that a single > will not append but overwrite the file. For appending you need >>:
awk -vD=$DISTANCE -vR=$RSSI 'BEGIN{print D,R}' >> $RSSI_CSV
Demo:
$ DISTANCE=20
$ RSSI=$(iwconfig wlan0 | awk -F'[ =]+' '/Signal level/ {print $7}')
$ awk -vD=$DISTANCE -vR=$RSSI 'BEGIN{print D,R}'
20 -47
Note: I believe you want comma separated values so:
$ awk -vD=$DISTANCE -vR=$RSSI 'BEGIN{print D","R}'
20,-47
However awk is overkill for printing variables just use good old echo:
$ echo "$DISTANCE,$RSSI"
20,-47
You don't need awk to print two shell variables.
printf "%s,%s\n" "$DISTANCE" "$RSSI" >> "$RSSI_CSV"
I have a program in C that I want to call by using awk in shell scripting. How can I do something like this?
From the AWK man page:
system(cmd)
executes cmd and returns its exit status
The GNU AWK manual also has a section that, in part, describes the system function and provides an example:
system("date | mail -s 'awk run done' root")
A much more robust way would be to use the getline() function of GNU awk to use a variable from a pipe. In form cmd | getline result, cmd is run, then its output is piped to getline. It returns 1 if got output, 0 if EOF, -1 on failure.
First construct the command to run in a variable in the BEGIN clause if the command is not dependant on the contents of the file, e.g. a simple date or an ls.
A simple example of the above would be
awk 'BEGIN {
cmd = "ls -lrth"
while ( ( cmd | getline result ) > 0 ) {
print result
}
close(cmd);
}'
When the command to run is part of the columnar content of a file, you generate the cmd string in the main {..} as below. E.g. consider a file whose $2 contains the name of the file and you want it to be replaced with the md5sum hash content of the file. You can do
awk '{ cmd = "md5sum "$2
while ( ( cmd | getline md5result ) > 0 ) {
$2 = md5result
}
close(cmd);
}1'
Another frequent usage involving external commands in awk is during date processing when your awk does not support time functions out of the box with mktime(), strftime() functions.
Consider a case when you have Unix EPOCH timestamp stored in a column and you want to convert that to a human readable date format. Assuming GNU date is available
awk '{ cmd = "date -d #" $1 " +\"%d-%m-%Y %H:%M:%S\""
while ( ( cmd | getline fmtDate) > 0 ) {
$1 = fmtDate
}
close(cmd);
}1'
for an input string as
1572608319 foo bar zoo
the above command produces an output as
01-11-2019 07:38:39 foo bar zoo
The command can be tailored to modify the date fields on any of the columns in a given line. Note that -d is a GNU specific extension, the *BSD variants support -f ( though not exactly similar to -d).
More information about getline can be referred to from this AllAboutGetline article at awk.freeshell.org page.
There are several ways.
awk has a system() function that will run a shell command:
system("cmd")
You can print to a pipe:
print "blah" | "cmd"
You can have awk construct commands, and pipe all the output to the shell:
awk 'some script' | sh
Something as simple as this will work
awk 'BEGIN{system("echo hello")}'
and
awk 'BEGIN { system("date"); close("date")}'
I use the power of awk to delete some of my stopped docker containers. Observe carefully how i construct the cmd string first before passing it to system.
docker ps -a | awk '$3 ~ "/bin/clish" { cmd="docker rm "$1;system(cmd)}'
Here, I use the 3rd column having the pattern "/bin/clish" and then I extract the container ID in the first column to construct my cmd string and passed that to system.
It really depends :) One of the handy linux core utils (info coreutils) is xargs. If you are using awk you probably have a more involved use-case in mind - your question is not very detailled.
printf "1 2\n3 4" | awk '{ print $2 }' | xargs touch
Will execute touch 2 4. Here touch could be replaced by your program. More info at info xargs and man xargs (really, read these).
I believe you would like to replace touch with your program.
Breakdown of beforementioned script:
printf "1 2\n3 4"
# Output:
1 2
3 4
# The pipe (|) makes the output of the left command the input of
# the right command (simplified)
printf "1 2\n3 4" | awk '{ print $2 }'
# Output (of the awk command):
2
4
# xargs will execute a command with arguments. The arguments
# are made up taking the input to xargs (in this case the output
# of the awk command, which is "2 4".
printf "1 2\n3 4" | awk '{ print $2 }' | xargs touch
# No output, but executes: `touch 2 4` which will create (or update
# timestamp if the files already exist) files with the name "2" and "4"
Update In the original answer, I used echo instead of printf. However, printf is the better and more portable alternative as was pointed out by a comment (where great links with discussions can be found).
#!/usr/bin/awk -f
BEGIN {
command = "ls -lh"
command |getline
}
Runs "ls -lh" in an awk script
You can call easily with parameters via the system argument.
For example, to kill jobs corresponding to a certain string (we can otherly of course) :
ps aux | grep my_searched_string | awk '{system("kill " $2)}'
I was able to have this done via below method
cat ../logs/em2.log.1 |grep -i 192.168.21.15 |awk '{system(`date`); print $1}'
awk has a function called system it enables you to execute any linux bash command within the output of awk.