Writing a shell wrapper script for awk - shell

I want to embed an awk script inside a shell script but I have trouble to do so as I don't know where to end a statement with a ; and where not.
Here's my script
#!/bin/sh
awk='
BEGIN {FS = ",?+" }
# removes all backspaces preceded by any char except _
function format() {
gsub("[^_]\b", "")
}
function getOptions() {
getline
format()
print
}
{
format()
if ($0 ~ /^SYNOPSIS$/ {
getOptions()
next
}
if ($0 /^[ \t]+--?[A-Za-z0-9]+/) {
print $0
}
}
END { print "\n" }'
path='/usr/share/man/man1'
list=$(ls $path)
for item in $list
do
echo "Command: $item"
zcat $path$item | nroff -man | awk "$awk"
done > opts
I'm using nawk by the way.
Thanks in advance

There are several things wrong, as far as I can see:
You don't close the multi-line string being assigned to $awk. You need a single quote on the line after END { ... }
You don't seem to actually use $awk anywhere. Perhaps you meant on the invocation of awk inside the do loop.
Once you fix those issues, awk is usually fairly forgiving about semicolons, but any problems in that regard don't have anything to do with using it inside a shell script.

These three lines:
path='/usr/share/man/man1'
list=$(ls $path)
for item in $list
Need to be changed into:
path='/usr/share/man/man1'
for item in $path/*
in case there are spaces in filenames and since ls is not intended to be used in this way.

i am not really sure what you meant, but if i understand you correctly, your showOpts.awk is that awk code at the beginning of your script, so you could do this
path='/usr/share/man/man1'
list=$(ls $path)
for item in $list
do
echo "Command: $item"
zcat $path$item | nroff -man | nawk ' BEGIN {FS = ",?+" }
# removes all backspaces preceded by any char except _
function format() {
gsub("[^_]\b", "")
}
function getOptions() {
getline
format()
print
}
{
format()
if ($0 ~ /^SYNOPSIS$/ {
getOptions()
next
}
if ($0 /^[ \t]+--?[A-Za-z0-9]+/) {
print $0
}
}
END { print "\n" } '
done >> opts
and you should probably use >> instead of > .

Related

Appending result of function on another field into csv using shell script, awk

I have a csv file stored as a temporary variable in a shell script (*.sh).
Let's say the data looks like this:
Account,Symbol,Price
100,AAPL US,200
102,SPY US,500
I want to add a fourth column, "Type", which is the result of a shell function "foobar". Run from the command line or a shell script itself:
$ foobar "AAPL US"
"Stock"
$ foobar "SPY US"
"ETF"
How do I add this column to my csv, and populate it with calls to foobar which take the second column as an argument? To clarify, this is my ideal result post-script:
Account,Symbol,Price,Type
100,AAPL US,200,Common Stock
102,SPY US,500,ETF
I see many examples online involving such a column addition using awk, and populating the new column with fixed values, conditional values, mathematical derivations from other columns, etc. - but nothing that calls a function on another field and stores its output.
You may use this awk:
export -f foobar
awk 'BEGIN{FS=OFS=","} NR==1{print $0, "Type"; next} {
cmd = "foobar \"" $2 "\""; cmd | getline line; close(cmd);
print $0, line
}' file.csv
Account,Symbol,Price,Type
100,AAPL US,200,Common Stock
102,SPY US,500,ETF
#anubhavas answer is a good approach so please don't change the accepted answer as I'm only posting this as an answer as it's too big and in need of formatting to fit in a comment.
FWIW I'd write his awk script as:
awk '
BEGIN { FS=OFS="," }
NR==1 { type = "Type" }
NR > 1 {
cmd = "foobar \047" $2 "\047"
type = ((cmd | getline line) > 0 ? line : "ERROR")
close(cmd)
}
{ print $0, type }
' file.csv
to:
better protect $2 from shell expansion, and
protect from silently printing the previous value if/when cmd | getline fails, and
consolidate the print statements to 1 line so it's easy to change for all output lines if/when necessary
awk to the rescue!
$ echo "Account,Symbol,Price
100,AAPL US,200
102,SPY US,500" |
awk -F, 'NR>1{cmd="foobar "$2; cmd | getline type} {print $0 FS (NR==1?"Type":type)}'
Not sure you need to quote the input to foobar
Another way not using awk:
paste -d, input.csv <({ read; printf "Type\n"; while IFS=, read -r _ s _; do foobar "$s"; done; } < input.csv)

Bash: Adding the contents of a column to a variable

So I have a file that contains some text, but there are some lines that contain only "Overall>5". (the number after > could be any number from 0 to 5).
The code I wrote:
let sumOfReviews=0
while read line; do awk -F ">" '{if($1=="Overall") ((sumOfReviews+=$2))}'; done<$file
echo $sumOfReviews
I tried splitting these lines into 2 columns at ">" and I want to add the number in the second column (5 in this case) to a variable. The problem is when I print out the value of the variable at the end it shows 0. Any thoughts? Thanks!
If called from the console, the following seems to do the job:
awk -F'>' '/Overall>[0-9]+/ { total += $2 } END { print total }' exampleData.txt
If you want to call it from inside bash, you have to enclose it in $( ... ):
#!/bin/bash
total="$(awk -F'>' '/Overall>[0-9]+/ { total += $2 } END { print total }' exampleData.txt)"
# do something with `total` here.
You cannot simply use awk as some sort of syntax inside bash, awk is a separate programming language, it is invoked as a completely separate process. You can try to build in some bash parameters into awks source code though.
This can be done in a one line awk script.
awk 'BEGIN { FS=">"; sumOfReviews=0 } /^Overall>[0-5]/ { sumOfReviews+=$2 } END { print sumOfReviews }' < file
Explanation from Manpage:
An AWK program consists of a sequence of pattern-action statements and optional function definitions.
pattern { action statements }
In this case we have used the BEGIN pattern to set the file separator to ">" and the sumOfReviews variable to 0.
We use the /^Overall>[0-5]/ regular expression pattern to match lines beginning with "Overall>" followed by a number 0-5 and if true add the $2 field to the sumOfReviews variable.
Finally we use the END pattern to output the final sumOfReviews value.
Example solution in a bash shell script:
#!/bin/bash
noAuthors=4 # set to 4 for example
sumOfReviews=$(awk 'BEGIN { FS=">"; sumOfReviews=0 } /^Overall>[0-5]/
{ sumOfReviews+=$2 } END { print sumOfReviews }' < file)
echo $(($sumOfReviews/$noAuthors))
awk and bash are two separate programs; they don't share variables. All you need is a single awk script:
awk -F '>' '$1 == "Overall" {reviews += $2}; END {print reviews}' "$file"

Using awk to iterate unix command nm and sum output through multiple files

I am currently working on a script that will look through the output of nm and sum the values of column $1 using the following
read $filename
nm --demangle --size-sort --radix=d ~/object/$filename | {
awk '{ sum+= $1 } END { print "Total =" sum }'
}
I want to do the following for any number of files, looping through a directory to then output a summary of results. I want the result for each file and also the result of adding the first column of all the columns.
I am limited to using just bash and awk.
You need to put the read $filename in a while; do; done loop and feed the output of the entire loop to awk.
e.g.
while read filename ; do
nm ... $filename
done | awk '{print $0} { sum+=$1 } END { print "Total="sum}'
the awk {print $0} will print each file's line so you can see each one.
bash globstar option is for recursive file matching
you can use like **/*.txt at the end awk command
$ shopt -s globstar
$ awk '
BEGINFILE {
c="nm --demangle --size-sort --radix=d \"" FILENAME "\""
while ((c | getline) > 0) { fs+=$1; ts+=$1; }
printf "%45s %10'\''d\n",FILENAME, fs
close(c); fs=0; nextfile
} END {
printf "%30s %s\n", " ", "-----------------------------"
printf "%45s %10'\''d\n", "total", ts
}' **/*filename*

Bash find/replace and run command on matching group

I'm trying to do a dynamic find/replace where a matching group from the find gets manipulated in the replace.
testfile:
…
other text
base64_encode_SOMEPATH_ something
other(stuff)
text base64_encode_SOMEOTHERPATH_
…
Something like this:
sed -i "" -e "s/(base64_encode_(.*)_)/cat MATCH | base64/g" testfile
Which would output something like:
…
other text
U09NRVNUUklORwo= something
other(stuff)
text U09NRU9USEVSU1RSSU5HCg==
…
Updated per your new requirement. Now using GNU awk for the 3rd arg to match() for convenience:
$ awk 'match($0,/(.*)base64_encode_([^_]+)_(.*)/,arr) {
cmd = "base64 <<<" arr[2]
if ( (cmd | getline rslt) > 0) {
$0 = arr[1] rslt arr[3]
}
close(cmd)
} 1' file
…
other text
U09NRVNUUklORwo= something
other(stuff)
text U09NRU9USEVSU1RSSU5HCg==
…
Make sure you read and understand http://awk.info/?tip/getline if you're going to use getline.
If you can't install GNU awk (but you really, REALLY would benefit from having it so do try) then something like this would work with any modern awk:
$ awk 'match($0,/base64_encode_[^_]+_/) {
arr[1] = substr($0,1,RSTART-1)
arr[2] = arr[3] = substr($0,RSTART+length("base64_encode_"))
sub(/_.*$/,"",arr[2])
sub(/^[^_]+_/,"",arr[3])
cmd = "base64 <<<" arr[2]
if ( (cmd | getline rslt) > 0) {
$0 = arr[1] rslt arr[3]
}
close(cmd)
} 1' file
I say "something like" because you might need to tweak the substr() and/or sub() args if they're slightly off, I haven't tested it.
awk '!/^base64_encode_/ { print } /^base64_encode_/ { fflush(); /^base64_encode_/ { fflush(); sub("^base64_encode_", ""); sub("_$", ""); cmd = "base64" ; print $0 | cmd; close(cmd); }' testfile > testfile.out
This says to print non-matching lines unaltered.
Matching lines get altered with the awk function sub() to extract the string to be encoded, which is then piped to the base64 command, which prints the result to stdout.
The fflush call is needed so that all the previous output from awk has been flushed before the base64 output appears, ensuring lines aren't re-ordered.
Edit:
As pointed out in the comment, testing every line twice for matching a pattern and non-matching the same pattern isn't very good. This single action handles all lines:
{
if ($0 !~ "base64_encode_")
{
print;
next;
}
fflush();
sub("^.*base64_encode_", "");
sub("_$", "");
cmd = "base64";
print $0 | cmd;
close(cmd);
}

Overwriting a file in bash

I have a file, of which a part is shown below:
OUTPUT_FILENAME="out.Received.Power.x.0.y.1.z.0.41
X_TX=0
Y_TX=1
Z_TX=0.41
I would like to automatically change some part of it with BASH: every time i see OUTPUT_FILENAME i want to over write the name next to it and change it with a new one. Then i want to do the same with the values X_TX, Y_TX and Z_TX: delete the value next to it and rewrite a new one. For example instead of X_TX=0 i want X_TX=0.3 or viceversa.
Do you think it's possible?Maybe with grep or so..
You can use sed like this:
i.e. to replace X_TX= with X_TX=123 you can do:
sed -i -e 's/X_TX=.*/X_TX=123/g' /tmp/file1.txt
One option using awk. Your values are passed as variables to the awk script and substituted when exists a match:
awk -v outfile="str_outfile" -v x_tx="str_x" -v y_tx="str_y" -v z_tx="str_z" '
BEGIN { FS = OFS = "=" }
$1 == "OUTPUT_FILENAME" { $2 = outfile; print; next }
$1 == "X_TX" { $2 = x_tx; print $0; next }
$1 == "Y_TX" { $2 = y_tx; print $0; next }
$1 == "Z_TX" { $2 = z_tx; print $0; next }
' infile

Resources