setting the output field separator in awk - shell

I'n trying this statement in my awk script (in a file containing separate code, so not inline), script name: print-table.awk
BEGIN {FS = "\t";OFS = "," ; print "about to open the file"}
{print $0}
END {print "about to close stream" }
and running it this way from the shell
awk -f print-table.awk table
Where table is a tab separated file,
My goal is to declare the field separator (FS) and the output field separator (OFS) within the external function, and calling from the shell simply the
awk -f file input
without setting the field separator in the command line with -F"\t"
and without stdout it to a sed statement replacing the tab with a comma,
Any advise how can i do that?


You need to convince awk that something has changed to get it to reformat $0 using your OFS. The following works though there may be a more idiomatic way to do it.
BEGIN {FS = "\t";OFS = "," ; print "about to open the file"}
{$1=$1}1
END {print "about to close stream" }


You need to alter one of the field in awk:
awk 'BEGIN {FS="\t";OFS=","; print "about to open the file"} {$1=$1}1' file

Related

Changing value in entire column of a CSV file using awk

I have CSV file like example below. I wish to change value of mail in every line for the same mail with awk. I used
awk -F ";" '{$18=firstname.lastname#testdata.invali} {print}' example_source_20180619.csv > erm.csv
but got error invalid statement
PNR;GES-SL-SAP;VERT-KANAL-SL-SAP;DSTNR-SAP;BTRBL-FKT;SCHWPKT-TAETIG-SL;SCHWPKT-TAETIG-TXT;BTRBL-TITEL-TXT;ANREDE;NAME;VORNAME;STRASSE;PLZ;ORT;DIENST-TEL-NR;TELEFAX-NR;MOBIL-TEL-NR;E-MAIL-ADR;INTERNET-ADR;P34F-KZ;HD-ANL-BER-KZ;VERT-KANAL-SL;
0000000;0010;2100 ;00602;Referent ;99;Sonstige/kein Schw. ;ohne Titel ;Sir ;John ;Doe ;Paul-Keller-Str. 21 ;92318;Neumarkt i.d.OPf. ;phone;0941/phone;;mail#mail.com;http://web.de ;NO;NO;

awk -F ";" '{OFS=";"; $18="firstname.lastname#testdata.invali"; print;}'
Put strings inside ".
Separate commands using ;.
Set the output separator also to ; so the output is similar to the input.
I guess theres no point in substituting the email address in the first line, so I added a small if below:
awk -F ";" '{ OFS=";"; if (NR != 1) { $18="firstname.lastname#testdata.invali"; } print; }'

Appending result of function on another field into csv using shell script, awk

I have a csv file stored as a temporary variable in a shell script (*.sh).
Let's say the data looks like this:
Account,Symbol,Price
100,AAPL US,200
102,SPY US,500
I want to add a fourth column, "Type", which is the result of a shell function "foobar". Run from the command line or a shell script itself:
$ foobar "AAPL US"
"Stock"
$ foobar "SPY US"
"ETF"
How do I add this column to my csv, and populate it with calls to foobar which take the second column as an argument? To clarify, this is my ideal result post-script:
Account,Symbol,Price,Type
100,AAPL US,200,Common Stock
102,SPY US,500,ETF
I see many examples online involving such a column addition using awk, and populating the new column with fixed values, conditional values, mathematical derivations from other columns, etc. - but nothing that calls a function on another field and stores its output.

You may use this awk:
export -f foobar
awk 'BEGIN{FS=OFS=","} NR==1{print $0, "Type"; next} {
cmd = "foobar \"" $2 "\""; cmd | getline line; close(cmd);
print $0, line
}' file.csv
Account,Symbol,Price,Type
100,AAPL US,200,Common Stock
102,SPY US,500,ETF

#anubhavas answer is a good approach so please don't change the accepted answer as I'm only posting this as an answer as it's too big and in need of formatting to fit in a comment.
FWIW I'd write his awk script as:
awk '
BEGIN { FS=OFS="," }
NR==1 { type = "Type" }
NR > 1 {
cmd = "foobar \047" $2 "\047"
type = ((cmd | getline line) > 0 ? line : "ERROR")
close(cmd)
}
{ print $0, type }
' file.csv
to:
better protect $2 from shell expansion, and
protect from silently printing the previous value if/when cmd | getline fails, and
consolidate the print statements to 1 line so it's easy to change for all output lines if/when necessary

awk to the rescue!
$ echo "Account,Symbol,Price
100,AAPL US,200
102,SPY US,500" |
awk -F, 'NR>1{cmd="foobar "$2; cmd | getline type} {print $0 FS (NR==1?"Type":type)}'
Not sure you need to quote the input to foobar

Another way not using awk:
paste -d, input.csv <({ read; printf "Type\n"; while IFS=, read -r _ s _; do foobar "$s"; done; } < input.csv)

How to set the delimiter of csv file as ; instead of default coma(,) in a shell script

I am running the below shell script
awk -F ' *[[:alnum:]_]*: *' 'BEGIN {h="bla;blaa;blaaa;blaaaaa;blaaaaa1;bla111;bla2;bla3;blaaa3;bla4"; print h; n=split(h,F,/;/)}
function pr() {if(F[1] in A) {for(i=1;i<=n;i++)printf "%s%s",A[F[i]],(i<n)?";":RS}}
/insert_job/ {pr(); delete A}
{for(i in F){if($0~"^"F[i])A[F[i]]=$2}}
END {pr()}' infile > outfile.csv
The output file is output.csv, where coma is delimiter.Since one of the column contains values where coma is also there, the output.csv gets messed up.Please help me to understand how can i set the delimiter as ;(semicolon)

How to set FS to eof?

I want to read whole file not per lines. How to change field separator to eof-symbol?
I do:
awk "^[0-9]+∆DD$1[PS].*$" $(ls -tr)
$1 - param (some integer), .* - message that I want to print. There is a problem: message can contains \n. In that way this code prints only first line of file. How can I scan whole file not per lines?
Can I do this using awk, sed, grep? Script must have length <= 60 characters (include spaces).

Assuming you mean record separator, not field separator, with GNU awk you'd do:
gawk -v RS='^$' '{ print "<" $0 ">" }' file
Replace the print with whatever you really want to do and update your question with some sample input and expected output if you want help with that part too.
The portable way to do this, by the way, is to build up the record line by line and then process it in the END section:
awk '{rec = rec (NR>1?RS:"") $0} END{ print "<" rec ">" }' file
using nf = split(rec,flds) to create fields if necessary.

search a keyword in file and replace line next to it

I have a file which requires modification via a shell script.
Need to do the following:
1. search for a keyword in the file.
2. replace the next line to this keyword with my supplied line text.
for e.g., my file has the following text:
(some text)
(some text)
(text_to_search)
(text_to_replace)
(some text)
(some text)
(some text)
I need to search the file for and rewrite the file replace the line leaving the remaining content untouched.
How can this be done?
Regards

awk ' zap==1 {print "my new line goes here"; zap=0; next}
/my pattern/ {zap=1; print $0; next}
{print $0} ' infile > newfile
Assuming I got what you wanted....
var="this is the line of text to insert"
awk -v lin="$var" ' zap==1 {print lin ; zap=0; next}
/my pattern/ {zap=1; print $0; next}
{print $0} ' infile > newfile
the awk internal variable lin, defined: -v lin="$var" is named lin, lin comes from the external bash variable var.

sed -i '/pattern/r/dev/stdin' file
the script will wait for us to enter the newline and press ctrl +d interactively +

cat file
first
second
third
fourth
fiveth
set var = "inserted"
sed '/second/a\'$var file
first
second
inserted
third
fourth
fiveth

Resources