I am writing a small programme whose functionality is like following. It uses a file named hist1.dat and the first step is like this
awk '{if(NR>1) s+=$2*($1-x); x=$1}END{print s}' hist1.dat >int1.txt
The hist1.dat is like following
0.259990113102 4
0.261752339307 10
0.263514565512 15
0.265276791717 35
0.267039017922 58
0.268801244127 84
0.270563470333 147
0.272325696537 217
0.274087922742 316
0.275850148947 410
0.277612375152 583
0.279374601357 750
0.281136827562 881
0.282899053767 1004
0.284661279972 1241
The int1.txt contains a value 79.9504 . The second step uses this value and process it further like following
awk '{if(NR>1) s+=$2*($1-x); print $1,s/c1; x=$1}' c1="$(cat int1.txt)" hist1.dat >cutoff1_org.txt
The cutoff1_org.txt looks like following
0.321668030277 0.832661
0.323430256483 0.851265
0.325192482688 0.867267
0.326954708893 0.882167
0.328716935097 0.895634
0.330479161302 0.906677
0.332241387507 0.917587
0.334003613712 0.927175
0.335765839917 0.935287
0.337528066122 0.9432
0.339290292327 0.94968
0.341052518532 0.955675
0.342814744737 0.961097
0.344576970942 0.96555
0.346339197147 0.969231
The next script uses the cutoff1_org.txt file and tries to find a value in column 1 if the corresponding value in column 2 is close to 0.95. This works well too
awk -v c=2 -v t=0.95 'NR==1{d=$c-t;d=d<0?-d:d;v=$c;next}{m=$c-t;m=m<0?-m:m}m<d{d=m;v=$1}END{print v}' cutoff1_org.txt >final_cutoff1.txt
The next two scripts will just use values from final_cutoff1.txt. Something like following
awk '{ if ($1 >= cutoff1) print $1 }' cutoff1="$(cat final_cutoff1.txt)" hist1.dat >hist_oc1.dat
awk '{ if ($1 <= cutoff1) print $1 }' cutoff1="$(cat final_cutoff1.txt)" hist1.dat >hist_uc1.dat
Now I want to put inside a loop like following
for i in {1..22}; do
awk '{if(NR>1) s+=$2*($1-x); x=$1}END{print s}' hist${i}.dat >int${i}.txt
awk '{if(NR>1) s+=$2*($1-x); print $1,s/c${i}; x=$1}' c${i}="$(cat int${i}.txt)" hist${i}.dat >cutoff${i}_org.txt
awk -v c=2 -v t=0.95 'NR==1{d=$c-t;d=d<0?-d:d;v=$c;next}{m=$c-t;m=m<0?-m:m}m<d{d=m;v=$1}END{print v}' cutoff${i}_org.txt >final_cutoff${i}.txt
awk '{ if ($1 >= cutoff${i}) print $1 }' cutoff${i}="$(cat final_cutoff${i}.txt)" hist${i}.dat >hist_oc${i}.dat
awk '{ if ($1 <= cutoff${i}) print $1 }' cutoff${i}="$(cat final_cutoff${i}.txt)" hist${i}.dat >hist_uc${i}.dat
cat hist_oc${i}.dat |stdev >stat_oc${i}.txt
cat hist_uc${i}.dat |stdev >stat_uc${i}.txt
done
However I got an error like following
awk: cmd. line:1: {if(NR>1) s+=$2*($1-x); print $1,s/c${i}; x=$1}
awk: cmd. line:1: ^ syntax error
awk: cmd. line:1: {if(NR>1) s+=$2*($1-x); print $1,s/c${i}; x=$1}
awk: cmd. line:1: ^ syntax error
awk: cmd. line:1: { if ($1 >= cutoff${i}) print $1 }
awk: cmd. line:1: ^ syntax error
I think I have brain fade now. Seems like very easy error to fix. Can anyone please point out and help with it. Thanks a lot in advance
Related
cat file1
xizaoshuijiao #E0488_5#
chifandaqiu #E0488_3#
gongzuoyouxi #E0977_5#
cat file2
#E0488_3# #E0488_3#
#E0488_5# #E0488_5#
#E0977_3# #E0977_3#
#E0977_5# #E0977_5#
#E0977_6# #E0977_6#
Purpose:if $NF in file1 found in file2 $1, than replace $NF in file1 with file2 $2.otherwise, makes no change.
My code:
awk '\
NR==FNR{a[$1]=$1;b[$2]=$2;next}\
{if($NF in a)\
{$NF=b[FNR];print $0}\
else if!($NF in a)\
{print $0}\
}' file2 file1
Then it came error:
awk: cmd. line:5: else if!($NF in a)\
awk: cmd. line:5: ^ syntax error
awk: cmd. line:6: {print $0}\
awk: cmd. line:6: ^ syntax error
So it seems that "!" issue. because I want to print all content in file1(both changed line and unchanged line).How can I do it ?
you can rewrite it in this form
awk 'NR==FNR {a[$1]=$2; next}
$NF in a {$2=a[$1]}1' file2 file1
since your file2 has the same values for $1 and $2, it seems useless.
Since you want to print unconditionally, don't print in the condition block. Here 1 corresponds to {print} which is the same as {print $0}
Replace:
if!($NF in a)
With:
if(!($NF in a))
! is part of the test-condition and awk expects the test-condition to all be inside parens.
Here comes my code after verification.
awk '\
NR==FNR{a[$1]=$1;b[$2]=$2;next}\
{if($NF in a)\
{$NF=b[FNR];print $0}\
else # use else... it will work, no need else if... , but why ? How can I achieve it with else if !($NF in a)
{print $0}\
}' file2 file1
I have a CSV like this:
1,"Paris","3.57"
10,"Singapore","3.57"
211,"Sydney","3.28"
324,"Toronto Center","3.33"
I'd like to pad the first column with zeroes to get:
001,"Paris","3.57"
010,"Singapore","3.57"
211,"Sydney","3.28"
324,"Toronto Center","3.33"
I tried to assign the first column to the output of printf with awk:
awk '{ $1 = printf("%03d", $1); print }' my.csv
But it gives me a syntax error :
awk: cmd. line:1: { $1 = printf("%03d", $1); print }
awk: cmd. line:1: ^ syntax error
It doesn't work either if I quote the printf function.
How could I do that?
If you want just to format the text of one field then you can use sprintf of awk.
awk '{ $1=sprintf("%03d", $1)}1' csvfile
Or standard way:
awk '{printf "%03d %s\n", $1,$2}' csvfile
As per update by OP in question:
awk 'BEGIN{FS=OFS=","}{ $1=sprintf("%03d", $1)}1' csvfile
printf is not a function, it is a keyword, and its result cannot be assigned.
To return a formatted string, use sprintf (which is a function):
awk -F, -v OFS=, '{ $1 = sprintf("%03d", $1) } 1' file
It is necessary to set FS (via -F) and OFS so that when awk reformats the line, the field separators remain intact.
As pointed out in the comments, using %d can potentially lead to problems when the input starts with a 0, as numbers with a leading 0 are interpreted as octal. This can break on input like 08 because 8 is outside of the octal range (0-7).
One way to get around this is to use %03.0f, which interprets the input as a floating point value, with the output precision set to 0:
awk -F, -v OFS=, '{ $1 = sprintf("%03f.0", $1) } 1' file
(the second 0 in the format specifier can in fact be omitted)
awk '{printf("%03d", $1) ; print " "$2}' my.csv
I have a csv file like the following example:
fieldname1: "This is field1", "id":55, fieldname2: "This is field2", "id":66
I would like to replace the fourth field from ""id":66" to ""id":72" using the awk command. I have tried it the following way but am getting a syntax error:
awk -F, '{${4}="\"id\":999";}1' OFS=, rule.txt
The error is:
awk: {${4}="\"id\":999";}1
awk: ^ syntax error
awk: {${4}="\"id\":999";}1
awk: ^ syntax error
awk: cmd. line:1: {${4}="\"id\":999";}1
awk: cmd. line:1: ^ unexpected newline or end of string
Any suggestions for correct way of doing this?
You just need to say $4 instead of ${4}:
$ awk -F, '{$4="\"id\":999";}1' OFS=, file
# ^^
fieldname1: "This is field1", "id":55, fieldname2: "This is field2","id":999
If you want to give the value via a variable, use -v value="$bash_var" as usual:
$ awk -F, -v val=999 '{$4="\"id\":" val;}1' OFS=, file
# ^^^^^^^^^^ ^^^^^
fieldname1: "This is field1", "id":55, fieldname2: "This is field2","id":999
Note that ${ } is used in Bash to avoid confusion when using a variable $hello being confused with $hello_you when saying eg echo "$hello_you" -> in that case, you would say echo "${hello}_you" to define the scope of the name of the variable.
But in awk such thing shouldn't be necessary because you enclose the string part in double quotes:
$ awk 'BEGIN {a=23; print a"_b"}'
23_b
I have simple bash script:
#!/bin/sh
column=${1:-1}
awk ' {colawk='$column'+2; print $colawk}'
awk '(x=4; print $x)'
But I have received error:
awk: (x=4; print $x)
awk: ^ syntax error
awk: cmd. line:1: (x=4; print $x)
awk: cmd. line:1: ^ unexpected newline or end of string
Why? Code in the previous line works.
An AWK program is a series of pattern action pairs, written as:
condition { action }
where condition is typically an expression and action is a series of commands.
print is not expression but a statement, so it's a syntax error as expected.
Your problem is with using parentheses instead of braces. Try:
awk '{x=4; print $x}'
instead, as in the following transcript:
pax$ echo a b c d e | awk '(x=4; print $x)'
awk: cmd. line:1: (x=4; print $x)
awk: cmd. line:1: ^ syntax error
awk: cmd. line:2: (x=4; print $x)
awk: cmd. line:2: ^ unexpected newline or end of string
pax$ echo a b c d e | awk '{x=4; print $x}'
d
I am writing a shell script which needs to pull values out of a text file which looks like this:
app.full.name /warfilelocation/ warfilename
My shell script will be iterating over a list of application names and pulling out either the location or name using AWK. I have tested doing this on the command line using the following:
awk "\$1 ~/app.full.name/ { print $2 }" applications.txt
which returns what I would expect however when i put this in a shell script I start having issues.
I have a function that looks like this:
function get_location() {
local application=$1
awk "\$1 ~/^$application/ { print \$2 }" applications.txt
}
But when i call this function i get the following error:
awk: $1 ~/^app.full.name
awk: ^ unterminated regexp
awk: cmd. line:1: app.full.name
awk: cmd. line:1: ^ syntax error
awk: cmd. line:2: app.full.name/ { print $2 }
awk: cmd. line:2: ^ syntax error
Does anyone have any ideas what I am doing wrong here. I presume I am not escaping the variable correct but no matter what i try it doesnt seem to work.
Thanks in advance
Use this approach to make awk recognize shell variables:
awk -v "v1=$VAR1" -v "v2=$VAR2" '{print v1, v2}' input_file
Update
$ cat input
tinky-winky
dipsy
laa-laa
noo-noo
po
$ teletubby='po'
$ awk -v "regexp=$teletubby" '$0 ~ regexp' input
po
Note that anything could go into the shell-variable,
even a full-blown regexp, e.g ^d.*y. Just make sure to use single-quotes
to prevent the shell from doing any expansion.
The error messages seem to indicate that there is a stray newline at the end of $application, which gives the "line 2" error messages.
see this: using awk match() function
kent$ app=app.ful
kent$ echo "app.full.name /warfilelocation/ warfilename"|awk -v a=$app '{if(match($1,a))print $2}'
/warfilelocation/
It's hard to tell without knowing exactly the value of $application, but it seems like you have a strange character in $application, such as a " or a / or something like that.
$ export application=foo/bar
$ awk "\$1 ~/^$application/ { print \$1 }"
gawk: cmd. line:1: $1 ~/^foo/bar/ { print $1 }
gawk: cmd. line:1: ^ parse error
I would look at the exact value that you have in $application, and if it contains a /, escape it.
One way to do this would be to use:
$ export application=`echo foo/bar | sed -e 's;/;\\\\/;g'`
$ awk "\$1 ~/^$application/ { print \$1 }"