I have a large csv files , where few columns values are in hex. I need to convert them into decimal. The CSV files are very big. So If I process each row , then it takes a lot of time to execute the script. So I want to know how this can be done parallely by using awk command
If I process the code line by line it works.
I process the files like this.
while read -r line;
do
start_time=`echo "$line"|awk -F "," '{ print $1 }'`
end_time=`echo "$line"|awk -F "," '{ print $2 }'`
st_time=$((16#$start_time))
en_time=$((16#$end_time))
Then I echo the required fields to output file.
Sample Input file:
16a91f90539,16a91f931a9,e,0
16a91f90bab,16a91f931a9,e,0
Expected output:
1557227177273,1557227188649,e,0
1557227178923,1557227188649,e,0
I need to know how the statement "((16#$start_time))" , can be used in awk.
I tried
awk -F',' '{OFS=",";}{print '"(($1#16))"','$en_time',$3'
But this syntax does not work.
With GNU awk for strtonum() you don't need to spawn multiple shells on each input line:
$ awk 'BEGIN{FS=OFS=","} {for (i=1;i<=2;i++) $i=strtonum("0x"$i)} 1' file
1557227177273,1557227188649,e,0
1557227178923,1557227188649,e,0
You can execute system calls from withnig awk with system(...). Don't forget to close the command afterwards.
awk -F "," '{ cmd=sprintf("echo $((0x%s))\n", $1); system(cmd); close(cmd); }' input
(for some reason the system call does not work with $((16#...)) on my system, but does work with $((0x...)))
With getline you can assign the echo'ed output to a variable. See https://www.gnu.org/software/gawk/manual/html_node/Getline-Notes.html to get you started.
Related
I have a directory with many files in it and want to edit each file to only contain a select few columns.
I have the following code which will only print the first column
for i in /directory_path/*.txt; do awk -F "\t" '{ print $1 }' "$i"; done
but if I try to edit each file by adding >'$I' as below then I lose all the information in my files
for i in /directory_path/*.txt; do awk -F "\t" '{ print $1 }' "$i" > "$i"; done
However I want to be able to remove all but a select few columns in each file for example 1 and 3.
Given:
cat file
1 2 3
4 5 6
You can do in place editing with sed:
sed -i.bak -E 's/^([^[:space:]]*).*/\1/' file
cat file
1
4
If you want freedom to work with multiple columns and have in place editing, use GNU awk that also supports in place editing:
gawk -i inplace '{print $1, $3}' file
cat file
1 3
4 6
If you only have POSIX awk or wanted to use cut you generally do this:
Modify the file with awk, cut, sed, etc
Redirect the output to a temp file
Rename the temp file back to the original file name.
Like so:
awk '{print $1, $3}' file >tmp_file; mv tmp_file file
Or with cut:
cut -d ' ' -f 1,3 file >tmp_file; mv tmp_file file
To do a loop on files in a directory, you would do:
for fn in /directory_path/*.txt; do
awk -F '\t' '{ print $1 }' "$fn" >tmp_file
mv tmp_file "$fn"
done
Just to add a little more to #dawg's perfectly well working answer according to my use case.
I was dealing with CSVs, and standard CSV can have , in some values as long as it's in double quotes like for example, the below-mentioned row will be a valid CSV row.
col1,col2,col2
1,abc,"abc, inc"
But the command above was treating the , between the double quotes as delimiter too.
Also, the output file delimiter wasn't specified in the command.
These are the modifications I had to make for it handle the above two problems:
for fn in /home/ubuntu/dir/*.csv; do
awk -F ',' '{ FPAT = "([^,]*)|(\"[^\"]+\")"; OFS=","; print $1,$2 }' "$fn" >tmp_file
mv tmp_file "$fn"
done
The OSF delimiter will be the diameter of the output/result file.
The FPAT handles the case of , between quotation mark.
The regex and the information for that is mentioned ins awk's official documentation in section 4.7 Defining Fields by Content.
I was led to that solution through this answer.
I'm trying to parse some data from a dat file using awk, however I cant get rid of a linebreak that is being added to $0. I've tried gsub(/\n/,""), but it's done nothing.
Example below:
from dat file:
<A>1
<B>2
running:
awk '
BEGIN {FS = ">"; ORS=""; OFS=""}
/<A>/ {printf $2; printf$2}
' file.dat
currently gives me:
1
1
when I want:
11
I think you just want
awk -F '>' '/<A>/ { print $2 $2 }' file.dat
This being said, your code should work, as well; the problem was that your input file contained DOS style newlines, which can be removed with, for example, dos2unix. See How to convert DOS/Windows newline (CRLF) to Unix newline (\n) in a Bash script? for more ways to do it.
For a lab, I wrote a shell script that used awk to do some stuff. Rereading the lab's directions, it seems that I was supposed to write a self-contained awk script. I'm working on translating my bash script into awk, and I'm having a problem right now:
I want to save the output of an awk command to a new file, and then I want to use that output as input for another awk command.
In my bash script, I have this:
awk '/Blocked SPAM/' maillog > spamlog
cat spamlog | awk '{print $0}' RS=' '
It takes all the lines from maillog that contain the string "Blocked SPAM" and saves this to a new file titled spamlog. Then it opens spamlog and replaces every space character ' ' with a new line.
For my awk script, maillog is the file that is passed to the script from shell. My attempt at writing analogous code:
/Blocked SPAM/ > spamlog`
-f spamlog {print $0} RS=' '
I don't really know what I'm doing with my awk script since I'm having trouble finding useful resources for self-contained awk scripts.
awk '/Blocked SPAM/{ print > "spamlog"; gsub( " ","\n"); print }' maillog
Personally, I prefer to invoke that directly from a shell script, but you can easily make it an awk script by writing:
#!/usr/bin/awk -f
/Blocked SPAM/{ print > "spamlog"; gsub( " ","\n"); print }
Invoke that script with 'maillog' as an argument.
Using the following script to access CSV items.
#!/bin/bash
awk -F "|" 'NR > 0 {print $1}' UserAgents.csv
When running the script I am getting the correct output, i.e. the entire set of values in the first 'column' of the CSV are printed to the terminal. What I would like to add is to read these items one by one and perform some operation on them like concatenate it with a string, and then output them (to file, pipe, or terminal) one by one.
This should make it clear what your awk script is doing:
awk -F '|' '{
print NR, NF, $1, "with some trailing text"
}' UserAgents.csv
I know I can redirect awk's print output to another file from within a script, like this:
awk '{print $0 >> "anotherfile" }' 2procfile
(I know that's dummy example, but it's just an example...)
But what I need is to redirect output to another file, which has a dynamic name like this
awk -v MYVAR"somedinamicdata" '{print $0 >> "MYWAR-SomeStaticText" }' 2procfile
And the outpus should be redirected to somedinamicdata-SomeStaticText.
I know I can do it via:
awk '{print $0 }' 2procfile >> "$MYVAR-somedinamicdata"
But the problem is that it's a bigger awk script, and I have to output to several files depending on certain conditions (and this awk script is called from another bash, and it passes some dynamic variable via the -v switch... and son on.
Is it possible anyhow?
Thanks in advance.
i think
awk -v MYVAR="somedinamicdata" '{print $0 >> (MYVAR "-SomeStaticText") }' 2procfile
should do it. String concatenation in awk is just put one after another.