Shell script awk concatenate text with entire column - bash

I have a csv file Final.csv and i am trying to change its first coulmn and add a text it each column row
like this is the csv
Trialtime type Track time Notes Athlete
07:15:00 Warmup ABC 85.2 Initial warmup Jon
07:45:00 Sprint1 ABC 59.44 First Sprint Jon
08:30:00 TRIAL ABC 57.21 Final attempt Jon
08:00:00 Warmup ABC 120.51 Initial warmup Bill
08:40:05 Sprint1 ABC 61.35 First Sprint Bill
09:15:00 Sprint2 ABC 60.08 Second Sprint Bill
10:30:00 TRIAL ABC 60.37 Final attempt Bill
and i want to add a text to the trial time coulmn like this
Trialtime type Track time Notes Athlete
2012 07:15:00 Warmup ABC 85.2 Initial warmup Jon
2012 07:45:00 Sprint1 ABC 59.44 First Sprint Jon
2012 08:30:00 TRIAL ABC 57.21 Final attempt Jon
2012 08:00:00 Warmup ABC 120.51 Initial warmup Bill
2012 08:40:05 Sprint1 ABC 61.35 First Sprint Bill
2012 09:15:00 Sprint2 ABC 60.08 Second Sprint Bill
2012 10:30:00 TRIAL ABC 60.37 Final attempt Bill
i try to print each line row in that coulmn like this with awk
awk -F',' 'NR>1{print $1}1' OFS=',' $OutFileName
NR>1 is for not including the headers
but i want to add a text variable to each row in the coulmn
lets say
text="2012"
now i want to modify the trial time column rows with this 2012 appended to each row
what i wa doing was
awk -F',' 'NR>1{$1$text;}1' OFS=',' $OutFileName
but the text variable does not work inside awk what to do ?
and i also want to keep the headers.

awk -v txt="2012" 'NR==1||$0=txt FS $0' yourFile
For the input and output example, this will do what you want.

awk solution:
awk -v t="2012" 'BEGIN{ FS=OFS="\t" }NR>1{ $1=t" "$1 }1' file
The output:
Trialtime type Track time Notes Athlete
2012 07:15:00 Warmup ABC 85.2 Initial warmup Jon
2012 07:45:00 Sprint1 ABC 59.44 First Sprint Jon
2012 08:30:00 TRIAL ABC 57.21 Final attempt Jon
2012 08:00:00 Warmup ABC 120.51 Initial warmup Bill
2012 08:40:05 Sprint1 ABC 61.35 First Sprint Bill
2012 09:15:00 Sprint2 ABC 60.08 Second Sprint Bill
2012 10:30:00 TRIAL ABC 60.37 Final attempt Bill

Related

Shell: how to right justify columns for my file

I have a data file like below:
Jones,Bill,235 S. Williams St.,Denver,CO,80221,(303) 244-7989
Smith,Tom,404 Polk Ave.,Los Angeles,CA,90003,(213) 879-5612
I want each line be separated by ',' and right justify columns, just like below
Jones Bill 235 S. Williams St. Denver CO 80221 (303) 244-7989
Smith Tom 404 Polk Ave. Los Angeles CA 90003 (213) 879-5612
This is my code and it doesn't work. Pls help, thx.
while read line
do
echo "$line" | awk -F, '{for(i=1;i<=NF;i++)printf "%15s" $i}'
echo
done < "datafile.txt"
You should not pipe each line to awk with a loop. Loops are slow and it decreases readability (have a look at Why is using a shell loop to process text considered bad practice?). Awk reads lines by default, just provide the filename as an argument:
awk -F, '{for(i=1;i<=NF;i++){printf "%22s",$i};print ""}' datafile.txt
Notice %15s would not be enough to yield a nice table, so I increased it to %22s. And you missed the comma , in printf statement, that is why it your attempt failed.
You could also use column -ts "," datafile.txt, but that would left justify the output.
Awk itself has for loop. Using a shell for loop can work too.
$ cat file
Jones,Bill,235 S. Williams St.,Denver,CO,80221,(303) 244-7989
Smith,Tom,404 Polk Ave.,Los Angeles,CA,90003,(213) 879-5612
$ awk -F, '{ for (i=1; i<=NF; i++) printf("%20s", $i); printf "\n"; }' file
Jones Bill 235 S. Williams St. Denver CO 80221 (303) 244-7989
Smith Tom 404 Polk Ave. Los Angeles CA 90003 (213) 879-5612
$
$ while read line; do echo "$line" | awk -F, '{ for (i=1; i<=NF; i++) printf("%20s", $i); printf "\n"; }'; done < file
Jones Bill 235 S. Williams St. Denver CO 80221 (303) 244-7989
Smith Tom 404 Polk Ave. Los Angeles CA 90003 (213) 879-5612
$
With column and rev:
$ rev file | column -ts, | rev
Jones Bill 235 S. Williams St. Denver CO 80221 (303) 244-7989
Smith Tom 404 Polk Ave. Los Angeles CA 90003 (213) 879-5612
or with GNU column (for -R) and seq:
$ column -ts, -R $(seq -s, 999) file
Jones Bill 235 S. Williams St. Denver CO 80221 (303) 244-7989
Smith Tom 404 Polk Ave. Los Angeles CA 90003 (213) 879-5612

reading a file into an array in bash

Here is my code
#!bin/bash
IFS=$'\r\n'
GLOBIGNORE='*'
command eval
'array=($(<'$1'))'
sorted=($(sort <<<"${array[*]}"))
for ((i = -1; i <= ${array[-25]}; i--)); do
echo "${array[i]}" | awk -F "/| " '{print $2}'
done
I keep getting an error that says "line 5: array=($(<)): command not found"
This is my problem.
As a whole my code should read in a file as a command line argument, sort the elements, then print out column 2 of the last 25 lines. I haven't been able to test this far so if there's a problem there too any help would be appreciated.
This is some of what the file contains:
290729 123456
79076 12345
76789 123456789
59462 password
49952 iloveyou
33291 princess
21725 1234567
20901 rockyou
20553 12345678
16648 abc123
16227 nicole
15308 daniel
15163 babygirl
14726 monkey
14331 lovely
14103 jessica
13984 654321
13981 michael
13488 ashley
13456 qwerty
13272 111111
13134 iloveu
13028 000000
12714 michelle
11761 tigger
11489 sunshine
11289 chocolate
11112 password1
10836 soccer
10755 anthony
10731 friends
10560 butterfly
10547 purple
10508 angel
10167 jordan
9764 liverpool
9708 justin
9704 loveme
9610 fuckyou
9516 123123
9462 football
9310 secret
9153 andrea
9053 carlos
8976 jennifer
8960 joshua
8756 bubbles
8676 1234567890
8667 superman
8631 hannah
8537 amanda
8499 loveyou
8462 pretty
8404 basketball
8360 andrew
8310 angels
8285 tweety
8269 flower
8025 playboy
7901 hello
7866 elizabeth
7792 hottie
7766 tinkerbell
7735 charlie
7717 samantha
7654 barbie
7645 chelsea
7564 lovers
7536 teamo
7518 jasmine
7500 brandon
7419 666666
7333 shadow
7301 melissa
7241 eminem
7222 matthew
In Linux you can simply do a
sort -nbr file_to_sort | head -n 25 | awk '{print $2}'
read in a file as a command line argument, sort the elements, then
print out column 2 of the last 25 lines.
From that discription of the problem, I suggest:
#! /bin/sh
sort -bn $1 | tail -25 | awk '{print $2}'
As a rule, use the shell to operate on filenames, and never use the
shell to operate on data. Utilities like sort and awk are far
faster and more powerful than the shell when it comes to processing a
file.

awk multiple csv file skip headers

I am concatenating multiple csv files with awk and modifying the rows
but i do not want the headers to print i am taking care of the headers in another method i just want awk to skip the headers
here is my code
OutFileName="Final.csv" # Fix the output name
i=0 # Reset a counter
for filename in ./*.csv; do
if [ "$filename" != "$OutFileName" ] ; # Avoid recursion
then
if [[ $i -eq 0 ]] ; then
head -1 $filename > $OutFileName # Copy header if it is the first file
fi
text=$(echo $filename | cut -d '_' -f 2-) #spilliting the string to get the date
text=$(echo $text | cut -d '.' -f -1)
awk -F',' -v txt="$text" 'FNR>1||$0=txt" "$1 FS $0' OFS=',' $filename | cat >> $OutFileName # cocatinating with awk
i=$(( $i + 1 )) #increse number
fi
done
it gives an output like this
Trialtime type Track time Notes Athlete
Trialtime type Track time Notes Athlete
20170101 07:15:00 07:15:00 Warmup ABC 85.2 Initial warmup Jon
20170101 07:45:00 07:45:00 Sprint1 ABC 59.44 First Sprint Jon
20170101 08:30:00 08:30:00 TRIAL ABC 57.21 Final attempt Jon
20170101 08:00:00 08:00:00 Warmup ABC 120.51 Initial warmup Bill
20170101 08:40:05 08:40:05 Sprint1 ABC 61.35 First Sprint Bill
20170101 09:15:00 09:15:00 Sprint2 ABC 60.08 Second Sprint Bill
20170101 10:30:00 10:30:00 TRIAL ABC 60.37 Final attempt Bill
20170101 07:15:00 07:15:00 Warmup ABC 85.2 Initial warmup Jon
20170101 07:45:00 07:45:00 Sprint1 ABC 59.44 First Sprint Jon
20170101 08:30:00 08:30:00 TRIAL ABC 57.21 Final attempt Jon
20170101 08:00:00 08:00:00 Warmup ABC 120.51 Initial warmup Bill
20170101 08:40:05 08:40:05 Sprint1 ABC 61.35 First Sprint Bill
20170101 09:15:00 09:15:00 Sprint2 ABC 60.08 Second Sprint Bill
20170101 10:30:00 10:30:00 TRIAL ABC 60.37 Final attempt Bill
Trialtime type Track time Notes Athlete
201701023 07:15:00 07:15:00 Warmup ABC 85.2 Initial warmup Jon
201701023 07:45:00 07:45:00 Sprint1 ABC 59.44 First Sprint Jon
201701023 08:30:00 08:30:00 TRIAL ABC 57.21 Final attempt Jon
201701023 08:00:00 08:00:00 Warmup ABC 120.51 Initial warmup Bill
201701023 08:40:05 08:40:05 Sprint1 ABC 61.35 First Sprint Bill
201701023 09:15:00 09:15:00 Sprint2 ABC 60.08 Second Sprint Bill
201701023 10:30:00 10:30:00 TRIAL ABC 60.37 Final attempt Bill
201701023 07:15:00 07:15:00 Warmup ABC 85.2 Initial warmup Jon
201701023 07:45:00 07:45:00 Sprint1 ABC 59.44 First Sprint Jon
201701023 08:30:00 08:30:00 TRIAL ABC 57.21 Final attempt Jon
201701023 08:00:00 08:00:00 Warmup ABC 120.51 Initial warmup Bill
201701023 08:40:05 08:40:05 Sprint1 ABC 61.35 First Sprint Bill
201701023 09:15:00 09:15:00 Sprint2 ABC 60.08 Second Sprint Bill
201701023 10:30:00 10:30:00 TRIAL ABC 60.37 Final attempt Bill
you can see awk is taking the headers each time for each csv file is there a way to ignore the headers ?
changing || with {} and using explicit print
awk -F',' -v txt="$text" 'FNR>1 { print txt" "$1 FS $0 }'
In your code, just add NR!=1 &&:
OutFileName="Final.csv" # Fix the output name
i=0 # Reset a counter
for filename in ./*.csv; do
if [ "$filename" != "$OutFileName" ] ; # Avoid recursion
then
if [[ $i -eq 0 ]] ; then
head -1 $filename > $OutFileName # Copy header if it is the first file
fi
text=$(echo $filename | cut -d '_' -f 2-) #spilliting the string to get the date
text=$(echo $text | cut -d '.' -f -1)
awk -F',' -v txt="$text" 'FNR>1||NR!=1 && $0=txt" "$1 FS $0' OFS=',' $filename | cat >> $OutFileName # cocatinating with awk
i=$(( $i + 1 )) #increse number
fi
done
This will skip the first line.

Merge 2 Files and Sort with Headers into a 3rd

I'm trying to take the 2 files below, and create a script to sort them into a new file of 3 column's with headers and some additional info.
I know the command to combine files and sort -- cat file1 file2 | sort > file3 but I don't know how to align the column's or add headings.
File 1
Dave 734.838.9800
Bob 313.123.4567
Carol 248.344.5576
Mary 313.449.1390
Ted 248-496-2204
Alice 616.556.4458
File 2
Bob Tuesday
Carol Monday
Ted Sunday
Alice Wednesday
Dave Thursday
Mary Saturday
Anticipated New File
Name On-Call Phone
Carol MONDAY 248.344.5576
Bob TUESDAY 313.123.4567
Alice WEDNESDAY 616.556.4458
Dave THURSDAY 734.838.9800
Mary SATURDAY 313.449.1390
Ted SUNDAY 248.496.2204

Combine text from two files, output to another [duplicate]

This question already has answers here:
Inner join on two text files
(5 answers)
Closed 1 year ago.
i'm having a bit of a problem and i've been searching allll day. this is my first Unix class don't be to harsh.
so this may sound fairly simple, but i can't get it
I have two text files
file1
David 734.838.9801
Roberto‭ ‬313.123.4567
Sally‭ ‬248.344.5576
Mary‭ ‬313.449.1390
Ted‭ ‬248.496.2207
Alice‭ ‬616.556.4458
Frank‭ ‬634.296.1259
file2
Roberto Tuesday‭ ‬2
Sally Monday‭ ‬8
Ted Sunday‭ ‬16
Alice Wednesday‭ ‬23
David Thursday‭ ‬10
Mary Saturday‭ ‬14
Frank Friday‭ ‬15
I am trying to write a script using a looping structure that will combine both files and come out with the output below as a separate file
output:
Name On-Call Phone Start Time
Sally Monday 248.344.5576 8am
Roberto Tuesday 313.123.4567 2am
Alice‭ Wednesday‭ 616.556.4458‭ 11pm
David‭ Thursday‭ 734.838.9801‭ 10am
Frank‭ Friday‭ 634.296.1259‭ 3pm
Mary‭ Saturday‭ 313.449.1390‭ 2pm
Ted‭ ‬ Sunday‭ 248.496.2207‭ 4pm
This is what i tried( i know its horrible)
echo " Name On-Call Phone Start Time"
file="/home/xubuntu/date.txt"
file1="/home/xubuntu/name.txt"
while read name2 phone
do
while read name day time
do
echo "$name $day $phone $time"
done<"$file"
done<"$file1"
any help would be appreciated
First, sort the files using sort and then use this command:
paste file1 file2 | awk '{print $1,$4,$2,$5}'
This will bring you pretty close. After that you have to figure out how to format the time from the 24 hour format to the 12 hour format.
If you want to avoid using sort separately, you can bring in a little more complexity like this:
paste <(sort file1) <(sort file2) | awk '{print $1,$4,$2,$5}'
Finally, if you have not yet figured out how to print the time in 12 hour format, here is your full command:
paste <(sort file1) <(sort file2) | awk '{"date --date=\"" $5 ":00:00\" +%I%P" |& getline $5; print $1 " " $4 " " $2 " " $5 }'
You can use tabs (\t) in place of spaces as connectors to get a nicely formatted output.
In this case join command will also work,
join -1 1 -2 1 <(sort file1) <(sort file2)
Description
-1 -> file1
1 -> first field of file1 (common field)
-2 -> file2
1 -> first field of file2 (common field)
**cat file1**
David 734.838.9801
Roberto 313.123.4567
Sally 248.344.5576
Mary 313.449.1390
Ted 248.496.2207
Alice 616.556.4458
Frank 634.296.1259
**cat file2**
Roberto Tuesday 2
Sally Monday 8
Ted Sunday 16
Alice Wednesday 23
David Thursday 10
Mary Saturday 14
Frank Friday 15
output
Alice 616.556.4458 Wednesday 23
David 734.838.9801 Thursday 10
Frank 634.296.1259 Friday 15
Mary 313.449.1390 Saturday 14
Roberto 313.123.4567 Tuesday 2
Sally 248.344.5576 Monday 8
Ted 248.496.2207 Sunday 16

Resources