I have a file with records in such a type-
,laac_repo,cntrylist,idlist,domlist,typelist
1,22DE17,BA,S6CD6728,24JA13,6A
2,12FE18,AA,S6FD7688,25DA15,7D
3,22DE17,BA,S6CD6728,24JA13,6A
4,12FE18,AA,S6FD7688,25DA15,7D
I want to remove duplicate records considering 4th column which has "S6CD6728" these type of record and skipping first row which is
",laac_repo,cntrylist,idlist,domlist,type list"
I have tried
awk '{a[$4]++}!(a[$4]-1)' filename
And also tried
awk 'FNR > 1 {a[$4]++}!(a[$4]-1)' filename
The expected output is-
,laac_repo,cntrylist,idlist,domlist,typelist
1,22DE17,BA,S6CD6728,24JA13,6A
2,12FE18,AA,S6FD7688,25DA15,7D
P.S file has more than 10 million records, please suggest solution w.r.t that.( If any script given much appreciated, instead of single command).
What about this:
awk -F, 'FNR>1 && \!seen[$4]++' filename
1,22DE17,BA,S6CD6728,24JA13,6A
2,12FE18,AA,S6FD7688,25DA15,7D
awk -F, '\!seen[$4]++' filename
,laac_repo,cntrylist,idlist,domlist,typelist
1,22DE17,BA,S6CD6728,24JA13,6A
2,12FE18,AA,S6FD7688,25DA15,7D
For example i have the following file:
4:Oscar:Ferk:Florida
14:Steven:Pain:Texas
7:Maya:Ross:California
and so on...
It has an unknown number of lines because you can keep adding more to it.
I'm writing a script where you can edit the name by giving in the id of the person and the new name you want to give it as parameters.
What i am trying to do is use awk to find the line and then change the name on that specific line and update the file. I'm having trouble because my code updates every single column value to the given one.
My current code is:
getID=$1
getName=$2
awk -v varID="$getID" -F':' '$0~varID' file.dat | awk -v varName="$getName" -F':' '$2=varName' file.dat > tmp && mv tmp file.dat
Help is really appreciated, thank you kindly in advance.
You may use this awk:
getID=14 # change to getID="$1"
getName='John K' # change to getName="$2"
awk -v id="$getID" -v name="$getName" 'BEGIN{FS=OFS=":"} $1==id{$2=name} 1' file
4:Oscar:Ferk:Florida
14:John K:Pain:Texas
7:Maya:Ross:California
Im trying to create a bash file to do the following task
1- I have a file name called "ORDER.CSV" need to make a copy of this file and append the date/time to file name - This I was able to get done
2- Need to edit a particular field in the new csv file I created above. Column DY and row 2. This have not been able to do. I need to insert date bash script is run in this row. Needs to be in this format DDMMYY
3- then have system upload to SFTP. This I believe I got it figured out as shown below.
#!/usr/bin/env bash
Im able to get this step done with below command
# Copies order.csv and appends file name date/time
#cp /mypath/SFTP/order.csv /mypath/SFTP/orders.`date +"%Y%m%d%H%M%S"`.csv
Need help to echo new file name
echo "new file name "
Need help to edit field under Colum DY Row 2. Need to insert current date inthis format MMDDYYYY
awk -v r=2 -v DY=3 -v val=1001 -F, 'BEGIN{OFS=","}; NR != r; NR == r {$c = val;
print}'
This should connect to SFTP, which it does with out issues.
sshpass -p MyPassword sftp -o "Port 232323"
myusername#mysftpserver.com
Need to pass new file that was created and put into SFTP server.
put /incoming/neworder/NEWFILEName.csv
Thanks
I guess that's what you want to do...
echo -e "h1,h2,h3,h4\n1,2,3,4" |
awk -v r=2 -v c=3 -v v=$(date +"%d%m%y") 'BEGIN{FS=OFS=","} NR==r{$c=v}1'
h1,h2,h3,h4
1,2,120617,4
to find the column index from the column name (not tested)
... | awk -v r=2 -v h="DY" -v v=$(date +"%d%m%y") '
BEGIN {FS=OFS=","}
NR==1 {c=$(NF+1); for(i=1;i<=NF;i++) if($i==h) {c=i; break}}
NR==r {$c=v}1'
the risk of doing this is the column name may not match, in that case this will add the value as a new column.
I have splited a file into multiple text files using below command -
awk '{print $2 > $1"_npsc.txt"}' complete.txt
I want to store all the output generated text files to another directory. How I can achieve this ? Please help.
You could do something like:
awk '{print $2 > "path/to/directory"$1"_npsc.txt"}' complete.txt
Just make sure that you create the director first (and replace path/to/directory with a the path that you like)
i want to parse a csv file in a shell script. I want to input the name of the file at the prompt. like :
somescript.sh filename
Can it be done?
Also, I will read user input to display a particular a particular data number in the csv.
For example, say the csv file has 10 values in each line:
1,2,3,4,5,6,7,8,9,10
And I want to read the 5th value. how can I do it?
And multiple lines are involved.
Thanks.
If your file is really in such a simple format (just commas, no spaces), then cut -d, -f5 would do the trick.
#!/bin/sh
awk -F, "NR==$2{print \$$3}" "$1"
Usage:
./test.sh FILENAME LINE_NUMBER FIELD_NUMBER