Im trying to create a bash file to do the following task
1- I have a file name called "ORDER.CSV" need to make a copy of this file and append the date/time to file name - This I was able to get done
2- Need to edit a particular field in the new csv file I created above. Column DY and row 2. This have not been able to do. I need to insert date bash script is run in this row. Needs to be in this format DDMMYY
3- then have system upload to SFTP. This I believe I got it figured out as shown below.
#!/usr/bin/env bash
Im able to get this step done with below command
# Copies order.csv and appends file name date/time
#cp /mypath/SFTP/order.csv /mypath/SFTP/orders.`date +"%Y%m%d%H%M%S"`.csv
Need help to echo new file name
echo "new file name "
Need help to edit field under Colum DY Row 2. Need to insert current date inthis format MMDDYYYY
awk -v r=2 -v DY=3 -v val=1001 -F, 'BEGIN{OFS=","}; NR != r; NR == r {$c = val;
print}'
This should connect to SFTP, which it does with out issues.
sshpass -p MyPassword sftp -o "Port 232323"
myusername#mysftpserver.com
Need to pass new file that was created and put into SFTP server.
put /incoming/neworder/NEWFILEName.csv
Thanks
I guess that's what you want to do...
echo -e "h1,h2,h3,h4\n1,2,3,4" |
awk -v r=2 -v c=3 -v v=$(date +"%d%m%y") 'BEGIN{FS=OFS=","} NR==r{$c=v}1'
h1,h2,h3,h4
1,2,120617,4
to find the column index from the column name (not tested)
... | awk -v r=2 -v h="DY" -v v=$(date +"%d%m%y") '
BEGIN {FS=OFS=","}
NR==1 {c=$(NF+1); for(i=1;i<=NF;i++) if($i==h) {c=i; break}}
NR==r {$c=v}1'
the risk of doing this is the column name may not match, in that case this will add the value as a new column.
Related
I have many csv files that look like this:
data/0.Raw/20190401_data.csv
(Only the date in the middle of the filename changes)
I want to concatenate all these files together, but add the date as a new column in the data to be able to distinguish between the different files after merging.
I wrote a bash script that adds the full path and filename as a column in each file, and then merges into a master csv. However, I am having trouble getting rid of the path and the extension to only keep the date portion
The bash script
#! /bin/bash
mkdir data/1.merged
for i in "data/0.Raw/"*.csv; do
awk -F, -v OFS=, 'NR==1{sub(/\_data.csv$/, "", FILENAME) } NR>1{ $1=FILENAME }1' "$i" |
column -t > "data/1.merged/"${i/"data/0.Raw/"/""}""
done
awk 'FNR > 1' data/1.merged/*.csv > data/1.merged/all_files
rm data/1.merged/*.csv
mv data/1.merged/all_files data/1.merged/all_files.csv
using "sub" I was able to remove the "_data.csv" part, but as a result the column gets added as "data/0.Raw/20190401" - that is, I am having trouble removing both the part before the date as well as the part after the date.
I tried replacing sub with gensub to regex match everything except the 8 digits in the middle but that does not seem to work either.
Any ideas on how to solve this?
Thanks!
You can process and concatenate all the files with a single awk call:
awk '
FNR == 1 {
date = FILENAME
gsub(/.*\/|_data\.csv$/,"",date)
next
}
{ print date "," $0 }
' data/0.Raw/*_data.csv > all_files.csv
However, I am having trouble getting rid of the path and the extension
to only keep the date portion
Then take look at basename command
basename NAME [SUFFIX]
Print NAME with any leading directory components removed. If
specified, also remove a trailing SUFFIX.
Example
basename 'data/0.Raw/20190401_data.csv' _data.csv
gives output
20190401
For example i have the following file:
4:Oscar:Ferk:Florida
14:Steven:Pain:Texas
7:Maya:Ross:California
and so on...
It has an unknown number of lines because you can keep adding more to it.
I'm writing a script where you can edit the name by giving in the id of the person and the new name you want to give it as parameters.
What i am trying to do is use awk to find the line and then change the name on that specific line and update the file. I'm having trouble because my code updates every single column value to the given one.
My current code is:
getID=$1
getName=$2
awk -v varID="$getID" -F':' '$0~varID' file.dat | awk -v varName="$getName" -F':' '$2=varName' file.dat > tmp && mv tmp file.dat
Help is really appreciated, thank you kindly in advance.
You may use this awk:
getID=14 # change to getID="$1"
getName='John K' # change to getName="$2"
awk -v id="$getID" -v name="$getName" 'BEGIN{FS=OFS=":"} $1==id{$2=name} 1' file
4:Oscar:Ferk:Florida
14:John K:Pain:Texas
7:Maya:Ross:California
I am trying to create a script that will create a log file in which there are two columns: first is date of editing and second is path to textfile, which was edited. How can I change the first column of a line, but keep the second column containing the path? I have tried
awk '$2=="$path{$1=$date}"' logfile.txt where $path contains path of the file, but that doesn't change the date.
Thanks in advance.
logfile:
20.03.18.19.08.56 /home/ubuntu/Desktop
now lets say i edited something and now the logfile should look like:
21.03.18.19.08.56 /home/ubuntu/Desktop
you can pass the values as awk variables
$ awk -v date="$date" -v path="$path" '$2==path{$1=date}1' file > newfile
I have a CSV file sample named RecordCountTest.csv that looks like this:
Date Time Shift Record
26-06-2016 15:23:01 Shift2 000002
23-06-2016 09:06:24 Shift1 000001
When a GPIO pin wired button is pressed a bash script command is run. I would like a simple AWK or Bash script that I can call after the command is run that can document that it happened and track when. I would like to do this by inserting a row at the top (but under the header) with the current date (DD-MM-YYYY), the current time (HH:MM:SS), the shift is a variable determined by the bash script (and I will post that question in 90 minutes if I can't figure it out before then), and then increment the record by one and saving the file as RecordCountTest.CSV. I would prefer SED if possible as that is what I am trying to learn currently and any suggestions that might help me understand the syntax more, but will accept anything that can work in this bash script including AWK.
awk to the rescue!
$ gawk -v OFS="\t" -v s="${shift}" 'NR==2{print strftime("%d-%m-%Y"),
strftime("%H:%M:%S"),
s,
sprintf("%06d",$4+1)}
{$1=$1}1' file > temp && mv temp file
if your awk doesn't support strftime function, you can revert to bash
$ awk -v OFS="\t" -v s="${shift}"
-v d="$(date +"%d-%m-%Y"$'\t'"%H:%M:%S")"
'NR==2{print d, s, sprintf("%06d",$4+1)}
{$1=$1}1' file > temp && mv temp file
I have a huge text file delimited with comma.
19429,(Starbucks),390 Provan Walk,Glasgow,G34 9DL,-4.136909,55.872982
The first one is a unique id. I want the user to enter the id and enter a value for one of the following 6 fields in order to be replaced. Also, i'm asking him to enter a 2-7 value in order to identify which field should be replaced.
Now i've done something like this. I am checking every line to find the id user entered and then i'm replacing the value.
awk -F ',' -v elem=$element -v id=$code -v value=$value '{if($1==id) {if(elem==2) { $2=value } etc }}' $path
Where $path = /root/clients.txt
Let's say user enters "2" in order to replace the second field, and also enters "Whatever". Now i want "(Starbucks)" to be replaced with "Whatever" What i've done work fine but does not save the change into the file. I know that awk is not supposed to do so, but i don't know how to do it. I've searched a lot in google but still no luck.
Can you tell me how i'm supposed to do this? I know that i can do it with sed but i don't know how.
Newer versions of GNU awk support inplace editing:
awk -i inplace -v elem="$element" -v id="$code" -v value="$value" '
BEGIN{ FS=OFS="," } $1==id{ $elem=value } 1
' "$path"
With other awks:
awk -v elem="$element" -v id="$code" -v value="$value" '
BEGIN{ FS=OFS="," } $1==id{ $elem=value } 1
' "$path" > /usr/tmp/tmp$$ &&
mv /usr/tmp/tmp$$ "$path"
NOTES:
Always quote your shell variables unless you have an explicit reason not to and fully understand all of the implications and caveats.
If you're creating a tmp file, use "&&" before replacing your original with it so you don't zap your original file if the tmp file creation fails for any reason.
I fully support replacing Starbucks with Whatever in Glasgow - I'd like to think they wouldn't have let it open in the first place back in my day (1986 Glasgow Uni Comp Sci alum) :-).
awk is much easier than sed for processing specific variable fields, but it does not have in-place processing. Thus you might do the following:
#!/bin/bash
code=$1
element=$2
value=$3
echo "code is $code"
awk -F ',' -v elem=$element -v id=$code -v value=$value 'BEGIN{OFS=",";} /^'$code',/{$elem=value}1' mydb > /tmp/mydb.txt
mv /tmp/mydb.txt ./mydb
This finds a match for a line starting with code followed by a comma (you could also use ($1==code)), then sets the elemth field to value; finally it prints the output, using the comma as output field separator. If nothing matches, it just echoes the input line.
Everything is written to a temporary file, then overwrites the original.
Not very nice but it gets the job done.