I have these files in mydir:
APPLE_STORE_iphone12.csv
APPLE_STORE_iphonex.csv
APPLE_STORE_ipad.csv
APPLE_STORE_imac.csv
Need to rename the files after a matching pattern "APPLE_STORE_".
Required O/P
APPLE_STORE_NY_iphone12_20210107140443.csv
APPLE_STORE_NY_iphonex_20210107140443.csv
APPLE_STORE_NY_ipad_20210107140443.csv
APPLE_STORE_NY_imac_20210107140443.csv
Here is what I tried:
filelist=/mydir/APPLE_STORE_*.csv
dtstamp=`date +%Y%m%d%H%M%S`
location='NY'
for file in ${filelist}
do
filebase=${file%.csv}
mv ${file} ${filebase}_${location}_${dtstamp}.csv
done
This is giving me name like APPLE_STORE_imac_NY_20210107140443.csv
Another (maybe not so elegant) way is to first explicitly divide the filename in its parts using awk with separator "_" and then build it up again as needed. Your script could then look like:
#!/bin/bash
filelist=./APPLE_STORE_*.csv
dtstamp=`date +%Y%m%d%H%M%S`
location='NY'
for file in ${filelist}
do
filebase=${file%.csv}
part1=`echo ${filebase} | awk -v FS="_" '{print $1}'`
part2=`echo ${filebase} | awk -v FS="_" '{print $2}'`
part3=`echo ${filebase} | awk -v FS="_" '{print $3}'`
mv ${file} ${part1}_${part2}_${location}_${part3}_${dtstamp}.csv
done
I tested it successfully.
You are so close.
destfile="$(echo $file | sed -e 's/^APPLE_STORE/APPLE_STORE_${location}/' -e 's/\.csv$/${dtstamp}.csv/')"`
mv "$file" "$destfile"
...or something like that.
Related
There is a large log file of 10GB, and formatted as following:
node123`1493000000`POST /api/info`app_id=123&token=123&sign=abc
node456`1493000000`POST /api/info`app_id=456&token=456&sign=abc
node456`1493000000`POST /api/info`token=456&app_id=456&sign=abc
node456`1493000000`POST /api/info`token=456&sign=abc&app_id=456
Now I want to get unique app_ids from the log file. For example, the expected result of the log file above should be:
123
456
I do that with shell script awk -F 'app_id=' '{print $2}' $filename | awk -F '&' '{print $1}' | sort | uniq, and is there a more efficient way?
If your log file's name is log_file.txt,you can use these commands:
grep -Po "(?<=&app_id=)[0-9]+" log_file.txt
awk -F "[&=]" '{print $4}' log_file.txt
Change the logfile name
awk '{print $17" "$18" "$19" "$20}' log.txt |sort -k1|uniq >> z #apache
# filename on line number(0-9) awk result
while read x;
do
echo $x
grep "$x" log.txt | wc -l
done < z
I have 66 files in my folder and I want to rename all of them by using their current names' parts.
For example;
My file: Subscriber_ID_List_2015_11_14.csv
I want it like this -> TheFile_20151114.csv
I tried the code below but it did not work. How could I do this?
Thank you
#!/bin/bash
FILES=/my/path/
for f in $FILES
do
cd /my/path/
b=`ls "$f" |awk -F" " '{print $5}' |cut -c6-9`
c=`ls "$f" |awk -F" " '{print $5}' |cut -c11-12`
d=`ls "$f" |awk -F" " '{print $5}' |cut -c14-15`
cp "$f" TheFile_${b}${c}${d}.csv
done
Can u try this one.
#!/bin/bash
for files in `ls Subscriber_ID_List_*.csv`
do
cd /my/path/
datePrefix=`echo $files | awk -F '.' '{ print $1}'| awk -F '_' '{ print $4$5$6}'`
cp "$files" "TheFile_${datePrefix}.csv"
done
FILES=/my/path/
This puts only the path into the variable, not the filenames. Provided that you want to process all files in /my/path/, you can use FILES=/my/path/*.
b=`ls "$f" |awk -F" " '{print $5}' |cut -c6-9`
c=`ls "$f" |awk -F" " '{print $5}' |cut -c11-12`
d=`ls "$f" |awk -F" " '{print $5}' |cut -c14-15`
This doesn't take into account that $f still contains /my/path/, uses space as a field separator without discernible cause, and cuts at the wrong character positions. Here's an easier approach:
#!/bin/bash
FILES=my/path/*
for f in $FILES
do newf=`sed 's/Subscriber_ID_List_\(....\)_\(..\)_\(..\).csv/TheFile_\1\2\3.csv/' <<<$f`
cp "$f" "$newf"
done
$ cat file
11 asasaw121
12 saasks122
13 sasjaks22
$ cat no
while read line
do
var=$(awk '{print $1}' $line)
echo $var
done<file
$ cat yes
while read line
do
var=$(echo $line | awk '{print $1}')
echo $var
done<file
$ sh no
awk: can't open file 11
source line number 1
awk: can't open file 12
source line number 1
awk: can't open file 13
source line number 1
$ sh yes
11
12
13
Why doesn't the first one work? What does awk expect to find in $1 in it? I think understanding this will help me avoid numerous scripting problems.
awk always expects a file name as input
In following, $line is string not a file.
var=$(awk '{print $1}' $line)
You could say (Note double quotes around variable)
var=$(awk '{print $1}' <<<"$line")
Why doesn't the first one work?
Because of this line:
var=$(awk '{print $1}' $line)
Which assumes $line is a file.
You can make it:
var=$(echo "$line" | awk '{print $1}')
OR
var=$(awk '{print $1}' <<< "$line")
awk '{print $1}' $line
^^ awk expects to see a file path or list of file paths here
what it is getting from you is the actual file line
What you want to do is pipe the line into awk as you do in your second example.
You got the answers to your specific questions but I'm not sure it's clear that you would never actually do any of the above.
To print the first field from a file you'd either do this:
while IFS= read -r first rest
do
printf "%s\n" "$first"
done < file
or this:
awk '{print $1}' file
or this:
cut -d ' ' -f1 <file
The shell loop would NOT be recommended.
I have a file file.txt having the following structure:-
./a/b/c/sdsd.c
./sdf/sdf/wer/saf/poi.c
./asd/wer/asdf/kljl.c
./wer/asdfo/wer/asf/asdf/hj.c
How can I get only the c file names from the path.
i.e., my output will be
sdsd.c
poi.c
kljl.c
hj.c
You can do this simpy with using awk.
set field seperator FS="/" and $NF will print the last field of every record.
awk 'BEGIN{FS="/"} {print $NF}' file.txt
or
awk -F/ '{print $NF}' file.txt
Or, you can do with cut and unix command rev like this
rev file.txt | cut -d '/' -f1 | rev
You can use basename command:
basename /a/b/c/sdsd.c
will give you sdsd.c
For a list of files in file.txt, this will do:
while IFS= read -r line; do basename "$line"; done < file.txt
Using sed:
$ sed 's|.*/||g' file
sdsd.c
poi.c
kljl.c
hj.c
The most simple one ($NF is the last column of current line):
awk -F/ '{print $NF}' file.txt
or using bash & parameter expansion:
while read file; do echo "${file##*/}"; done < file.txt
or bash with basename :
while read file; do basename "$file"; done < file.txt
OUTPUT
sdsd.c
poi.c
kljl.c
hj.c
Perl solution:
perl -F/ -ane 'print $F[#F-1]' your_file
Also you can use sed:
sed 's/.*[/]//g' your_file
I am trying to grep out the lines in a file where the third field matches certain criteria.
I tried using grep but had no luck in filtering out by a field in the file.
I have a file full of records like this:
12794357382;0;219;215
12795287063;0;220;215
12795432063;0;215;220
I need to grep only the lines where the third field is equal to 215 (in this case, only the third line)
Thanks a lot in advance for your help!
Put down the hammer.
$ awk -F ";" '$3 == 215 { print $0 }' <<< $'12794357382;0;219;215\n12795287063;0;220;215\n12795432063;0;215;220'
12795432063;0;215;220
grep:
grep -E "[^;]*;[^;]*;215;.*" yourFile
in this case, awk would be easier:
awk -F';' '$3==215' yourFile
A solution in pure bash for the pre-processing, still needing a grep:
while read line; do
OLF_IFS=$IFS; IFS=";"
line_array=( $line )
IFS=$OLD_IFS
test "${line_array[2]}" = 215 && echo "$line"
done < file | grep _your_pattern_
Simple egrep (=grep -E)
egrep ';215;[0-d][0-d][0-d]$' /path/to/file
or
egrep ';215;[[:digit:]]{3}$' /path/to/file
How about something like this:
cat your_file | while read line; do
if [ `echo "$line" | cut -d ";" -f 3` == "215" ]; then
# This is the line you want
fi
done
Here is the sed version to grep for lines where 3rd field is 215:
sed -n '/^[^;]*;[^;]*;215;/p' file.txt
Simplify your problem by putting the 3rd field at the beginning of the line:
cut -d ";" -f 3 file | paste -d ";" - file
then grep for the lines matching the 3rd field and remove the 3rd field at the beginning:
grep "^215;" | cut -d ";" -f 2-
and then you can grep for whatever you want. So the complete solution is:
cut -d ";" -f 3 file | paste -d ";" - file | grep "^215;" | cut -d ";" -f 2- | grep _your_pattern_
Advantage: Easy to understand; drawback: many processes.