Rename Multiple Files - Shell script - shell

I have 66 files in my folder and I want to rename all of them by using their current names' parts.
For example;
My file: Subscriber_ID_List_2015_11_14.csv
I want it like this -> TheFile_20151114.csv
I tried the code below but it did not work. How could I do this?
Thank you
#!/bin/bash
FILES=/my/path/
for f in $FILES
do
cd /my/path/
b=`ls "$f" |awk -F" " '{print $5}' |cut -c6-9`
c=`ls "$f" |awk -F" " '{print $5}' |cut -c11-12`
d=`ls "$f" |awk -F" " '{print $5}' |cut -c14-15`
cp "$f" TheFile_${b}${c}${d}.csv
done

Can u try this one.
#!/bin/bash
for files in `ls Subscriber_ID_List_*.csv`
do
cd /my/path/
datePrefix=`echo $files | awk -F '.' '{ print $1}'| awk -F '_' '{ print $4$5$6}'`
cp "$files" "TheFile_${datePrefix}.csv"
done

FILES=/my/path/
This puts only the path into the variable, not the filenames. Provided that you want to process all files in /my/path/, you can use FILES=/my/path/*.
b=`ls "$f" |awk -F" " '{print $5}' |cut -c6-9`
c=`ls "$f" |awk -F" " '{print $5}' |cut -c11-12`
d=`ls "$f" |awk -F" " '{print $5}' |cut -c14-15`
This doesn't take into account that $f still contains /my/path/, uses space as a field separator without discernible cause, and cuts at the wrong character positions. Here's an easier approach:
#!/bin/bash
FILES=my/path/*
for f in $FILES
do newf=`sed 's/Subscriber_ID_List_\(....\)_\(..\)_\(..\).csv/TheFile_\1\2\3.csv/' <<<$f`
cp "$f" "$newf"
done

Related

Rename file as per fixed patter in UNIX

I have these files in mydir:
APPLE_STORE_iphone12.csv
APPLE_STORE_iphonex.csv
APPLE_STORE_ipad.csv
APPLE_STORE_imac.csv
Need to rename the files after a matching pattern "APPLE_STORE_".
Required O/P
APPLE_STORE_NY_iphone12_20210107140443.csv
APPLE_STORE_NY_iphonex_20210107140443.csv
APPLE_STORE_NY_ipad_20210107140443.csv
APPLE_STORE_NY_imac_20210107140443.csv
Here is what I tried:
filelist=/mydir/APPLE_STORE_*.csv
dtstamp=`date +%Y%m%d%H%M%S`
location='NY'
for file in ${filelist}
do
filebase=${file%.csv}
mv ${file} ${filebase}_${location}_${dtstamp}.csv
done
This is giving me name like APPLE_STORE_imac_NY_20210107140443.csv
Another (maybe not so elegant) way is to first explicitly divide the filename in its parts using awk with separator "_" and then build it up again as needed. Your script could then look like:
#!/bin/bash
filelist=./APPLE_STORE_*.csv
dtstamp=`date +%Y%m%d%H%M%S`
location='NY'
for file in ${filelist}
do
filebase=${file%.csv}
part1=`echo ${filebase} | awk -v FS="_" '{print $1}'`
part2=`echo ${filebase} | awk -v FS="_" '{print $2}'`
part3=`echo ${filebase} | awk -v FS="_" '{print $3}'`
mv ${file} ${part1}_${part2}_${location}_${part3}_${dtstamp}.csv
done
I tested it successfully.
You are so close.
destfile="$(echo $file | sed -e 's/^APPLE_STORE/APPLE_STORE_${location}/' -e 's/\.csv$/${dtstamp}.csv/')"`
mv "$file" "$destfile"
...or something like that.

bulk write in Unix using shell script

Is there any way to write bulk data in a file in shell script instead of writing line by line code in file?
In below script, I want to write difference between arrival time and generation time of files in test.csv file.
########################################################
echo "Starting the Execution for Time difference\n";
############################################################
# Functions used across the script
datediff() {
Unixtime=`echo $1 $2 $3 $4`
Filetime=`echo $5 $6 $7 $8`
echo $Unixtime;
echo $Filetime;
d1=`date -d "$Unixtime" +%s`
d2=`date -d "$Filetime" +%s`
echo $d1;
echo $d2;
TIME_DIFF=`expr $d1 - $d2`
TIME_DIFF=`expr $TIME_DIFF / 60`
echo $TIME_DIFF;
echo "$Unixtime,$Filetime,$TIME_DIFF,$9" >> ../test.csv
}
rm -f ../test.csv;
for i in `ls -1 | grep -v 'DelayCheck.s*'`
do
DayMonth=`ls -lrt $i | awk '{print $7" "$6" "}'`
Year=`ls --full-time $i | awk '{print $6}' | cut -c1-4`
HourMin=`ls -lrt $i | awk '{print " "$8}'`
timeA=`echo $DayMonth $Year $HourMin`
FileYearMonDay=`ls -ltr $i | awk '{print $9}' | awk -F'--' '{print $3}' | cut
-c2-9`
timeB1=`date -d $FileYearMonDay +'%d %b %Y'`
timeB2=`echo $i | awk -F'--' '{print substr($3,10,13)}' | sed -e
's/../:&/2g'`
timeB=`echo $timeB1 $timeB2`
echo "Time A is $timeA";
echo " Time b is $timeB";
datediff $timeA $timeB $i
done
echo $?;
script is working fine, but the problem is there is over 100k files. So script performance is bad.
I had tried to search is there any way to write bulk data in a file but I didn't find any solution.

How to remove the usage of temp file and read data from the command itself

I have a shell script and I need help to make it efficient. I am using temp files to store and read the data, but I need to read the data in memory.
It collects metrics from the Postgres database using a command and fetches the metrics. My current script fetches the metrics to a temp file, then reads from it.
I want to stop using temp files and use memory instead.
The script works, I just need help to automate more and get rid of reading data from temp files.
List item
INPUT=`mktemp`
#/usr/pgsql-9.5/bin/pgbench -c1 -j1 -t 1000 -S man > $INPUT
TESTTIME=15 #seconds
echo "Waiting $TESTTIME seconds..."
/usr/pgsql-9.5/bin/pgbench -c1 -j1 -T $TESTTIME -r man > $INPUT
OLDIFS=$IFS
IFS=" "
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
tps=`cat $INPUT |awk '/^tps/ {print $3}' |awk -F'.' '{print $1}' |head -n1`
update_l=`cat $INPUT |awk '/UPDATE/ {print $1}' |tail -n1`
select_l=`cat $INPUT |awk '/SELECT/ {print $1}' |tail -n1`
insert_l=`cat $INPUT |awk '/INSERT/ {print $1}' |tail -n1`
echo ${PLOTTER_PREFIX}.tps $tps kv
echo ${PLOTTER_PREFIX}.update_latency $update_l kv
echo ${PLOTTER_PREFIX}.select_latency $select_l kv
echo ${PLOTTER_PREFIX}.insert_latency $insert_l kv
#{ while read line; do
# # statsite_buildData ${PLOTTER_PREFIX}.latency average ${latency average} kv
# echo ${PLOTTER_PREFIX}.${line} kv
# done } < $INPUT
statsite_sendData
#echo $Test
IFS=$OLDIFS
rm -f $INPUT
You can capture the output of the command to a variable, like so:
output=$(/usr/pgsql-9.5/bin/pgbench -c1 -j1 -T $TESTTIME -r man)
Then just use echo instead of cat and substitute $INPUT with the variable name.
tps=`echo "$output" | awk '/^tps/ {print $3}' | awk -F'.' '{print $1}' |head -n1`
update_l=`echo "$output" | awk '/UPDATE/ {print $1}' | tail -n1`
...
I would also suggest using $() instead of surrounding commands with backticks. So the above would become:
tps=$(echo "$output" | awk '/^tps/ {print $3}' | awk -F'.' '{print $1}' |head -n1)
update_l=$(echo "$output" | awk '/UPDATE/ {print $1}' | tail -n1)
...

Find TXT files and show Total Count of records of each file and Size of each file

I need to find row Count and size of each TXT files.
It needs to search all the directories and just show result as :
FileName|Cnt|Size
ABC.TXT|230|23MB
Here is some code:
v_DIR=$1
echo "the directory to cd is "$1
x=`ls -l $0 | awk '{print $9 "|" $5}'`
y=`awk 'END {print NR}' $0`
echo $x '|' $y
Try something like
find -type f -name '*.txt' -exec bash -c 'lines=$(wc -l "$0" | cut -d " " -f1); size=$(du -h "$0" | cut -f1); echo "$0|$lines|$size"' {} \;

extract information regarding : size && time && row_count in one line shell script

Hey every one! I am pretty new for shell script and I am stuck
I need to extract information regarding: file_name && size && time && row_count and I want it do in one command line. I tried like this :
ls -l * && wc -l file.txt && du -ks file.txt | cut -f1| awk '{print $5" " $6 " " $7 " "$8 " " $9 " "$1 " "$2}'
but is not working properly
I also tried do in loop but i dont know how extract from there
for file in `ls -ltr /export/home/oracle/dbascripts/scripts`
do
[[ -f $file ]] && echo $file | awk '{print $3}'
done
Then I want to redirect to file like this >> for sql loader purpose.
Thanks in advance!
This could be a start if you have GNU find and GNU coreutils (most Linux distribution will do):
for i in /my/path/*; do
find "$i" ! -type d -printf '%p %TY-%Tm-%Td %TH:%TM:%TS %s '
wc -l <"$i"
done
/my/path/* should be modified to reflect the files you want to probe.
Also keep in mind that this one-liner has a few major issues if any directories are specified. This should be safer in that regard:
for i in *; do
if [[ -d "$i" ]]; then
continue
fi
find "$i" -printf '%p %TY-%Tm-%Td %TH:%TM:%TS %s '
wc -l <"$i"
done
You will want to see the manual page for GNU find to understand this better.
EDIT:
There is at least other faster way, using join and bash process substitution, but it's a bit ugly and somewhat harder to make safe and work the kinks out of.
ExtractInformation()
{
timesep="-"
sep="|"
dot=":"
sec="00"
lcount=`wc -l < $fname`
modf_time=`ls -l $fname`
f_size=`echo $modf_time | awk '{print $5}'`
time_month=`echo $modf_time | awk '{print $6}'`
time_day=`echo $modf_time | awk '{print $7}'`
time_hrmin=`echo $modf_time | awk '{print $8}'`
time_hr=`echo $time_hrmin | cut -d ':' -f1`
time_min=`echo $time_hrmin | cut -d ':' -f2`
time_year=`date '+%Y'`
time_param="DD-MON-YYYY HH24:MI:SS"
time_date=$time_day$timesep$time_month$timesep$time_year" "$time_hrmin$dot$sec
result=$fname$sep$time_date$sep$f_size$sep$lcount$sep$time_param
sqlresult=`echo $result | awk '{FS = "|" ;q=sprintf("%c", 39); print "INSERT INTO SIP_ICMS_FILE_T(f_name, f_date_time,f_size,f_row_count) VALUES (" q $1 q ", TO_DATE("q $2 q,q $5 q "),"$3","$4");";}'`
echo $sqlresult>>data.sql
echo "Reading data....."
}
UploadData()
{
#ss=`sqlplus -s a/a#adb #data.sql
#set serveroutput on
#set feedback off
#set echo off`
echo "loading with sql Loader....."
}
f_data=data.sql
[[ -f $f_data ]] && rm data.sql
for fname in * ;
do
if [[ -f $fname ]] then
ExtractInformation
fi
UploadData
#Zipdata
done

Resources