Change file name suffix(es) (using sed ?) - shell

I'd like to change the file name suffix from files (using a bash script), but sometimes there are files with one period and some with two.
Now I use this:
new_file=`echo ${file} | sed 's/\(.*\.log.*\)'${suf}'/\1.'${num}'/'`
Where 'new_file' is the new file name, 'file' the original file name, '${suf}' the file's suffix and ${num} a new number.
So some.log must become some.log.1 and some.log.1 must become some.log.2. With my code some.log becomes some.log.1, but some.log.1 remains some.log.1.
I hope I'm clear enough. I appreciate any advice (even not using sed).
Update:
#paxdiablo. Something went wrong testing I think.
Now I use this piece of code as test;
#!/usr/bin/bash
shft() {
for suff in {6..1} ; do
if [[ -f "$1.${suff}" ]] ; then
((nxt = suff + 1))
echo Moving "$1.${suff}" to "$1.${nxt}"
mv -f "$1.${suff}" "$1.${nxt}"
fi
done
echo Moving "$1" to "$1.1"
mv -f "$1" "$1.1"
}
clear
folder=~/logs/*.log
for i in {1..20}; do
echo ${i}> ~/logs/some.log
for fspec in ${folder} ; do
shft "${fspec}"
done
done
Every thing works fine now. Sorry for the confusion.

If you're looking to roll over log files, and depending on how complex you need to get, I've used the following segment before:
#!/usr/bin/bash
# rollover.sh
# Rolls over log files in the current directory.
# *.log.8 -> *.log.9
# *.log.7 -> *.log.8
# : : :
# *.log.1 -> *.log.2
# *.log -> *.log.1
shft() {
# Change this '8' to one less than your desired maximum rollover file.
# Must be in reverse order for renames to work (n..1, not 1..n).
for suff in {8..1} ; do
if [[ -f "$1.${suff}" ]] ; then
((nxt = suff + 1))
echo Moving "$1.${suff}" to "$1.${nxt}"
mv -f "$1.${suff}" "$1.${nxt}"
fi
done
echo Moving "$1" to "$1.1"
mv -f "$1" "$1.1"
}
for fspec in *.log ; do
shft "${fspec}"
#date >"${fspec}" #DEBUG code
done
This will automatically roll over log files up to version 9 although you can just change the suff for loop to allow more.
With that DEBUG added so new files are created automatically for testing, the following transcript shows it in action:
pax> touch qq.log ; ./rollover.sh
Moving "qq.log" to "qq.log.1"
pax> touch "has spaces.log" ; ./rollover.sh
Moving "has spaces.log" to "has spaces.log.1"
Moving "qq.log.1" to "qq.log.2"
Moving "qq.log" to "qq.log.1"
pax> ll *log*
-rw-r--r-- 1 pax None 30 2010-09-11 20:39 has spaces.log
-rw-r--r-- 1 pax None 0 2010-09-11 20:39 has spaces.log.1
-rw-r--r-- 1 pax None 30 2010-09-11 20:39 qq.log
-rw-r--r-- 1 pax None 30 2010-09-11 20:38 qq.log.1
-rw-r--r-- 1 pax None 0 2010-09-11 20:38 qq.log.2
The good thing about this script is that it's easily configurable to handle a large amount of history (by changing the {8..1} bit), handles names with spaces, and handles gaps relatively robustly if log files go missing.

To rotate logs, you should really use logrotate.
If you can't rely on logrotate being available, here's a way do it inside the shell. To keep things simple, I'll assume that nothing else (including another instance of your script) will try to rename the log files while your script is running.
The easiest approach is to recursively rename log N+1 before actually renaming log N to N+1. The shell can perform all the necessary arithmetic, you don't need sed here. Note that while recursive functions are possible in a POSIX shell, there are no local variables other than the positional parameters (many shells offer local variables as an extension).
#!/bin/sh
## Move "$1.$2" to "$1.$(($2+1))", first rotating the target as well.
rotate () {
if [ -e "$1.$(($2+1))" ]; then rotate "$1" $(($2+1)); fi
mv -- "$1.$2" "$1.$(($2+1))"
}
for x; do
## Break each argument into FILE.NUMBER or just FILE.
suffix=${x##*.}
case $suffix in
*[!0-9]*)
if [ -e "$x.0" ]; then rotate "$x" 0; fi
mv -- "$x" "$x.0";;
*) rotate "${x%.*}" "$suffix";;
esac
done
Regarding what you've written, note that echo ${file} is bad for two reasons: most importantly, if ${file} contains any special characters such as white space, the shell will interpret them; also, with some shells, echo itself will interpret backslashes and possibly a leading -. So you should always write printf %s "$file" instead.

Related

Renaming files with numbered 0-padding names while leaving pre-renamed ones alone

I have several .jpg files in serveral folders about 20K actually. The filename are different like 123.jpg, abc.jpg, ab12.jpg. What is need is to rename all those files using bash script with leading 0 pattern.
I had used one code down below and while I do this the everytime I add new files the previous files get renamed again. Could anyone help me out from this situation and this would be really help full. I have searched entire web for this and not find one :(
#!/bin/bash
num=0
for i in *.jpg
do
a=`printf "%05d" $num`
mv "$i" "filename_$a.jpg"
let "num = $(($num + 1))"
done
To provide a concrete example of the problem, consider:
touch foo.jpg bar.jpg baz.jpg
The first time this script is run, bar.jpg is renamed to filename_00000.jpg; baz.jpg is renamed to filename_00001.jpg; foo.jpg is renamed to filename_00002.jpg. This behavior is acceptable.
If someone then runs:
touch a.jpg
...and runs the script again, then it renames a.jpg to filename_00000.jpg, renames filename_00000.jpg (now a.jpg, as the old version got overwritten!) to filename_00001.jpg, renames filename_00001.jpg to filename_00002.jpg, etc.
How can I make the program leave the files already matching filename_#####.jpg alone, and rename new files to have numbers after the last one that already exists?
#!/bin/bash
shopt -s extglob # enable extended globbing -- regex-like syntax
prefix="filename_"
# Find the largest-numbered file previously renamed
num=0 # initialize counter to 0
for f in "$prefix"+([[:digit:]]).jpg; do # Iterate only over names w/ prefix/suffix
f=${f#"$prefix"} # strip the prefix
f=${f%.jpg} # strip the suffix
if (( 10#$f > num )); then # force base-10 evaluation
num=$(( 10#$f ))
fi
done
# Second pass: Iterate over *all* names, and rename the ones that don't match the pattern
for i in *.jpg; do
[[ $i = "$prefix"+([[:digit:]]).jpg ]] && continue # Skip files already matching pattern
printf -v a '%05d' "$num" # More efficient than subshell use
until mv -n -- "$i" "$prefix$a.jpg"; do # "--" forces parse of "$i" as name
[[ -e "$i" ]] || break # abort if source file disappeared
num=$((num + 1)) # if we couldn't rename, increment num
printf -v a '%05d' "$num" # ...and try again with the next name
done
num=$((num + 1)) # modern POSIX math syntax
done
Note the use of mv -n to prevent overwrites -- that way two copies of this script running at the same time won't overwrite each others' files.

Add time stamp to file which has at least one row in UNIX

I have list of files at a location ${POWERCENTER_FILE_DIR} .
The files consist of row header and values.
MART_Apple.csv
MART_SAMSUNG.csv
MART_SONY.csv
MART_BlackBerry.csv
Requirements:
select only those files which has atleast 1 row.
Add time stamp to the files which has at least 1 row.
For example:
If all the files except MART_BlackBerry.csv has atleast one row then my output files names should be
MART_Apple_20170811112807.csv
MART_SAMSUNG_20170811112807.csv
MART_SONY_20170811112807.csv
Code tried so far
#!/bin/ksh
infilename=${POWERCENTER_FILE_DIR}MART*.csv
echo File name is ${infilename}
if [ wc -l "$infilename"="0" ];
then
RV=-1
echo "input file name cannot be blank or *"
exit $RV
fi
current_timestamp=`date +%Y%m%d%H%M%S`
filename=`echo $infilename | cut -d"." -f1 `
sftpfilename=`echo $filename`_${current_timestamp}.csv
cp -p ${POWERCENTER_FILE_DIR}$infilename ${POWERCENTER_FILE_DIR}$sftpfilename
RV=$?
if [[ $RV -ne 0 ]];then
echo Adding timestamp to ${POWERCENTER_FILE_DIR}$infilename failed ... Quitting
echo Return Code $RV
exit $RV
fi
Encountering errors like:
line 3: [: -l: binary operator expected
cp: target `MART_Apple_20170811121023.csv' is not a directory
failed ... Quitting
Return Code 1
to be frank, i am not able to apprehend the errors nor i am sure i am doing it right. Beginner in unix scripting.Can any experts guide me where to the correct way.
Here's an example using just find, sh, mv, basename, and date:
find ${POWERCENTER_FILE_DIR}MART*.csv ! -empty -execdir sh -c "mv {} \$(basename -s .csv {})_\$(date +%Y%m%d%H%M%S).csv" \;
I recommend reading Unix Power Tools for more ideas.
When it comes to shell scripting there is rarely a single/one/correct way to accomplish the desired task.
Often times you may need to trade off between readability vs maintainability vs performance vs adhering-to-some-local-coding-standard vs shell-environment-availability (and I'm sure there are a few more trade offs). So, fwiw ...
From your requirement that you're only interested in files with at least 1 row, I read this to also mean that you're only interested in files with size > 0.
One simple ksh script:
#!/bin/ksh
# define list of files
filelist=${POWERCENTER_FILE_DIR}/MART*.csv
# grab current datetime stamp
dtstamp=`date +%Y%m%d%H%M%S`
# for each file in our list ...
for file in ${filelist}
do
# each file should have ${POWERCENTER_FILE_DIR} as a prefix;
# uncomment 'echo' line for debugging purposes to display
# the contents of the ${file} variable:
#echo "file=${file}"
# '-s <file>' => file exists and size is greater than 0
# '! -s <file>' => file doesn't exist or size is equal to 0, eg, file is empty in our case
#
# if the file is empty, skip/continue to next file in loop
if [ ! -s ${file} ]
then
continue
fi
# otherwise strip off the '.csv'
filebase=${file%.csv}
# copy our current file to a new file containing the datetime stamp;
# keep in mind that each ${file} already contains the contents of the
# ${POWERCENTER_FILE_DIR} variable as a prefix; uncomment 'echo' line
# for debug purposes to see what the cp command looks like:
#echo "cp command=cp ${file} ${filebase}.${dtstamp}.csv"
cp ${file} ${filebase}.${dtstamp}.csv
done
A few good resources for learning ksh:
O'Reilly: Learning the Korn Shell
O'Reilly: Learning the Korn Shell, 2nd Edition (includes the newer ksh93)
at your UNIX/Linux command line: man ksh
A simplified script would be something like
#!/bin/bash
# Note I'm using bash above, can't guarantee (but I hope) it would work in ksh too.
for file in ${POWERCENTER_FILE_DIR}/*.csv # Check Ref [1]
do
if [ "$( wc -l "$file" | grep -Eo '^[[:digit:]]+' )" -ne 0 ] # checking at least one row? Check Ref [2]
then
mv "$file" "${file%.csv}$(date +'%Y%m%d%H%M%S').csv" # Check Ref [3]
fi
done
References
File Globbing [1]
Command Substitution [2]
Parameter Substitution [3]

load multiple files from different folders with same file name in bash-scipt

So far i am able to read in the files from a single folder on my Ubuntu using:
for i in /path/to/files/Folder1/*.pcd
do
if [ ! -z $last_i ]
then
./vapp $last_i $i
fi
last_i="$i"
done
this will read all files in Folder1. I Also have folder 2 and 3 (i.e. Folder2, Folder3). Inside each folder are several 100 files which are simply numbered such as 0000.pcd, 0001.pcd ... 0129.pcd... and so on.
I have tried to use
/path/to/files/Folder{1..3}/*.pcd
The problem is that it takes now all files from one folder and processes two files within, than goes through all files in this folder the same way before moving on to the next folder.
What i really want is to take from each of my three folders the ith filename e.g. 000i.pcd and pass it (including the path) to my application to do some calculations.
effectively I want to do this:
./vapp /Folder1/000i.pcd /Folder2/000i.pcd /Folder3/000i.pcd
Using native bash features alone, with its extended glob features. Run the script from /path/to/files/
#!/bin/bash
shopt -s globstar nullglob dotglob
i=0
end=129
while [ "$i" -le "$end" ]
do
# Generating the file-name to be generated, the numbers 0-129
# with 4 character padding, taken care by printf
file="$(printf "%04d.pcd" $i)"
# The ** pattern enabled by globstar matches 0 or more directories,
# allowing the pattern to match to an arbitrary depth in the current
# directory.
fileList=( **/"$file" )
# Remember to un-comment the below line and see if the required files
# are seen by doing
# printf "%s\n" "${fileList[#]}"
# and run the executable below
./vapp "$(printf "%s " "${fileList[#]}")"
done
The usage of extglob features are re-used from this wonderful answer.
I have solved my problem yesterday in a similar way to what Inian suggested. Except that my way is the manual way!
It works for me but I agree that Inian's way is much more flexible. Here is my solution anyway:
j=0
leadingZeros1="00000000"
leadingZeros2="0000000"
leadingZeros3="000000"
fol0="lidar03_00/"
fol1="lidar03_01/"
fol2="lidar03_02/"
ext=".pcd"
basepath="/my/path/"
for i in /my/path/lidar03_00/*.pcd
do
if [ ! -z $last_i ]
((j+=1))
then
if [ $j -le 9 ]
then
mypath1=$basepath$fol0$leadingZeros1$j$ext
mypath2=$basepath$fol1$leadingZeros1$j$ext
mypath3=$basepath$fol2$leadingZeros1$j$ext
fi
sleep .009
if [ $j -ge 10 ]
then
if [ $j -le 99 ]
then
mypath1=$basepath$fol0$leadingZeros2$j$ext
mypath2=$basepath$fol1$leadingZeros2$j$ext
mypath3=$basepath$fol2$leadingZeros2$j$ext
fi
if [ $j -ge 100 ]
then
if [ $j -le 999 ]; then
mypath1=$basepath$fol0$leadingZeros3$j$ext
mypath2=$basepath$fol1$leadingZeros3$j$ext
mypath3=$basepath$fol2$leadingZeros3$j$ext
fi;
fi
fi
#echo $mypath1
#echo $mypath2
#echo $mypath3
./vapp -i $mypath1 $mypath2 $mypath3 .txt"
fi
last_i="$i"
done
I admit, it is not the nicest solution but it fixes my problem for now. If i need to do this again I will probably do it Inian's way.
Thanks all for the help.
I have tried to avoid the nested ifs using logical AND (&&) but somehow it didn't work and I didn't wanted to spend more time on this. Thus the clumsy way...

Bash script to compare files

I have a folder with a ton of old photos with many duplicates. Sorting it by hand would take ages, so I wanted to use the opportunity to use bash.
Right now I have the code:
#!/bin/bash
directory="~/Desktop/Test/*"
for file in ${directory};
do
for filex in ${directory}:
do
if [ $( diff {$file} {$filex} ) == 0 ]
then
mv ${filex} ~/Desktop
break
fi
done
done
And getting the exit code:
diff: {~/Desktop/Test/*}: No such file or directory
diff: {~/Desktop/Test/*:}: No such file or directory
File_compare: line 8: [: ==: unary operator expected
I've tried modifying working code I've found online, but it always seems to spit out some error like this. I'm guessing it's a problem with the nested for loop?
Also, why does it seem there are different ways to call variables? I've seen examples that use ${file}, "$file", and "${file}".
You have the {} in the wrong places:
if [ $( diff {$file} {$filex} ) == 0 ]
They should be at:
if [ $( diff ${file} ${filex} ) == 0 ]
(though the braces are optional now), but you should allow for spaces in the file names:
if [ $( diff "${file}" "${filex}" ) == 0 ]
Now it simply doesn't work properly because when diff finds no differences, it generates no output (and you get errors because the == operator doesn't expect nothing on its left-side). You could sort of fix it by double quoting the value from $(…) (if [ "$( diff … )" == "" ]), but you should simply and directly test the exit status of diff:
if diff "${file}" "${filex}"
then : no difference
else : there is a difference
fi
and maybe for comparing images you should be using cmp (in silent mode) rather than diff:
if cmp -s "$file" "$filex"
then : no difference
else : there is a difference
fi
In addition to the problems Jonathan Leffler pointed out:
directory="~/Desktop/Test/*"
for file in ${directory};
~ and * won't get expanded inside double-quotes; the * will get expanded when you use the variable without quotes, but since the ~ won't, it's looking for files under an directory actually named "~" (not your home directory), it won't find any matches. Also, as Jonathan pointed out, using variables (like ${directory}) without double-quotes will run you into trouble with filenames that contain spaces or some other metacharacters. The better way to do this is to not put the wildcard in the variable, use it when you reference the variable, with the variable in double-quotes and the * outside them:
directory=~/"Desktop/Test"
for file in "${directory}"/*;
Oh, and another note: when using mv in a script it's a good idea to use mv -i to avoid accidentally overwriting another file with the same name.
And: use shellcheck.net to sanity-check your code and point out common mistakes.
If you are simply interested in knowing if two files differ, cmp is the best option. Its advantages are:
It works for text as well as binary files, unlike diff which is for text files only
It stops after finding the first difference, and hence it is very efficient
So, your code could be written as:
if ! cmp -s "$file" "$filex"; then
# files differ...
mv "$filex" ~/Desktop
# any other logic here
fi
Hope this helps. I didn't understand what you are trying to do with your loops and hence didn't write the full code.
You can use diff "$file" "$filex" &>/dev/null and get the last command result with $? :
#!/bin/bash
SEARCH_DIR="."
DEST_DIR="./result"
mkdir -p "$DEST_DIR"
directory="."
ls $directory | while read file;
do
ls $directory | while read filex;
do
if [ ! -d "$filex" ] && [ ! -d "$file" ] && [ "$filex" != "$file" ];
then
diff "$file" "$filex" &>/dev/null
if [ "$?" == 0 ];
then
echo "$filex is a duplicate. Copying to $DEST_DIR"
mv "$filex" "$DEST_DIR"
fi
fi
done
done
Note that you can also use fslint or fdupes utilities to find duplicates

Bash Script Nested loop error

Code:
#! /bin/bash
while [ 1 -eq 1 ]
do
while [ $(cat ~/Genel/$(ls -t1 ~/Genel | head -n1)) != $(cat ~/Genel/$(ls -t1 ~/Genel | head -n1)) ]
$(cat ~/Genel/$(ls -t1 ~/Genel | head -n1)) > /tmp/cmdb;obexftp -b $1 -B 6 -p /tmp/cmdb
done
done
This code give me this error:
btcmdserver: 6: Syntax error: "done" unexpected (expecting "do")
Your second while loop is missing a do keyword.
Looks like you didn't close your while condition ( the [ has no matching ]), and that your loop has no body.
You cannot compare whole files like that. Anyway, you seem to be comparing a file to itself.
#!/bin/bash
while true
do
newest=~/Gene1/$(ls -t1 ~/Gene1 | head -n 1)
while ! cmp "$newest" "$newest" # huh? you are comparing a file to itself
do
# huh? do you mean this:
cat "$newest" > /tmp/cmdb
obexftp -b $1 -B 6 -p /tmp/cmdb
done
done
This has the most troubling syntax errors and antipatterns fixed, but is virtually guaranteed to not do anything useful. Hope it's still enough to get you a little bit closer to your goal. (Stating it in the question might help, too.)
Edit: If you are attempting to copy the newest file every time a new file appears in the directory you are watching, try this. There's still a race condition; if multiple new files appear while you are copying, you will miss all but one of them.
#!/bin/sh
genedir=$HOME/Gene1
previous=randomvalue_wehavenobananas
while true; do
newest=$(ls -t1 "$genedir" | head -n 1)
case $newest in
$previous) ;; # perhaps you should add a sleep here
*) obexftp -b $1 -B 6 -p "$genedir"/"$newest"
previous="$newest" ;;
esac
done
(I changed the shebang to /bin/sh mainly to show that this no longer contains any bashisms. The main change was to use ${HOME} instead of ~.)
A more robust approach would be to find all the files which have appeared since the previous time you copied, and copy them over. Then you could run this a little less aggressively (say, once per 5 minutes maybe, instead of the spin lock you have here, with no sleep at all between iterations). You could use a sentinel file in the watch directory to keep track of when you last copied some files, or just run a for loop over the ls -t1 output until you see a file you have seen before. (Note the comment about the lack of robustness with parsing ls output, though.)

Resources