Renaming files with a specific scheme - bash

I have a FTP folder receiving files from a remote camera. The camera stores the video file name always as ./rec_YYYY-MM-DD_HH-MM.mkv. The video files are stored all in the same folder, the root folder from the FTP server.
I need to move these files to another folder, with this new scheme:
Remove rec_ from the file name.
Change date format to DD-MM-YY.
Remove date from the file name and make it a folder instead, where that same file and all the others in the same date will be stored in.
Final file path would be: ./DD-MM-YYYY/HH-MM.mkv.
The process would continue to all the files, putting them in the folder corresponding to the day it was created.
Summing up: ./rec_YYYY-MM-DD_HH-MM.mkv >> ./DD-MM-YYYY/HH-MM.mkv. The same should apply to all files that are in the same folder.
As I can't make it happen directly from the camera, this needs to be done with Bash on the server that is receiving the files.
So far, what I got is script, which would get the file's creation date and use it to make a folder, and then get creation time to move the file with the new name, based on it's creation time.:
for f in *.mp4
do
mkdir "$f" "$(date -r "$f" +"%d-%m-%Y")"
mv -n "$f" "$(date -r "$f" +"%d-%m-%Y/%H-%M-%S").mp4"
done
I'm getting this output (with testfile 1.mp4):
It creates the folder based on the file's creation date;
it renames the file to it's creation time;
Then, it returns mkdir: cannot create directory ‘1.mp4’: File exists
If two or more files, only one gets renamed and moved as described. The others stay the same and terminal returns:
mkdir: cannot create directory ‘1.mp4’: File exists
mkdir: cannot create directory ‘2.mp4’: File exists
mkdir: cannot create directory ‘12-12-2018’: File exists
Could someone help me out? Better suggestions? Thanks!

Honestly I would just use Perl or Python for this. Here's how to embed either in a shell script.
Here's a perl script that doesn't use any libraries, even ones that ship with Perl (so it'll work without extra packages on distributions like CentOS that don't ship with the entire Perl library). The perl script launches one new process per file in order to perform the copy.
perl -e '
while (<"*.m{p4,kv}">) {
my $path = $_;
my ($prefix, $year, $month, $day, $hour, $minute, $ext) =
split /[.-_]/, $path;
my $sec = q[00];
die "unexpected prefix ($prefix) in $path"
unless $prefix eq q[rec];
die "unexpected extension ($ext) in $path"
unless $ext eq q[mp4] or $ext eq q[mkv];
my $dir = "$day-$month-$year";
my $name = "$hour-$min-$sec" . q[.] . $ext;
my $destpath = $dir . q[/] . $name;
die "$dir . $name is unexpectedly a directory" if (-d $dir);
system("cp", "--", $path, $destpath);
}
'
Here's a Python example, it's compatible with either Python 2 or Python 3 but does use the standard library. The Python script does not spawn any additional processes.
python3 -c '
import os.path as path
import re
from glob import iglob
from itertools import chain
from os import mkdir
from shutil import copyfile
for p in chain(iglob("*.mp4"), iglob("*.mkv")):
fields = re.split("[-]|[._]", p)
prefix = fields[0]
year = fields[1]
month = fields[2]
day = fields[3]
hour = fields[4]
minute = fields[5]
ext = fields[6]
sec = "00"
assert prefix == "rec"
assert ext in ["mp4", "mkv"]
directory = "".join([day, "-", month, "-", year])
name = "".join([hour, "-", minute, "-", sec, ".", ext])
destpath = "".join([directory, "/", name])
assert not path.isdir(destpath)
try:
mkdir(directory)
except FileExistsError:
pass
copyfile(src=p, dst=destpath)
'
Finally, here's a bash solution. It splits paths using -, ., and _ and then extracts various subfields by indexing into $# inside a function. The indexing trick is portable, although regex substitution on variables is a bash extension.
#!/bin/bash
# $1 $2 $3 $4 $5 $6 $7 $8
# path rec YY MM DD HH MM ext
process_file() {
mkdir "$5-$4-$3" &> /dev/null
cp -- "$1" "$5-$4-$3"/"$6-$7-00.$8"
}
for path in *.m{p4,kv}; do
[ -e "$path" ] || continue
# NOTE: two slashes are needed in the substitution to replace everything
# read -a ARRAYVAR <<< ... reads the words of a string into an array
IFS=' ' read -a f <<< "${path//[-_.]/ }"
process_file "$path" "${f[#]}"
done

If you cd /to/some/directory/containing_your_files then you could use the following script
#!/usr/bin/env bash
for f in rec_????-??-??_??-??.m{p4,kv} ; do
dir=${f:4:10} # skip 4 chars ('rec_') take 10 chars ('YYYY_MM_DD')
fnm=${f:15} # skip 15 chars, take the remainder
test -d "$dir" || mkdir "$dir"
mv "$f" "$dir"/"$fnm"
done
note ① that I have not exchanged the years and the days, if you absolutely need to do the swap you can extract the year like this, year=${dir::4} etc and ② that this method of parameter substitution is a Bash-ism, e.g., it doesn't work in dash.

your problem is: mkdir creates folder but you are giving filename for folder creation.
if you want to use fileName for folder creation then use it without extension.
the thing is you are trying to create folder with the already existing fileName

Related

Bash/sh: Move Folder + subfolder(s) reclusively rename files if they exist [duplicate]

This question already has answers here:
Extract filename and extension in Bash
(38 answers)
Closed 1 year ago.
I'm trying to create a bash script that will move all files recursively from a source folder to a target folder, and simply rename files if they already exist. Similar to the way M$ Windows does, when a file exists it auto-renames it with "<filemame> (X).<ext>", etc. except for ALL files.
I've create the below, which works fine for almost all scenarios except when a folder has a (.) period in its name and a file within that folder has no extension (no period in its name).
eg a folder-path-file such as: "./oldfolder/this.folder/filenamewithoutextension"
I get (incorrectly):
"./newfolder/this (1).folder/filenamewithoutextension"
if "./newfolder/this.folder/filenamewithoutextension" already exist in the target location (./newfolder),
instead of correctly naming the new file: "./oldfolder/this.folder/filenamewithoutextension (1)"
#!/bin/bash
source=$1 ; target=$2 ;
if [ "$source" != "" ] && [ "$target" != "" ] ; then
#recursive file search
find "$source" -type f -exec bash -c '
#setup variables
oldfile="$1" ; osource='"${source}"' ; otarget='"${target}"' ;
#set new target filename with target path
newfile="${oldfile/${osource}/${otarget}}" ;
#check if file already exists at target
[ -f "${newfile}" ] && {
#get the filename and fileextension for numbering - ISSUE HERE?
filename="${newfile%/}" ; newfileext="${newfile##*.}" ;
#compare filename and file extension for missing extension
if [ "$filename" == "$newfileext" ] ; then
#filename has no ext - perhaps fix the folder with a period issue here?
newfileext="" ;
else
newfileext=".$newfileext" ;
fi
#existing files counter
cnt=1 ; while [ -f "${newfile%.*} (${cnt})${newfileext}" ] ; do ((cnt+=1)); done
#set new filename with counter - New Name created here *** Needs re-work, as folder with a period = fail
newfile="${newfile%.*} (${cnt})${newfileext}";
}
#show mv command
echo "mv \"$oldfile\" \"${newfile}\""
' _ {} \;
else
echo "Requires source and target folders";
fi
I suspect the issue is, how to properly identify the filename and extension, found in this line:
filename="${newfile%/}" ; newfileext="${newfile##*.}" which doesn't identify a filename properly (files are always after the last /).
Any suggestion on how to make it work properly?
UPDATED: Just some completion notes - Issues fixes with:
Initially Splitting each full path filename: path - filename - (optional ext)
Reconstructing the full path filename: path - filename - counter - (optional ext)
fixed the file move to ensure directory structure exists with mkdir -p (mv does not create new folders if they do not exist in the target location).
Maybe you could try this instead?
filename="${newfile##*/}" ; newfileext="${filename#*.}"
The first pattern means: remove the longest prefix (in a greedy way) up to the last /.
The second one: remove the prefix up to the first dot (the greedy mode seems unnecessary here) − and as you already noted, in case the filename contains no dot, you will get newfileext == filename…
Example session:
newfile='./oldfolder/this.folder/filenamewithoutextension'
filename="${newfile##*/}"; newfileext="${filename#*.}"
printf "%s\n" "$filename"
#→ filenamewithoutextension
printf "%s\n" "$newfileext"
#→ filenamewithoutextension
newfile='./oldfolder/this.folder/file.tar.gz'
filename="${newfile##*/}"; newfileext="${filename#*.}"
printf "%s\n" "$filename"
#→ file.tar.gz
printf "%s\n" "$newfileext"
#→ tar.gz

Renaming two files in multiple folders by adding suffix and prefix

I have multiple folders where two files are present.
For example, 123.jpg, 456.jpg under folder ABC. I want to rename the files to IT1_ABC_123.v1.jpg and IT2_ABC_456.v1.jpg. Similarly, other folders have two files.
How can I do this in shell or Perl?
Try this, using shell and perl:
mkdir /tmp/test; cd $_
mkdir ABC DEF
touch {ABC,DEF}/{123,456}.jpg #creates four files, two in each directory
find|perl -nlE's,((.*)/(.+))/((123|456).jpg),$1/IT#{[++$n]}_$3_$4,&&say"$&\n$_\n"'
./ABC/123.jpg
./ABC/IT1_ABC_123.jpg
./ABC/456.jpg
./ABC/IT2_ABC_456.jpg
./DEF/123.jpg
./DEF/IT3_DEF_123.jpg
./DEF/456.jpg
./DEF/IT4_DEF_456.jpg
Now, after confirming this is what you want, replace the say with a rename:
find|perl -nlE's,((.*)/(.+))/((123|456).jpg),$1/IT#{[++$n]}_$3_$4, and rename$&,$_'
The new filenames:
find -type f
./ABC/IT1_ABC_123.jpg
./ABC/IT2_ABC_456.jpg
./DEF/IT3_DEF_123.jpg
./DEF/IT4_DEF_456.jpg
This will find filenames with 123.jpg or 456.jpg and rename them.
s,,, is the search-replace and it returns 1 (the number of changes it made) which again leads to the right side of the and being done (the rename).
Filenames that doesn't match 123.jpg or 456.jpg isn't renamed since s,,, will return 0 and the and is "short cutted" since it then logically cannot be true with a false (0) left side. So then the rename is not executed.
This variant does the same, but might be easier to read:
find|perl -nlE 'rename$&,$_ if s,((.*)/(.+))/((123|456).jpg),$1/IT#{[++$n]}_$3_$4,'
I have found this pattern useful in many cases of mass renamings. Also, dedicated software for mass renaming with GUIs exists, which for some might be easier to use.
Rewritten as a program abc.pl, it could be:
#!/usr/bin/perl
while(<>){
chomp;
next if not s,((.*)/([A-Z]{3}))/(\d{3}\.jpg),$1/IT#{[++$n]}_$3_$4,;
print "Found: $&\nNew name: $_\n\n";
#rename $&, $_;
}
Run:
find|perl abc.pl
You can do this in core Perl using the File::Find, File::Basename, and File::Copy modules. You can test it out with the script below. It won't make any changes until you uncomment the line with the "move" function.
#! perl
use strict;
use warnings;
use File::Basename;
use File::Copy;
use File::Find;
my $root_dir = '/path/to/main/folder';
# Recursively searches for all files below the $root_dir
my #fileset;
find(
sub {
# Get the absolute file path
my $path = $File::Find::name;
# Only capture the path if not a directory
# You can add any number of conditions here
if (!-d $path) {
push #fileset, $path;
}
},
$root_dir
);
# set the IT counter in new file name
my $int = 1;
# list of all possible file suffixes to have fileparse() look for. It will
# capture the end of the file path verbatim (including the period) if it's
# in this array
my #suffixes = ('.jpg', '.txt');
my $previous_dir;
foreach my $old_path (#fileset) {
# split apart the basename of the file, the directory path, and the file suffix
my ($basename, $parent_dir, $suffix) = fileparse($old_path, #suffixes);
# strip off trailing slash so fileparse() will capture parent dir name correctly
$parent_dir =~ s{[/]$}{};
# capture just the name of the parent directory
my $parent_name = fileparse($parent_dir);
# Assemble the new path
my $new_path = $parent_dir . '/IT' . $int . '_'
. $parent_name . '_' . "$basename.v1" . $suffix;
# Move the file to rename (this is safer than using rename() for cross-platform)
# move $old_path, $new_path;
print "OLD PATH: $old_path\n";
print "NEW PATH: $new_path\n\n";
# Reset counter when dir changes
if (!$previous_dir) {
$previous_dir = $parent_dir; # set previous_dir on first loop
}
elsif($previous_dir ne $parent_dir) {
$previous_dir = $parent_dir; # update previous_dir to check next loop
$int = 0; # reset counter
}
$int++; # iterate the counter
}
Edit 2018-07-12: I've updated the answer to show how to reset the counter when the directory changes by evaluating the current path with the one used in the previous loop and updating accordingly. This is not tested so it may need some adjustments.
Given the abc/def examples given, the output should look something like this:
OLD PATH: /path/to/main/folder/abc/123.jpg
NEW PATH: /path/to/main/folder/abc/IT1_abc_123.v1.jpg
OLD PATH: /path/to/main/folder/abc/456.txt
NEW PATH: /path/to/main/folder/abc/IT2_abc_456.v1.jpg
OLD PATH: /path/to/main/folder/def/123.jpg
NEW PATH: /path/to/main/folder/def/IT1_def_123.v1.jpg
OLD PATH: /path/to/main/folder/def/456.jpg
NEW PATH: /path/to/main/folder/def/IT2_def_456.v1.jpg

Bash script to copy a folder's contents and overwrite destination?

I'm just barely familiar with bash scripts, and am trying to achieve being able to specify a source location (a directory with files / sub-directories) and a list of multiple destination directories, along with a list of files / directories from the source to exclude when copying & replacing.
The script should loop through each destination location, copy the files from the source and replace them in the destination location, excluding any files / directories listed in the script to skip.
In PHP it would be something like:
$source = '/home/user1/public_html';
$destinations = array(
'user2',
'user3',
'user4'
);
$exclude = array(
'/uploads/custom',
'/config/conf.php'
);
foreach ($destinations as $destination) {
$dir = '/home/' .$destination. '/public_html';
if (is_dir($dir)) {
exec('cp -R ' .$source. '/* ' .$dir);
// and somehow exclude overwriting any files / directories matching what's in the $exclude array
}
}
So, my question is, how would a bash script with similar functionality be written?
Here's how I ended up doing it:
#!/bin/bash
while IFS= read -r line || [[ $line ]]; do
if [[ -d "$line" ]]; then
rsync -rv --exclude-from "exclude.txt" /home/source/public_html/ "$line"
else
echo "Directory $line does not exist!"
fi
done < "accounts.txt"
This reads from a file called "accounts.txt" that has all the destination account directories in it, one per line. Ex:
/home/user1/public_html
/home/myblog/public_html
It loops through each one, and if the directory exists, it copies the files from /home/source/public_html to the destination. It also reads from a file called "exclude.txt" that includes, one per line, any files or directories to exclude. Ex:
.git
/uploads/custom
/config/conf.php

Looping through folders and renaming all files in each folder sequentially with folder name

I have a series of subfolders, each with images on them.
Structure looks like
/stuff/
IMG_012.JPG
IMG_013.JPG
IMG_014.JPG
/morestuff/
IMG_022.JPG
IMG_023.JPG
IMG_024.JPG
I would like to write a script on my mac terminal to loop through each folder and rename the images sequentially including the folder name. So, the above structure would look like:
/stuff/
stuff_1.JPG
stuff_2.JPG
stuff_3.JPG
/morestuff/
morestuff_1.JPG
morestuff_1.JPG
morestuff_1.JPG
I orignally tried creating a Automator workflow and using variables, but had difficulty assigning the folder name as the variable and getting the loop to work.
I'm hoping there is an elegant solution with the terminal.
Any ideas?
This should work nicely for you. Save it in your HOME directory as "RenameImages", then make it executable like this:
cd
chmod +x RenameImages
Then you can run it (-exec) it on every directory (-type d) like this:
find . -type d -exec ./RenameImages {} \;
Here is the script:
#!/bin/bash
# Ignore case, i.e. process *.JPG and *.jpg
shopt -s nocaseglob
shopt -s nullglob
# Go to where the user says
echo Processing directory $1
cd "$1" || exit 1
# Get last part of directory name
here=$(pwd)
dir=${here/*\//}
i=1
for image in *.JPG
do
echo mv "$image" "${dir}${i}.jpg"
((i++))
done
At the moment it does nothing except show you what it would do. If you like what it is doing, simply remove the word echo from the 3rd to last line and run it again.
I would just like to throw out another way to do this since you mentioned you tried using Automator. This file search and process software: http://www.softpedia.com/get/File-managers/JFileProcessor.shtml https://github.com/stant/jfileprocessor
Will let you search for files with glob or regex, in subfolders to X or all depth, by name, size, date. You can save to a List window or file. Then you can run a groovy (think java) script to do whatever you want to the list of files; zip or tar them, modify the list strings like sed, delete, move, copy files, grep or ls -l them, whatever. In your case you can modify an existing groovy example to do something like:
int numItems = defaultComboBoxModel.getSize();
System.out.println( "defaultComboBoxModel.getSize() num of items =" + numItems + "=" );
String str = "";
for( int i = 0; i < numItems; i++ )
{
str = defaultComboBoxModel.getElementAt( i ).toString();
System.out.println( "file of index =" + i + " str =" + str + "=" );
String cmd = "mv " + str + " " + i + ".jpg"; // or whatever
def list = cmd.execute().text // this stuff just captures output and write to a log file
list.eachLine{
outFile << it;
}
outFile << System.getProperty("line.separator") + "-------------------------------------" + System.getProperty("line.separator");
}
It will also let you massage your list like add to, delete from, subtract one list from another.

How to find a specific files recursively in the directory, rename it by prefixing sub-directory name, and move it to different directory

I am perl noob, and trying to do following:
Search for files with specific string in a directory recursively. Say string is 'abc.txt'
The file can be in two different sub-directories, say dir_1 or dir_2
Once the file is found, if it is found in dir_1, rename it to dir_1_abc.txt. If it is in dir_2, then rename it to dir_2_abc.txt.
Once all the files have been found and renamed, move them all to a new directory named, say dir_3
I don't care if I have to use any module to accomplish this. I have been trying to do it using File::Find::Rule and File::copy, but not getting the desired result. Here is my sample code:
#!/usr/bin/perl -sl
use strict;
use warnings;
use File::Find::Rule;
use File::Copy;
my $dir1 = '/Users/macuser/ParentDirectory/logs/dir_1'
my $dir2 = '/Users/macuser/ParentDirectory/logs/dir_2'
#ideally I just want to define one directory but because of the logic I am using in IF
#statement, I am specifying two different directory paths
my $dest_dir = '/Users/macuser/dir_3';
my(#old_files) = find(
file => (),
name => '*abc.txt',
in => $dir1, $dir2 ); #not sure if I can give two directories, works with on
foreach my $old_file(#old_files) {
print $old_file; #added this for debug
if ($dest_dir =~ m/dir_1/)
{
print "yes in the loop";
rename ($old_file, "dir_1_$old_file");
print $old_file;
copy "$old_file", "$dest_dir";
}
if ($dest_dir =~ m/dir_2/)
{
print "yes in the loop";
rename ($old_file, "dir_2_$old_file");
print $old_file;
copy "$old_file", "dest_dir";
}
}
The code above does not change the file name, instead when I am printing $old_file inside if, it spits the whole directory path, where the file is found, and it is prefixing the path with dir_1 and dir_2 respectively. Something is horribly wrong. Please help simply.
If you have bash ( I assume in OSX it is available), you can do this in a few lines (usually I put them in one line).
destdir="your_dest_dir"
for i in `find /Users/macuser/ParentDirectory/logs -type f -iname '*abc.txt' `
do
prefix=`dirname $i`
if [[ $prefix = *dir_1* ]] ; then
prefix="dir_1"
fi
dest="$destdir/${prefix}_`basename $i`"
mv "$i" "$dest"
done
The advantage of this method is that you can have many sub dirs under logs and you don't need to specify them. you can search for files like blah_abc.txt, tada_abc.txt too. If you want a exact match just juse abc.txt, instead of *abc.txt.
If the files can be placed in the destination as you rename them, try this:
#!/usr/bin/perl
use strict;
use File::Find;
use File::Copy;
my $dest_dir = '/Users/macuser/dir_3';
foreach my $dir ('/Users/macuser/ParentDirectory/logs/dir_1', '/Users/macuser/ParentDirectory/logs/dir_2') {
my $prefix = $dir; $prefix =~ s/.*\///;
find(sub {
move($File::Find::name, "$dest_dir/${prefix}_$_") if /abc\.txt$/;
}, $dir);
}
If you need to do all the renaming first and then move them all, you could either remember the list of files you have to move or you can make two passes making sure the pattern on the second pass is still OK after the initial rename in the first pass.

Resources