Inside a folder I have a file structure, the files have all a name whose format
name-randomstring.extension
eg
./dir1/aaa-5h34jk5hk.js
./dir2/bbb-5yh45uh9h.css
./dir3/ccc-uiero6tio.js
./dir3/ddd-7y8fygfre.css
. . .
with a bash script I would like to rename them recursively; so to eliminate the -randomstring from every file
./dir1/aaa.js
./dir2/bbb.css
./dir3/ccc.js
./dir3/ddd.css
. . .
You can do this with a native bash loop:
shopt -s globstar # enable ** for recursive expansion
for file in **/*{j,cs}; do
# remove everything up to the last . to get extension
ext=${file##*.}
# remove everything after the last - and concat with extension
new_name=${file%-*}$ext
# -- prevents weird filenames being interpreted as options
mv -- "$file" "$new_name"
done
shopt -u globstar # disable ** if you don't want it anymore e.g. in a script
if you do not mind using Perl then you can do anything you like.
for example for renaming all files with this structure:
dir > find .
.
./00_file.txt
./01_file.txt
./02_file.txt
./dir1
./dir1/00_file.txt
./dir1/01_file.txt
./dir1/02_file.txt
./dir2
./dir2/00_file.txt
./dir2/01_file.txt
./dir2/02_file.txt
./dir3
./dir3/00_file.txt
./dir3/01_file.txt
./dir3/02_file.txt
./find.log
then save your list in a file like: find.log now you can use Perl
dir > perl -lne '-f && ($old=$_) && s/file/ABCD/g && print "$old => $_"' find.log
./00_file.txt => ./00_ABCD.txt
./01_file.txt => ./01_ABCD.txt
./02_file.txt => ./02_ABCD.txt
./dir1/00_file.txt => ./dir1/00_ABCD.txt
./dir1/01_file.txt => ./dir1/01_ABCD.txt
./dir1/02_file.txt => ./dir1/02_ABCD.txt
./dir2/00_file.txt => ./dir2/00_ABCD.txt
./dir2/01_file.txt => ./dir2/01_ABCD.txt
./dir2/02_file.txt => ./dir2/02_ABCD.txt
./dir3/00_file.txt => ./dir3/00_ABCD.txt
./dir3/01_file.txt => ./dir3/01_ABCD.txt
./dir3/02_file.txt => ./dir3/02_ABCD.txt
how it works?
read the file
let only file; not directory or other stuffs with: -f
then save the old name: ($old=$_)
then doing substitution with: s///g operator
finally rename the old file to new one that is $_
NOTE I used print and you should use rename like this:
dir > perl -lne '-f && ($old=$_) && s/file/ABCD/g && rename $old, $_' find.log
dir > find .
.
./00_ABCD.txt
./01_ABCD.txt
./02_ABCD.txt
./dir1
./dir1/00_ABCD.txt
./dir1/01_ABCD.txt
./dir1/02_ABCD.txt
./dir2
./dir2/00_ABCD.txt
./dir2/01_ABCD.txt
./dir2/02_ABCD.txt
./dir3
./dir3/00_ABCD.txt
./dir3/01_ABCD.txt
./dir3/02_ABCD.txt
./find.log
NOTE Since the find returns list of all files and sub-directories, with this technique Perl renames all the files! Thus first use print then rename. And your pattern can be: -.*(?=\.). In fact:
s/-.*(?=\.)//g
Related
I have a FTP folder receiving files from a remote camera. The camera stores the video file name always as ./rec_YYYY-MM-DD_HH-MM.mkv. The video files are stored all in the same folder, the root folder from the FTP server.
I need to move these files to another folder, with this new scheme:
Remove rec_ from the file name.
Change date format to DD-MM-YY.
Remove date from the file name and make it a folder instead, where that same file and all the others in the same date will be stored in.
Final file path would be: ./DD-MM-YYYY/HH-MM.mkv.
The process would continue to all the files, putting them in the folder corresponding to the day it was created.
Summing up: ./rec_YYYY-MM-DD_HH-MM.mkv >> ./DD-MM-YYYY/HH-MM.mkv. The same should apply to all files that are in the same folder.
As I can't make it happen directly from the camera, this needs to be done with Bash on the server that is receiving the files.
So far, what I got is script, which would get the file's creation date and use it to make a folder, and then get creation time to move the file with the new name, based on it's creation time.:
for f in *.mp4
do
mkdir "$f" "$(date -r "$f" +"%d-%m-%Y")"
mv -n "$f" "$(date -r "$f" +"%d-%m-%Y/%H-%M-%S").mp4"
done
I'm getting this output (with testfile 1.mp4):
It creates the folder based on the file's creation date;
it renames the file to it's creation time;
Then, it returns mkdir: cannot create directory ‘1.mp4’: File exists
If two or more files, only one gets renamed and moved as described. The others stay the same and terminal returns:
mkdir: cannot create directory ‘1.mp4’: File exists
mkdir: cannot create directory ‘2.mp4’: File exists
mkdir: cannot create directory ‘12-12-2018’: File exists
Could someone help me out? Better suggestions? Thanks!
Honestly I would just use Perl or Python for this. Here's how to embed either in a shell script.
Here's a perl script that doesn't use any libraries, even ones that ship with Perl (so it'll work without extra packages on distributions like CentOS that don't ship with the entire Perl library). The perl script launches one new process per file in order to perform the copy.
perl -e '
while (<"*.m{p4,kv}">) {
my $path = $_;
my ($prefix, $year, $month, $day, $hour, $minute, $ext) =
split /[.-_]/, $path;
my $sec = q[00];
die "unexpected prefix ($prefix) in $path"
unless $prefix eq q[rec];
die "unexpected extension ($ext) in $path"
unless $ext eq q[mp4] or $ext eq q[mkv];
my $dir = "$day-$month-$year";
my $name = "$hour-$min-$sec" . q[.] . $ext;
my $destpath = $dir . q[/] . $name;
die "$dir . $name is unexpectedly a directory" if (-d $dir);
system("cp", "--", $path, $destpath);
}
'
Here's a Python example, it's compatible with either Python 2 or Python 3 but does use the standard library. The Python script does not spawn any additional processes.
python3 -c '
import os.path as path
import re
from glob import iglob
from itertools import chain
from os import mkdir
from shutil import copyfile
for p in chain(iglob("*.mp4"), iglob("*.mkv")):
fields = re.split("[-]|[._]", p)
prefix = fields[0]
year = fields[1]
month = fields[2]
day = fields[3]
hour = fields[4]
minute = fields[5]
ext = fields[6]
sec = "00"
assert prefix == "rec"
assert ext in ["mp4", "mkv"]
directory = "".join([day, "-", month, "-", year])
name = "".join([hour, "-", minute, "-", sec, ".", ext])
destpath = "".join([directory, "/", name])
assert not path.isdir(destpath)
try:
mkdir(directory)
except FileExistsError:
pass
copyfile(src=p, dst=destpath)
'
Finally, here's a bash solution. It splits paths using -, ., and _ and then extracts various subfields by indexing into $# inside a function. The indexing trick is portable, although regex substitution on variables is a bash extension.
#!/bin/bash
# $1 $2 $3 $4 $5 $6 $7 $8
# path rec YY MM DD HH MM ext
process_file() {
mkdir "$5-$4-$3" &> /dev/null
cp -- "$1" "$5-$4-$3"/"$6-$7-00.$8"
}
for path in *.m{p4,kv}; do
[ -e "$path" ] || continue
# NOTE: two slashes are needed in the substitution to replace everything
# read -a ARRAYVAR <<< ... reads the words of a string into an array
IFS=' ' read -a f <<< "${path//[-_.]/ }"
process_file "$path" "${f[#]}"
done
If you cd /to/some/directory/containing_your_files then you could use the following script
#!/usr/bin/env bash
for f in rec_????-??-??_??-??.m{p4,kv} ; do
dir=${f:4:10} # skip 4 chars ('rec_') take 10 chars ('YYYY_MM_DD')
fnm=${f:15} # skip 15 chars, take the remainder
test -d "$dir" || mkdir "$dir"
mv "$f" "$dir"/"$fnm"
done
note ① that I have not exchanged the years and the days, if you absolutely need to do the swap you can extract the year like this, year=${dir::4} etc and ② that this method of parameter substitution is a Bash-ism, e.g., it doesn't work in dash.
your problem is: mkdir creates folder but you are giving filename for folder creation.
if you want to use fileName for folder creation then use it without extension.
the thing is you are trying to create folder with the already existing fileName
I have multiple folders where two files are present.
For example, 123.jpg, 456.jpg under folder ABC. I want to rename the files to IT1_ABC_123.v1.jpg and IT2_ABC_456.v1.jpg. Similarly, other folders have two files.
How can I do this in shell or Perl?
Try this, using shell and perl:
mkdir /tmp/test; cd $_
mkdir ABC DEF
touch {ABC,DEF}/{123,456}.jpg #creates four files, two in each directory
find|perl -nlE's,((.*)/(.+))/((123|456).jpg),$1/IT#{[++$n]}_$3_$4,&&say"$&\n$_\n"'
./ABC/123.jpg
./ABC/IT1_ABC_123.jpg
./ABC/456.jpg
./ABC/IT2_ABC_456.jpg
./DEF/123.jpg
./DEF/IT3_DEF_123.jpg
./DEF/456.jpg
./DEF/IT4_DEF_456.jpg
Now, after confirming this is what you want, replace the say with a rename:
find|perl -nlE's,((.*)/(.+))/((123|456).jpg),$1/IT#{[++$n]}_$3_$4, and rename$&,$_'
The new filenames:
find -type f
./ABC/IT1_ABC_123.jpg
./ABC/IT2_ABC_456.jpg
./DEF/IT3_DEF_123.jpg
./DEF/IT4_DEF_456.jpg
This will find filenames with 123.jpg or 456.jpg and rename them.
s,,, is the search-replace and it returns 1 (the number of changes it made) which again leads to the right side of the and being done (the rename).
Filenames that doesn't match 123.jpg or 456.jpg isn't renamed since s,,, will return 0 and the and is "short cutted" since it then logically cannot be true with a false (0) left side. So then the rename is not executed.
This variant does the same, but might be easier to read:
find|perl -nlE 'rename$&,$_ if s,((.*)/(.+))/((123|456).jpg),$1/IT#{[++$n]}_$3_$4,'
I have found this pattern useful in many cases of mass renamings. Also, dedicated software for mass renaming with GUIs exists, which for some might be easier to use.
Rewritten as a program abc.pl, it could be:
#!/usr/bin/perl
while(<>){
chomp;
next if not s,((.*)/([A-Z]{3}))/(\d{3}\.jpg),$1/IT#{[++$n]}_$3_$4,;
print "Found: $&\nNew name: $_\n\n";
#rename $&, $_;
}
Run:
find|perl abc.pl
You can do this in core Perl using the File::Find, File::Basename, and File::Copy modules. You can test it out with the script below. It won't make any changes until you uncomment the line with the "move" function.
#! perl
use strict;
use warnings;
use File::Basename;
use File::Copy;
use File::Find;
my $root_dir = '/path/to/main/folder';
# Recursively searches for all files below the $root_dir
my #fileset;
find(
sub {
# Get the absolute file path
my $path = $File::Find::name;
# Only capture the path if not a directory
# You can add any number of conditions here
if (!-d $path) {
push #fileset, $path;
}
},
$root_dir
);
# set the IT counter in new file name
my $int = 1;
# list of all possible file suffixes to have fileparse() look for. It will
# capture the end of the file path verbatim (including the period) if it's
# in this array
my #suffixes = ('.jpg', '.txt');
my $previous_dir;
foreach my $old_path (#fileset) {
# split apart the basename of the file, the directory path, and the file suffix
my ($basename, $parent_dir, $suffix) = fileparse($old_path, #suffixes);
# strip off trailing slash so fileparse() will capture parent dir name correctly
$parent_dir =~ s{[/]$}{};
# capture just the name of the parent directory
my $parent_name = fileparse($parent_dir);
# Assemble the new path
my $new_path = $parent_dir . '/IT' . $int . '_'
. $parent_name . '_' . "$basename.v1" . $suffix;
# Move the file to rename (this is safer than using rename() for cross-platform)
# move $old_path, $new_path;
print "OLD PATH: $old_path\n";
print "NEW PATH: $new_path\n\n";
# Reset counter when dir changes
if (!$previous_dir) {
$previous_dir = $parent_dir; # set previous_dir on first loop
}
elsif($previous_dir ne $parent_dir) {
$previous_dir = $parent_dir; # update previous_dir to check next loop
$int = 0; # reset counter
}
$int++; # iterate the counter
}
Edit 2018-07-12: I've updated the answer to show how to reset the counter when the directory changes by evaluating the current path with the one used in the previous loop and updating accordingly. This is not tested so it may need some adjustments.
Given the abc/def examples given, the output should look something like this:
OLD PATH: /path/to/main/folder/abc/123.jpg
NEW PATH: /path/to/main/folder/abc/IT1_abc_123.v1.jpg
OLD PATH: /path/to/main/folder/abc/456.txt
NEW PATH: /path/to/main/folder/abc/IT2_abc_456.v1.jpg
OLD PATH: /path/to/main/folder/def/123.jpg
NEW PATH: /path/to/main/folder/def/IT1_def_123.v1.jpg
OLD PATH: /path/to/main/folder/def/456.jpg
NEW PATH: /path/to/main/folder/def/IT2_def_456.v1.jpg
I got a folder with the following zip files :
13162.zip 14864.zip 19573.zip 20198.zip
In console, when i run :
cd my_folder; echo `ls *{.zip,.ZIP}`
I got the following output (which is perfect) :
ls: cannot access *.ZIP: No such file or directory
13162.zip 14864.zip 19573.zip 20198.zip
Now when in ruby i try the same :
cmd= "cd my_folder; echo `ls {*.zip,*.ZIP}`";
puts `#{cmd}`
It only display :
ls: cannot access {*.zip,*.ZIP}: No such file or directory
=> nil
I try this solution :
Getting output of system() calls in Ruby
But it seem not work in my case.
How can i get the same output in ruby and in shell ?
Ruby Only
You can use Dir.glob with File::FNM_CASEFOLD for case-insensitive search :
Dir.chdir 'my_folder' do
Dir.glob('*.zip', File::FNM_CASEFOLD).each do |zip_file|
puts zip_file
end
end
#=>
# 19573.zip
# 13162.zip
# 14864.zip
# 20198.zip
# 12345.zIp
Ruby + bash
You can use find for case-insensitive search :
paths = `find my_folder -maxdepth 1 -iname '*.zip'`.split
#=> ["my_folder/19573.zip", "my_folder/13162.zip", "my_folder/14864.zip", "my_folder/20198.zip", "my_folder/12345.zIp"]
-printf '%P' can also be used to only display the filenames :
files = `find my_folder -maxdepth 1 -iname '*.zip' -printf '%P\n'`.split
#=> ["19573.zip", "13162.zip", "14864.zip", "20198.zip", "12345.zIp"]
I think this should work directly on the terminal:
echo 'system("ls *ZIP,*zip")' | ruby
or create a ruby file with the following contents
system("cd my_folder; ls {*.zip,*.ZIP}")
and then execute it. Once you write ls, you don't need echo!
I'm just barely familiar with bash scripts, and am trying to achieve being able to specify a source location (a directory with files / sub-directories) and a list of multiple destination directories, along with a list of files / directories from the source to exclude when copying & replacing.
The script should loop through each destination location, copy the files from the source and replace them in the destination location, excluding any files / directories listed in the script to skip.
In PHP it would be something like:
$source = '/home/user1/public_html';
$destinations = array(
'user2',
'user3',
'user4'
);
$exclude = array(
'/uploads/custom',
'/config/conf.php'
);
foreach ($destinations as $destination) {
$dir = '/home/' .$destination. '/public_html';
if (is_dir($dir)) {
exec('cp -R ' .$source. '/* ' .$dir);
// and somehow exclude overwriting any files / directories matching what's in the $exclude array
}
}
So, my question is, how would a bash script with similar functionality be written?
Here's how I ended up doing it:
#!/bin/bash
while IFS= read -r line || [[ $line ]]; do
if [[ -d "$line" ]]; then
rsync -rv --exclude-from "exclude.txt" /home/source/public_html/ "$line"
else
echo "Directory $line does not exist!"
fi
done < "accounts.txt"
This reads from a file called "accounts.txt" that has all the destination account directories in it, one per line. Ex:
/home/user1/public_html
/home/myblog/public_html
It loops through each one, and if the directory exists, it copies the files from /home/source/public_html to the destination. It also reads from a file called "exclude.txt" that includes, one per line, any files or directories to exclude. Ex:
.git
/uploads/custom
/config/conf.php
I am perl noob, and trying to do following:
Search for files with specific string in a directory recursively. Say string is 'abc.txt'
The file can be in two different sub-directories, say dir_1 or dir_2
Once the file is found, if it is found in dir_1, rename it to dir_1_abc.txt. If it is in dir_2, then rename it to dir_2_abc.txt.
Once all the files have been found and renamed, move them all to a new directory named, say dir_3
I don't care if I have to use any module to accomplish this. I have been trying to do it using File::Find::Rule and File::copy, but not getting the desired result. Here is my sample code:
#!/usr/bin/perl -sl
use strict;
use warnings;
use File::Find::Rule;
use File::Copy;
my $dir1 = '/Users/macuser/ParentDirectory/logs/dir_1'
my $dir2 = '/Users/macuser/ParentDirectory/logs/dir_2'
#ideally I just want to define one directory but because of the logic I am using in IF
#statement, I am specifying two different directory paths
my $dest_dir = '/Users/macuser/dir_3';
my(#old_files) = find(
file => (),
name => '*abc.txt',
in => $dir1, $dir2 ); #not sure if I can give two directories, works with on
foreach my $old_file(#old_files) {
print $old_file; #added this for debug
if ($dest_dir =~ m/dir_1/)
{
print "yes in the loop";
rename ($old_file, "dir_1_$old_file");
print $old_file;
copy "$old_file", "$dest_dir";
}
if ($dest_dir =~ m/dir_2/)
{
print "yes in the loop";
rename ($old_file, "dir_2_$old_file");
print $old_file;
copy "$old_file", "dest_dir";
}
}
The code above does not change the file name, instead when I am printing $old_file inside if, it spits the whole directory path, where the file is found, and it is prefixing the path with dir_1 and dir_2 respectively. Something is horribly wrong. Please help simply.
If you have bash ( I assume in OSX it is available), you can do this in a few lines (usually I put them in one line).
destdir="your_dest_dir"
for i in `find /Users/macuser/ParentDirectory/logs -type f -iname '*abc.txt' `
do
prefix=`dirname $i`
if [[ $prefix = *dir_1* ]] ; then
prefix="dir_1"
fi
dest="$destdir/${prefix}_`basename $i`"
mv "$i" "$dest"
done
The advantage of this method is that you can have many sub dirs under logs and you don't need to specify them. you can search for files like blah_abc.txt, tada_abc.txt too. If you want a exact match just juse abc.txt, instead of *abc.txt.
If the files can be placed in the destination as you rename them, try this:
#!/usr/bin/perl
use strict;
use File::Find;
use File::Copy;
my $dest_dir = '/Users/macuser/dir_3';
foreach my $dir ('/Users/macuser/ParentDirectory/logs/dir_1', '/Users/macuser/ParentDirectory/logs/dir_2') {
my $prefix = $dir; $prefix =~ s/.*\///;
find(sub {
move($File::Find::name, "$dest_dir/${prefix}_$_") if /abc\.txt$/;
}, $dir);
}
If you need to do all the renaming first and then move them all, you could either remember the list of files you have to move or you can make two passes making sure the pattern on the second pass is still OK after the initial rename in the first pass.