I need to rename files which are in this structure:
Dir1
--file1
--file2
-- ...
Dir2
--file1
--file2
-- ...
...
Dir62
--file1101
--file1102
-- ...
The new names would be 1_01,1_02 in 1 dir and 2_01,2_02 in 2nd dir and so on...
is there a way to do it in a single go...
Currently, I am using:
ls | cat -n | while read n f; do mv "$f" "10_$n.png"; done
Which work in 1 dir at a time...
Any better way, please?
If you run this command, it will use GNU Parallel to start a new bashshell in each of the directories in parallel, and run ls in each one in parallel independently:
parallel --dry-run -k 'cd {} && ls' ::: */
Sample Output
cd Dir01/ && ls
cd Dir02/ && ls
cd Dir78/ && ls
If you remove the --dry-run it will do it for real.
So, instead of running ls, let's now look at using the rename command in each of the directories. The following will rename all the files in a directory with sequentially increasing numbers ($N):
rename --dry-run '$_=$N' *
Sample Output
'file87' would be renamed to '1'
'file88' would be renamed to '2'
'file89' would be renamed to '3'
'fred' would be renamed to '4'
All the foregoing suggests the command you want would be:
parallel --dry-run -k 'cd {} && rename --dry-run "s/.*/{#}_\$N/" *' ::: */
You can run it as it is and it will just show you what it is going to do, without actually doing anything.
If you like the look of that, remove the first --dry-run and run it again and it will actually go into each subdirectory and do a dry-run of the rename, again without actually changing anything.
If you still like the look of the command, make a small copy of your files somewhere in a temporary directory and try removing both the --dry-run parameters ands if it lives up to your needs.
ls -1 -d ./*/ | cat -n | xargs -I % bash -c 'echo "%" | while read dirnum dirname; do { ls "${dirname}" | cat -n | while read filenum filename; do { mv -v "${dirname}${filename}" "${dirnum}_${filenum}.png"; }; done }; done'
We create a directory structure with mkdir and touch:
mkdir Dir{1,2,3,4,5}
touch Dir{1,2,3,4,5}/file{1,2,3}
Which gives the result:
1_1.png
1_2.png
1_3.png
2_1.png
2_2.png
2_3.png
3_1.png
3_2.png
3_3.png
4_1.png
4_2.png
4_3.png
5_1.png
5_2.png
5_3.png
Related
touch source
$ echo dest.{000000..999999} | tr ' ' '\n' | while read dest ; do echo cp -v source $dest ; done
cp -v source dest.000000
cp -v source dest.000001
cp -v source dest.000002
cp -v source dest.000003
cp -v source dest.000004
cp -v source dest.000005
cp -v source dest.000006
cp -v source dest.000007
cp -v source dest.000008
cp -v source dest.000009
...
Well, this is gonna take forever, mainly because each copy invokes a new cp process.
Let's try with xargs:
$ echo dest.{000000..999999} | xargs -n 1000 cp source
cp: target 'dest.000999' is not a directory
Yeah, right, when giving multiple arguments, cp assumes that n-1 arguments are source files, and the nth argument is a destination directory.
I need a command that works differently:
mycp source dest1 dest2 dest3 ...
How could I achieve this, without invoking a new process for each copy?
(based on the suggestion by Cyrus)
This works:
function multi-cp () {
local source="$1"
shift
tee "${#}" < "$source" > /dev/null
}
echo dest.{000000..999999} | xargs -n 1000 | while read -r destinations ; do
multi-cp source $destinations
done
We use while because xargs can not call functions (there are ways around this, but they have other problems). We still use xargs to split the arguments in manageable chunks.
This assumes that the arguments have no spaces (which is the case, since we are in control).
Goal
Separate files into directories according to their filenames, run a Bash script that reverse sorts them and assembles the content into one file (I know steps to achieve this are already documented on Stack Overflow, but please keep reading...)
Problem
Scripts work on all files but two
State
Root directory
dos-18-1-18165-03-for-sql-server-2012---15-june-2018.html
dos-18-1-18165-03-for-sql-server-2016---15-june-2018.html
dos-18-1-18176-03-for-sql-server-2012---10-july-2018.html
dos-18-1-18197-01-for-sql-server-2012---23-july-2018.html
dos-18-1-18197-01-for-sql-server-2016---23-july-2018.html
dos-18-1-18232-01-for-sql-server-2012---21-august-2018.html
dos-18-1-18232-01-for-sql-server-2016---21-august-2018.html
dos-18-1-18240-01-for-sql-server-2012---5-september-2018.html
dos-18-1-18240-01-for-sql-server-2016---5-september-2018.html
dos-18-2-release-notes.html
dos-18-2-known-issues.html
Separate the files into directories according to their SQL Server version or name
ls | grep "^dos-18-1.*2012.*" | xargs -i cp {} dos181-2012
ls | grep "^dos-18-1.*2016.*" | xargs -i cp {} dos181-2016
ls | grep ".*notes.*" | xargs -i cp {} dos-18-2-release-notes
ls | grep ".*known.*" | xargs -i cp {} dos-18-2-known-issues
Result (success)
/dos181-2012:
dos-18-1-18165-03-for-sql-server-2012---15-june-2018.html
dos-18-1-18176-03-for-sql-server-2012---10-july-2018.html
dos-18-1-18197-01-for-sql-server-2012---23-july-2018.html
dos-18-1-18232-01-for-sql-server-2012---21-august-2018.html
dos-18-1-18240-01-for-sql-server-2012---5-september-2018.html
/dos181-2016:
dos-18-1-18165-03-for-sql-server-2016---15-june-2018.html
dos-18-1-18197-01-for-sql-server-2016---23-july-2018.html
dos-18-1-18232-01-for-sql-server-2016---21-august-2018.html
dos-18-1-18240-01-for-sql-server-2016---5-september-2018.html
/dos-18-2-known-issues
dos-18-2-known-issues.html
/dos-18-2-release-notes
dos-18-2-release-notes.html
Variables (all follow this pattern)
dos181-2012.sh
file="dos181-2012"
export
dos-18-2-known-issues
file="dos-18-2-known-issues"
export
Reverse sort and assemble (assumes /$file exists; after testing all lines of code I believe this is where the problem lies):
cat $( ls "$file"/* | sort -r ) > "$file"/"$file".html
Result (success and failure)
dos181-2012.html has the correct content in the correct order.
dos-18-2-known-issues.html is empty.
What I have tried
I tried to ignore the two files in the command:
cat $( ls "$file"/* -i (grep ".*known.*" ) | sort -r ) > "$file"/"$file".html
Result: The opposite occurs
dos181-2012.html is empty
dos-18-2-known-issues.html is not empty
Thank you
I am completely baffled. Why do these scripts work on some files but not others? (I can share more information about the file contents if that will help, but the file contents are nearly identical.) Thank you for any insights.
first off, you question is quite incomplete. You start great, showing the input files and directories. But then you talk about variables and $files, but you do not show the code from which these originate. So I based my answer on the explanation in the first paragraph and what I deduced from the rest of the question.
I did this:
#!/bin/bash
cp /etc/hosts dos-18-1-18165-03-for-sql-server-2012---15-june-2018.html
cp /etc/hosts dos-18-1-18165-03-for-sql-server-2016---15-june-2018.html
cp /etc/hosts dos-18-1-18176-03-for-sql-server-2012---10-july-2018.html
cp /etc/hosts dos-18-1-18197-01-for-sql-server-2012---23-july-2018.html
cp /etc/hosts dos-18-1-18197-01-for-sql-server-2016---23-july-2018.html
cp /etc/hosts dos-18-1-18232-01-for-sql-server-2012---21-august-2018.html
cp /etc/hosts dos-18-1-18232-01-for-sql-server-2016---21-august-2018.html
cp /etc/hosts dos-18-1-18240-01-for-sql-server-2012---5-september-2018.html
cp /etc/hosts dos-18-1-18240-01-for-sql-server-2016---5-september-2018.html
cp /etc/hosts dos-18-2-release-notes.html
cp /etc/hosts dos-18-2-known-issues.html
DIRS='dos181-2012 dos181-2016 dos-18-2-release-notes dos-18-2-known-issues'
for DIR in $DIRS
do
if [ ! -d $DIR ]
then
mkdir $DIR
fi
done
cp dos-18-1*2012* dos181-2012
cp dos-18-1*2016* dos181-2016
cp *notes* dos-18-2-release-notes
cp *known* dos-18-2-known-issues
for DIR in $DIRS
do
/bin/ls -c1r $DIR >$DIR.html
done
The cp commands are just to create the files with something in them.
You did not specify how the directory names were produced, so I went with the easy option and listed them in a variable ($DIRS). These could be built based on the filenames, but you did not mention that.
Then created the directories (first for).
Then 4 cp commands. Your code is very complicated for something so basic. cp, like rm;mv;ls;... can do wildcard expansion, so there is no need for complex grep and xargs to copy files around.
Finally in the last for loop, list the files (ls), in 1 column (-c1, strictly output formatting), reversed the sort order (-r). The result of that ls is sent to a ".html" file of the same name as the directory.
I would like to:
Find latest modified file in a folder
Change some files in the folder
Find all files modified after file of step 1
Run a script on these files from step 2
This this where I've end up:
#!/bin/bash
var=$(find /home -type f -exec stat \{} --printf="%y\n" \; |
sort -n -r |
head -n 1)
echo $var
sudo touch -d $var /home/foo
find /home/ -newer /home/foo
Can anybody help me in achieving these actions ?
Use inotifywait instead to monitor files and check for changes
inotifywait -m -q -e modify --format "%f" {Path_To__Monitored_Directory}
Also, you can make it output to file, loop over it's contents and run your script on every entry.
inotifywait -m -q -e modify --format "%f" -o {Output_File} {Path_To_Monitored_Directory}
sample output:
file1
file2
Example
We are monitoring directory named /tmp/dir which contains file1 and file2.
The following script which monitor the whole directory and echo the file name:
#!/bin/bash
while read ch
do
echo "File modified= $ch"
done < <(inotifywait -m -q -e modify --format "%f" /tmp/dir)
Run this script and modify file1 echo "123" > /tmp/dir/file1, the script will output the following:
File modified= file1
Also you can look at this stackoverflow answer
How in bash would I remove all directories except the newest five created directories (by date created time)?
This is being used by a build script, and we want to clear up old builds. Thanks.
Edit: Date modified time is also fine.
It's not trivial to get a files creation time. See here for reference: https://unix.stackexchange.com/questions/20460/how-do-i-do-a-ls-and-then-sort-the-results-by-date-created
But if your happy to the last modification time instead (which should be fine here?) then something like this one-liner should do.
ls -dt */ | tail -n +6 | xargs rmdir
ls -d */ list directories -t lists them in order
tail -n +6 prints all but the last five lines
xargs rmdir calls rm -r on each of those dirs (or you can use rm -r if they're non-empty)
It could be done with ls but parsing ls output is not safe.
As there is no shell builtin to retrieve file time a solution using perl.
perl -e '
#A = sort {(stat($a))[9]<=>(stat($b))[9]} <*/>;
rmdir for splice (#A,0,-10)
'
Test
mkdir tmp && cd tmp
for d in {A..Z}; do mkdir "$d"; done
ls
perl -e '
#A = sort {(stat($a))[9]<=>(stat($b))[9]} <*/>;
rmdir for splice (#A,0,-10)
'
ls
EDIT: splice(#A,0,-10) could be changed to #A[0..$#A-10], maybe easier to read. To get all elements except the last 10.
I accidentally unzipped files into a wrong directory, actually there are hundreds of files... now the directory is messed up with the original files and the wrongly unzip files. I want to pick the unzipped files and remove them using shell script, e.g.
$unzip foo.zip -d test_dir
$cd target_dir
$ls test_dir | rm -rf
nothing happened, no files were deleted, what's wrong with my command ? Thanks !
The following script has two main benefits over the other answers thus far:
It does not require you to unzip a whole 2nd copy to a temp dir (I just list the file names)
It works on files that may contain spaces (parsing ls will break on spaces)
while read -r _ _ _ file; do
arr+=("$file")
done < <(unzip -qql foo.zip)
rm -f "${arr[#]}"
Right way to do this is with xargs:
$find ./test_dir -print | xargs rm -rf
Edited Thanks SiegeX to explain to me OP question.
This 'read' wrong files from test dir and remove its from target dir.
$unzip foo.zip -d /path_to/test_dir
$cd target_dir
(cd /path_to/test_dir ; find ./ -type f -print0 ) | xargs -0 rm
I use find -0 because filenames can contain blanks and newlines. But if not is your case, you can run with ls:
$unzip foo.zip -d /path_to/test_dir
$cd target_dir
(cd /path_to/test_dir ; ls ) | xargs rm -rf
before to execute you should test script changing rm by echo
Try
for file in $( unzip -qql FILE.zip | awk '{ print $4 }'); do
rm -rf DIR/YOU/MESSED/UP/$file
done
unzip -l list the content with a bunch of information about the zipped files. You just have to grep the file name out of it.
EDIT: using -qql as suggested by SiegeX
The following worked for me (bash)
unzip -l filename.zip | awk '{print $NF}' | xargs rm -Rf
Do this:
$ ls test_dir | xargs rm -rf
You need ls test_dir | xargs rm -rf as your last command
Why:
rm doesn't take input from stdin so you can't pipe the list of files to it. xargs takes the output of ls command and presents it to rm as input so that it can delete it.
Compacting the previous one. Run this command in the /DIR/YOU/MESSED/UP
unzip -qql FILE.zip | awk '{print "rm -rf " $4 }' | sh
enjoy