I have about 50 files in a directory that contain spaces, apostrophes, etc. How can I go about mass-renaming them to remove the apostrophes and replaces spaces with underscores?
I can do
ls | grep '*.txt' | xargs ....
but I'm not sure what to do in the xargs bit
I use ren-regexp, which is a Perl script that lets you mass-rename files very easily.
You'd do something like ren-regexp 's/ /_/g' *.txt.
$ ls -l
total 16
-rw-r--r-- 1 marc marc 7 Apr 11 21:18 That's a wrap.txt
-rw-r--r-- 1 marc marc 6 Apr 11 21:18 What's the time.txt
$ ren-regexp "s/\'//g" "s/ /_/g" *.txt
That's a wrap.txt
1 Thats a wrap.txt
2 Thats_a_wrap.txt
What's the time.txt
1 Whats the time.txt
2 Whats_the_time.txt
$ ls -l
total 16
-rw-r--r-- 1 marc marc 7 Apr 11 21:18 Thats_a_wrap.txt
-rw-r--r-- 1 marc marc 6 Apr 11 21:18 Whats_the_time.txt
Related
This question already has answers here:
How to ignore xargs commands if stdin input is empty?
(7 answers)
Closed 1 year ago.
me#new840:~/.config$ ls -al
total 196
drwx------ 40 me me 4096 Aug 22 22:08 .
drwxr-xr-x 52 me me 4096 Aug 22 12:12 ..
drwxr-xr-x 2 me me 4096 Jul 23 04:36 autostart
drwxr-xr-x 2 me me 4096 Jul 25 00:30 bcompare
drwx------ 20 me me 4096 Aug 22 22:37 Code
drwx------ 2 me me 4096 Aug 22 22:50 dconf
drwxr-xr-x 3 me me 4096 Aug 15 07:19 deadbeef
-rw------- 1 me me 1131 Nov 21 2016 dleyna-server-service.conf
me#new840:~/.config$ find /home/me/.config ! -user "me" -type f -print0 | xargs -0 ls -Al
total 188
drwxr-xr-x 2 me me 4096 Jul 23 04:36 autostart
drwxr-xr-x 2 me me 4096 Jul 25 00:30 bcompare
drwx------ 20 me me 4096 Aug 22 22:37 Code
drwx------ 2 me me 4096 Aug 22 22:56 dconf
drwxr-xr-x 3 me me 4096 Aug 15 07:19 deadbeef
-rw------- 1 me me 1131 Nov 21 2016 dleyna-server-service.conf
Actually,the result of find /home/me/.config ! -user "me" -type f -print0 | xargs -0 ls -Al should be empty.
How to avoid ls . when find | xargs ls has no result?
GNU xargs has the -r option to avoid doing anything if it receives no input, but this is not portable.
With find, the solution is simple: use -exec instead of xargs.
find /home/me/.config ! -user "me" -type f -exec ls -Al {} +
Tangentially, don't use ls in scripts.
Xargs supports the options -r or --no-run-if-empty. Then the ls command is not run when nothing is found.
Note that find supports the option -ls. Depending on what you want to see, you can skip xargs entirely.
Note: Assuming GNU environment.
I want my ls -l command to list both files and directories together rather than separating them. I also want a case insensitive list. For example, the following commands create the directories a and C and also the file b.txt:
% mkdir a C
% touch b.txt
Then I list them
tyler#Tylers-MacBook-Pro test % ls -l
total 0
drwxr-xr-x 2 tyler staff 64 Feb 12 12:06 C
drwxr-xr-x 2 tyler staff 64 Feb 12 12:06 a
-rw-r--r-- 1 tyler staff 0 Feb 12 12:06 b.txt
Note how the order is C, a, b.txt. I want it to list: a, b.txt, C (like this):
tyler#Tylers-MacBook-Pro test % ls -l
total 0
drwxr-xr-x 2 tyler staff 64 Feb 12 12:06 a
-rw-r--r-- 1 tyler staff 0 Feb 12 12:06 b.txt
drwxr-xr-x 2 tyler staff 64 Feb 12 12:06 C
How do I do this case insensitive list that doesn't separate files and directories.
Combined with sort, this should be what's required :
ls -l | sort -f -k 9,9
-f -k 9,9 means sort insensitively (-f) by 9th column (-k 9,9).
I have done these commands to concatenate the files into one file:
$ ls -1 | wc -l
16916
$ ls -1 *.txt | wc -l
16916
$ ls -lh | head -1
total 93M
$ cat *.txt > ../nectar_3.txt
$ ls -lh ../nectar_3.txt
-rw-r--r-- 1 llopis llopis 52M May 25 16:03 ../nectar_3.txt
Why is the resulting file size half of the sum of the size of all files? The only explanation I can found is about rounding in the ls -lh command, but I couldn't find anything (using ls -lk outputs almost the same 92.76953125M)
The total is rounded, and is not guaranteed to be accurate:
Simple example:
marc#panic$ ls -lk
total 24
-rw-r--r-- 1 marc marc 6000 May 25 08:39 test1.txt
-rw-r--r-- 1 marc marc 7000 May 25 08:39 test2.txt
-rw-r--r-- 1 marc marc 8000 May 25 08:39 test3.txt
Three simple files, total size = 21,000 bytes, yet the total shows 24.
I'm using bash shell.
Hi,ppl
Would be glad if someone could provide some kind of advice, because googling around yielded some answers
but couldn't still get the script to work.
I'am new to using bash script and got a script to modify because it was failing to copy
a large number of files from and input directory to an output directory after the files were processed.
Description:
We have a bunch of pdf's in a large directory.
We process a file called filename.pdf, after it's processed an additional file is created called filename.pdf.marker
Then both files filename.pdf.marker and filename.pdf shoud be moved from input/in directory to directory output/out.
We work with about 10 -15 thousands of files.
The script should do the following:
select all .marker file names
move.marker files from input/in directory to directory output/out (done in separate line)
remove the .marker from the selected filename,
move the file filename.pdf to the output/out directory
Old script (didn't work for a larger number of files) :
FILELIST=$(ls ${V04}/*.pdf.marker 2> /dev/null | sort)
for FILEMARKER in ${FILELIST}; do
FILENAME=${V04}/$(basename $FILEMARKER .marker)
mv ${FILENAME} ${VLOGDIR}/.
mv ${FILENAME}.marker ${VLOGDIR}/.
done
Because of that I needed to use xargs command.
Problem:
I managed to move the .marker files in a separate line.
Now i need to move the .pdf files with this script line.
find /input/in -iname "*.marker" -print0 | xargs -0 -r -I {} mv `basename {} .marker` /output/out
My problem lies in the part: `basename {} .marker`
Why isn't the string filename.pdf extracted from the string filename.pdf.marker, and substituted into the mv command ?
Any help i's welcome ;)
UPDATED
Corrected description of what script should do: Both filetypes .pdf
and .pdf.marker should be moved in my script not copied.
Added old script that didn't work well for larger amount of files.
The problem is that the command in backticks is executed once before xargs is ever invoked.
The fix is a bit harder, not least because your step 2 says 'copy' but the previous description suggests 'move'. I'd probably create a simple script to be invoked by xargs:
find /input/in -name '*.marker' -print0 | xargs -0 mover.sh
The contents of mover.sh might be:
for mrk_source in "$#"
do
pdf_source=$(echo "$mrk_source" | sed 's/\.marker$//')
mrk_target=$(echo "/output/out/$mrk_source" | sed 's%/input/in%%')
pdf_target=$(echo "/output/out/$pdf_source" | sed 's%/input/in%%')
mv "$mrk_source" "$mrk_target"
mv "$pdf_source" "$pdf_target"
done
Note that this code preserves any directory structure under /input/in but assumes that the corresponding directory exists under /output/out (without checking). It would be possible to alter the code to flatten any directory structure, or to create the directories as needed (exercise for the reader). There is a small sleight-of-hand going on in the file name manipulation in the two xxx_target assignment lines; I think it will work OK for relative names as well as absolute names, but be a little cautious with that part (test before using, in other words).
tripleee commented:
The echo and sed invocations are very brittle -- for example, echo on some platforms will interpret backslashes in the filename as escape sequences. Fortunately, you can use the shell's substitution mechanisms to mv "${mrk_source#.marker}" /output/out instead. (Why would you want to calculate the destination file name, when all you need to give to mv is the destination directory?)
I explained the destination file name - preserving sub-directories, so /input/in/dir1/abc.pdf goes to /output/out/dir1/abc.pdf; if you want to flatten the directory structure (or there is no directory structure), then simply specifying the destination is sufficient.
The problem with echo 'should not' be a problem in the sense that the original design of echo was simple and all the later additional ... baggage simply makes what should be utterly reliable into something horrendously unreliable. That said, there could be problems with names containing backticks, $(...) and so on. There are no problems with backticks or $(...) in the names. There is a problem with backslashes in the name.
$ mkdir -p input/in output/out
$ for name in a b 'c d' 'e f g' '$(cat x)' '`cat y`' 'a\\nb'
> do
> cp /dev/null input/in/"$name.pdf"
> cp /dev/null "input/in/$name.pdf.marker"
> done
$ ls -lR [io]*
input:
total 0
drwxr-xr-x 16 jleffler staff 544 Aug 22 00:45 in
input/in:
total 0
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\\nb.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\\nb.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf.marker
output:
total 0
drwxr-xr-x 2 jleffler staff 68 Aug 22 00:45 out
output/out:
$ find input/in -name '*.marker' -print0 | xargs -0 sh mover.sh
mv: rename input/in/a\nb.pdf to ./output/out/a
b.pdf: No such file or directory
$ ls -lR [io]*
input:
total 0
drwxr-xr-x 3 jleffler staff 102 Aug 22 00:46 in
input/in:
total 0
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\\nb.pdf
output:
total 0
drwxr-xr-x 15 jleffler staff 510 Aug 22 00:46 out
output/out:
total 0
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\nb.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf.marker
$
Using the Bash built-ins is sensible; I'm still stuck in the 1980s on occasion, and need reminding of that.
Solution that works with backslashes etc
for mrk_source in "$#"
do
pdf_source=${mrk_source%.marker}
mrk_target=${mrk_source/\/input\/in/\/output\/out}
pdf_target=${pdf_source/\/input\/in/\/output\/out}
mv "$mrk_source" "$mrk_target"
mv "$pdf_source" "$pdf_target"
done
With the same set of input files, this code works cleanly:
EDIT: As pointed out in the comments, this will not work if there are spaces in the filenames. In that case see #Jonathan Leffler's answer (even if there are no spaces now, you should probably use his version anyway, to avoid breakage when there suddenly are spaces...).
Since the command is expanded before it is executed, you can't use it that way. The command you'll give xargs would look like this:
xargs -0 -r -I {} mv {} /output/out
Since it tries to remove any path components, and the a .marker suffix, from the string {}.
I'd say you want to use a loop in this case:
for f in $(find /input/in -iname "*.marker"); do
mv `basename $f .marker` /output/out
done
With GNU Parallel you should be able to do:
ls "$V04"/*.pdf.marker | parallel -q mv {.} {} "$VLOGDIR"
This will work even if $V04 and $VLOGDIR contains ' " space \t.
Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ
The problem with the old script was, that you try to catch all entries in one var, which have size limits.
You can solve that, if you must not sort the entries in this way:
ls -1 "${V04}/*.pdf,.marker | while read FM;
do
mv "${FM}" "${VLOGDIR}/"
mv "${V04}/$(basename "${FM}" .marker)" "${VLOGDIR}/"
done;
The backticks are executed at evaluation time, not when xargs runs. Perhaps try something like this?
find /input/in -iname "*.marker" -print0 |
xargs -r0 -i sh -c 'mv `basename "{}" .marker` /output/out; mv "{}" /output/out'
Edit: The shell is still problematic here; if the file name contains double quotes, it will not parse correctly. Using a separate script might be better:
find /input/in -iname "*.marker" -exec ./myscript {} \;
where myscript contains the simple moving commands:
#!/bin/sh
mv `basename "$1" .marker` /output/out
mv "$1" /output/out
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Delete all but the most recent X files in bash
I have a script to create incremental backups daily and I need to delete all backups but last 5.
For example, I have this folders:
drwxr-xr-x 4 root root 4096 Oct 29 01:10 2010-10-29
drwxr-xr-x 4 root root 4096 Oct 30 01:10 2010-10-30
drwxr-xr-x 4 root root 4096 Oct 31 01:10 2010-10-31
drwxr-xr-x 4 root root 4096 Nov 1 01:10 2010-11-01
drwxr-xr-x 4 root root 4096 Nov 2 01:10 2010-11-02
drwxr-xr-x 4 root root 4096 Nov 3 01:10 2010-11-03
drwxr-xr-x 4 root root 4096 Nov 4 01:10 2010-11-04
drwxr-xr-x 4 root root 4096 Nov 5 01:10 2010-11-05
drwxr-xr-x 4 root root 4096 Nov 6 01:10 2010-11-06
drwxr-xr-x 4 root root 4096 Nov 7 01:10 2010-11-07
drwxr-xr-x 4 root root 4096 Nov 8 01:10 2010-11-08
And I need to maintain only the last 5 directories and delete the others. After command execute, I need to have only this:
drwxr-xr-x 4 root root 4096 Nov 4 01:10 2010-11-04
drwxr-xr-x 4 root root 4096 Nov 5 01:10 2010-11-05
drwxr-xr-x 4 root root 4096 Nov 6 01:10 2010-11-06
drwxr-xr-x 4 root root 4096 Nov 7 01:10 2010-11-07
drwxr-xr-x 4 root root 4096 Nov 8 01:10 2010-11-08
I don't need to delete previous to 5 days, I need to delete all except 5 last directories :)
Now I'm using:
find /backup/increment -maxdepth 1 -type d -mtime +5 -exec rm -rf {} \;
But I need to improved not based in time :)
EDIT: This is an example for a server that do backups all days, but I need an script that delete all folders previous to last 5 because my computer do backups at 00:10 at night, but not all nights the backup is done it, because my computer isn't working all days, and I need to have always the last 5 backups :)
use the tail command to print lines starting with the n th line (Option -n +N):
rm `ls -t | tail -n +6`
ls -t outputs the current directory sorted by time. tail -n +6 takes al lines starting with the 6th line. Quoting with backticks feeds the result of the pipe into the rm command.
OLD SOLUTION, not correct ...
use the head command, which prints the first n lines of some output:
rm `ls -t1 | head -n 5`
ls -t outputs the current directory sorted by time. head -n 5 takes the first five entries of the previous output. Quoting with backticks feeds the result of the pipe into the rm command.
Please try out first before applying to live data :) ...
The first thing that came to my mind. It's not elegant:
a=0;
for i in `ls -t`;
do
a=`expr $a + 1`;
if [ $a -gt 5 ]; then
echo "removing $i";
rm -rf $i
fi;
done
create two dummy files with the start and the end date
touch -t 1010290000 before
touch -t 2011042359 after
find all the files between the 2 dummy files and "rm -rf" the result
find . -newer before \! -newer after -exec rm -rf {} \;
ls -tr | perl -ne '{#files = <>; print #files[0..$#files-5 ]}' | xargs -n1 echo rm -rf
You would remove the echo before the rm -rf to get it to work.
The trick will be the -t option to ls, which sorts by modification time, from newest to oldest.
A really naive solution, using a temporary file, might go this way:
ls -t > /tmp/file_list
num_files_to_keep=5
# wc -l gets the line count of a file
# first word of wc output is the actual line count, and that's all we need, so
# delete everything after the space.
num_files=`wc -l /tmp/file_list | sed "s/ .*//"`
#if appropriate you should add a check for num_files < num_files_to_keep
num_files_to_delete=$(( $num_files - $num_files_to_keep ))
rm `tail -n $num_files_to_delete /tmp/file_list`