I need to search through many files in a directory for a list of keywords and add a prefix to all of them. For example, if various files in my directory contained the terms foo, bar, and baz, I would need to change all instances of these terms to: prefix_foo, prefix_bar, and prefix_baz.
I'd like to write a shell script to do this so I can avoid doing the search one keyword at a time in SublimeText (there are a lot of them). Unfortunately, my shell-fu is not that strong.
So far, following this advice, I have created a file called "replace.sed" with all of the terms formatted like this:
s/foo/prefix_foo/g
s/bar/prefix_bar/g
s/baz/prefix_baz/g
The terminal command it suggests to use with this list is:
sed -f replace.sed < old.txt > new.txt
I was able to adapt this to replace instances within the file (instead of creating a new file) by setting up the following script, which I called inline.sh:
#!/bin/sh -e
in=${1?No input file specified}
mv $in ${bak=.$in.bak}
shift
"$#" < $bak > $in
Putting it all together, I ended up with this command:
~/inline.sh old.txt sed -f replace.sed
I tried this and it works, for one file at a time. How would I adapt this to search and replace through all of the files in my entire directory?
for f in *; do
[[ -f "$f" ]] && ~/inline.sh "$f" sed -f ~/replace.sed
done
In a script:
#!/bin/bash
files=`ls -1 your_directory | egrep keyword`
for i in ${files[#]}; do
cp ${i} prefix_${i}
done
This will, of course, leave the originals where they are.
Related
I have a very large selection of files eg.
foo_de.vtt, foo_en.vtt, foo_es.vtt, foo_fr.vtt, foo_pt.vtt, baa_de.vtt, baa_en.vtt, baa_es.vtt, baa_fr.vtt, baa_pt.vtt... etc.
I have created a tab separated file, filenames.txt containing the current string and replacement string eg.
foo 1000
baa 1016
...etc
I want to rename all of the files to get the following:
1000_de.vtt, 1000_en.vtt, 1000_es.vtt, 1000_fr.vtt, 1000_pt.vtt, 1016_de.vtt, 1016_en.vtt, 1016_es.vtt, 1016_fr.vtt, 1016_pt.vtt
I know I can use a utility like rename to do it manually term by term eg:
rename 's/foo/1000/g' *.vtt
could i chain this into an awk command so that it could run through the filenames.txt?
or is there an easier way to do it just in awk? I know I can rename with awk such as:
find . -type f | awk -v mvCmd='mv "%s" "%s"\n' \
'{ old=$0;
gsub(/foo/,"1000");
printf mvCmd,old,$0;
}' | sh
How can I get awk to process filenames.txt and do all of this in one go?
This question is similar but uses sed. I feel that being tab separated this should be quite easy in awk?
First ever post so please be gentle!
Solution
Thanks for all your help. Ultimately I was able to solve by adapting your answers to the following:
while read new old; do
rename "s/$old/$new/g" *.vtt;
done < filenames.txt
I'm assuming that the strings in the TSV file are literals (not regexes nor globs) and that the part to be replaced can be located anywhere in the filenames.
With that said, you can use mv with shell globs and bash parameter expansion:
#!/bin/bash
while IFS=$'\t' read -r old new
do
for f in *"$old"*.vtt
do
mv "$f" "${f/"$old"/$new}"
done
done < file.tsv
Or with GNU rename (more performant):
while IFS=$'\t' read -r old new
do
rename "$old" "$new" *"$old"*.vtt
done < file.tsv
This might work for you (GNU sed and rename):
sed -E 's#(.*)\t(.*)#rename -n '\''s/\1/\2/'\'' \1*#e' ../file
This builds a script which renames the files in the current directory using file to match and replace parts of the filenames.
Once you are happy with the results, remove the -n and the renaming will be enacted.
I'm trying to create a bash script based on a input file (list.txt). The input File contains a list of files with absolute path. The output should be a bash script (move.sh) which moves the files to another location, preserve the folder structure, but changing the target folder name slightly before.
the Input list.txt File example looks like this :
/In/Folder_1/SomeFoldername1/somefilename_x.mp3
/In/Folder_2/SomeFoldername2/somefilename_y.mp3
/In/Folder_3/SomeFoldername3/somefilename_z.mp3
The output file (move.sh) should looks like this after creation :
mv "/In/Folder_1/SomeFoldername1/somefilename_x.mp3" /gain/Folder_1/
mv "/In/Folder_2/SomeFoldername2/somefilename_y.mp3" /gain/Folder_2/
mv "/In/Folder_3/SomeFoldername3/somefilename_z.mp3" /gain/Folder_3/
The folder structure should be preserved, more or less.
after executing the created bash script (move.sh), the result should looks like this :
/gain/Folder_1/somefilename_x.mp3
/gain/Folder_2/somefilename_y.mp3
/gain/Folder_3/somefilename_z.mp3
What I've done so far.
1. create a list of files with absolute path
find /In/ -iname "*.mp3" -type f > /home/maars/mp3/list.txt
2. create the move.sh script
cp -a /home/maars/mp3/list.txt /home/maars/mp3/move.sh
# read the list and split the absolute path into fields
while IFS= read -r line;do
fields=($(printf "%s" "$line"|cut -d'/' --output-delimiter=' ' -f1-))
done < /home/maars/mp3/move.sh
# add the target path based on variables at the end of the line
sed -i -E "s|\.mp3|\.mp3"\"" /gain/"${fields[1]}"/|g" /home/maars/mp3/move.sh
sed -i "s|/In/|mv "\""/In/|g" /home/maars/mp3/move.sh
The script just use the value of ${fields[1]}, which is Folder_1 and put this in all lines at the end. Instead of Folder_2 and Folder_3.
The current result looks like
mv "/In/Folder_1/SomeFoldername1/somefilename_x.mp3" /gain/Folder_1/
mv "/In/Folder_2/SomeFoldername2/somefilename_y.mp3" /gain/Folder_1/
mv "/In/Folder_3/SomeFoldername3/somefilename_z.mp3" /gain/Folder_1/
rsync is not an option since I need the full control of files to be moved.
What could I do better to solve this issue ?
EDIT : #Socowi helped me a lot by pointing me in the right direction. After I did a deep dive into the World of Regex, I could solve my Issues. Thank you very much
The script just use the value of ${fields[1]}, which is Folder_1 and put this in all lines at the end. Instead of Folder_2 and Folder_3.
You iterate over all lines and update fields for every line. After you finished the loop, fields retains its value (from the last line). You would have to move the sed commands into your loop and make sure that only the current line is replaced by sed. However, there's a better way – see down below.
What could I do better
There are a lot of things you could improve, for instance
Creating the array fields with mapfile -d/ fields instead of printf+cut+($()). That way, you also wouldn't have problems with spaces in paths.
Use sed only once instead of creating the array fields and using multiple sed commands. You can replace step 2 with this small script:
cp -a /home/maars/mp3/list.txt /home/maars/mp3/move.sh
sed -i -E 's|^/[^/]*/([^/]*).*$|mv "&" "/gain/\1"|' /home/maars/mp3/move.sh
However, the best optimization would be to drop that three step approach and use only one script to find and move the files:
find /In/ -iname "*.mp3" -type f -exec rename -n 's|^/.*?/(.*?)/.*/(.*)$|/gain/$1/$2|' {} +
The -n option will print what will be renamed without actually renaming anything . Remove the -n when you are happy with the result. Here is the output:
rename(/In/Folder_1/SomeFoldername1/somefilename_x.mp3, /gain/Folder_1/somefilename_x.mp3)
rename(/In/Folder_2/SomeFoldername2/somefilename_y.mp3, /gain/Folder_2/somefilename_y.mp3)
rename(/In/Folder_3/SomeFoldername3/somefilename_z.mp3, /gain/Folder_3/somefilename_z.mp3)
It's not builtin to bash, but the mmv command is nice for this kind of mv where you need to use wildcards in paths. Something like the following should work:
mmv "in/*/*/*" "#1/#3"
Note that this won't create the directories for you - but in your example above it looks like these already exist?
I need to create a file that lists all the files in a folder into a text file, along with a comma and the number 15 after. For example
My folder has video.mp4, video2.mp4, picture1.jpg, picture2.jpg, picture3.png
I need the text file to read as follows:
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
picture3.png,15
No spaces, just filename.ext,15 on each line. I am using a raspberry pi. I am aware that the command ls > filename.txt would put all the file names into a folder, but how would I get a ,15 after every line?
Thanks
bash one-liner:
for f in *; do echo "$f,15" >> filename.txt; done
To avoid opening the output file on each iteration you may redirect the entire output with > filename.txt:
for f in *; do echo "$f,15"; done > filename.txt
$ printf '%s,15\n' *
picture1.jpg,15
picture2.jpg,15
picture3.png,15
video.mp4,15
video2.mp4,15
This will work if those are the only files in the directory. The format specifier %s,15\n will be applied to each of printf's arguments (the names in the current directory) and they will be outputted with ,15 appended (and a newline).
If there are other files, then the following would work too, regardless of whether there are files called like this or not:
$ printf '%s,15\n' video.mp4 video2.mp4 picture1.jpg picture2.jpg "whatever this is"
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
whatever this is,15
Or, on all MP4, PNG and JPEG files:
$ printf '%s,15\n' *.mp4 *.jpg *.png
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
picture3.png,15
Then redirect this to a file with printf ...as above... >output.txt.
If you're using Bash, then this will not make use of any external utility, as printf is built into the shell.
You need to do something like this:
#!/bin/bash
for i in $(ls folder_name); do
echo $i",15" >> filename.txt;
done
It's possible to do this in one line, however, if you want to create a script, consider code readability in the long run.
Edit 1: better solution
As #CristianRamon-Cortes suggested in the comments below, you should not rely on the output of ls because of the problems explained in this discussion: why not parse ls. As such, here's how you should write the script instead:
#!/bin/bash
cd folder_name
for i in *; do
echo $i",15" >> filename.txt;
done
You can skip the part cd folder_name if you are already in the folder.
Edit 2: Enhanced solution:
As suggested by #kusalananda, you'd better do the redirection after done to avoid opening the file in each iteration of the for loop, so the script will look like this:
#!/bin/bash
cd folder_name
for i in *; do
echo $i",15";
done > filename.txt
Just 1 command line using 2 msr commands recusively (-r) search specific files:
msr -rp your-dir1,dir2,dirN -l -f "\.(mp4|jpg|png)$" -PAC | msr -t .+ -o '$0,15' -PIC > save-file.txt
If you want to sort by time, add --wt to first command like: msr --wt -l -rp your-dirs
Sort by size? Add --sz but only the prior one is effective if use both --sz and --wt.
If you want to exclude some directory, add like: --nd "^(test|garbage)$"
remove tail \r\n in save-file.txt : msr -p save-file.txt -S -t "\s+$" -o "" -R
See msr.exe / msr.gcc48 etc in my open project https://github.com/qualiu/msr tools directory.
A solution without a loop:
ls | xargs -i echo {},15 > filename.txt
I have a lot of files which have a certain pattern:
some123_name4.with5.number01-02_and6-other7.stuff.txt
some123_name4.with5.number05-06_and6-other7.stuff.txt
some123_name4.with5.number11-12_and6-other7.stuff.txt
and I would like to rename them keeping the part in the middle number??-??. For example like:
different45_start.keep76.number01-02_but.change34_rest.txt
different45_start.keep76.number05-06_but.change34_rest.txt
different45_start.keep76.number11-12_but.change34_rest.txt
I have played around with expr, %% and ? but I didn't even manage to extract the number??-?? part of the filename.
This ought to do it (replace with actual patterns)
#!/bin/bash
for f in some123* ; do
mv $f `echo $f | sed -e 's/some123_name4.with5/different45_start.keep76/' -e 's/and6-other7.stuff/but.change34_rest/'`
done
May I suggest you use regexp'es to extract your numbers from the old name into the new name? Then it's just a question about
creating a new subdirectory (just in case you make a mistake)
using "ls" to list the file names (with options for 1 (one) name per line, not following down into subdirs)
iterating over the file names
In each iteration,
set the new name
run the copy commande "cp" using the old and the new names (but as a trick, copy down into your new subdirectory)
All in all, something like this:
mkdir NEW
ls -1d some* \
| while read FILE; do
NEWFILE=`echo "$FILE" \
| sed 's|^some12\\([0-9]\\)_name\\([0-9]\\)[.]with\\([0-9]\\)[.]number\\([0-9][0-9]-[0-9][0-9]\\)_and\\([0-9]\\)-other\\([0-9]\\)[.]stuff[.]txt$|different\\2\\3_start.keep\\6\\5.number\\4_but.change\\1\\2_rest.txt|'`
cp "$FILE" NEW/"$NEWFILE"
done
As you can see, due to the backticks (`) you have to use extra backslashes in the regexp.
Does this help you, as a start?
a possible solution using expr looks like the following:
for f in *number??-??*; do
fixedPart=$(expr "$f" : '.*\(number[0-9][0-9]-[0-9][0-9]\).*')
newName="different45_start.keep76.${fixedPart}_but.change34_rest.txt"
mv "$f" "$newName"
done
I have a few files that I want to copy and rename with the new file names generated by adding a fixed string to each of them.
E.g:
ls -ltr | tail -3
games.txt
files.sh
system.pl
Output should be:
games_my.txt
files_my.sh
system_my.pl
I am able to append at the end of file names but not before *.txt.
for i in `ls -ltr | tail -10`; do cp $i `echo $i\_my`;done
I am thinking if I am able to save the extension of each file by a simple cut as follows,
ext=cut -d'.' -f2
then I can append the same in the above for loop.
do cp $i `echo $i$ext\_my`;done
How do I achieve this?
You can use the following:
for file in *
do
name="${file%.*}"
extension="${file##*.}"
cp $file ${name}_my${extension}
done
Note that ${file%.*} returns the file name without extension, so that from hello.txt you get hello. By doing ${file%.*}_my.txt you then get from hello.txt -> hello_my.txt.
Regarding the extension, extension="${file##*.}" gets it. It is based on the question Extract filename and extension in bash.
If the shell variable expansion mechanisms provided by fedorqui's answer look too unreadable to you, you also can use the unix tool basename with a second argument to strip off the suffix:
for file in *.txt
do
cp -i "$file" "$(basename "$file" .txt)_my.txt"
done
Btw, in such cases I always propose to apply the -i option for cp to prevent any unwanted overwrites due to typing errors or similar.
It's also possible to use a direct replacement with shell methods:
cp -i "$file" "${file/.txt/_my.txt}"
The ways are numerous :)