I know there are many of these questions out there already, but my situation is unique enough that none of them have helped me thus far.
I'm writing a script that will reformat my (poorly tagged) music library. It's currently in the form of Music/Genre/Artist/Album/##-Title.wav.
From the Music directory, I run my script1, which calls find -name "*.wav" -exec script2 {} ${other_args}. Script2 parses the path to variables, creates the appropriate output directory if it doesn't exist, and then calls ffmpeg to do the compression.
ex: ($sourcefilepath contains {} from find)
ffmpeg -i "$sourcefilepath" "$path/$to/$outfile.$extension"
If my file name or path contains no spaces, it works like a charm.
If it contains spaces, ffmpeg fails, with the following error:
Blues/Country Blues/Rhythm, Country And Blues/08-Natalie Cole & Reba
McEntire Since I Fell For You.wav: No such file or directory
Note: that's the source file it can't find. The output file appears to be working correctly, even with spaces. That's not the full command I'm running (it has some -metadata flags too, but that also appears to be working correctly. I'm debugging with the command I wrote above).
The interesting thing here is that ffmpeg is NOT interpreting $sourcefilepath as two args, it just can't find the file I specified. Running the following from the Music directory works.
ffmpeg -i Blues/Country\ Blues/Rhythm\,\ Country\ And\ Blues/08-Natalie\ Cole\ \&\ Reba\ McEntire\ \ \ Since\ I\ Fell\ For\ You.wav output.flac
I'm kind of stuck here. The other information on the web involves getting a command to see a variable with a space as one arg instead of multiple. I appear to have done that, but ffmpeg can't find my file now.
Normally I'd just use find -name "*.wav" -exec ffmpeg -i {} ${args} \; but I need to process the file path to extract my metadata.
I'm fairly new to shell scripting; does anybody know what's going on here?
My psychic powers suggest that at some point in your code, you're doing something along the lines of:
something=$(echo $filename | something)
This should instead have been:
something=$(echo "$filename" | something)
Here's a script demonstrating why this is a problem:
#!/bin/sh
filename="Reba McEntire Since I fell"
echo "$filename" # Correct, quoted
echo $filename # Incorrect, unquoted
It outputs:
Reba McEntire Since I fell
Reba McEntire Since I fell
You'll notice that
the first line has three spaces after the artist, as in your correct command.
the second line has one space after the artist, as in your error message.
This is due to unquoted variable expansion undergoing wordsplitting. echo then concatenates the arguments with single spaces. The net result is multiple whitespace coalescing into one space.
Related
A bit lowly a query but here goes:
bash shell script. POSIX, Mint 21
I just want one/any (mp3) file from a directory. As a sample.
In normal execution, a full run, the code would be such
for f in *.mp3 do
#statements
done
This works fine but if I wanted to sample just one file of such an array/glob (?) without looping, how might I do that? I don't care which file, just that it is an mp3 from the directory I am working in.
Should I just start this for-loop and then exit(break) after one statement, or is there a neater way more tailored-for-the-job way?
for f in *.mp3 do
#statement
break
done
Ta (can not believe how dopey I feel asking this one, my forehead will hurt when I see the answers )
Since you are using Linux (Mint) you've got GNU find so one way to get one .mp3 file from the current directory is:
mp3file=$(find . -maxdepth 1 -mindepth 1 -name '*.mp3' -printf '%f' -quit)
-maxdepth 1 -mindepth 1 causes the search to be restricted to one level under the current directory.
-printf '%f' prints just the filename (e.g. foo.mp3). The -print option would print the path to the filename (e.g. ./foo.mp3). That may not matter to you.
-quit causes find to exit as soon as one match is found and printed.
Another option is to use the Bash : (colon) command and $_ (dollar underscore) special variable:
: *.mp3
mp3file=$_
: *.mp3 runs the : command with the list of .mp3 files in the current directory as arguments. The : command ignores its arguments and does nothing.
mp3file=$_ sets the value of the mp3file variable to the last argument supplied to the previous command (:).
The second option should not be used if the number of .mp3 files is large (hundreds or more) because it will find all of the files and sort them by name internally.
In both cases $mp3file should be checked to ensure that it really exists (e.g. [[ -e $mp3file ]]) before using it for anything else, in case there are no .mp3 files in the directory.
I would do it like this in POSIX shell:
mp3file=
for f in *.mp3; do
if [ -f "$f" ]; then
mp3file=$f
break
fi
done
# At this point, the variable mp3file contains a filename which
# represents a regular file (or a symbolic link) with the .mp3
# extension, or empty string if there is no such a file.
The fact that you use
for f in *.mp3 do
suggests to me, that the MP3s are named without to much strange characters in the filename.
In that case, if you really don't care which MP3, you could:
f=$(ls *.mp3|head)
statement
Or, if you want a different one every time:
f=$(ls *.mp3|sort -R | tail -1)
Note: if your filenames get more complicated (including spaces or other special characters), this will not work anymore.
Assuming you don't have spaces in your filenames, (and I don't understand why the collective taboo is against using ls in scripts at all, rather than not having spaces in filenames, personally) then:-
ls *.mp3 | tr ' ' '\n' | sed -n '1p'
I'm writing a small piece of code that checks for .mov files in a specific folder over 4gb and writes it to a log.txt file by name (without an extension). I'm then reading the names into a while loop line by line which signals some archiving and copying commands.
Consider a file named abcdefg.mov (new) and a corresponding folder somewhere else named abcdefg_20180525 (<-*underscore timestamp) that also contains a file named abcedfg.mov (old).
When reading in the filename from the log.txt, I strip the extension to store the variable "abcdefg" ($in1) and i'm using that variable to locate a folder elsewhere that contains that matching string at the beginning.
My problem is with how the mv command seems to support a wild card in the "source" string, but not in the "destination" string.
For example i can write;
mv -f /Volumes/Myshare/SourceVideo/$in1*/$in1.mov /Volumes/Myshare/Archive
However a wildcard on the destination doesn't work in the same way. For example;
mv -f /Volumes/Myshare/Processed/$in1.mov Volumes/Myshare/SourceVideo/$in1*/$in1.mov
Is there an easy fix here that doesn't involve using another method?
Cheers for any help.
mv accepts a single destination path. Suppose that $in1 is abcdfg, and that $in1* expands to abcdefg_20180525 and abcdefg_20180526. Then the command
mv -f /dir1/$in1 /dir2/$in1*/$in1.mov
will be equivalent to:
mv -f /dir1/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
mv -f /dir1/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
mv -f /dir2/abcdefg_20180525/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
Moreover, because the destination file is the same in all three cases, the first two files will be overwritten by the third.
You should create a precise list and do a precise copy instead of using wild cards.
This is what I would probably do, generate a list of results in a file with FULL path information, then read those results in another function. I could have used arrays but I wanted to keep it simple. At the bottom of this script is a function call to scan for files of EXT mp4 (case insensitive) then writes the results to a file in tmp. then the script reads the results from that file in another function and performs some operation (mv etc.). Note, if functions are confusing , you can just remove the function name { } and name calls and it becomes a normal script again. functions are really handy, learn to love them!
#!/usr/bin/env bash
readonly SIZE_CHECK_LIMIT_MB="10M"
readonly FOLDER="/tmp"
readonly DESTINATION_FOLDER="/tmp/archive"
readonly SAVE_LIST_FILE="/tmp/$(basename $0)-save-list.txt"
readonly EXT="mp4"
readonly CASE="-iname" #change to -name for exact ext type upper/lower
function find_files_too_large() {
> ${SAVE_LIST_FILE}
find "${FOLDER}" -maxdepth 1 -type f "${CASE}" "*.${EXT}" -size +${SIZE_CHECK_LIMIT_MB} -print0 | while IFS= read -r -d $'\0' line ; do
echo "FOUND => $line"
echo "$line" >> ${SAVE_LIST_FILE}
done
}
function archive_large_files() {
local read_file="${SAVE_LIST_FILE}"
local write_folder="$DESTINATION_FOLDER"
if [ ! -s "${read_file}" ] || [ ! -f "${read_file}" ] ;then
echo "No work to be done ... "
return
fi
while IFS= read -r line ;do
echo "mv $line $write_folder" ;sleep 1
done < "${read_file}"
}
# MAIN (this is where the script starts) We just call two functions.
find_files_too_large
archive_large_files
it might be easier, i think, to change the filenames to the folder name initially. So abcdefg.mov would be abcdefg_timestamp.mov. I can always strip the timestamp from the filename easy enough after its copied to the right location. I was hoping i had a small syntax issue but i think there is no easy way of doing what i thought i could...
I think you have a basic misunderstanding of how wildcards work here. The mv command doesn't support wildcards at all; the shell expands all wildcards into lists of matching files before they get passed to the mv command as wildcards. Furthermore, the mv command doesn't know if the list of arguments it got came from wildcards or not, and the shell doesn't know anything about what the command is going to do with them. For instance, if you run the command grep *, the grep command just gets a list of names of files in the current directory as arguments, and will treat the first of them as a regex pattern ('cause that's what the first argument to grep is) to search the rest of the files for. If you ran mv * (note: don't do this!), it will interpret all but the last filename as sources, and the last one as a destination.
I think there's another source of confusion as well: when the shell expands a string containing a wildcard, it tries to match the entire thing to existing files and/or directories. So when you use Volumes/Myshare/SourceVideo/$in1*/$in1.mov, it looks for an already-existing file in a matching directory; AIUI the file isn't there yet, there's no match. What it does in that case is pass the raw (unexpanded) wildcard-containing string to mv as an argument, which looks for that exact name, doesn't find it, and gives you an error.
(BTW, should there be a "/" at the front of that pattern? I assume so below.)
If I understand the situation correctly, you might be able to use this:
mv -f /Volumes/Myshare/Processed/$in1.mov /Volumes/Myshare/SourceVideo/$in1*/
Since the filename isn't supplied in the second string, it doesn't look for existing files by that name, just directories with the right prefix; mv will automatically retain the filename from the source.
However, I'll echo #Sergio's warning about chaos from multiple matches. In this case, it won't overwrite files (well, it might, but for other reasons), but if it gets multiple matching target directories it'll move all but the last one into the last one (along with the file you meant to move). You say you're 100% certain this won't be a problem, but in my experience that means that there's at least a 50% chance that something you'd never have thought of will go ahead and make it happen anyway. For instance, is it possible that $in1 could wind up empty, or contain a space, or...?
Speaking of spaces, I'd also recommend double-quoting all variable references. You want the variables inside double-quotes, but the wildcards outside them (or they won't be expanded), like this:
mv -f "/Volumes/Myshare/Processed/$in1.mov" "/Volumes/Myshare/SourceVideo/$in1"*/
I can remove file extensions if I know the extensions, for example to remove .txt from files:
foreach file (`find . -type f`)
mv $file `basename $file .txt`
end
However if I don't know what kind of file extension to begin with, how would I do this?
I tried:
foreach file (`find . -type f`)
mv $file `basename $file .*`
end
but it wouldn't work.
What shell is this? At least in bash you can do:
find . -type f | while read -r; do
mv -- "$REPLY" "${REPLY%.*}"
done
(The usual caveats apply: This doesn't handle files whose name contains newlines.)
You can use sed to compute base file name.
foreach file (`find . -type f`)
mv $file `echo $file | sed -e 's/^\(.*\)\.[^.]\+$/\1/'`
end
Be cautious: The command you seek to run could cause loss of data!
If you don't think your file names contain newlines or double quotes, then you could use:
find . -type f -name '?*.*' |
sed 's/\(.*\)\.[^.]*$/mv "&" "\1"/' |
sh
This generates your list of files (making sure that the names contain at least one character plus a .), runs each file name through the sed script to convert it into an mv command by effectively removing the material from the last . onwards, and then running the stream of commands through a shell.
Clearly, you test this first by omitting the | sh part. Consider running it with | sh -x to get a trace of what the shell's doing. Consider making sure you capture the output of the shell, standard output and standard error, into a log file so you've got a record of the damage that occurred.
Do make sure you've got a backup of the original set of files before you start playing with this. It need only be a tar file stored in a different part of the directory hierarchy, and you can remove it as soon as you're happy with the results.
You can choose any shell; this doesn't rely on any shell constructs except pipes and single quotes and double quotes (pretty much common to all shells), and the sed script is version neutral too.
Note that if you have files xyz.c and xyz.h before you run this, you'll only have a file xyz afterwards (and what it contains depends on the order in which the files are processed, which needn't be alphabetic order).
If you think your file names might contain double quotes (but not single quotes), you can play with the changing the quotes in the sed script. If you might have to deal with both, you need a more complex sed script. If you need to deal with newlines in file names, then it is time to (a) tell your user(s) to stop being silly and (b) fix the names so they don't contain newlines. Then you can use the script above. If that isn't feasible, you have to work a lot harder to get the job done accurately — you probably need to make sure you've got a find that supports -print0, a sed that supports -z and an xargs that supports -0 (installing the most recent GNU versions if you don't already have the right support in place).
It's very simple:
$ set filename=/home/foo/bar.dat
$ echo ${filename:r}
/home/foo/bar
See more in man tcsh, in "History substitution":
r
Remove a filename extension '.xxx', leaving the root name.
I have several folders with some files that I would like to rename from
Foo'Bar - Title
to
Title
I'm using OS X 10.7. I've looked at other solutions, but none that address recursion very well.
Any suggestions?
There are two parts to your problem: Finding files to operate on recursively, and renaming them.
For the first, if everything is exactly one level below the current directory, you can just list the contents of every directory in the current directory (as in Mattias Wadman's answer above), but more generally (and possibly more easy to understand, to boot), you can just use the find command.
For the second, you can use sed and work out how to get the quoting and piping right (which you should definitely eventually learn), but it's much simpler to use the rename command. Unfortunately, this one isn't built in on Mac, but you can install it with, e.g., Homebrew, or just download the perl script and sudo install -m755 rename /usr/local/bin/rename.
So, you can do this:
find . -exec rename 's|[^/]* - ||' {} +
If you want to do a "dry run" to make sure it's right, add the "-n" flag to rename:
find . -exec rename -n 's|[^/]* - ||' {} +
To understand how it works, you really should read the tutorial for find, and the manpage for rename, but breaking it down:
find . means 'find all files recursively under the current directory'.
You can add additional tests to filter things (e.g., -type f if you want to skip everything but regular files, or `-name '*Title' if you want to only change files that end in 'Title'), but that isn't necessary for your use.
-exec … + means to batch up the found files, and pass as many of them as possible in place of any {} in the command that appears in the '…'.
rename 's|[^/]* - ||' {} means for each file in {}, apply the perl expression s|[^/]* - || to the filename, and, if the result is different, rename it to that result.
s|[^/]* - || means to match the regular expression '[^/]* -' and replace the match with '' (the empty string).
[^/]* - means to match any string of non-slash characters that ends with ' - '. So, in './A/FooBar - Title', it'll match the 'FooBar -'.
I should mention that, when I have something complicated to do like this, if after a few minutes and a couple attempts to get it right with find/sed/awk/rename/etc., I still haven't got it, I often just code it up imperatively with Python and os.walk. If you know Python, that might be easier for you to understand (although more verbose and less simple), and easier for you to modify to other use cases, so if you're interested, ask for that.
Try this:
ls -1 * | while read f ; do mv "$f" "`echo $f | sed 's/^.* - //'`" ; done
I recommend you to add a echo before mv before running it to make sure the commands look ok. And as abarnert noted in the comments this command will only work for one directory at a time.
Detailed explanation of the various commands:
ls -1 * will output a line for each file (and directory) in the current directory (except .-files). So this will be expanded in to ls -1 file1 file2 ..., -1 to ls tells it to list the filename only and one file per line.
The output is then piped into while read f ; ... ; done which will loop while read f returns zero, which it does until it reaches end of file. read f reads one line at a time from standard input (which in this case is the output from ls -1 ...) and store it in the the variable specified, in this case f.
In the while loop we run a mv command with two arguments, first "$f" as the source file (note the quotes to handle filenames with spaces etc) and second the destination filename which uses sed and ` (backticks) to do what is called command substitution that will call the command inside the backticks and be replaced it with the output from standard output.
echo $f | sed 's/^.* - //' pipes the current file $f into sed that will match a regular expression and do substitution (the s in s/) and output the result on standard output. The regular expression is ^.* - which will match from the start of the string ^ (called anchoring) and then any characters .* followed by - and replace it with the empty string (the string between //).
I know you asked for batch rename, but I suggest you to use Automator.
It works perfectly, and if you create it as a service you will have the option in your contextual menu :)
After some trial and error, I came across this solution that worked for me to solve the same problem.
find <dir> -name *.<oldExt> -exec rename -S .<oldExt> .<newExt> {} \;
Basically, I leverage the find and rename utilities. The trick here is figuring out where to place the '{}' (which represents the files that need to be processed by rename) of rename.
P.S. rename is not a built-in linux utility. I work with OS X and used homebrew to install rename.
This is for the Apple platform. My end goal is to do a find and replace for a line inside of the firefox preference file "prefs.js" to turn off updates. I want to be able to do this for all accounts on the Mac, including the user template (didn't include that in the examples). So far I've been able to get a list of all the paths that have the prefs.js file with this:
find /Users -name prefs.js
I then put the old preference and new preference in variables:
oldPref='user_pref("app.update.enabled", false);'
newPref='user_pref("app.update.enabled", true);'
I then have a "for loop" with the sed command to replace the old preference with the new preference:
for prefs in `find /Users -name prefs.js`
do
sed "s/$oldPref/$newPref/g" "$prefs"
done
The problem I'm running into is that the "find" command returns the full paths with the stupid "Application Support" in the path name like this:
/Users/admin/Library/Application Support/Firefox/Profiles/437cwg3d.default/prefs.js
When the command runs, I get these errors:
sed: /Users/admin/Library/Application: No such file or directory
sed: Support/Firefox/Profiles/437cwg3d.default/prefs.js: No such file or directory
I'm assuming that I somehow need to get the "find" command to wrap the outputted path in quotes for the "sed" command to parse it correctly? I'm I on the right path? I've tried to pipe the find command into sed to wrap quotes, but I can't get anything to work correctly. Please let me know if I should go about this differently. Thank you.
You don't want to for prefs in ... on a list of files that are output from find. For a more complete explanation of why this is bad, see Greg's wiki page about parsing ls. You would only use a for loop in bash if you could match the files using a glob, which is difficult if you want to do it recursively.
It would be better, if you can swing it, to use find ... -exec ... instead. Perhaps something like:
find /Users -name prefs.js -exec sed -i.bak -e "s/$oldPref/$newPref/" {} \;
The sed command line is executed once for each file found by find. The {} gets replaced with the filename. Sed's -i option lets you run it in-place, rather than requiring stdin/stdout. Check the man page for usage details.
(Grain of salt: I'm basing this on my experience with linux)
I think it less to do with sed and more to do with the way the for loop array is formed. When the the results of find are converted to an array, the space between Application and Support is treated as a delimiter.
There are several ways to work around this, but the easiest is probably to change the IFS variable. The IFS variable is an internal variable that your command line interpreter uses to separate fields (more info). You can change the IFS variable of the environment before running the find command.
Modified example from here:
#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for f in `find /Users -name prefs.js`
do
echo "$f"
done
# restore $IFS
IFS=$SAVEIFS