I want to create a file (containing the dir name) in any sub dir of e.g music that has at least one but maybe more .mp3 file. I want one file created no matter if there is one or more .mp3 in that dir and the dir can have whitespace.. I tried something like this: for i in $(find . -name "*.mp3" -printf "%h\n"|sort -u); do echo "$i" ; done
This breaks the path into 2 lines where the whitespace was so:
./directory one
outputs as:
./directory
one
The construct $( ... ) in your
for x in $(find ... | ... | ... ) ; do ... ; done
executes whatever is in $( ... ) and passes the newline separated output that you would see in the terminal if you had executed the ... command from the shell prompt to the for construct as a long list of names separated by blanks, as in
% ls -d1 b*
bianco nodi.pdf
bin
b.txt
% echo $(ls -d1 b*)
bianco nodi.pdf bin b.txt
%
now, the for cycle assign to i the first item in the list, in my example bianco and of course it's not what you want...
This situation is dealt with this idiom, in which the shell reads ONE WHOLE LINE at a time
% ls -d1 b* | while read i ; do echo "$i" ; ... ; done
in your case
find . -name "*.mp3" -printf "%h\n" | sort -u | while read i ; do
echo "$i"
done
hth, ciao
Edit
My anser above catches the most common case of blanks inside the filename, but it still fails if one has blanks at the beginning or the end of the filename and it fails also if there are newlines embedded in the filename.
Hence, I'm modifying my answer quite a bit, according to the advice from BroSlow (see comments below).
find . -name "*.mp3" -printf "%h\0" | \
sort -uz | while IFS= read -r -d '' i ; do
...
done
Key points
find's printf now separates filenames with a NUL.
sort, by the -z option, splits elements to be sorted on NULs rather than on newlines.
IFS= stops completely the shell habit of splitting on generic whitespace.
read's option -d (this is a bashism) means that the input is split on a particular character (by default, a newline).
Here I have -d '' that bash sees as specifying NUL, where BroSlow had $'\0' that bash expands, by the rules of parameter expansion, to '' but may be clearer as one can see an explicit reference to the NUL character.
I like to close with "Thank you, BroSlow".
Related
i have a directory full of .png files with a random caracters in the middle of the filenames like
T1_021_É}ÉcÉjÉV_solid box.png
T1_091_ÉRÉjÉtÉ#Å[_City.png
T1_086_ÉnÉiÉ~ÉYÉL_holiday.png
I expect this after removing
T1_021_solid box.png
T1_091_City.png
T1_086_holiday.png
Thank you
Using for to collect the file lists and bash parameter expansion with substring removal, you can do the following in the directory containing the files:
for i in T1_*; do
beg="${i%_*_*}" ## trim from back to 2nd '_'
end="${i##*_}" ## trim from from through last '_'
mv "$i" "${beg}_$end" ## mv file to new name.
done
(note: you don't have to use variables beg and end you can just combing both parameter expansions to form the new filenaame, e.g. mv "$i" "${i%_*_*}_${i##*_}", up to you, but beg and end make things a bit more readable.)
Result
New file names:
$ ls -al T1_*
T1_021_solid
T1_086_holiday.png
T1_091_City.png
Just another way to approach it from bash only.
Using cut
You can use cut to remove the 3rd field using '_' as the delimiter with :
for i in T1_*; do
mv "$i" $(cut -d'_' -f-2,4- <<< "$i")
done
(same output)
The only drawback is the use of cut in the command substitution would require an additional subshell be spawned each iteration.
If the set of random characters have _ before and after
find . -type f -iname "T1_0*" 2>/dev/null | while read file; do
mv "${file}" "$(echo ${file} | cut -d'_' -f1,2,4-)"
done
Explanation:
Find all files that start with T1_
Read the list line by line using the while loop
Use _ as delimiter and cut the 3rd column
Use mv to rename
Filenames after renaming:
T1_021_solid box.png
T1_086_holiday.png
T1_091_City.png
I need to write a Bash script that achieve the following goals:
1) move the newest n pdf files from folder 1 to folder 2;
2) correctly handles files that could have spaces in file names;
3) output each file name in a specific position in a text file. (In my actual usage, I will use sed to put the file names in a specific position of an existing file.)
I tried to make an array of filenames and then move them and do text output in a loop. However, the following array cannot handle files with spaces in filename:
pdfs=($(find -name "$DOWNLOADS/*.pdf" -print0 | xargs -0 ls -1 -t | head -n$NUM))
Suppose a file has name "Filename with Space". What I get from the above array will have "with" and "Space" in separate array entries.
I am not sure how to avoid these words in the same filename being treated separately.
Can someone help me out?
Thanks!
-------------Update------------
Sorry for being vague on the third point as I thought I might be able to figure that out after achieving the first and second goals.
Basically, it is a text file that have a line start with "%comment" near the end and I will need to insert the filenames before that line in the format "file=PATH".
The PATH is the folder 2 that I have my pdfs moved to.
You can achieve this using mapfile in conjunction with gnu versions of find | sort | cut | head that have options to operate on NUL terminated filenames:
mapfile -d '' -t pdfs < <(find "$DOWNLOADS/*.pdf" -name 'file*' -printf '%T#:%p\0' |
sort -z -t : -rnk1 | cut -z -d : -f2- | head -z -n $NUM)
Commands used are:
mapfile -d '': To read array with NUL as delimiter
find: outputs each file's modification stamp in EPOCH + ":" + filename + NUL byte
sort: sorts reverse numerically on 1st field
cut: removes 1st field from output
head: outputs only first $NUM filenames
find downloads -name "*.pdf" -printf "%T# %p\0" |
sort -z -t' ' -k1 -n |
cut -z -d' ' -f2- |
tail -z -n 3
find all *.pdf files in downloads
for each file print it's modifition date %T with the format specifier # that means seconds since epoch with fractional part, then print space, filename and separate with \0
Sort the null separated stream using space as field separator using only first field using numerical sort
Remove the first field from the stream, ie. creation date, leaving only filenames.
Get the count of the newest files, in this example 3 newest files, by using tail. We could also do reverse sort and use head, no difference.
Don't use ls in scripts. ls is for nice formatted output. You could do xargs -0 stat --printf "%Y %n\0" which would basically move your script forward, as ls isn't meant to be used for scripts. Just that I couldn't make stat output fractional part of creation date.
As for the second part, we need to save the null delimetered list to a file
find downloads ........ >"$tmp"
and then:
str='%comment'
{
grep -B$((2**32)) -x "$str" "$out" | grep -v "$str"
# I don't know what you expect to do with newlines in filenames, but I guess you don't have those
cat "$tmp" | sed -z 's/^/file=/' | sed 's/\x0/\n/g'
grep -A$((2**32)) -x "$str" "$out"
} | sponge "$out"
where output is the output file name
assuming output file name is stored in variable "$out"
filter all lines before the %comment and remove the line %comment itself from the file
output each filename with file= on the beginning. I also substituted zeros for newlines.
the filter all lines after %comment including %comment itself
write the output for outfile. Remember to use a temporary file.
Don't use pdf=$(...) on null separated inputs. You can use mapfile to store that to an array, as other answers provided.
Then to move the files, do smth like
<"$tmp" xargs -0 -i mv {} "$outdir"
or faster, with a single move:
{ cat <"$tmp"; printf "%s\0" "$outdir"; } | xargs -0 mv
or alternatively:
<"$tmp" xargs -0 sh -c 'outdir="$1"; shift; mv "$#" "$outdir"' -- "$outdir"
Live example at turorialspoint.
I suppose following code will be close to what you want:
IFS=$'\n' pdfs=($(find -name "$DOWNLOADS/*.pdf" -print0 | xargs -0 -I ls -lt "{}" | tail -n +1 | head -n$NUM))
Then you can access the output through ${pdfs[0]}, ${pdfs[1]}, ...
Explanations
IFS=$'\n' makes the following line to be split only with "\n".
-I option for xargs tells xargs to substitute {} with filenames so it can be quoted as "{}".
tail -n +1 is a trick to suppress an error message saying "xargs: 'ls' terminated by signal 13".
Hope this helps.
Bash v4 has an option globstar, after enabling this option, we can use ** to match zero or more subdirectories.
mapfile is a built-in command, which is used for reading lines into an indexed array variable. -t option removes a trailing newline.
shopt -s globstar
mapfile -t pdffiles < <(ls -t1 **/*.pdf | head -n"$NUM")
typeset -p pdffiles
for f in "${pdffiles[#]}"; do
echo "==="
mv "${f}" /dest/path
sed "/^%comment/i${f}=/dest/path" a-text-file.txt
done
for f in $( tasks/1_uniq/ -type f -follow -print | sed -r 's/[[:blank:]]+/ /g' ); do
md5sum $f
done
And this is what is printing where it finds a space.
tasks/1_uniq/two/6/66/test/me
&
my
friends
I cant manage to escape the spaces properly.
this is due to word splitting, without quotes after expansions it is split by characters in '$IFS' (space tab and newline), by double quoting the whole expansion will be taken as an argument. It seems you want to split by newlines. It can be done easily with read;
while read filepath; do
md5sum "$filepath" # note the double quotes to avoid word splitting
done < <( tasks/1_uniq/ -type f -follow -print | sed -r 's/[[:blank:]]+/ /g' )
I don't understand sed command which modifies filenames, it will give wrong filenames, another option
find ... -exec md5sum {} +
where ... is replaced with options
From your original code, it seems that your goal is to print out the md5 checksums for all files under a directory. In that case, you can simply use rhash
rhash -r -M dir/
-r for recursive and -M for md5 hash sum
Need help on writing a bash script that will rename files that are being outputted as file name.suffix.date I need these files to be rewritten as name.date.suffix instead.
Edited:
Changed suffix from date to ~
Here's what I have so far:
find . -type f -name "*.~" -print0 | while read -d $'\0' f
do
new=`echo "$f" | sed -e "s/~//"`
mv "$f" "$new"
done
This changes the suffix back to original but can't figure out how to get the date to be named before the extension (fname??)
You can use regular expression matching to pull apart the original file name:
find . -type f -name "*.~" -print0 | while read -d $'\0' f
do
dir=${f%/*}
fname=${f##*/}
[[ $fname =~ (.+)\.([^.]+)\.([^.]+)\.~$ ]] || continue
name=${BASH_REMATCH[1]}
suffix=${BASH_REMATCH[2]}
d=${BASH_REMATCH[3]}
mv "$f" "$dir/$name.$d.$suffix"
done
Bash-only solution:
while IFS=. read -r -u 9 -d '' name suffix date tilde
do
mv "${name}.${suffix}.${date}.~" "${name}.${date}.${suffix}"
done 9< <(find . -type f -name "*.~" -print0)
Notes:
-d '' gives you the same result as -d $'\0'
Splits file names by the dots while reading them. Of course this means it would break if there are dots anywhere else.
Should otherwise work with pretty much any filenames, including those containing space, newlines and other funny business.
create a list of the files first and redirect to a file.
ls > fileList.txt
Open the file and read line by line in Perl. Use a regex to match the parts of the files and capture them like this
my ($fileName,$suffix,$date)=($WholeFileName=~/(.*)\.(.*)\.(.*)/);
This should capture the three seperate variables for you. Now all you need to do is move the old file to the new file name. The new file name will be a concatenation of the above three variables that you have got. $newFileName=$fileName. ".".$date.".".$suffix. If you have a sample fileName post a comment and I can reply with a short script. Perl is not the only way. You could just use bash or awk and find alternate ways to do this.
cut each part of your filenames:
FIN=$(echo test.12345.ABCDEF | sed -e 's/[a-zA-Z0-9]*[\\.][a-zA-Z0-9]*[\\.]//')
DEBUT=$(echo test.12345.ABCDEF | sed -e 's/[\\.][a-zA-Z0-9]*[\\.][a-zA-Z0-9]*//')
MILIEU=$(echo test.12345.ABCDEF | sed -e 's/'${FIN}'//' -e 's/'${DEBUT}'//' -e 's/[\.]*//g')
paste each part as expected:
echo ${DEBUT}.${FIN}.${MILIEU}
rename --no-act 's/\(name-regex\).\(suffix-regex\).\(date-regex\)/\1.\3.\2' *
Tweak the three regexes to fit your file names, and remove --no-act when you're happy with the result to actually rename the files.
I'm grabbing all the filenames from a directory and I want to create a comma delimited string of these filenames so that I can pass that string as an argument to an application. This is my code snippet:
if [[ -n $(ls | grep lpt) ]]; then
for files in $(find . -maxdepth 1 -type f); do
#parse output into variable fileList
done
fi
How do I accomplish this?
Think easier:
find . -maxdepth 1 -type f -printf '%P,' | sed -e 's/,$/\n/'
The sed expression replaces the terminal , by a linebreak.
You should use the one-liners shown in the other answers, but in order to fill in your script, you could do:
if [[ -n $(ls | grep lpt) ]]; then
for file in $(find . -maxdepth 1 -type f); do
#parse output into variable fileList
fileList="$file,$fileList"
done
fi
#now remove the trailing comma from the fileList
fileList=$(sed 's/,$//' <<< "$fileList")
(Note: your for-loop won't work correctly if your filenames have spaces in them)
Instead of your loop, you could use find's -exec option, along with shell expansion:
fileList=$(find . -maxdepth 1 -type f -exec echo -n "{}," \; | sed 's/,$//')
The sed bit is just to remove the trailing comma. sed is used to edit input streams, i.e. here, it gets piped text from find and is editing what it's getting. Since the command given leaves an extra , at the end, sed uses its substitution command (s) to get rid of it. The form is:
s/EXPRESSION/REPLACEMENT/
So ,$ means "a comma at the end of the line, since $ means "at the end of the line", and the nothingness between the second and third slashes means it gets replaced by nothing.
As far as the \; in find, that's just a requirement for using -exec, so it knows when it's done reading commands, and it's in the man page. :)