I've got an external drive with over 1TB of project files on it. I need to reformat this drive so I can reorganize it, however before I do that I need to transfer everything. The issue is I'm on a Mac and the drive is formatted as NTFS so all I can do is copy from it. I have tried to simply just copy and paste in Finder but the drive seems to lock up after roughly 15 min of copying that way. So I decided to write a bash script to iterate through the all 1000+ files one at a time. This seems to work for files that are without spaces but skips when it hits one.
Here is what I've hacked together so far.. I'm not too advanced in bash so any suggestions would be great on how you handle the spaces.
quota=800
size=`du -sg /Users/work/Desktop/TEMP`
files="/Volumes/Lacie/EXR_files/*"
for file in $files
do
if [[ ${size%%$'\t'*} -lt $quota ]];
then
echo still under quota;
cp -v $file /Users/work/Desktop/TEMP_EXR;
du -sg /Users/work/Desktop/TEMP_EXR;
else
echo over quota;
fi
done
(I'm checking for directory size because I'm having to split this temporary copy onto a few different place before I copy it all back onto the one reformatted drive.)
Hope I'm not misunderstanding. If you have problem with space character in filename, quote it. If you want bash to expand parameters inside it, use double quote.
cp -v "$file" /Users/work/Desktop/TEMP_EXR
You can put all the file names in an array, then iterate over that.
quota=800
size=`du -sg /Users/work/Desktop/TEMP`
files=( /Volumes/Lacie/EXR_files/* )
for file in "${files[#]}"
do
if [[ ${size%%$'\t'*} -lt $quota ]];
then
echo still under quota;
cp -v "$file" /Users/work/Desktop/TEMP_EXR;
du -sg /Users/work/Desktop/TEMP_EXR;
else
echo over quota;
fi
done
The two things to note are 1) quoting the array expansion in the for list, and 2) quoting $file for the cp command.
Related
I have a hundreds of image files in a structure like this:
path/to/file/100/image1.jpg
path/to/file/9999/image765.jpg
path/to/file/333/picture2.jpg
I'd like to remove the 4th part of the path (100,9999,333, ...) so that I get this:
path/to/file/image1.jpg
path/to/file/image765.jpg
path/to/file/picture2.jpg
In this case the image file names have no duplicates and the the target directory could be named entirely different if this makes things easier (e.g. target could be "another/path/to/the/images/image1.jpg"
The solution might be some combination of find/cut/rename command.
How can I do this in bash?
Since you only have "hundreds" of files, it's quite possible that you don't need to do anything special, and can just write:
mv path/to/file/*/*.jpg path/to/file/
But depending on the number of files and lengths of their names, this may turn out to be more than the kernel will let you pass to a single command, in which case you may need to write a for-loop instead:
for file in path/to/file/*/*.jpg ; do
mv "$file" path/to/file/
done
(Of course, this assumes you have mv on your path. There's no Bash builtin for renaming a file, so any approach will depend on what else is available on your system. If you don't have mv, you'll need to adjust the above accordingly.)
I recommend using ruakh's solution if it will work, but if you need to explicitly test for those numeric directories, here's an alternative.
I'm just using echo to pipe the list of names in, and to show the mv at the end, but you could use find (example in a comment) and remove the echo on the mv to make it live.
IFS=/
echo "path/to/file/100/image1.jpg
path/to/file/9999/image765.jpg
path/to/file/333/picture2.jpg" |
# find path/to/file -name "*.jpg" |
while read -r orig
do this=""
read -a line <<< "$orig"
for sub in "${line[#]}"
do if [[ "$sub" =~ ^[0-9]+$ ]]
then continue
else this="$this$sub/"
fi
done
old="${line[*]}"
echo mv "$old" "${this%/}"
done
mv path/to/file/100/image1.jpg path/to/file/image1.jpg
mv path/to/file/9999/image765.jpg path/to/file/image765.jpg
mv path/to/file/333/picture2.jpg path/to/file/picture2.jpg
This question already has answers here:
Rename filename to another name
(3 answers)
Closed 7 years ago.
Let´s say I have a bunch of files named something like this: bsdsa120226.nai bdeqa140223.nai and I want to rename them to 120226.nai 140223.nai. How can i achieve this using the script below?
#!/bin/bash
name1=`ls *nai*`
names=`ls *nai*| grep -Po '(?<=.{5}).+'`
for i in $name1
do
for y in $names
do
mv $i $y
done
done
Solution:
name1=`ls *nai*`
for i in $name1
do
y=$(echo "$i" | grep -Po '(?<=.{5}).+')
mv $i $y
done
This:
#!/bin/bash
shopt -s extglob nullglob
for file in *+([[:digit:]]).nai; do
echo mv -nv -- "$file" "${file##+([^[:digit:]])}"
done
Remove the echo if you're happy with the mv commands.
Note. This solution does not assume that there are 5 leading characters to delete. It will delete all the leading non-numeric characters.
Using only bash, you could do this:
for file in *nai* ; do
echo mv -- "$file" "${file:5}"
done
(Remove the echo when satisfied with the output.)
Avoid ls in scripts, except for displaying information. Use plain globbing instead.
See also How do I do string manipulations in bash? for more string manipulation techniques.
Your script can't work with that structure: if you have 5 files, it will call mv five times for the first file (once for each element in the second list), five times for the second, etc. You'd need to iterate over the two sets of names in lockstep. (It also doesn't deal with things like whitespace in filenames.)
You would be better off using rename (prename on some systems) since that allows you to use Perl regular expressions to do the renaming, along the lines of:
prename 's/^.{5}//' *.nai
The reason your script is not behaving is that, for every source file, you're attempting to rename it to every target file.
If you need to limit yourself to using that script, you need to work out the single target file for each source file, something like:
#!/bin/bash
for i in *.nai; do
y=$(echo "$i" | cut -c6-)
mv "$i" "$y"
done
If your system has rename tool, it's better to go with the simple rename command,
rename 's/^.{5}//' *.nai
It just remove the first 5 characters from the file name.
OR
for i in *.nai; do mv "$i" $(grep -oP '(?<=^.{5}).+' <<< "$i"); done
Thank you in advance for any help, this is coursework so further reading/ pointers is greatly appreciated.
I asked a question the other day relating to my own delete/trash/restore scripts and I have completed delete and trash as well as giving delete a backup text file for Restore to use later on.
However, instead of giving me errors, the Restore script just kinda stops in the console. Like when I type # ~/Restore -n the cursor skips to the next line without the usual # and I have to close it manually. Likewise without the -n option. The -n option should ask for a new location to restore to, and without it should restore to the files original location.
I'll post my script, see what y'all think.
#!/bin/bash
if [ "$1" == "-n" ]
then cd ~/rubbish
restore= grep $2 ~/store
filename= basename "$restore"
echo "Type the files new location"
read location
location1 = "readlink -f $location"
mv -i $filename "$location1" /$filename
else cd ~/rubbish
restore= grep $2 ~/store
filename= basename "$restore"
mv -i $filename "$location1" $location
fi
so, ~/rubbish is my own created directory to act as a recycle bin and ~/store is my text file which appends the deleted files readlink details on deletion. I can post the whole 3 scripts if necessary?
Many thanks!
If you call ~/Restore -n it will go to the if part and do a grep $2 ~/store. Since there is no parameter $2 it will result in grep ~/store, which tells grep to search for "~/store" in the input coming from standard input.
That's why your script stops and waits for input.
You can either test for a second parameter or enclose $2 in double quotes to make sure grep gets the correct number of parameters. Better yet, do both: 1. test for a second parameter and 2. enclose $2 in double quotes.
Some more points:
Don't put spaces around =
enclose commands in backticks `, if you want to capture the output
And no spaces between directory and filename
So, you should presumably write
restore=`grep "$2" ~/store`
filename=`basename "$restore"`
echo "Type the files new location"
read location
location1=`readlink -f "$location"`
mv -i $filename "$location1/$filename"
I suggest you look at bash info and follow the "Books and Resources".
I wrote one of these quite some time ago which I still use today. I don't have a restore script because I wrote it so that you could open your desktop trash can, right click and select "Restore". In other words it follows the Linux "trash info" standard.
http://wiki.linuxquestions.org/wiki/Scripting#KDE4_Command_Line_Trash_Can
I've got a directory with a few thousand files in it, named things like:
filename.ext
filename (1).ext
filename (2).ext
otherfile.ext
otherfile (1).ext
etc.
Most of the files with bracketed numbers are duplicates of the original, but in some cases they're not.
How can I keep my original files, delete the duplicates, but not lose the files that are different?
I know that I could rm *\).ext, but that obviously doesn't make sure that files match the original.
I'm using OS X, so I have a md5 program that functions sort of like md5sum in Linux, though it puts the hash at the end of the line instead of the beginning. I was thinking I could use an awk script to take the output of md5 *.ext | awk 'some script', find duplicates by md5, and delete them, but the command line is too long (bash: /sbin/md5: Argument list too long).
And I don't know what to write in the script. I was thinking of storing things in an array with this:
awk '{a[$NF]++} a[$NF]>1{sub(/).*/,""); sub(/.*(/,""); system("rm " $0);}'
But that always seems to delete my original.
What am I doing wrong? How do I do it right?
Thanks.
Your awk script deletes original files because when you sort your files, . (period) sorts after (space). SO the first file that's seen is numbered, not the original, and subsequent checks (including the one against the original) compare files to the first numbered one.
Not only does rm *\).txt fail to match the original, it loses files that may not have an original in the first place.
I wouldn't do this quite this way. Rather than checking every numbered file and verifying whether it matches an original, you can go through your list of originals, then delete the numbered files that match them.
Instead:
$ for file in *[^\)].txt; do echo "-- Found: $file"; rm -v $(basename "$file" .txt)\ \(*\).txt; done
You can expand this to check MD5's along the way. But it's more code, so I'll break it into multiple lines, in a script:
#!/bin/bash
shopt -s nullglob # Show nothing if a fileglob matches no files
for file in *[^\)].ext; do
md5=$(md5 -q "$file") # The -q option gives you only the message digest
echo "-- Found: $file ($md5)"
for duplicate in $(basename "$file" .ext)\ \(*\).ext; do
if [[ "$md5" = "$(md5 -q "$duplicate")" ]]; then
rm -v "$duplicate"
fi
done
done
As an alternative, you can probably get away with doing this a little more simply, with less CPU overhead than calculating MD5 digests. Unix and Linux have a shell tool called cmp, which is like diff without the output. So:
#!/bin/bash
shopt -s nullglob
for file in *[^\)].ext; do
for duplicate in $(basename "$file" .ext)\ \(*\).ext; do
if cmp "$file" "$duplicate"; then
rm -v "$file"
fi
done
done
If you don't need to use AWK, you could maybe do something simpler in bash:
for file in *\([0-9]*\)*; do
[ -e "$(echo "$file" | sed -e 's/ ([0-9]\+)//')" ] && rm "$file"
done
Hope this helps a little =)
I am writing a simple shell script to make automated backups, and I am trying to use basename to create a list of directories and them parse this list to get the first and the last directory from the list.
The problem is: when I use basename in the terminal, all goes fine and it gives me the list exactly as I want it. For example:
basename -a /var/*/
gives me a list of all the directories inside /var without the / in the end of the name, one per line.
BUT, when I use it inside a script and pass a variable to basename, it puts single quotes around the variable:
while read line; do
dir_name=$(echo $line)
basename -a $dir_name/*/ > dir_list.tmp
done < file_with_list.txt
When running with +x:
+ basename -a '/Volumes/OUTROS/backup/test/*/'
and, therefore, the result is not what I need.
Now, I know there must be a thousand ways to go around the basename problem, but then I'd learn nothing, right? ;)
How to get rid of the single quotes?
And if my directory name has spaces in it?
If your directory name could include spaces, you need to quote the value of dir_name (which is a good idea for any variable expansion, whether you expect spaces or not).
while read line; do
dir_name=$line
basename -a "$dir_name"/*/ > dir_list.tmp
done < file_with_list.txt
(As jordanm points out, you don't need to quote the RHS of a variable assignment.)
Assuming your goal is to populate dir_list.tmp with a list of directories found under each directory listed in file_with_list.txt, this might do.
#!/bin/bash
inputfile=file_with_list.txt
outputfile=dir_list.tmp
rm -f "$outputfile" # the -f makes rm fail silently if file does not exist
while read line; do
# basic syntax checking
if [[ ! ${line} =~ ^/[a-z][a-z0-9/-]*$ ]]; then
continue
fi
# collect targets using globbing
for target in "$line"/*; do
if [[ -d "$target" ]]; then
printf "%s\n" "$target" >> $outputfile
fi
done
done < $inputfile
As you develop whatever tool will process your dir_list.tmp file, be careful of special characters (including spaces) in that file.
Note that I'm using printf instead of echo so that targets whose first character is a hyphen won't cause errors.
This might work
while read; do
find "$REPLY" >> dir_list.tmp
done < file_with_list.txt