In my external HDD I have two partitions, one is for Mac and the other for Windows (FAT32). Since my Mac partition is almost full due to Time Machine backup, I want to move some of my old folders (in which are movies) from the Mac partition to the Windows partition. However, the FAT32 file system only allows each file less than 4GB. But my some of the folders contain files larger than 4G. I don't want to manually go through each folder , check the size and then copy & paste the folders of small size.
So my question is:
What is the command for moving all the folders (including the sub-directories) less than 4GB to the new partition? Does it have anything to do with the options of mv command?
Thanks
--- Update 12/7/2014---
I ran find . -mindepth 1 -type d -exec bash -c 'f="$1";set $(du -bs "$f"); \ [[ $1 -lt 4294967296 ]] && echo mv "$f" /dest-dir' - '{}' \; >> output.txt.
The following was the first a few lines of my output:
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.stZBByQJc0/Render
BASH=/bin/bash
BASH_ARGC=([0]="1")
BASH_ARGV=([0]=".")
BASH_EXECUTION_STRING=$'f="$1";set $(du -bs "$f"); \\\n [[ $1 -lt 4294967296 ]] && echo mv "$f" /Volumes/WIN_PANC/movies/'
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="3" [1]="2" [2]="53" [3]="1" [4]="release" [5]="x86_64-apple-darwin14")
BASH_VERSION='3.2.53(1)-release'
CLICOLOR=1
COLORFGBG='15;0'
They are not the folders I want to move. Am I doing right?
You can use this find command to list directories that have files greater than 4GB:
find . -mindepth 1 -type d -exec bash -c 'f="$1"; read s _ < <(du -s "$f"); \
[[ $s -lt 4194304 ]] && echo mv "$f" /dest-dir' - '{}' \;
Remove echo before mv command once you're satisfied with the listing.
Using the following codes can do this for you (for files >4G
#! /bin/bash
my_files=`ls --almost-all -1v -s -A --block-size=G|sort|sed -e 's#^[0-4]*G##g' -e '$ s#.*##g'`
echo "$my_files" >> my_files.txt
while read -r file; do
echo "MOVING FILE : $file"
mv "$file" "destination_location"
sleep 0.5
done < my_files.txt
rm -rf my_files.txt
Note: change your directory to where all your files to be copied are present in a terminal, then you can run script from the same terminal. Ensure you replace "destination_location" with the directory you want to move the file to inside the codes. Afterwards execute script.
Note: You will have to change your directory and run the codes in each directory.
Related
I needed to move a large s3 bucket to a local file store for a variety of reasons, and the files were stored as 160,000 directories with subdirectories.
As this is just far too many folders to look at with something like a gui FTP interface, I'd like to move the 160,000 root directories into, say, 320 directories - 500 directories in each.
I'm a newbie at bash scripting, and I just wrote this up, but I'm scared I'm going to mangle the whole thing and have to redo the transfer. I tested with [[ "$i" -ge 3 ]]; and some directories with subdirectories and it looked like it worked okay, but I'm quite nervous. Do not want to retransfer all this data.
i=0;
j=0;
for file in *; do
if [[ -d "$file" && ! -L "$file" ]];
then
((i++))
echo "directory $file is being written to assets_$j";
mv $file ./assets_$j/;
if [[ "$i" -ge 499 ]];
then
((j++));
((i=0));
fi
fi;
done
Thanks for the help!
find all the directories in the current folder.
Read a count of the folders.
Exec mv for each chunk
find . -mindepth 1 -maxdepth 1 -type d |
while IFS= readarray -n10 -t files && ((${#files[#]})); do
dest="./assets_$((j++))/"
echo mkdir -v -p "$dest"
echo mv -v "${files[#]}" "$dest";
done
On the condition that assets_1, assets_2, etc. do not exist in the working directory yet:
dirs=(./*/)
for (( i=0,j=1; i<${#dirs[#]}; i+=500,j++ )); do
echo mkdir ./assets_$j/
echo mv "${dirs[#]:i:500}" ./assets_$j/
done
If you're happy with the output, remove echos.
A possible way, but you have no control on the counter, is:
find . -type d -mindepth 1 -maxdepth 1 -print0 \
| xargs -0 -n 500 sh -c 'echo mkdir -v ./assets_$$ && echo mv -v "$#" ./assets_$$' _
This gets the counter of assets from the PID which only recycles when the wrap-around is reached (Linux PID recycling)
The order which findreturns is slight different then the glob * (Find command default sorting order)
If you want to have the sort order alphabetically, you can add a simple sort:
find . -type d -mindepth 1 -maxdepth 1 -print0 | sort -z \
| xargs -0 -n 500 sh -c 'echo mkdir -v ./assets_$$ && echo mv -v "$#" ./assets_$$' _
note: remove the echo if you are pleased with the output
I'm using
find . -mindepth 1 -maxdepth 1 -type d -exec ./mvsmalldirs.sh {} \;
to pass each subdirectory in Movies to this shell script:
SIZE=$(du -sb "$1" | cut -f1)
if [[ $SIZE -lt 5000000 ]]; then
mv -t "$1" ../Moved/
read -n 1 -s -r -p "Press any key to continue"
fi
At first I was having problems with spaces in the directory name but now that I've (I think) figured that out, something else really weird is happening... as soon as I run the find command, the script deletes the destination directory. If I'm barking up the wrong tree, could someone please tell me the right way to move all directories with a size of less than 5MB?
This command is guilty:
mv -t "$1" ../Moved/
The -t option must be followed by the target directory, which is ../Moved in your case, and not "$1" which is the source, as far as I understand. I suppose you mean:
mv -t ../Moved/ "$1"
I came across scenario using shell script. I need to delete the folder and Zip if it has the same name. Can any one please help me in this .
Example
Below is the directory path in which script need to search the same name in the directory (here it needs to print and delete example and example.zip)
Path:/tmp/test/
/tmp/test/example
/tmp/test/example.zip
/tmp/test/zack
You can use script as below:
#!/bin/bash
parent_dir=/tmp/test/
for i in `ls /tmp`
do
if [[ $i != *.zip ]]; then
if [[ -f /tmp/$i.zip ]]; then
array=$array" "$i
fi
fi
done
array=( $array )
for i in ${array[#]}
do
rm -r $parent_dir$i
rm -r $parent_dir$i".zip"
done
This should do a trick, even suitable for a cronjob:
find $DIR_PATH -type d -exec sh -c '[ -f "{}.zip" ] && rm -fr {}.zip {}' \;
Just set DIR_PATH to where youre searching
I havent been able to find an answer that best suites my needs, and I appologize if someone is able to find it easily.
I have a script that works to move files into folders based on their names. It worked perfectly until I realized that The files where missing their extension once I fixed this (another script was responsible for the file naming based on an email subject line) Once I fixed this problem It then started making a folder for each file. Is there anyway I can make this script drop everything in the folder name before the first (.)
Here is the script
#!/bin/bash
#folder script
#Benjamin D. Schran
MAIN_DIR=/PGHWH1/Photos
cd $MAIN_DIR
find . -maxdepth 1 -type f > SCRIPT_LOG1
find . -name '* *' | while read fname
do
new_fname=`echo $fname | tr " " "_"`
if [ -e $new_fname ]
then
echo "File $new_fname already exists. Not replacing $fname"
else
echo "Creating new file $new_fname to replace $fname"
mv "$fname" $new_fname
fi
done
find . -maxdepth 1 -type f | while read file;
do
f=$(basename "$file")
f1=${f%.*}
if [ -d "$f1" ];
then
mv "$f" "$f1"
else
mkdir "$f1"
chmod 777 "$f1"
mv "$f" "$f1"
fi
done
SCRIPTLOG=Script_log.$(date +%Y-%m-%d-%H-%M)
find . -type f > SCRIPT_LOG2
cd /PGHWH1/bin
sh scriptlog.sh > $SCRIPTLOG.html
mv $SCRIPTLOG.html /PGHWH1/log
rm $MAIN_DIR/SCRIPT_LOG1 $MAIN_DIR/SCRIPT_LOG2
What I need it to do is to take a files that is
Filename-date.%.jpg
and make
Foldername-date
then move the files of
Filename-date.1.jpg
Filename-date.2.jpg
Filename-date.3.jpg
to the appropriate folder
Foldername-date
but the current output is
Foldername-date.1
Foldername-date.2
Foldername-date.3
Any help at all would be appreciated
The following lines do the job in my bash:
#first create a tmp file with unique directory names
ls *.jpg | awk -F'.' '{print $1}' | uniq > dirs
#second create the directories
mkdir -p `cat dirs`
#third move the files
for i in `cat dirs`; do mv $i*.jpg $i/; done
#(optionally) remove the tmp file
rm dirs
I have the situation, where a template directory - containing files and links (!) - needs to be copied recursively to a destination directory, preserving all attributes. The template directory contains any number of placeholders (__NOTATION__), that need to be renamed to certain values.
For example template looks like this:
./template/__PLACEHOLDER__/name/__PLACEHOLDER__/prog/prefix___FILENAME___blah.txt
Destination becomes like this:
./destination/project1/name/project1/prog/prefix_customer_blah.txt
What I tried so far is this:
# first create dest directory structure
while read line; do
dest="$(echo "$line" | sed -e 's#__PLACEHOLDER__#project1#g' -e 's#__FILENAME__#customer#g' -e 's#template#destination#')"
if ! [ -d "$dest" ]; then
mkdir -p "$dest"
fi
done < <(find ./template -type d)
# now copy files
while read line; do
dest="$(echo "$line" | sed -e 's#__PLACEHOLDER__#project1#g' -e 's#__FILENAME__#customer#g' -e 's#template#destination#')"
cp -a "$line" "$dest"
done < <(find ./template -type f)
However, I realized that if I want to take care about permissions and links, this is going to be endless and very complicated. Is there a better way to replace __PLACEHOLDER__ with "value", maybe using cp, find or rsync?
I suspect that your script will already do what you want, if only you replace
find ./template -type f
with
find ./template ! -type d
Otherwise, the obvious solution is to use cp -a to make an "archive" copy of the template, complete with all links, permissions, etc, and then rename the placeholders in the copy.
cp -a ./template ./destination
while read path; do
dir=`dirname "$path"`
file=`basename "$path"`
mv -v "$path" "$dir/${file//__PLACEHOLDER__/project1}"
done < <(`find ./destination -depth -name '*__PLACEHOLDER__*'`)
Note that you'll want to use -depth or else renaming files inside renamed directories will break.
If it's very important to you that the directory tree is created with the names already changed (i.e. you must never see placeholders in the destination), then I'd recommend simply using an intermediate location.
First copy with rsync, preserving all the properties and links etc.
Then change the placeholder strings in the destination filenames:
#!/bin/bash
TEMPL="$PWD/template" # somewhere else
DEST="$PWD/dest" # wherever it is
mkdir "$DEST"
(cd "$TEMPL"; rsync -Hra . "$DEST") #
MyRen=$(mktemp)
trap "rm -f $MyRen" 0 1 2 3 13 15
cat >$MyRen <<'EOF'
#!/bin/bash
fn="$1"
newfn="$(echo "$fn" | sed -e 's#__PLACEHOLDER__#project1#g' -e s#__FILENAME__#customer#g' -e 's#template#destination#')"
test "$fn" != "$newfn" && mv "$fn" "$newfn"
EOF
chmod +x $MyRen
find "$DEST" -depth -execdir $MyRen {} \;