bash script for putting thousands of files, into separate folders - bash

in a directory, i have thousands of files.
i use the following bash snippet, in order to make these thousands of files, be put in folders, that have each 1000 files,
and i face two issues:
a) now each folder has a prefix of dir_ while i would like it to have a name, that will have 6 digits, if less than 6 in the folder name, leading zeros should be added appropriately.
b) current script, puts first folder, into the last one, for example, i have dir_400325 as the last folder, and in there,i find the bash script i have run, and the dir_1000 folder, which is the first folder created. How could i change this, so the first folder, is not stored into the last one?
#!/bin/bash
c=0; d=1000; mkdir -p dir_${d}
for file in *
do
if [ $c -eq 1000 ]
then
d=$(( d + 1000 )); c=0; mkdir -p dir_${d}
fi
mv "$file" dir_${d}/
c=$(( c + 1 ))
done

You can use printf and a format string to generate your 6-digit directory name with leading zeros (%06d), demonstrated in a shell:
bash-4.4$ d=1001
bash-4.4$ dir_name=$(printf "/path/to/%06d" $d)
bash-4.4$ echo $dir_name
/path/to/001001
Using an absolute path may help ensure the files end up where you're expecting them and not in some subfolder of your current working directory.
#!/bin/bash
c=0
d=1000
dir_name=$(printf "/path/to/%06d" $d)
mkdir -p $dir_name
for file in *
do
if [ $c -eq 1000 ]
then
c=0
d=$(( d + 1000 ))
dir_name=$(printf "/path/to/%06d" $d)
mkdir -p $dir_name
fi
if [[ -f "$file" ]]
then
mv "$file" $dir_name
c=$(( c + 1 ))
fi
done

Related

How to list files with words exceeding n characters in all subdirectories

I have to write a shell script that creates a file containing the name of each text files from a folder (given as parameter) and it's subfolders that contain words longer than n characters (read n from keyboard).
I wrote the following code so far :
#!/bin/bash
Verifies if the first given parameter is a folder:
if [ ! -d $1 ]
then echo $1 is not a directory\!
exit 1
fi
Reading n
echo -n "Give the number n: "
read n
echo "You entered: $n"
Destination where to write the name of the files:
destinatie="destinatie"
the actual part that i think it makes me problems:
nr=0;
#while read line;
#do
for fisier in `find $1 -type f`
do
counter=0
for word in $(<$fisier);
do
file=`basename "$fisier"`
length=`expr length $word`
echo "$length"
if [ $length -gt $n ];
then counter=$(($counter+1))
fi
done
if [ $counter -gt $nr ];
then echo "$file" >> $destinatie
fi
done
break
done
exit
The script works but it does a few more steps that i don't need.It seems like it reads some files more than 1 time. If anyone can help me please?
Does this help?
egrep -lr "\w{$n,}" $1/* >$destinatie
Some explanation:
\w means: a character that words consist of
{$n,} means: number of consecutive characters is at least $n
Option -l lists files and does not print the grepped text and -r performs a recursive scan on your directory in $1
Edit:
a bit more complete version around the egrep command:
#!/bin/bash
die() { echo "$#" 1>&2 ; exit 1; }
[ -z "$1" ] && die "which directory to scan?"
dir="$1"
[ -d "$dir" ] || die "$dir isn't a directory"
echo -n "Give the number n: "
read n
echo "You entered: $n"
[ $n -le 0 ] && die "the number should be > 0"
destinatie="destinatie"
egrep -lr "\w{$n,}" "$dir"/* | while read f; do basename "$f"; done >$destinatie
This code has syntax errors, probably leftovers from your commented-out while loop: It would be best to remove the last 3 lines: done causes the error, break and exit are unnecessary as there is nothing to break out from and the program always terminates at its end.
The program appears to output files multiple times because you just append to $destinatie. You could simply delete that file when you start:
rm "$destinatie"
You echo the numbers to stdout (echo "$length") and the file names to $destinatie (echo "$file" >> $destinatie). I do not know if that is intentional.
I found the problem.The problem was the directory in which i was searching.Because i worked on the files from the direcotry and modified them , it seems that there remained some files which were not displayed in file explorer but the script would find them.i created another directory and i gived it as parameter and it works. Thank you for your answers
.

Using shell script, how do I put numbers in formats like 000?

I have large number of files which have file names in the format of XXX_name_YYY.out with YYY and YYY being numbers. I want to use a loop to move all files starting with XXX_name to a folder with the name 'XXX_name'. I am very new to shell scripting and only code a bit in C.
I would do something like this but the format of the numbers does not match the numbers in the file names.
c=1
while[c -le 100]
do
d=1
mkdir "$c"_name
while[d - le 100]
do
mv "$c"_name_"$d".out "$c"_name/"$c"_name_"$d".out
(( d++ ))
done
(( c++ ))
done
for FILE in [0-9][0-9][0-9]_name_[0-9][0-9][0-9].out; do
DIR=${FILE%_*.out}
[[ -d $DIR ]] || mkdir "$DIR" && echo mv "$FILE" $DIR/"
done
Remove echo when you're sure it works already.

bash mv every n files into different subfolders with consecutive names

I am trying to move n files in a directory to subdirectories. To be clear, lets say I have a folder named 1999.
I have 364 files under /1999 which are datafiles for each day. I want to move these files into subfolders named as weeks (1-52). In this case, I want to move 1st-7th files to folder named "1" ,then 8th-14th to folder named "2" and move files 358th-364th to folder named "52". So I need a loop of 7s :). How can I do this in bash or ksh?
Here's a pseudocode:
Get timestamp of the beginning of the year 1999 as FIRST_TS
Get timestamp of the end of the year 1999 as LAST_TS
For each file in /1999 as FILE
Get timestamp of FILE as FILE_TS
SUBFOLDER = (Integer) 52 * ((FILE_TS - FIRST_TS) / (LAST_TS - FIRST_TS)) + 1
Create folder named as /1999/SUBFOLDER if it doesn't exist
Move FILE to folder SUBFOLDER
End
Tools you can use:
* find (find /1999 -type f -maxdepth 1 ...)
* date (date -r "$FILE" '+%s')
Update:
#!/bin/bash
shopt -s extglob
YEAR=1999
ROOT=/$YEAR
for FILE in "$ROOT"/"$YEAR"_???.dat; do
N=${FILE##*_*(0)}; N=${N%.dat}; N=$(( (N - 1) / 7 + 1 ))
DEST=$ROOT/$N
[[ -d $DEST ]] || mkdir -p "$DEST" || {
echo "Failed to create destination directory \"$DEST\"." >&2
exit 1
}
mv -v -- "$FILE" "$DEST" || {
echo "Failed to move \"$FILE\" to \"$DEST\"." >&2
exit 1
}
done
This works for me:
#!/bin/bash
COUNTER=0
while [ $COUNTER -lt 52 ]; do
mkdir $((COUNTER + 1)) # Leave this out if the directories already exist.
mv `seq $((COUNTER*7 + 1)) $((COUNTER*7 + 7)) | xargs print "1999_%03d.dat "` $((COUNTER + 1))
COUNTER=$((COUNTER + 1))
done
But unless you want to stay a "total newbie" for ever you should try coming up with it on your own! If you can completely get everything this snippet is doing you'll be well on the way to understanding shell scripting, so try and read this code, play with it, and see what you can learn from it.

Renaming Multiples Files with Different Names in a Directory using shell

I've found most of the questions of this kind where the change in name has been same for the entire set of files in that directory.
But i'm here presented with a situation to give a different name to every file in that directory or just add a different prefix.
For Example, I have about 200 files in a directory, all of them with numbers in their filename. what i want to do is add a prefix of 1 to 200 for every file. Like 1_xxxxxxxx.png,2_xxxxxxxx.png...........200_xxxxxxxx.png
I'm trying this, but it doesnt increment my $i everytime, rather it gives a prefix of 1_ to every file.
echo "renaming files"
i=1 #initializing
j=ls -1 | wc -l #Count number of files in that dir
while [ "$i" -lt "$j" ] #looping
do
for FILE in * ; do NEWFILE=`echo $i_$FILE`; #swapping the file with variable $i
mv $FILE $NEWFILE #doing the actual rename
i=`expr $i+1` #increment $i
done
Thanks for any suggestion/help.
To increment with expr, you definitely need spaces( expr $i + 1 ), but you would probably be better off just doing:
echo "renaming files"
i=1
for FILE in * ; do
mv $FILE $((i++))_$FILE
done
i=1
for f in *; do
mv -- "$f" "${i}_$f"
i=$(($i + 1))
done

Bash - replacing targeted files with a specific file, whitespace in directory names

I have a large directory tree of files, and am using the following script to list and replace a searched-for name with a specific file. Problem is, I don't know how to write the createList() for-loop correctly to account for whitespace in a directory name. If all directories don't have spaces, it works fine.
The output is a list of files, and then a list of "cp" commands, but reports directories with spaces in them as individual dirs.
aindex=1
files=( null )
[ $# -eq 0 ] && { echo "Usage: $0 filename" ; exit 500; }
createList(){
f=$(find . -iname "search.file" -print)
for i in $f
do
files[$aindex]=$(echo "${i}")
aindex=$( expr $aindex + 1 )
done
}
writeList() {
for (( i=1; i<$aindex; i++ ))
do
echo "#$i : ${files[$i]}"
done
for (( i=1; i<$aindex; i++ ))
do
echo "/usr/bin/cp /cygdrive/c/testscript/TheCorrectFile.file ${files[$filenumber]}"
done
}
createList
writeList
Replace your entire script with these four lines:
find . -iname "search.file" | while read file
do
/usr/bin/cp /cygdrive/c/testscript/TheCorrectFile.file "$file"
done
(But isn't cp in /bin? Oh, on Cygwin it's in both. It's more portable, though, to use /bin.)

Resources