bash mv every n files into different subfolders with consecutive names - bash

I am trying to move n files in a directory to subdirectories. To be clear, lets say I have a folder named 1999.
I have 364 files under /1999 which are datafiles for each day. I want to move these files into subfolders named as weeks (1-52). In this case, I want to move 1st-7th files to folder named "1" ,then 8th-14th to folder named "2" and move files 358th-364th to folder named "52". So I need a loop of 7s :). How can I do this in bash or ksh?

Here's a pseudocode:
Get timestamp of the beginning of the year 1999 as FIRST_TS
Get timestamp of the end of the year 1999 as LAST_TS
For each file in /1999 as FILE
Get timestamp of FILE as FILE_TS
SUBFOLDER = (Integer) 52 * ((FILE_TS - FIRST_TS) / (LAST_TS - FIRST_TS)) + 1
Create folder named as /1999/SUBFOLDER if it doesn't exist
Move FILE to folder SUBFOLDER
End
Tools you can use:
* find (find /1999 -type f -maxdepth 1 ...)
* date (date -r "$FILE" '+%s')
Update:
#!/bin/bash
shopt -s extglob
YEAR=1999
ROOT=/$YEAR
for FILE in "$ROOT"/"$YEAR"_???.dat; do
N=${FILE##*_*(0)}; N=${N%.dat}; N=$(( (N - 1) / 7 + 1 ))
DEST=$ROOT/$N
[[ -d $DEST ]] || mkdir -p "$DEST" || {
echo "Failed to create destination directory \"$DEST\"." >&2
exit 1
}
mv -v -- "$FILE" "$DEST" || {
echo "Failed to move \"$FILE\" to \"$DEST\"." >&2
exit 1
}
done

This works for me:
#!/bin/bash
COUNTER=0
while [ $COUNTER -lt 52 ]; do
mkdir $((COUNTER + 1)) # Leave this out if the directories already exist.
mv `seq $((COUNTER*7 + 1)) $((COUNTER*7 + 7)) | xargs print "1999_%03d.dat "` $((COUNTER + 1))
COUNTER=$((COUNTER + 1))
done
But unless you want to stay a "total newbie" for ever you should try coming up with it on your own! If you can completely get everything this snippet is doing you'll be well on the way to understanding shell scripting, so try and read this code, play with it, and see what you can learn from it.

Related

Creating files in succession

How would one go about creating a script for creating 25 empty files in succession? (I.e 1-25, 26-51, 52-77)
I can create files 1-25 but I’m having trouble figuring out how to create a script that continues that process from where it left off, every time I run the script.
#!/bin/bash
higher=$( find files -type f -exec basename {} \; | sort -n | tail -1 )
if [[ "$higher" == "" ]]
then
start=1
end=25
else
(( start = higher + 1 ))
(( end = start + 25 ))
fi
echo "$start --> $end"
for i in $(seq $start 1 $end)
do
touch files/"$i"
done
I put my files in a directory called "files".
hence the find on directory "files".
for each file found, I run a basename on it. That will return only integer values, since the files all have a number filename.
sort -n puts them in order.
tail -1 extracts the highest number.
if there are no files, higher will be empty, so the indexes will be 1 and 25.
otherwise, they will be higher + 1, and higher + 26.
I used seq for the for loop to avoid problems with variables inside a range definition (you did {1..25})
#! /usr/bin/env bash
declare -r base="${1:-base-%d.txt}"
declare -r lot="${2:-25}"
declare -i idx=1
declare -i n=0
printf -v filename "${base}" ${idx}
while [[ -e "${filename}" ]]; do
idx+=1
printf -v filename "${base}" "${idx}"
done
while [[ $n -lt $lot ]]; do
printf -v filename "${base}" ${idx}
if [[ ! -e "${filename}" ]]; then
> "$filename"
n+=1
fi
idx+=1
done
This script accepts two optional parameters.
The first is the basename of your future files with a %d token automatically replaced by the file number. Default value is base-%d.txt;
The number of file to create. Default value is 25.
How script works:
Variable declarations
base: file basename (constant)
lot: number of file to create (constant)
idx: search index
n: counter for new files
Search files already created from 1
The loop stop at first hole in the numbering
Loop to create empty files
The condition in the loop allows to fill in the numbering holes
> filename create an empty file

bash script for putting thousands of files, into separate folders

in a directory, i have thousands of files.
i use the following bash snippet, in order to make these thousands of files, be put in folders, that have each 1000 files,
and i face two issues:
a) now each folder has a prefix of dir_ while i would like it to have a name, that will have 6 digits, if less than 6 in the folder name, leading zeros should be added appropriately.
b) current script, puts first folder, into the last one, for example, i have dir_400325 as the last folder, and in there,i find the bash script i have run, and the dir_1000 folder, which is the first folder created. How could i change this, so the first folder, is not stored into the last one?
#!/bin/bash
c=0; d=1000; mkdir -p dir_${d}
for file in *
do
if [ $c -eq 1000 ]
then
d=$(( d + 1000 )); c=0; mkdir -p dir_${d}
fi
mv "$file" dir_${d}/
c=$(( c + 1 ))
done
You can use printf and a format string to generate your 6-digit directory name with leading zeros (%06d), demonstrated in a shell:
bash-4.4$ d=1001
bash-4.4$ dir_name=$(printf "/path/to/%06d" $d)
bash-4.4$ echo $dir_name
/path/to/001001
Using an absolute path may help ensure the files end up where you're expecting them and not in some subfolder of your current working directory.
#!/bin/bash
c=0
d=1000
dir_name=$(printf "/path/to/%06d" $d)
mkdir -p $dir_name
for file in *
do
if [ $c -eq 1000 ]
then
c=0
d=$(( d + 1000 ))
dir_name=$(printf "/path/to/%06d" $d)
mkdir -p $dir_name
fi
if [[ -f "$file" ]]
then
mv "$file" $dir_name
c=$(( c + 1 ))
fi
done

navigating directories and sub-directories and moving the files using shell

Directory structure
MyDirectory
-data/
-DATA-TT_20160714_soe_test_testbill_52940_1.lst
-output/
-DATA-TT_20160714_soe_test_testbill_52940_1.pdf
-Backup/
enter code here
#!/bin/bash
for i in $( ls ); do
echo $i
#cd $i
#cd $i/data/
echo $i
cd $i/data/
echo $i/data/
$(ls *1.lst)
cd ../output
$(ls *1.pdf)
done
I need to navigate in the directories and the sub directories where
the
input and output files are kept. These file have
date in YYYYMMDD format which I need to compare with current date.
If the difference is greater than 1 month I need to
zip those file and move them to Backup directory. The "DATA-TT" part
is constant.
Can anyone help me in this.There may be many directories with same sub directories structure.Example MyDirectory1,MyDirectory2,MyDirectory3
Have to leave for a meeting. I'll leave you the script I've been working on to help you:
#!/bin/sh
# in your case filename would be the variable used in the for loop
filename=$(find DATA-TT*)
# Part gets the date from the filename
part=$(echo $filename | grep -Eo '[[:digit:]]{8}')
echo "$filename -> $part"
# limit_date is current date -30 days
limit_date=$(date +'%Y%m%d' -d "now 30 days")
echo $cur_date
# this part compares and echoes, you can try it and replace echoes with ziping, moving, etc.
if [ $part -ge $limit_date ]
then
echo "part is older than 1 month"
else
echo "part had not yet reached 1 month age"
fi
This one for the data directory the same for the output
#!/bin/bash
list=($(find data -name "DATA-TT_*"))
limit=`date +%Y%m%d -d "-30 days"`
for i in ${list[#]}; do
filedate=$(echo $i| cut -d'_' -f 2)
if [ "$limit" -gt "$filedate" ]; then
mv $i backup
fi
done

script for checking directories and file contains a word

I have the following file "nbr_chk.txt" ,this file contains numbers that can be in any directory (from 1 to 10). the sub directory where they are is the 5th and 6th number
nbr_chk.txt
612345678
623456789
634567890
I want to make a script using that file and do the following:
for i in 'cat nbr_chk.txt'
do
ls -lrtd /*/d5d6/i to find the directory
if there is more than one directory print the directories
if there is only 1 directory use it and find if there is a file that contains the word " test" and print number xxxxxxxxx
done
edited 1:
For example the number 612345678 could be in the following directory /05/45/612345678
but could be also in the directory /09/45/612345678.
For that reason I need to do a ls -lrtd /*/.... to find the directory.
In case there are more than one directory I need to create an error message
d5 means the 5th digits and d6 the 6th digits of the number .
if the number is 612300012 digit 5 =0 and digit 6=0 and the I would have to use
ls -lrtd /*/00/612300012
If it was in an other language than Unix I would know how to do it but here I'm lost.
thanks
The specification isn't too clear, but I take some guesses and come up with this:
#!/bin/bash
for i in $(cat nbr_chk.txt)
do
dirName=d${i:4:1}d${i:5:1}
grep -q " test" "$dirName"/* && echo "$i"
done
This cannot solve all issues you've got because some are still unclear (to me at least). Please elaborate on the not-met specifics so I can fit them in the solution.
I think you may be looking for something like this:
#!/bin/bash
for i in $(cat nbr_chk.txt)
do
dir=d${i:4:1}d${i:5:1}
echo Checking $dir...
dirlist=$(find . -type d -name "$dir")
ndirs=$(find . -type d -name "$dir" | wc -l)
if [ $ndirs -gt 1 ]; then
echo $dirlist
fi
if [ $ndirs -eq 1 ]; then
cd $dirlist
grep -q " test" * 2> /dev/null && echo $i
fi
done

Renaming Multiples Files with Different Names in a Directory using shell

I've found most of the questions of this kind where the change in name has been same for the entire set of files in that directory.
But i'm here presented with a situation to give a different name to every file in that directory or just add a different prefix.
For Example, I have about 200 files in a directory, all of them with numbers in their filename. what i want to do is add a prefix of 1 to 200 for every file. Like 1_xxxxxxxx.png,2_xxxxxxxx.png...........200_xxxxxxxx.png
I'm trying this, but it doesnt increment my $i everytime, rather it gives a prefix of 1_ to every file.
echo "renaming files"
i=1 #initializing
j=ls -1 | wc -l #Count number of files in that dir
while [ "$i" -lt "$j" ] #looping
do
for FILE in * ; do NEWFILE=`echo $i_$FILE`; #swapping the file with variable $i
mv $FILE $NEWFILE #doing the actual rename
i=`expr $i+1` #increment $i
done
Thanks for any suggestion/help.
To increment with expr, you definitely need spaces( expr $i + 1 ), but you would probably be better off just doing:
echo "renaming files"
i=1
for FILE in * ; do
mv $FILE $((i++))_$FILE
done
i=1
for f in *; do
mv -- "$f" "${i}_$f"
i=$(($i + 1))
done

Resources