I'm looking for a script to change an existing folder structure quickly on my synology nas using ssh from YYYY/MM/DD/ to YYYY-MM-DD/ so nested to flat but struggling to find one or any examples probably due to me not searching for the correct terminology.
I did start to use exiftool and go through moving each item but the collection is taking ages.
for example an image01 that currently resides in say 2020/01/01/images01.jpg needs to move to 2020-01-01/images.jpg
image and video files only currently live in the day folders.
For example, you can do this with this command:
find . -mindepth 3 -type d |awk -F"/" 'system("mv " $2"/"$3"/"$4" "$2"-"$3"-"$4" && rmdir "$2"/"$3" && rmdir "$2" 2>/dev/null")'
Example input folder structure:
./2020/06/12/img1.jpg
./2020/06/12/file2.mpg
./2020/05/10/img2.jpg
./2020/05/10/img1.jpg
Result:
./2020-05-10/img1.jpg
./2020-05-10/img2.jpg
./2020-06-12/img1.jpg
./2020-06-12/file2.mpg
Explanation:
find . searches in the current directory
-mindepth 3 only searches at level 3 nesting
-type d only search for directories
| creates a pipe, directing the find result to AWK
-F"/" sets the AWK field separator to /
system command is executed for each line which transfers files to the new directory and deletes unnecessary old directories
&& do next command only when previous ones were successful
2>/dev/null directs stderr to the void so that you won't see any errors trying to delete a non-empty "year" directory
You cannot use rm -rf because the year directory may still contain other directories, hence using rmdir twice.
You can resolve the stderr redirection more elegantly by testing whether the directory of the year is empty before deleting it,
I do not put this test so as not to obscure the idea of action unnecessarily.
Related
I have been trying to use the command line to delete all files in all subdirectories with the name s_1_1102_c.jpg.
This question is similar to what I need How to remove folders with a certain name but it is removing directories and I only want to delete the files with the name s_1_1102_c.jpg.
I will need to remove this file from 260 subdirectories under the L001 directory. My directory structure is like this:
L001
C5.1
s_1_1101_a.jpg
s_1_1101_c.jpg
s_1_1101_g.jpg
s_1_1101_t.jpg
s_1_1102_a.jpg
s_1_1102_c.jpg
s_1_1102_g.jpg
s_1_1102_t.jpg
s_1_1103_a.jpg
s_1_1103_c.jpg
s_1_1103_g.jpg
s_1_1103_t.jpg
C6.1
s_1_1101_a.jpg
s_1_1101_c.jpg
s_1_1101_g.jpg
s_1_1101_t.jpg
s_1_1102_a.jpg
s_1_1102_c.jpg
s_1_1102_g.jpg
s_1_1102_t.jpg
s_1_1103_a.jpg
s_1_1103_c.jpg
s_1_1103_g.jpg
s_1_1103_t.jpg
Ultimately I need to remove several files from all subdirectories (s_1_1101_g.jpg, s_1_1101_t.jpg, s_1_1102_a.jpg, s_1_1102_c.jpg, s_1_1102_g.jpg, s_1_1102_t.jpg). So maybe there is a way to provide a list of the file names I need to delete.
How can I delete these files?
find . -name "s_1_1102_c.jpg" -exec rm -f {} \;
Note: This will find and delete the file in any subdirectory of the current one. So you could execute it in L001 or wherever else you want to do this.
for i in s_1_1101_g.jpg s_1_1101_t.jpg s_1_1102_a.jpg s_1_1102_c.jpg s_1_1102_g.jpg s_1_1102_t.jpg; do
echo rm L001/*/"$i";
done
If output looks fine, remove echo.
The final method I used to delete my files was given by #Peter - Reinstate Monica
for f in s_1_1101_t.jpg s_1_1102_a.jpg s_1_1102_c.jpg s_1_1102_g.jpg s_1_1102_t.jpg s_1_1103_a.jpg s_1_1103_c.jpg s_1_1103_g.jpg s_1_1103_t.jpg s_1_1104_a.jpg s_1_1104_c.jpg s_1_1104_g.jpg s_1_1104_t.jpg s_1_2101_g.jpg s_1_2101_t.jpg s_1_2102_a.jpg s_1_2102_c.jpg s_1_2102_g.jpg s_1_2102_t.jpg s_1_2103_a.jpg s_1_2103_c.jpg s_1_2103_g.jpg s_1_2103_t.jpg s_1_2104_g.jpg s_1_2104_t.jpg; do find /hpc/home/L001 -name $f -delete; done
I was concerned that my file list would be too long but it worked in this situation.
I am very new to bash scripting. I need to perform same operation on 300 files. I have tried the following so far:
empty_files=$(find ./ -type f -empty)
echo "$empty_files"
Now I have the full path to all the 300 files, stored in variable empty_files.
Now what I want to do is, for each file in the result
go to it's parent's parent directory,
then go to the other child (sibling of earlier file's parent directory) and find a file by name 'abc.js' inside it
Now, inside abc.js, find a particular word (object property) "user"
now on the new line, insert a line of js code. (exact same code for all files)
Please let me know if it's possible from mac command line.
Thanks
You can use a for loop:
for file in "${empty_files[#]}"
do
... code that uses "$file"
done
You could also pipe directly from find into the loop:
find . -type f -empty | while read -r file
do
... code that uses "$file"
done
This version should work for any filenames that don't contain newline.
I was wondering how to move a number of folders to another one, according to the filename of a file inside each of the folders.
I mean, let's assume I have a big amount of folders, each one with a name starting by 'folder*', each one containing 3 files. Specifically one the files contains a string which might be '-100', '-200' or '-300' for example.
I want to move the folders containing the files according to this strings, and put them in a folder called 'string'. For example, to put every folder containing a file which contains the string '-100' into the folder 'FOLDER1'I'm trying something like:
find folder* -name '100' -exec mv {} folder* FOLDER1
but it returns -bash: /usr/bin/find: Argument list too long.
How can I pass less arguments to find at every step so I don't get this.
Thank in advance.
Best.
Using your example, and running in the topmost folder containing all the folders, I believe that what you need is this:
grep -rlw folder* -e "-100" | xargs -I % mv % FOLDER1
I have a following data dir:
root/A/1
root/A/2
root/B/1
root/B/2
root/B/3
root/C/1
root/C/2
And I want to convert it into following file structure:
root2/I/A/1
root2/I/A/2
root2/I/B/1
root2/I/B/2
root2/I/B/3
root2/II/C/1
root2/II/C/2
Purpose of doing it is I want to run some script which takes home folder (root here) and runs on it. And I want to run it in parallel on many folders(I, II) to speed up the process.
Simple assumption about file and folder name is that all are alphanumeric, even no period or underscore.
Edit: I tried following:
for i in `seq 1 30`; do mkdir -p "root2/folder$i"; find root -type f | head -n 4000 | xargs -i cp "{}" "root2/folder$i"; done
Problem is that it creates something like following, which is not what i wanted.
root2/I/1
root2/I/2
root2/I/1
root2/I/2
root2/I/3
root2/II/1
root2/II/2
You may wish to use a lesser known command called dirsplit, the usual application of which is to split a directory into multiple directories for burning purposes.
Use it like below :
dirsplit -m -s 300M /root/ -p /backup/folder1
Options implies below stuff :
-m|--move Move files to target dirs
-e 2 special exploration mode, 2 means files in directory are put together
-p prefix to be attached to each directory created, in you case I, II etc
-s Maximum size allowed for each new folder created.
For more information see :
dirsplit -H
Is there a way to stop wget from creating empty directories? Most of the files I need are found at one level of depth, i.e. in folder 2 of /1/2/, but I need to use infinite recursion because sometimes the file I need is at 1/2/3/ or deeper. Or at least, I need infinite recursion for the time being, until I figure out the maximum depth of where the files of interest are located.
Right now I'm using
wget -nH --cut-dirs=3 -rl 0 -A "*assembly*.txt" ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria
Which gets all the files I need, but I am left with a bunch of empty directories. I would prefer the directory structure /bacteria/organism/*assembly*.txt, but if creating multiple subdirectories cannot be avoided, I want to at least stop wget from creating empty directories. I can, of course, remove the empty directories after running wget, but I want to stop wget from creating them in the first place if possible
Short answer: you can't prevent the directories from being created.
You can do post-processing on the directories though:
find bacteria/ -type d -empty -exec rmdir {} \;
Looking at a bunch of these directories (including the very busy one for e. coli) it appears, as you said, that the only files matching *assembly*.txt are stored in the first directory below bacteria. Unless there's some variation to this rule, you could just do this:
wget -nH --cut-dirs=2 -rl 2 -A "*assembly*.txt" ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria
BTW if you want your directory structure to start at bacteria/ you'll need to change --cut-dirs to 2 instead of 3.