Bash - Copy files iteratively - bash

I want to copy a list of files, called dti_fin_fa, from one folder to another.
These files are scattered in different folders.
Controls
└───C01
│ └───difusion
│ └───Deterministic
│ └───dti_fin_fa.nii
└───C02
│ └───difusion
│ └───Deterministic
│ └───dti_fin_fa.nii
└───C03
│ └───difusion
│ └───Deterministic
│ └───dti_fin_fa.nii
I want to select and copy all the dti_fin_fa, keeping the folder structure, so, in my new directory, I would have the folder distribution just seen above. The problem is that in this folders, (deterministic, difusion, etc), there are lots of other files I don´t want to copy, so I can´t just copy the main folder (C01 etc)
This is what I have:
#!/bin/bash
DIR="/media/Batty/Analysis"; cd "$DIR" || exit
for group in Controls; do
for folder in $group/*; do
for dti in $folder/difussion/Deterministic/dti_fin_fa.nii; do
echo $dti
cp $dti /media/Roy/Analysis/Controls/ --verbose
done;
done;
done;
The problem is that this code copies each of the dti_fin_fa.nii images into /media/Roy/Analysis/Controls/, hence keeping just the last one, instead of creating all the other subfolders.
Could something like this work?
cp $folder/difusion/Deterministic/$dti /media/Roy/Analysis/Patients/ --verbose

Iterate through each control, make the equivalent directory and copy only the file you want:
for group in "Control"/* "Patients"/*; do
orig="$group/difusion/Deterministic"
dest="/media/Roy/Analysis/Controls/$(dirname group)/$orig"
mkdir -p "$dest"
cp "$orig/dti_fin_fa.nii" "$dest/"
done
I suppose you execute this from the DIR directory of your example. So here orig is the relative path to the image folder, dest is the absolute path to your destination directory, including the preserved original folder structure. mkdir -p ensures that such end directory exists and finally the image is copied.

Related

linux - batch move files into a directory and rename those files according to sequential syntax in that directory

I have two directories - A and B - that contain a bunch of photo files. Directory A is where I keep photos long-term, and the files inside are sequentially named "Photo-1.jpg, Photo-2.jpg, etc.".
Directory B is where I upload new photos to from my camera, and the naming convention is whatever the camera names the file. I figured out how to run some operations on Directory B to ensure everything is in .jpg format as needed (imagemagik convert), remove duplicate files (fdupes), etc.
My goal now is to move the files from B to A, and end up with the newly-added files in A sequentially named according to A's naming convention described above.
I know how to move the files into A, and then to batch rename everything in A after the new files have been added (which would theoretically occur every night), but I'm guessing there must be a more efficient way of moving the files from B to A without re-naming all 20,000+ photos every night, just because a few new files were added.
I guess my question is two parts - 1) I found a solution that works (us mv to rename all photos every night), is there any downside to this? and 2) If there is a downside and a more elegant method exists, can anyone help with a script that would look at whatever the highest number that exists in A, then re-name the files, appending onto that number, in B as they are moved over to A?
Thank you!
This bash script will only move and rename the new files from DiretoryB into your DirectoryA path. It also handles file names with spaces and/or any other odd characters in their name in DirectoryB
#!/bin/bash
aPath="./photos-A"
bPath="./photos-B"
aPattern="Photo-"
lNum=$(find $aPath -type f -name "*.jpg" -printf "%f\n" | \
awk -F'[-.]' '{if($2>m)m=$2}END{print m}')
while IFS= read -r -d $'\0' photo; do
mv "$photo" "$aPath/$aPattern$((++lNum)).jpg"
done < <(find $bPath -type f -name "*.jpg" -print0)
Note
The command to find the last numbered photo, aka $lNum will run over all 20K+ files, but it should be fairly quick. If it's not, you can always run this once and store the latest number into a file and read from that file.
Proof of Concept
$ tree photos-A/
photos-A/
├── Photo-1.jpg
├── Photo-2.jpg
├── Photo-3.jpg
├── Photo-5.jpg
├── Photo-6.jpg
├── Photo-7.jpg
└── Photo-8.jpg
0 directories, 7 files
$ tree photos-B/
photos-B/
├── bar.jpg
├── baz\ with\ spaces.jpg
└── foo.jpg
0 directories, 3 files
$ ./mvphoto.sh
$ tree photos-A/
photos-A/
├── Photo-10.jpg
├── Photo-11.jpg
├── Photo-1.jpg
├── Photo-2.jpg
├── Photo-3.jpg
├── Photo-5.jpg
├── Photo-6.jpg
├── Photo-7.jpg
├── Photo-8.jpg
└── Photo-9.jpg
0 directories, 10 files

Moving files into new subdirectory, except for existing folders?

A Redditor had written me a great script that allowed me to move all the files in a series of folders into a new subdirectory in all those folders called New. However, I also have pre-existing folders (namely just 1 called "Journals") that have had their files moved into a subdirectory called New, as well.
How would I modify the following script (on Windows) to not touch any folders within the folders, or perhaps not touch any folder called Journals?
For example, the current directory looks like:
Parent/Folder name/tons of files/
Parent/Folder name/Journals/tons of files
(folder name = random string of alphanumeric numbers in the thousands). Each folder has a ton of files, and a folder called Journals.
I would like:
Parent/randomstring folder/New/tons of files/
Parent/randomstring folder/Journals/tons of files
The code they wrote for me:
# Run from search root
cd "O:\..."
# Change this to taste
export NEWDIR=New
find . | egrep '(mp4$|jpg$|png$)' |
while read FILE
do
BASEDIR=$(dirname "$FILE")
mkdir "$BASEDIR/$NEWDIR" > /dev/null 2>&1
mv "$FILE" "$BASEDIR/$NEWDIR"
done
This code would do the following:
Parent/randomstring folder/New/tons of files/
Parent/randomstring folder/Journals/new/tons of files

Bash (Mac) - Moving Randomized Files to a Multiple Other Folders with a "Fill Limit"

I have a couple of hundred files in one folder, and I'd like to randomly move them to a number of different folders with a bash script - however, I'd like to fill each of those destination folders only up to a given capacity.
I'm thinking the right way to approach this is to assign two arrays, one containing all destination folders and one containing all files. Then I can randomly take a file from the filesarr and place it in a destination folder. My question is, how can I limit the number of files placed in each destination folder? So say I'm looking for ten files per destination folder - how can I move the first ten files from filesarr to the first folder in foldersarr, then move the next ten to the second folder in foldersarr, until all files have been moved? I know I should probably use a counter here, but my current attempt (below) is not doing the trick.
filesarr=(/Path/to/files/*) # this is the array of files to shuffle
foldersarr=(/Path/to/destination/folders/) # array of folders to move into
foldercount=0 # set it to 0
for afolder in "${foldersarr[#]}"; do
if [[ "$foldercount" -gt 10 ]]; then
echo "$foldercount files in folder, exiting and moving to next folder"
exit 1
else
for afile in "${filesarr[#]}"; do # do loop length(array) times; once for each file
length=${#filesarr[#]}
randomi=$(( $RANDOM % $length )) # select a random index
filename=${filesarr[$randomi]}
mv ${filename} ${foldersarr[#]}
echo "moving '$filename'"
foldercount=$((foldercount+1))
unset -v "filesarr[$randomi]" # unset after moved
array=("${filesarr[#]}") # remove NULL elements introduced by unset; copy array
done
fi
done
My current directory structure consists of all the files in a "holding" directory, and all the destination folders where I'd like to move them in a separate folder.
rootfolder
│
├── holding
│ ├── dywd.pdf
│ ├── ... (approx. 200 files)
│ └── kjfwekfjnwe.pdf
│
└── destinations
├── folder01
├── ...
└── folder10
I'd like to end up with this:
rootfolder
│
├── holding
│
└── destinations
├── folder01
│ ├── lwkejdwe.pdf
│ ├── ...
│ └── (ten files in this folder)
├── ...
│
└── folderXX
├── qwuoe.pdf
├── ...
└── (ten files in this folder)
something like this, (not tested)
dirs=(..) # array of dirs
dir_length=${#dirs[#]}
find -maxdepth 1 -type f | # or any other list of files
shuf |
while c=0 IFS= -r file;
do mv "$file" "{dirs[c++%$dir_length]}";
done
this will round robin moving files to target directories. The randomness is generated with shuf, no need to maintain the list of files separately.
You could create some "buckets" variables and fill each one with the same amount of file names, eg divide all your files in scope into these buckets. Then when done, write each bucket into a separate folder

How to find all empty folders and untracked files when using git?

Based on this post and this post ,
git ls-files --others --exclude-standard can list all untracked files.
But I test it, cannot list empty folder(both tracked and not tracked).
For example,cannot list empty folder archiver folder as below:
.
├── admin.php
├── api
│   ├── index.htm
│   └── remote
│   └── mod
│   ├── index.htm
│   ├── mod_cron.php
│   └── mod_index.php
└── archiver folder
Then my question is: how to list all untracked files and empty folders?
TL;DR: just look for empty directories. You can safely remove them—well, "safe" depends on your own software, but as far as Git is concerned, it's safe. (Watch out for missing files—see the definition of "missing" below—which may remove a directory that Git might want later, but that's sort of OK, because Git will just create it again.)
On a Unix / Linux system (edited to correct lost word in transcription):
find . -name .git -prune -o -type d -empty -print
(at the top level of the work-tree) will find the empty directories.
Long(ish)
Git is not interested in folders / directories. There's no such thing as an untracked folder in the same way that there's no such thing as a tracked folder: Git only cares about files. Specifically, a file is either in the index, or not in the index, and if it's not in the index, it's untracked.
When you use the various options to list untracked files (which tend to skip over ones that are untracked-and-ignored since you normally want that), Git will, sometimes, aggregate together all the files that are in some folder, notice that there are no tracked files in that folder, and report them using the aggregated notation. You can stop this with, e.g., git status --untracked-mode=all; then you'll get the individual file names.
Note that it's possible to have some file that is tracked, yet missing. For instance, suppose sub/README.txt is a tracked file, and actually exists. Then we run rm sub/README.txt. The file sub/README.txt remains in Git's index, and will be in the next commit, but it's missing. If that was the only file in sub in your work-tree, sub is now empty, and you can remove it with rmdir sub. Even though sub/README.txt remains missing (and sub is missing too!), that does not affect the next commit: it will still contain sub/README.txt, because that file is in the index. (Using git rm --cached sub/README.txt, you can remove it from the index too, if that's what you wanted.)
If and when Git goes to copy sub/README.txt back out of the index into the work-tree, Git will, at this point, discover that there is no sub. Git will merely shrug its metaphorical shoulders and create the directory sub, and then put sub/README.txt into it. So this is why Git is not interested in folders / directories: they're just boring and dull, required only when needed to hold files, created on demand.
If you want Git to create a directory, you need to store a file in it. Since programs managed by Git need to be able to ignore the file named .gitignore, this is a very good file name to stick into such a directory. You can write * into that file, and add it to your commits, so that Git will create the directory and write a .gitignore file there containing *, and will thus ignore all additional untracked files within that directory automatically.
Side note: In general, when Git pulls the last file out of some directory, it will remove the directory too, but occasionally I've seen it leave some behind. (Of course, it has to leave the directory behind if it still contains some untracked files. Note that git clean -fd will remove the empty directories, though it also removes the untracked files.)
git ls-files --others --exclude-standard> not_tracked
find . -depth -empty -type d \( ! -regex '.*/\..*' \) >> not_tracked
Please check my answer,I spent 2 days for it.
The command git clean does exactly what you want.

How to update one file in a zip archive

Is it possible to replace a file in a zip file without unzipping?
The file to update is an XML file that resides in a huge zip archive. To update this XML file, I have to unzip the archive, delete the old XML file, add the new one and then rezip. This takes a considerable amount of time. So want to be able to replace that one XML through a script. I already have the one that checks for updates on the XML I have.
using zip command
Sorry, I would use the zip command to do things like that but the problem is the script is actually for an android phone and zip is not a command I can use unfortunately sorry I left that out. I would have used zip definitely if i could but I only have unzip for droid and then there is tar in busybox but tar doesn't do what I need
Try the following:
zip [zipfile] [file to update]
An example:
$ zip test.zip test/test.txt
updating: test/test.txt (stored 0%)
I've found the Linux zip file to be cumbersome for replacing a single file in a zip. The jar utility from the Java Development Kit may be easier. Consider the common task of updating WEB/web.xml in a JAR file (which is just a zip file):
jar -uf path/to/myapp.jar -C path/to/dir WEB-INF/web.xml
Here, path/to/dir is the path to a directory containing the WEB-INF directory (which in turn contains web.xml).
From zip(1):
When given the name of an existing zip archive, zip will replace identically named entries in the zip archive or add entries for new names.
So just use the zip command as you normally would to create a new .zip file containing only that one file, except the .zip filename you specify will be the existing archive.
Use the update flag: -u
Example:
zip -ur existing.zip myFolder
This command will compress and add myFolder (and it's contents) to the existing.zip.
Advanced Usage:
The update flag actually compares the incoming files against the existing ones and will either add new files, or update existing ones.
Therefore, if you want to add/update a specific subdirectory within the zip file, just update the source as desired, and then re-zip the entire source with the -u flag. Only the changed files will be zipped.
If you don't have access to the source files, you can unzip the zip file, then update the desired files, and then re-zip with the -u flag. Again, only the changed files will be zipped.
Example:
Original Source Structure
ParentDir
├── file1.txt
├── file2.txt
├── ChildDir
│ ├── file3.txt
│ ├── Logs
│ │ ├── logs1.txt
│ │ ├── logs2.txt
│ │ ├── logs3.txt
Updated Source Structure
ParentDir
├── file1.txt
├── file2.txt
├── ChildDir
│ ├── file3.txt
│ ├── Logs
│ │ ├── logs1.txt
│ │ ├── logs2.txt
│ │ ├── logs3.txt
│ │ ├── logs4.txt &lt-- NEW FILE
Usage
$ zip -ur existing.zip ParentDir
> updating: ParentDir/ChildDir/Logs (stored 0%)
> adding: ParentDir/ChildDir/Logs/logs4.txt (stored 96%)
I know this is old question, but I wanted to do the same. Update a file in zip archive. And none of the above answers really helped me.
Here is what I did. Created temp directory abc. Copied file.zip to abc and extracted the file in that directory. I edited the file I wanted to edit.
Then while being in abc, ran the following command
user#host ~/temp/abc $ zip -u file.zip
updating: content/js/ (stored 0%)
updating: content/js/moduleConfig.js (deflated 69%)
-u switch will look for changed/new files and will add to the zip archive.
You can use:
zip -u file.zip path/file_to_update
There is also the -f option that will freshen the zip file. It can be used to update ALL files which have been updated since the zip was generated (assuming they are in the same place within the tree structure within the zip file).
If your file is named /myfiles/myzip.zip all you have to do is
zip -f /myfiles/myzip.zip
From the side of ZIP archive structure - you can update zip file without recompressing it, you will just need to skip all files which are prior of the file you need to replace, then put your updated file, and then the rest of the files. And finally you'll need to put the updated centeral directory structure.
However, I doubt that most common tools would allow you to make this.
7zip (7za) can be used for adding/updating files/directories nicely:
Example:
Replacing (regardless of file date) the MANIFEST.MF file in a JAR file. The /source/META-INF directory contains the MANIFEST.MF file that you want to put into the jar (zip):
7za a /tmp/file.jar /source/META-INF/
Only update (does not replace the target if the source is older)
7za u /tmp/file.jar /source/META-INF/
yes its possible.
on linux based systems just install zip and you can call it in the command line. have a look at the manpage: http://linux.die.net/man/1/zip
but in my personal experience, if possible and compression is not so important, this works better with plain tar files and tar.

Resources