I am trying to use a single line command in terminal to find and copy all the files of a certain type in one directory of my computer to another directory. I can do this right now using the below command:
find ./ -name '*.fileType' -exec cp -prv '{}' '/destination_directory/' ';'
The problem I'm having is that if a file that is being copied has the same name as a file that was previously copied, it will replace the previously copied file.
To remedy this, I would like to edit my command such that the files are numbered as they are copied to the new directory.
so the output should look something like this:
Original Files
cat.txt
dog.txt
dog.txt
Copied Files
cat1.txt
dog2.txt
dog3.txt
Edit:
The list of commands I can work with are linked here: https://ss64.com/osx/
Specifically for the cp command: https://ss64.com/osx/cp.html
-Note: --backup and -b are not available (it seems) for this version of cp
You are looking for the --backup option of the cp command. E.g.:
find ./ -name '*.fileType' -exec cp --backup=t -prv '{}' '/destination_directory/' ';'
Edit: If you are stuck with MacOS's cp you can emulate --backup's behaviour in a script:
#!/bin/bash
set -e
# First parameter: source directory
srcdir=$1
# Second parameter: destination directory
destdir=$2
# Print all filenames separated by '\0' in case you have strange
# characters in the names
find "$srcdir" -type f -print0 |
# Split the input using '\0' as separator and put the current line
# into the $file variable
while read -d $'\0' file; do
# filename = just the name of the file, without dirs
filename=$(basename "$file")
# if destdir does not have a file named filename
if [ \! -f "$destdir/$filename" ]; then
cp -pv "$file" "$destdir/$filename";
continue;
fi
# Otherwise
suffix=1
# Find the first suffix number that is free
while [ -f "$destdir/$filename.$suffix" ]; do
suffix=$(($suffix + 1))
done
cp -pv "$file" "$destdir/$filename.$suffix"
done
Related
I have a fairly large number of directories (500+), each directory (and possible sub-directories) contains 4 or more zip files.
I managed to piece together a bash script that unzips the compressed files while maintaining zip filename as directory and all the directory hierarchy.
For example: If I have a zip file called 100011_test123.zip, and it contains 10 files. The script will uncompress all the files into 100011_test123/ directory.
The occurrence of numbers 100010 before the underscore in the filename/directoryname is totally random.
Here's the actual bash script:
#!/bin/bash
cd <directory-with-large-number-of-zip-files>
find . -name "*.zip" | while read filename; do unar -d -o "`dirname "$filename"`" "$filename"; done;
find . -name "*.zip" -type f -delete
Now I would like to update the script in order to remove the 100010_ from the .zip filename without tampering with the directory structure/hierarchy (I guess there's a way to rename the zip files before using unar command) and then uncompress the files into a directory without 100010_ at the beginning.
I have been stuck with this for more than 3 days. Any insights on this would be highly appreciated.
Thank you.
With all zip files at the same level, you don't need find, but a regular filename pattern globbing will do to iterate each zip archive.
And with bash's globstar option, you can also find the zip archives inside sub-directories
#!/usr/bin/env bash
shopt -s nullglob # Prevents iterating if no filename match
shopt -s globstar # ./**/ Allow searching inside sub-directories
# Set the basedir if you want all output directories at same place
#basedir="$PWD"
for zipfile in ./**/*.zip; do
# Extract the base directory containing the archive
zipdir="${zipfile%/*}"
# Extract the base name without the directory path
basename="${zipfile##*/}"
# Remove the .zip extension
# 100011_test123.zip -> 100011_test123
extensionless="${basename%.zip}"
# Remove everything before and first underscore 100011_
# 100011_test123 -> test123
outputdir="${basedir:-$zipdir}/${extensionless#*_}"
# Create output directory or continue with next archive
# mkdir -p test123
mkdir -p "$outputdir" || continue
# Unzip the zipfile into the outputdir and remove the zipfile if successful
# unrar -d -o test123 100011_test123.zip && rm -f -- 100011_test123.zip
unar -d -o "$outputdir" "$zipfile" && rm -f -- "$zipfile"
done
You need to parse directory name and filename first for each entry. Please check the ${fullpath%/*} and ${fullpath##*/} for this purpose. And awk for splitting filename with '_' and getting second part of it.
You can try following code.
#!/bin/bash
# cd directory
zip_files=($(find . -name "*.zip"))
for fullpath in "${zip_files[#]}"; do
echo "Processing: "$fullpath""
DIRNAME="${fullpath%/*}"
FILENAME="${fullpath##*/}"
NEW_FILENAME="`echo $FILENAME | awk -F'_' '{print $NF}'`"
echo " DIRNAME="$DIRNAME
echo " NEW_FILENAME="$NEW_FILENAME
mv $fullpath "$DIRNAME/$NEW_FILENAME"
# call unar command
unar -d -o $DIRNAME $NEW_FILENAME
# delete file if you want
done
I need to copy all files of a specific type to a folder, which I'm doing like this:
find ./ -name '*.gql' -exec cp -prv '{}' '/path/to/dir/' ';'
But if there are two files with a identical name, although located in different subfolders, some files would be overwritten.
Is it possible to keep all files, which are copied? Maybe renaming the copied file or is it possible to keep the folder structure in the target directory?
c.f. https://ss64.com/osx/cp.html : Historic versions of the cp utility had a -r option...its use is strongly discouraged, as it does not correctly copy special files, symbolic links, or fifo's. You can use -n to prevent overwrites, but more complex logic will likely require custom code.
dest=/path/to/dir
while IFS= read -r -d '' filename # read null-delimited list from find
do basename="${filename##*/}"
if [[ -e "$dest/$basename" ]]
then ctr=0
while [[ -e "$dest/$basename.$((++ctr))" ]]; do : ; done # find available name
mv "$dest/$basename" "$dest/$basename.$ctr"
fi
cp -pv "$filename" "$dest/$basename"
done < <( find ./ -name '*.gql' -type f -print0 ) # null-delimit
I am working on Linux machine.
I have a lot of files named the same, with a directory structure like this:
P45_input_foo/result.dat
P45_input_bar/result.dat
P45_input_tar/result.dat
P45_input_cool/result.dat ...
It is difficult to copy them one by one. I want to copy them into another folder named as data with similar folder names and file names:
/data/foo/result.dat
/data/bar/result.dat
/data/tar/result.dat
/data/cool/result.dat ...
In stead of copy them one by one what I should do?
Using a for loop in bash :
# we list every files following the pattern : ./<somedirname>/<any file>
# if you want to specify a format for the folders, you could change it here
# i.e. for your case you could write 'for f in P45*/*' to only match folders starting by P45
for f in */*
do
# we strip the path of the file from its filename
# i.e. 'P45_input_foo/result.dat' will become 'P45_input_foo'
newpath="${f%/*}"
# mkdir -p /data/${newpath##*_} will create our new data structure
# - /data/${newpath##*_} extract the last chain of character after a _, in our example, 'foo'
# - mkdir -p will recursively create our structure
# - cp "$f" "$_" will copy the file to our new directory. It will not launch if mkdir returns an error
mkdir -p /data/${newpath##*_} && cp "$f" "$_"
done
the ${newpath##*_} and ${f%/*} usage are part of Bash string manipulation methods. You can read more about it here.
You will need to extract the 3rd item after "_" :
P45_input_foo --> foo
create the directory (if needed) and copy the file to it. Something like this (not tested, might need editing):
STARTING_DIR="/"
cd "$STARTING_DIR"
VAR=$(ls -1)
while read DIR; do
TARGET_DIR=$(echo "$DIR" | cut -d'_' -f3)
NEW_DIR="/data/$DIR"
if [ ! -d "$NEW_DIR" ]; then
mkdir "$NEW_DIR"
fi
cp "$DIR/result.dat" "$NEW_DIR/result.dat"
if [ $? -ne 0 ];
echo "ERROR: encountered an error while copying"
fi
done <<<"$VAR"
Explanation: assuming all the paths you've mentioned are under root / (if not change STARTING_PATH accordingly). With ls you get the list of the directories, store the output in VAR. Pass the content of VAR to the while loop.
A bit of find and with a few bash tricks, the below script could do the trick for you. Remember to run the script without the mv and see if "/data/"$folder"/" is the actual path that you want to move the file(s).
#!/bin/bash
while IFS= read -r -d '' file
do
fileNew="${file%/*}" # Everything before the last '\'
fileNew="${fileNew#*/}" # Everything after the last '\'
IFS="_" read _ _ folder <<<"$fileNew"
mv -v "$file" "/data/"$folder"/"
done < <(find . -type f -name "result.dat" -print0)
The history of this problem is:
I have millions of files and directories on a NAS system. I found a count of 1,095,601 empty (0 byte) files. These files used to have data but were destroyed by a predecessor not using the correct toolsets to migrate data between an XSAN and this Isilon NAS.
The files were media production data, like fonts, pdfs and image files. They are no longer useful beyond the history of their existence. Before I proceed to delete them, the production user's need a record of which files used to exist, so when they browse a project folder, they can use the unaffected files but then refer to a text file in the same directory which records which files used to also be there and thus provide reason as to why certain reference files are broken.
So how do I find files across multiple directories and delete them but first output their filename to a text file which would be saved to each relevant path location?
I am thinking along the lines of:
for file in $(find . -type f -size 0); do
echo "$file" >> /PATH/TO/FOUND/FILE/PARENT/DIR/deletedFiles.txt -print0 |
xargs -0 rm ;
done
To delete each empty file while leaving behind a file called deletedFiles.txt which contains the names of the deleted files, try:
PATH=/bin:/usr/bin find . -empty -type f -execdir bash -c 'printf "%s\n" "$#" >>deletedFiles.txt' none {} + -delete
How it works
PATH=/bin:/usr/bin
This sets a temporary but secure path.
find .
This starts find looking in the current directory
-empty
This tells find to only look for empty files
-type f
This restricts find to looking for regular files.
-execdir bash -c 'printf "%s\n" "$#" >>deletedFiles.txt' none {} +
In each directory that contains an empty file, this adds the name of each empty file to the file deletedFiles.txt.
Notice the peculiar use of none in the command:
bash -c 'printf "%s\n" "$#" >>deletedFiles.txt' none {} +
When this command is run, bash will execute the string printf "%s\n" "$#" >>deletedFiles.txt and the arguments that follow that string are assigned to the positional parameters: $0, $1, $2, etc. When we use $#, it does not include $0. It, as is usual, expands to $1, $2, .... Thus, we add the placeholder none so that the placeholder is assigned is the $0, which we will ignore, and the complete list of file names are assigned to "$#".
-delete
This deletes each empty file.
Why not simply
find . -type f -size 0 -exec rm -v + |
sed -e 's%^removed .\./%%' -e 's/.$//' >deletedFiles.txt
If your find is too old to support -exec ... + you'll need to revert to -exec rm -v {} \; or refactor to
find . -type f -size 0 -print0 |
xargs -r -0 rm -v |
sed -e 's%^removed .\./%%' -e 's/.$//' >deletedFiles.txt
The brief sed script is to postprocess the output from rm -v which looks like
removed ‘./bar’
removed ‘./foo’
(with some funny quote characters around the file name) on my system. If you are fine with that output, of course, just omit the sed script from the pipeline.
If you know in advance which directories contain empty files, you can run the above snippet individually in those directories. Assuming you saved the snippet above as a script (with a proper shebang and execute permissions) named find-empty, you could simply use
for path in /path/to/first /path/to/second/directory /path/to/etc; do
cd "$path" && find-empty
done
This will only work if you have absolute paths (if not, you can run the body of the loop in a subshell by adding parentheses around it).
If you want to inspect all the directories in a tree, change the script to print to standard output instead (remove >deletedFiles.txt from the script) and try something like
find /path/to/tree -type d -exec sh -c '
t=$(mktemp -t find-emptyXXXXXXXX)
cd "$1" &&
find-empty | grep . >"$t" &&
mv "$t" deletedFiles.txt ||
rm "$t"' _ {} \;
This uses a temporary file so as to avoid updating the timestamp of directories which do not contain any empty files. The grep . is used purely for side effect; if any (non-empty) lines are printed, it will return success, whereas otherwise, it will report failure; this way, we know whether or not to move the temporary file to the target directory.
With prompting from #JonathanLeffler I have succeeded with the following:
#!/bin/bash
## call this script with: find . -type f -empty -exec handleEmpty.sh {} +
for file in "$#"
do
file2="$(basename "$file")"
echo "$file2" >> "$(dirname "$file")"/deletedFiles.txt
rm "$file"
done
This means I retain a trace of the removed files in a deletedFiles.txt flag file in each respective directory for the users to see when files are missing. That way, they can pursue going back to archive CD's to retrieve these deleted files, which are hopefully not 0 byte files.
Thanks to #John1024 for the suggestion of using the empty flag rather than size.
I was thinking if using a BASH script is possible without manually copying each file that is in this parent directory
"/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.0.sdk
/System/Library/PrivateFrameworks"
So in this folder PrivateFrameworks, there are many subfolders and in each subfolder it consists of the file that I would like to copy it out to another location. So the structure of the path looks like this:
-PrivateFrameworks
-AccessibilityUI.framework
-AccessibilityUI <- copy this
-AccountSettings.framework
-AccountSettings <- copy this
I do not want the option of copying the entire content in the folder as there might be cases where the folders contain files which I do not want to copy. So the only way I thought of is to copy by the file extension. However as you can see, the files which I specified for copying does not have an extension(I think?). I am new to bash scripting so I am not familiar if this can be done with it.
To copy all files in or below the current directory that do not have extensions, use:
find . ! -name '*.*' -exec cp -t /your/destination/dir/ {} +
The find . command looks for all files in or below the current directory. The argument -name '*.*' would restrict that search to files that have extensions. By preceding it with a not (!), however, we get all files that do not have an extension. Then, -exec cp -t /your/destination/dir/ {} + tells find to copy those files to the destination.
To do the above starting in your directory with the long name, use:
find "/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.0.sdk/System/Library/PrivateFrameworks" ! -name '*.*' -exec cp -t /your/destination/dir/ {} +
UPDATE: The unix tag on this question has been removed and replaced with a OSX tag. That means we can't use the -t option on cp. The workaround is:
find . ! -name '*.*' -exec cp {} /your/destination/dir/ \;
This is less efficient because a new cp process is created for every file moved instead of once for all the files that fit on a command line. But, it will accomplish the same thing.
MORE: There are two variations of the -exec clause of a find command. In the first use above, the clause ended with {} + which tells find to fill up the end of command line with as many file names as will fit on the line.
Since OSX lacks cp -t, however, we have to put the file name in the middle of the command. So, we put {} where we want the file name and then, to signal to find where the end of the exec command is, we add a semicolon. There is a trick, though. Because bash would normally consume the semicolon itself rather than pass it on to find, we have to escape the semicolon with a backslash. That way bash gives it to the find command.
sh SCRIPT.sh copy-from-directory .extension copy-to-directory
FROM_DIR=$1
EXTENSION=$2
TO_DIR=$3
USAGE="""Usage: sh SCRIPT.sh copy-from-directory .extension copy-to-directory
- EXAMPLE: sh SCRIPT.sh PrivateFrameworks .framework .
- NOTE: 'copy-to-directory' argument is optional
"""
## print usage if less than 2 args
if [[ $# < 2 ]]; then echo "${USAGE}" && exit 1 ; fi
## set copy-to-dir default args
if [[ -z "$TO_DIR" ]] ; then TO_DIR=$PWD ; fi
## DO SOMETHING...
## find directories; find target file;
## copy target file to copy-to-dir if file exist
find $FROM_DIR -type d | while read DIR ; do
FILE_TO_COPY=$(echo $DIR | xargs basename | sed "s/$EXTENSION//")
if [[ -f $DIR/$FILE_TO_COPY ]] ; then
cp $DIR/$FILE_TO_COPY $TO_DIR
fi
done