I have directory with content (example)
/dir1/a/b/c/file1
/dir1/a/b/c/file2
/dir1/a/d/file3
/dir1/a/e/file4
/dir1/f/dir3/
/dir1/f/dir4/
...
I have list of files and directories, which can be removed - for example file1,file3 and dir3
I would like to move(move, not copy nor tar them - files are large and i need to do it in short time) them to another directory /dir2 (on the same filesystem), but - preserving subdirectories:
/dir1/a/b/c/file1 -> /dir2/a/b/c/file1
/dir1/a/d/file3 -> /dir2/a/d/file3
/dir1/f/dir3/ -> /dir2/f/dir3/
Is there any better way than for each file and directory(for directories skipping last part) create directory in dir2(using mkdir -p/install -d) and then moving it into?
one of simplest solutions is using rsync, with list of files in
--include-from, and with --remove-source-files. But - it copy files, and then remove then - i need to avoid copying - for large files it
take too much time.
If you are comfortable with rsync, you can use it just to list the files and then process that list with this short shell script:
cd dir1
rsync --files-from list --list-only --no-implied-dirs . / |
while read mode size date time path
do
dest=$dir2/`dirname $path` # $dir2 must be an absolute path
mkdirhier $dest
mv $path $dest
done
I have tried this code with example you mentioned above and it worked okay.. Please test it before you use it. In second line, you have to put all file names in a plain text file and provide it's path. My file contents are shown below
#!/bin/ksh
c_file="Path_to_the_file_containing_list_for_movement"
while IFS= read v_line
do
v_fullfilepath=$(find $1 -name "$v_line")
v_dirname=$(dirname $v_fullfilepath)
v_target_path=${v_dirname/$1\//$2/}
mkdir -p "$v_target_path"
mv $v_fullfilepath $v_target_path
#echo $v_line " " $v_fullfilepath " " $v_dirname " " $v_target_path
done <"$c_file"
This was my file contents,
file1
file3
dir3
Related
I have a bash script I'm trying to write
I have 2 base directories:
./tmp/serve/
./src/
I want to go through all the directories in ./tmp and copy the *.html files into the same folder path in ./src
i.e
if I have a html file in ./tmp/serve/app/components/help/ help.html -->
copy to ./src/app/components/help/ And recursively do this for all subdirectories in ./tmp/
NOTE: the folder structures should exist so just need to copy them only. If it doesn't then hopefully it could create the folder for me (not what I want) but with GIT I can track these folders to manually handle those loose html files.
I got as far as
echo $(find . -name "*.html")\n
But not sure how to actually extract the file path with pwd and do what I need to, maybe it's not a one liner and needs to be done with some vars.
something like
for i in `echo $(find /tmp/ -name "*.html")\n
do
cp -r $i /src/app/components/help/
done
going so far to create the directories would take some more time for me.
I'll try to do it on my own and see if I come up with something
but for argument sake if you do run pwd and get a response the pseudo code for that:
pwd
get response
if that directory does not exist in src create that directory
copy all the original directories contents into the new folder at /src/$newfolder
(possibly running two for loops, one to check the directory tree, and then one to go through each original directory, copying all the html files)
You process substitution to loop the output from your find command and create the destination directory(ies) and then copy the file(s):
#!/bin/bash
# accept first parameters to script as src_dir and dest values or
# simply use default values if no parameter(s) passed
src_dir=${1:-/tmp/serve}
dest=${2-src}
while read -r orig_path ; do
# To replace the first occurrence of a pattern with a given string,
# use ${parameter/pattern/string}
dest_path="${orig_path/tmp\/serve/${dest}}"
# Use dirname to remove the filename from the destination path
# and create the destination directory.
dest_dir=$(dirname "${dest_path}")
mkdir -p "${dest_dir}"
cp "${orig_path}" "${dest_path}"
done < <(find "${src_dir}" -name '*.html')
This script copy .html files from src directory to des directory (create the subdirectory if they do not exist)
Find the files, then remove the src directory name and copy them into the destination directory.
#!/bin/bash
for i in `echo $(find src/ -name "*.html")`
do
file=$(echo $i | sed 's/src\///g')
cp -r --parents $i des
done
Not sure if you must use bash constructs or not, but here is a GNU tar solution (if you use GNU tar), which IMHO is the best way to handle this situation because all the metadata for the files (permissions, etc.) are preserved:
find ./tmp/serve -name '*.html' -type f -print0 | tar --null -T - -c | tar -x -v -C ./src --strip-components=3
This finds all the .html files (-type f) in the ./tmp/serve directory and prints them nul-terminated (-print0), then sends these filenames via stdin to tar as nul-terminated literals (--null) for inclusion (-T -), creating (-c) an archive which is then sent to another tar instance which extracts (-x) the archive printing its contents along the way (optional: -v), changing directory to the destination (-C ./src) before commencing and stripping (--strip-components=3) the ./tmp/serve/ prefix from the files. (You could also cd ./tmp/serve beforehand, using find . instead, and change -C to ../../src.)
I have a txt file with a list of files (approximately 500) for example:
file_0_hard.msOut
file_1_hard.msOut
file_10_hard.msOut
.
.
.
file_1000_hard.msOut
I want to delete all those files whose name is not in the txt file. All of these files are in the same directory. How can I do this using bash where I read the text file and then delete all those files in the directory that are not in the text file. Help would be appreciated.
Along the lines of user1934428
There is something to say for this solution. But since we have linux at our disposal with a strong filesystem in use I hope. we can make hardlinks; The only requirement for that the destination is on the same filesystem.
So along those lines:
make a directory to store the files you want to keep.
hardlink (ln {file} {target}) ; as this does not cost extra disk space, it only stores the inode number in the new directory file.
remove all files
move the files back from their origin.
And actually this would be about the same as:
mv {files} {save spot}
remove all files
mv {save spot}/{files} back
Which does pretty much the same thing. Then again; it is a nice way to learn about the power of a hardlink.
you may try this :
cd path/dir
for f in *; do
if ! grep -Fxq "$f" pathToFile/file.txt; then
rm -r "$f"
else
printf "exists-- %s \n" ${f}
fi
done
In case you are wondering (as I did) what -Fxq means in plain English:
F: Affects how PATTERN is interpreted (fixed string instead of a regex)
x: Match whole line
q: Shhhhh... minimal printing
Assuming the directory in question is mydir
set -e
cd mydir
tmpdir=/tmp/x$$ # adapt this to your taste
mv $(<list.txt) $tmpdir
cd ..
rm -r mydir
mkdir mydir
mv $tmpdir/* mydir
rm -r $tmpdir
Basically, instead to delete those files you want to keep, you safe them, then delete everything, and then restore them. For your case, this is probably faster than doing the other way around.
UPDATE:
As Michiel commented, it is advisable that you place your tmpdir in the same file system as mydir.
I have lots of projects archived under a directory tree, some of which have a .git folder in them.
What I'd like to do is recursively copy those files and directories to a new destination, keeping the current structure - EXCEPT for those directories containing a .git folder, in which case the script should run a command (let's say "echo", I'll change it later) followed by the folder name, without creating or copying it.
Any help would be much appreciated.
Edit: I'll try to explain myself better: I need to copy every single file and directory, except for those containing .git, which should be skipped and their path should be passed to another command. In this example, path a/b/c/d and its subfolders should be skipped entirely and a/b/c/d should be displayed using echo (just for brevity, I'll replace it with a different command later):
a
a/b
a/b/c
a/b/c/d/.git
a/b/c/d/e
a/b/c/d/f/g
a/b/c/e
a/b/d
a/c
b
b/c
...
IIUC, the following find one-liner will do the job:
find . -type d -mindepth 1 -maxdepth 1 -exec sh -c "test -e '{}/.git' && echo not copy '{}' || cp -r -- '{}' /tmp/copy-here " \;
I have a whole bunch of files in source folder need to be updated to destination folder. the layout in source folder is flat, no sub directory such as:
src\a.h
\a.c
\b.h
\b.c
\c.h
\c.c
The destination folder contains multiple layers deep sub directory such as:
dst\App\a.h
\App\a.c
\USBD\CDC\b.h
\USBD\CDC\b.c
\USBH\CORE\FUNCTION\c.h
\USBH\CORE\FUNCTION\c.h
I need to search all the files in the "dst" directory recursively, if any of them match any of the file in the "src" directory, then rename the one in "dst" as XX.sav (XX is the original name) and copy the one from "dst" to "src".
thanks
Assuming that you actually want to copy from src to dst, that src and dst are siblings, and that the command is run from the common parent directory, try:
find dst -type f -exec sh -c 'test -f src/${0##*/} &&
{ mv $0 $0.sav && cp src/${0##*/} $0; }' {} \;
If the final two assumptions are not correct, try (untested):
find /path/to/dst -type f -exec sh -c 'test -f $1/${0##*/} &&
{ mv $0 $0.sav && cp $1/${0##*/} $0; }' {} /path/to/src \;
You should be careful: if there are any existing .sav files in dst they may be overwritten, and if there are any .sav files in src you may wind up with .sav.sav files in dst that are duplicates of the related .sav file that this command is itself creating. Use mv -i to address these issues if they are relevant. If the number of existing files is high, mv -i may not be an appropriate solution.
There are several steps to finding a solution to your problem.
First of all, you need to iterate over all files in src folder to make sure that you know what to look for in dst. This can be done easily in bash like this:
for filename in src/*
do
echo "$filename" # simple sanity check
done
Now that you know what you're looking for, you can use command find to try to find the files in dst. Since this command enables recursive look-ups itself, you don't need to do anything special. However, you'll probably want to look for a name of a file, not the whole path also containing directory structure. Something like this should work.
for filename in src/*
do
pattern=$(basename $filename) # remove folder from filename
for another_filename in $(find dst -name "$pattern")
do
echo "$filename -> $another_filename" # another sanity check
done
done
Now that you have a file from src matched with a file from dst, you can simply use cp and mv to achieve what you need.
This is nowhere near the shortest possible solution, but it should be easy to understand and improve once you figure out the commands you need.
i have to copy all the file from source directory to destination directory , but skip all file with extension ".txt" and not to the replace the file if its already present in destination directory
example
source directory
/a/aone.js
/a/atwo.js
/b/bone.txt
/b/btwo.js
destination directory
/a/atwo.js
then it should only copy
/a/aone.js
/b/btwo.js
and skip "/a/atwo.js" because its already present in destination folder
and skip "/b/bone.txt" because its extension is ".txt"
i tried this command but this does not work
find /path/to/source/ \( ! -name "*.txt" \) -type f | cp -n /path/to/destination/ -R
cp -n /path/to/source/*(!*.txt) /path/to/destination/ -R
Assuming you can use rsync, (vaz is verbose, archive and compress - I believe the other options are self explanatory)
rsync -vaz --exclude "*.txt" /path/to/source/ /path/to/destination/
Why make it difficult. You were on the right track. A simple:
cp -an /path/to/source/*.[^t*] /path/to/destination
will copy all files from source, except those whose extension begins with a t to destination. It will do so without overwriting existing files in destination. This presumes that files do not have more than one dot. If so, then a few more lines of code will be needed.
The following will illustrate use of the above:
$ md tmp
$ md a
$ md b
$ touch a/a.{j,k,l,txt}
$ ls -1 a
a.j
a.k
a.l
a.txt
$ cp -an a/a*.[^t*] b
$ ls -1 b
a.j
a.k
a.l
using cp, you must match the proper directory depth. If you have another intervening directory, then simply add an additional wildcard. For example:
$ ls -1 dat/*/*.[^t*]
dat/a/a.j
dat/a/a.k
dat/a/a.l
dat/b/a.j
dat/b/a.k
dat/b/a.l
If your directory structure gets more complex, then go with find or rsync. Both are excellent tools and rsync can handle both local and network transfers. cp is the right tool for small jobs, but when more flexibility is needed, then grab a bigger hammer.