I have a bash script I'm trying to write
I have 2 base directories:
./tmp/serve/
./src/
I want to go through all the directories in ./tmp and copy the *.html files into the same folder path in ./src
i.e
if I have a html file in ./tmp/serve/app/components/help/ help.html -->
copy to ./src/app/components/help/ And recursively do this for all subdirectories in ./tmp/
NOTE: the folder structures should exist so just need to copy them only. If it doesn't then hopefully it could create the folder for me (not what I want) but with GIT I can track these folders to manually handle those loose html files.
I got as far as
echo $(find . -name "*.html")\n
But not sure how to actually extract the file path with pwd and do what I need to, maybe it's not a one liner and needs to be done with some vars.
something like
for i in `echo $(find /tmp/ -name "*.html")\n
do
cp -r $i /src/app/components/help/
done
going so far to create the directories would take some more time for me.
I'll try to do it on my own and see if I come up with something
but for argument sake if you do run pwd and get a response the pseudo code for that:
pwd
get response
if that directory does not exist in src create that directory
copy all the original directories contents into the new folder at /src/$newfolder
(possibly running two for loops, one to check the directory tree, and then one to go through each original directory, copying all the html files)
You process substitution to loop the output from your find command and create the destination directory(ies) and then copy the file(s):
#!/bin/bash
# accept first parameters to script as src_dir and dest values or
# simply use default values if no parameter(s) passed
src_dir=${1:-/tmp/serve}
dest=${2-src}
while read -r orig_path ; do
# To replace the first occurrence of a pattern with a given string,
# use ${parameter/pattern/string}
dest_path="${orig_path/tmp\/serve/${dest}}"
# Use dirname to remove the filename from the destination path
# and create the destination directory.
dest_dir=$(dirname "${dest_path}")
mkdir -p "${dest_dir}"
cp "${orig_path}" "${dest_path}"
done < <(find "${src_dir}" -name '*.html')
This script copy .html files from src directory to des directory (create the subdirectory if they do not exist)
Find the files, then remove the src directory name and copy them into the destination directory.
#!/bin/bash
for i in `echo $(find src/ -name "*.html")`
do
file=$(echo $i | sed 's/src\///g')
cp -r --parents $i des
done
Not sure if you must use bash constructs or not, but here is a GNU tar solution (if you use GNU tar), which IMHO is the best way to handle this situation because all the metadata for the files (permissions, etc.) are preserved:
find ./tmp/serve -name '*.html' -type f -print0 | tar --null -T - -c | tar -x -v -C ./src --strip-components=3
This finds all the .html files (-type f) in the ./tmp/serve directory and prints them nul-terminated (-print0), then sends these filenames via stdin to tar as nul-terminated literals (--null) for inclusion (-T -), creating (-c) an archive which is then sent to another tar instance which extracts (-x) the archive printing its contents along the way (optional: -v), changing directory to the destination (-C ./src) before commencing and stripping (--strip-components=3) the ./tmp/serve/ prefix from the files. (You could also cd ./tmp/serve beforehand, using find . instead, and change -C to ../../src.)
Related
I want to copy all files with specific extensions recursively in bash.
****editing****
I've written the full script. I have list of names in a csv file, I'm iterating through each name in that list, then creating a directory with that same name somewhere else, then I'm searching in my source directory for the directory with that name, inside it there are few files with endings of xlsx,tsv,html,gz and I'm trying to copy all of them into the newly created directory.
sample_list_filepath=/home/lists/papers
destination_path=/home/ds/samples
source_directories_path=/home/papers_final/new
cat $sample_list_filepath/sample_list.csv | while read line
do
echo $line
cd $source_directories_path/$line
cp -r *.{tsv,xlsx,html,gz} $source_directories_path/$line $destination_path
done
This works, but it copies all the files there, with no discrimination for specific extension.
What is the problem?
An easy way to solve your problem is to use find and regex :
find src/ -regex '.*\.\(tsv\|xlsx\|gz\|html\)$' -exec cp {} dest/ \;
find look recursively in the directory you specify (in my example it's src/), allows you to filter with -regex and to apply a command for matching results with -exec
For the regex part :
.*\.
will take the name of the file and the dot before extension,
\(tsv\|xlsx\|gz\|html\)$
verify the extension with those you want.
The exec block is what you do with files you got from regex
-exec cp {} dest/ \;
In this case, you copy what you got ({} meaning) to the destination directory.
I have directory with content (example)
/dir1/a/b/c/file1
/dir1/a/b/c/file2
/dir1/a/d/file3
/dir1/a/e/file4
/dir1/f/dir3/
/dir1/f/dir4/
...
I have list of files and directories, which can be removed - for example file1,file3 and dir3
I would like to move(move, not copy nor tar them - files are large and i need to do it in short time) them to another directory /dir2 (on the same filesystem), but - preserving subdirectories:
/dir1/a/b/c/file1 -> /dir2/a/b/c/file1
/dir1/a/d/file3 -> /dir2/a/d/file3
/dir1/f/dir3/ -> /dir2/f/dir3/
Is there any better way than for each file and directory(for directories skipping last part) create directory in dir2(using mkdir -p/install -d) and then moving it into?
one of simplest solutions is using rsync, with list of files in
--include-from, and with --remove-source-files. But - it copy files, and then remove then - i need to avoid copying - for large files it
take too much time.
If you are comfortable with rsync, you can use it just to list the files and then process that list with this short shell script:
cd dir1
rsync --files-from list --list-only --no-implied-dirs . / |
while read mode size date time path
do
dest=$dir2/`dirname $path` # $dir2 must be an absolute path
mkdirhier $dest
mv $path $dest
done
I have tried this code with example you mentioned above and it worked okay.. Please test it before you use it. In second line, you have to put all file names in a plain text file and provide it's path. My file contents are shown below
#!/bin/ksh
c_file="Path_to_the_file_containing_list_for_movement"
while IFS= read v_line
do
v_fullfilepath=$(find $1 -name "$v_line")
v_dirname=$(dirname $v_fullfilepath)
v_target_path=${v_dirname/$1\//$2/}
mkdir -p "$v_target_path"
mv $v_fullfilepath $v_target_path
#echo $v_line " " $v_fullfilepath " " $v_dirname " " $v_target_path
done <"$c_file"
This was my file contents,
file1
file3
dir3
I have a whole bunch of files in source folder need to be updated to destination folder. the layout in source folder is flat, no sub directory such as:
src\a.h
\a.c
\b.h
\b.c
\c.h
\c.c
The destination folder contains multiple layers deep sub directory such as:
dst\App\a.h
\App\a.c
\USBD\CDC\b.h
\USBD\CDC\b.c
\USBH\CORE\FUNCTION\c.h
\USBH\CORE\FUNCTION\c.h
I need to search all the files in the "dst" directory recursively, if any of them match any of the file in the "src" directory, then rename the one in "dst" as XX.sav (XX is the original name) and copy the one from "dst" to "src".
thanks
Assuming that you actually want to copy from src to dst, that src and dst are siblings, and that the command is run from the common parent directory, try:
find dst -type f -exec sh -c 'test -f src/${0##*/} &&
{ mv $0 $0.sav && cp src/${0##*/} $0; }' {} \;
If the final two assumptions are not correct, try (untested):
find /path/to/dst -type f -exec sh -c 'test -f $1/${0##*/} &&
{ mv $0 $0.sav && cp $1/${0##*/} $0; }' {} /path/to/src \;
You should be careful: if there are any existing .sav files in dst they may be overwritten, and if there are any .sav files in src you may wind up with .sav.sav files in dst that are duplicates of the related .sav file that this command is itself creating. Use mv -i to address these issues if they are relevant. If the number of existing files is high, mv -i may not be an appropriate solution.
There are several steps to finding a solution to your problem.
First of all, you need to iterate over all files in src folder to make sure that you know what to look for in dst. This can be done easily in bash like this:
for filename in src/*
do
echo "$filename" # simple sanity check
done
Now that you know what you're looking for, you can use command find to try to find the files in dst. Since this command enables recursive look-ups itself, you don't need to do anything special. However, you'll probably want to look for a name of a file, not the whole path also containing directory structure. Something like this should work.
for filename in src/*
do
pattern=$(basename $filename) # remove folder from filename
for another_filename in $(find dst -name "$pattern")
do
echo "$filename -> $another_filename" # another sanity check
done
done
Now that you have a file from src matched with a file from dst, you can simply use cp and mv to achieve what you need.
This is nowhere near the shortest possible solution, but it should be easy to understand and improve once you figure out the commands you need.
i have to copy all the file from source directory to destination directory , but skip all file with extension ".txt" and not to the replace the file if its already present in destination directory
example
source directory
/a/aone.js
/a/atwo.js
/b/bone.txt
/b/btwo.js
destination directory
/a/atwo.js
then it should only copy
/a/aone.js
/b/btwo.js
and skip "/a/atwo.js" because its already present in destination folder
and skip "/b/bone.txt" because its extension is ".txt"
i tried this command but this does not work
find /path/to/source/ \( ! -name "*.txt" \) -type f | cp -n /path/to/destination/ -R
cp -n /path/to/source/*(!*.txt) /path/to/destination/ -R
Assuming you can use rsync, (vaz is verbose, archive and compress - I believe the other options are self explanatory)
rsync -vaz --exclude "*.txt" /path/to/source/ /path/to/destination/
Why make it difficult. You were on the right track. A simple:
cp -an /path/to/source/*.[^t*] /path/to/destination
will copy all files from source, except those whose extension begins with a t to destination. It will do so without overwriting existing files in destination. This presumes that files do not have more than one dot. If so, then a few more lines of code will be needed.
The following will illustrate use of the above:
$ md tmp
$ md a
$ md b
$ touch a/a.{j,k,l,txt}
$ ls -1 a
a.j
a.k
a.l
a.txt
$ cp -an a/a*.[^t*] b
$ ls -1 b
a.j
a.k
a.l
using cp, you must match the proper directory depth. If you have another intervening directory, then simply add an additional wildcard. For example:
$ ls -1 dat/*/*.[^t*]
dat/a/a.j
dat/a/a.k
dat/a/a.l
dat/b/a.j
dat/b/a.k
dat/b/a.l
If your directory structure gets more complex, then go with find or rsync. Both are excellent tools and rsync can handle both local and network transfers. cp is the right tool for small jobs, but when more flexibility is needed, then grab a bigger hammer.
I am writing the following script to copy *.nzb files to a folder to queue them for Download.
I wrote the following script
#!/bin/bash
#This script copies NZB files from Downloads folder to HellaNZB queue folder.
${DOWN}="/home/user/Downloads/"
${QUEUE}="/home/user/.hellanzb/nzb/daemon.queue/"
for a in $(find ${DOWN} -name *.nzb)
do
cp ${a} ${QUEUE}
rm *.nzb
done
it gives me the following error saying:
HellaNZB.sh: line 5: =/home/user/Downloads/: No such file or directory
HellaNZB.sh: line 6: =/home/user/.hellanzb/nzb/daemon.queue/: No such file or directory
Thing is that those directories exsist, I do have right to access them.
Any help would be nice.
Please and thank you.
Variable names on the left side of an assignment should be bare.
foo="something"
echo "$foo"
Here are some more improvements to your script:
#!/bin/bash
#This script copies NZB files from Downloads folder to HellaNZB queue folder.
down="/home/myusuf3/Downloads/"
queue="/home/myusuf3/.hellanzb/nzb/daemon.queue/"
find "${down}" -name "*.nzb" | while read -r file
do
mv "${file}" "${queue}"
done
Using while instead of for and quoting variables that contain filenames protects against filenames that contain spaces from being interpreted as more than one filename. Removing the rm keeps it from repeatedly producing errors and failing to copy any but the first file. The file glob for -name needs to be quoted. Habitually using lowercase variable names reduces the chances of name collisions with shell variables.
If all your files are in one directory (and not in multiple subdirectories) your whole script could be reduced to the following, by the way:
mv /home/myusuf3/Downloads/*.nzb /home/myusuf3/.hellanzb/nzb/daemon.queue/
If you do have files in multiple subdirectories:
find /home/myusuf3/Downloads/ -name "*.nzb" -exec mv {} /home/myusuf3/.hellanzb/nzb/daemon.queue/ +
As you can see, there's no need for a loop.
The correct syntax is:
DOWN="/home/myusuf3/Downloads/"
QUEUE="/home/myusuf3/.hellanzb/nzb/daemon.queue/"
for a in $(find ${DOWN} -name *.nzb)
# escape the * or it will be expanded in the current directory
# let's just hope no file has blanks in its name
do
cp ${a} ${QUEUE} # ok, although I'd normally add a -p
rm *.nzb # again, this is expanded in the current directory
# when you fix that, it will remove ${a}s before they are copied
done
Why don't you just use rm $(a}?
Why use a combination of cp and rm anyway, instead of mv?
Do you realize all files will end up in the same directory, and files with the same name from different directories will overwrite each other?
What if the cp fails? You'll lose your file.