bash: completely replace a folder with cp - bash

I have a question concerning the "cp"-command:
I have to copy folders to a directory. For me two different scenarios exist. In the first scenario folders with the same name don't exist. That's simple so far:
ls test/folder1/
file1 file2
ls test/destination/
cp -r -v test/folder1/ test/destination/
»test/folder1/“ -> »test/destination/folder1“
»test/folder1/file1“ -> »test/destination/folder1/file1“
»test/folder1/file2“ -> »test/destination/folder1/file2“
In the second scenario a folder with the same name exists. These files have different names than the ones in the source folder. What I actually want is that the folder in the destination directory will be completely replaced by the source folder (if files with the same name exist they should be overriden; if files don't exist in the source directory they should be deleted).
ls test/folder1/
file1 file2
ls test/destination/
folder1
ls test/destination/folder1/
file3 file4
cp -r -v test/folder1/ test/destination/
»test/folder1/file1“ -> »test/destination/folder1/file1“
»test/folder1/file2“ -> »test/destination/folder1/file2“
ls test/destination/folder1/
file1 file2 file3 file4
It would probably be possible with something like
if [ -d destination/$foldername ]
then
rm -r /destination/$foldername
cp-r -v test/$foldername/ test/destination/$foldername
else
cp-r -v test/$foldername/ test/destination/$foldername
fi
but I was wondering if there is a better solution to that.
Thanks already!

In you second case you want to use:
cp -av test/$foldername/* test/destination/$foldername
That will copy the contents of test/$foldername to test/destination/$foldername overwriting any files of the same name that exist in test/destination/$foldername. Additionally, to preserve any (heaven forbid) spaces in $foldername you can quote the path/filename, but NOT the *:
cp -av "test/$foldername/"* "test/destination/$foldername"

Related

batch files copy to another folder, and overwrite files that shared partial name

There are two folders, Folder A that contains a batch of log files, such as
Test10771_20181210141431.log
Test10771_20181210141431_141714370.jpg
Test2062_20181210135223.log
Test2118_20181210134852.log
There is another Folder B that contains another list of log files
Test10771_20181110142431.log
Test10771_20181110142431_141714370.jpg
Test2062_20181010132223.log
Test2118_20181010132852.log
Test3667_20181210142612.log
Test8461_20181210134434.log
I would like to copy files from Folder A to Folder B, if Test10771 exists in folder B, remove Test10771_20181110142431.log and Test10771_20181210141431_141714370.jpg in folder B, and copy Test10771_20181210141431.log in folder A to B.
How about:
#!/bin/bash
folder_a="./FolderA"
folder_b="./FolderB"
while read -r line; do
rm -f -- "$folder_b"/${line}*
cp -p -- "$folder_a"/${line}* "$folder_b"
done < <(for i in "$folder_a"/*; do
if [ -f "$i" ]; then
filename="${i##*/}"
prefix="${filename%%_*}"
echo "$prefix"
fi
done | sort | uniq)
After execution:
$ ls -1 FolderA
Test10771_20181210141431.log
Test10771_20181210141431_141714370.jpg
Test2062_20181210135223.log
Test2118_20181210134852.log
$ls -1 FolderB
Test10771_20181210141431.log
Test10771_20181210141431_141714370.jpg
Test2062_20181210135223.log
Test2118_20181210134852.log
Test3667_20181210142612.log
Test8461_20181210134434.log
The procedure is performed in two steps.
In the 1st step (for i .. loop), extract prefixes of the filenames in FolderA, and uniquify them.
In the 2nd step (while .. loop), remove files in FolderB which matches with the prefix and copy files in FolderA to FolderB.

Bash script to separate files into directories, reverse sort and print in an HTML file works on some files but not others

Goal
Separate files into directories according to their filenames, run a Bash script that reverse sorts them and assembles the content into one file (I know steps to achieve this are already documented on Stack Overflow, but please keep reading...)
Problem
Scripts work on all files but two
State
Root directory
dos-18-1-18165-03-for-sql-server-2012---15-june-2018.html
dos-18-1-18165-03-for-sql-server-2016---15-june-2018.html
dos-18-1-18176-03-for-sql-server-2012---10-july-2018.html
dos-18-1-18197-01-for-sql-server-2012---23-july-2018.html
dos-18-1-18197-01-for-sql-server-2016---23-july-2018.html
dos-18-1-18232-01-for-sql-server-2012---21-august-2018.html
dos-18-1-18232-01-for-sql-server-2016---21-august-2018.html
dos-18-1-18240-01-for-sql-server-2012---5-september-2018.html
dos-18-1-18240-01-for-sql-server-2016---5-september-2018.html
dos-18-2-release-notes.html
dos-18-2-known-issues.html
Separate the files into directories according to their SQL Server version or name
ls | grep "^dos-18-1.*2012.*" | xargs -i cp {} dos181-2012
ls | grep "^dos-18-1.*2016.*" | xargs -i cp {} dos181-2016
ls | grep ".*notes.*" | xargs -i cp {} dos-18-2-release-notes
ls | grep ".*known.*" | xargs -i cp {} dos-18-2-known-issues
Result (success)
/dos181-2012:
dos-18-1-18165-03-for-sql-server-2012---15-june-2018.html
dos-18-1-18176-03-for-sql-server-2012---10-july-2018.html
dos-18-1-18197-01-for-sql-server-2012---23-july-2018.html
dos-18-1-18232-01-for-sql-server-2012---21-august-2018.html
dos-18-1-18240-01-for-sql-server-2012---5-september-2018.html
/dos181-2016:
dos-18-1-18165-03-for-sql-server-2016---15-june-2018.html
dos-18-1-18197-01-for-sql-server-2016---23-july-2018.html
dos-18-1-18232-01-for-sql-server-2016---21-august-2018.html
dos-18-1-18240-01-for-sql-server-2016---5-september-2018.html
/dos-18-2-known-issues
dos-18-2-known-issues.html
/dos-18-2-release-notes
dos-18-2-release-notes.html
Variables (all follow this pattern)
dos181-2012.sh
file="dos181-2012"
export
dos-18-2-known-issues
file="dos-18-2-known-issues"
export
Reverse sort and assemble (assumes /$file exists; after testing all lines of code I believe this is where the problem lies):
cat $( ls "$file"/* | sort -r ) > "$file"/"$file".html
Result (success and failure)
dos181-2012.html has the correct content in the correct order.
dos-18-2-known-issues.html is empty.
What I have tried
I tried to ignore the two files in the command:
cat $( ls "$file"/* -i (grep ".*known.*" ) | sort -r ) > "$file"/"$file".html
Result: The opposite occurs
dos181-2012.html is empty
dos-18-2-known-issues.html is not empty
Thank you
I am completely baffled. Why do these scripts work on some files but not others? (I can share more information about the file contents if that will help, but the file contents are nearly identical.) Thank you for any insights.
first off, you question is quite incomplete. You start great, showing the input files and directories. But then you talk about variables and $files, but you do not show the code from which these originate. So I based my answer on the explanation in the first paragraph and what I deduced from the rest of the question.
I did this:
#!/bin/bash
cp /etc/hosts dos-18-1-18165-03-for-sql-server-2012---15-june-2018.html
cp /etc/hosts dos-18-1-18165-03-for-sql-server-2016---15-june-2018.html
cp /etc/hosts dos-18-1-18176-03-for-sql-server-2012---10-july-2018.html
cp /etc/hosts dos-18-1-18197-01-for-sql-server-2012---23-july-2018.html
cp /etc/hosts dos-18-1-18197-01-for-sql-server-2016---23-july-2018.html
cp /etc/hosts dos-18-1-18232-01-for-sql-server-2012---21-august-2018.html
cp /etc/hosts dos-18-1-18232-01-for-sql-server-2016---21-august-2018.html
cp /etc/hosts dos-18-1-18240-01-for-sql-server-2012---5-september-2018.html
cp /etc/hosts dos-18-1-18240-01-for-sql-server-2016---5-september-2018.html
cp /etc/hosts dos-18-2-release-notes.html
cp /etc/hosts dos-18-2-known-issues.html
DIRS='dos181-2012 dos181-2016 dos-18-2-release-notes dos-18-2-known-issues'
for DIR in $DIRS
do
if [ ! -d $DIR ]
then
mkdir $DIR
fi
done
cp dos-18-1*2012* dos181-2012
cp dos-18-1*2016* dos181-2016
cp *notes* dos-18-2-release-notes
cp *known* dos-18-2-known-issues
for DIR in $DIRS
do
/bin/ls -c1r $DIR >$DIR.html
done
The cp commands are just to create the files with something in them.
You did not specify how the directory names were produced, so I went with the easy option and listed them in a variable ($DIRS). These could be built based on the filenames, but you did not mention that.
Then created the directories (first for).
Then 4 cp commands. Your code is very complicated for something so basic. cp, like rm;mv;ls;... can do wildcard expansion, so there is no need for complex grep and xargs to copy files around.
Finally in the last for loop, list the files (ls), in 1 column (-c1, strictly output formatting), reversed the sort order (-r). The result of that ls is sent to a ".html" file of the same name as the directory.

Shell: Copy list of files with full folder structure stripping N leading components from file names

Consider a list of files (e.g. files.txt) similar (but not limited) to
/root/
/root/lib/
/root/lib/dir1/
/root/lib/dir1/file1
/root/lib/dir1/file2
/root/lib/dir2/
...
How can I copy the specified files (not any other content from the folders which are also specified) to a location of my choice (e.g. ~/destination) with a) intact folder structure but b) N folder components (in the example just /root/) stripped from the path?
I already managed to use
cp --parents `cat files.txt` ~/destination
to copy the files with an intact folder structure, however this results in all files ending up in ~/destination/root/... when I'd like to have them in ~/destination/...
I think I found a really nice an concise solution by using GNU tar:
tar cf - -T files.txt | tar xf - -C ~/destination --strip-components=1
Note the --strip-components option that allows to remove an arbitrary number of path components from the beginning of the file name.
One minor problem though: It seems tar always "compresses" the whole content of folders mentioned in files.txt (at least I couldn't find an option to ignore folders), but that is most easily solved using grep:
cat files.txt | grep -v '/$' > files2.txt
This might not be the most graceful solution - but it works:
for file in $(cat files.txt); do
echo "checking for $file"
if [[ -f "$file" ]]; then
file_folder=$(dirname "$file")
destination_folder=/destination/${file_folder#/root/}
echo "copying file $file to $destination_folder"
mkdir -p "$destination_folder"
cp "$file" "$destination_folder"
fi
done
I had a look at cp and rsync, but it looks like they would benefit more if you to cd into /root first.
However, if you did cd to the correct directory before hand, you could always run it as a subshell so that you would be returned to your original location once the subshell has finished.

Moving files from one directory to another preserving subdirectories

I have directory with content (example)
/dir1/a/b/c/file1
/dir1/a/b/c/file2
/dir1/a/d/file3
/dir1/a/e/file4
/dir1/f/dir3/
/dir1/f/dir4/
...
I have list of files and directories, which can be removed - for example file1,file3 and dir3
I would like to move(move, not copy nor tar them - files are large and i need to do it in short time) them to another directory /dir2 (on the same filesystem), but - preserving subdirectories:
/dir1/a/b/c/file1 -> /dir2/a/b/c/file1
/dir1/a/d/file3 -> /dir2/a/d/file3
/dir1/f/dir3/ -> /dir2/f/dir3/
Is there any better way than for each file and directory(for directories skipping last part) create directory in dir2(using mkdir -p/install -d) and then moving it into?
one of simplest solutions is using rsync, with list of files in
--include-from, and with --remove-source-files. But - it copy files, and then remove then - i need to avoid copying - for large files it
take too much time.
If you are comfortable with rsync, you can use it just to list the files and then process that list with this short shell script:
cd dir1
rsync --files-from list --list-only --no-implied-dirs . / |
while read mode size date time path
do
dest=$dir2/`dirname $path` # $dir2 must be an absolute path
mkdirhier $dest
mv $path $dest
done
I have tried this code with example you mentioned above and it worked okay.. Please test it before you use it. In second line, you have to put all file names in a plain text file and provide it's path. My file contents are shown below
#!/bin/ksh
c_file="Path_to_the_file_containing_list_for_movement"
while IFS= read v_line
do
v_fullfilepath=$(find $1 -name "$v_line")
v_dirname=$(dirname $v_fullfilepath)
v_target_path=${v_dirname/$1\//$2/}
mkdir -p "$v_target_path"
mv $v_fullfilepath $v_target_path
#echo $v_line " " $v_fullfilepath " " $v_dirname " " $v_target_path
done <"$c_file"
This was my file contents,
file1
file3
dir3

replacing files in another directory

I have two directories structured as follows:
dir1/a/file1
dir1/a/b/file2
dir1/a/c/d/file3
and
dir2/a/file4
dir2/a/b/file5
dir2/a/c/d/file6
I want to copy all the files in the subdirectories under dir1 to dir2, but keep the files that are currently in dir2, in other words I want to resulting structure to look like:
dir2/a/file1
dir2/a/file4
dir2/a/b/file2
dir2/a/b/file5
dir2/a/c/d/file3
dir2/a/c/d/file6
Is there a simple way to do this using bash?
You could start with
cd dir1
cp -rpuv * ../dir2/
Before:
$ find dir2/
dir2/
dir2/a
dir2/a/file4
dir2/a/c
dir2/a/c/d
dir2/a/c/d/file6
dir2/a/b
dir2/a/b/file5
After:
$ find dir2/
dir2/
dir2/a
dir2/a/file1
dir2/a/file4
dir2/a/c
dir2/a/c/d
dir2/a/c/d/file3
dir2/a/c/d/file6
dir2/a/b
dir2/a/b/file2
dir2/a/b/file5
Note that -p preserves permissions, -v make copy verbose and -u only updates files (doing what the question suggests: keep the files already in dir2)

Resources