Often after unzipping a file I end up with a directory containing nothing but another directory (e.g., mkdir foo; cd foo; tar xzf ~/bar.tgz may produce nothing but a bar directory in foo). I wanted to write a script to collapse that down to a single directory, but if there are dot files in the nested directory it complicates things a bit.
Here's a naive implementation:
mv -i $1/* $1/.* .
rmdir $1
The only problem here is that it'll also try to move . and .. and ask overwrite ./.? (y/n [n]). I can get around this by checking each file in turn:
IFS=$'\n'
for file in $1/* $1/.*; do
if [ "$file" != "$1/." ] && [ "$file" != "$1/.." ]; then
mv -i $file .
fi
done
rmdir $1
But this seems like an inelegant workaround. I tried a cleaner method using find:
for file in $(find $1); do
mv -i $file .
done
rmdir $1
But find $1 will also give $1 as a result, which gives an error of mv: bar and ./bar are identical.
While the second method seems to work, is there a better way to achieve this?
Turn on the dotglob shell option, which allows the your pattern to match files beginning with ..
shopt -s dotglob
mv -i "$1"/* .
rmdir "$1"
First, consider that many tar implementations provide a --strip-components option that allows you to strip off that first path. Not sure if there is a first path?
tar -tf yourball.tar | awk -F/ '!s[$1]++{print$1}'
will show you all the first-level contents. If there is only that one directory, then
tar --strip-components=1 -tf yourball.tar
will extract the contents of that directory in tar into the current directory.
So that's how you can avoid the problem altogether. But it's also a solution to your immediate problem. Having extracted the files already, so you have
foo/bar/stuff
foo/bar/.otherstuff
you can do
tar -cf- foo | tar --strip-components=2 -C final_destination -xf-
The --strip-components feature is not part of the POSIX specification for tar, but it is on both the common GNU and OSX/BSD implementations.
Related
I am writing a bash script that pulls files from another server to the current directory. The issue is that I get a lot of files and I only need ~3 of them; however all 3 might not be there.
For example, make server all:
server call --> file1.txt file2.txt file3.xls file4.json .... (etc)
Then compress files with tar:
tar zcf needed_files.tgz file4.json file23.doc *.txt
But file4.json was not there, so I would expect tar to compress file23.doc and all .txt files but the script fails with:
tar: file4.json: Cannot stat: No such file or directory
I have tried other combinations of tar commands like czvf but no luck.
tar should successfully compress the existing files despite the "no such file or directory" errors.
Anyway, you could also use nullglob in combination with extglob #() to get only the existing files:
shopt -s extglob nullglob
files=( "fileA"#() "fileB"#() *.txt )
(( ${#files[#]} )) && tar zcf needed_files.tgz -- "${files[#]}"
Try an extended glob.
shopt -s extglob # set extended globbing on
if echo file[1234].+(txt|xls|json) | grep -vq '\['
then tar cvzf needed_files.tgz file[1234].+(txt|xls|json)
else echo No matching files for extglob 'file[1234].+(txt|xls|json)'
fi
If matching files exist, it will list them.
If not, it will literally echo back the pattern.
grepping out the pattern metacharacters tells you whether there are any files in the set. If they do exist, use the same glob to provide the files to tar, and it will receive exactly the set of matching files. If they don't, the condition test lets you skip it.
Of course, it breaks if you make files with [ in the names, etc...
Or, you could do it in a loop....
for f in file[1234].+(txt|xls|json)
do if [[ -e "$f" ]]
then [[ -e needed_files.tar ]] && c=r || c=c
tar ${c}vf needed_files.tar "$f"
fi
done
Not perfect, but might suit your tastes better.
Neither is a great solution, but one of them ought to get you rolling.
tar zcf needed_files.tgz $(ls -d file4.json file23.doc *.txt 2>/dev/null)
Notice that prints only existing files
ls -d file4.json file23.doc *.txt 2>/dev/null
Also you can use --ignore-failed-read option, but it will also ignore other read errors.
Consider a list of files (e.g. files.txt) similar (but not limited) to
/root/
/root/lib/
/root/lib/dir1/
/root/lib/dir1/file1
/root/lib/dir1/file2
/root/lib/dir2/
...
How can I copy the specified files (not any other content from the folders which are also specified) to a location of my choice (e.g. ~/destination) with a) intact folder structure but b) N folder components (in the example just /root/) stripped from the path?
I already managed to use
cp --parents `cat files.txt` ~/destination
to copy the files with an intact folder structure, however this results in all files ending up in ~/destination/root/... when I'd like to have them in ~/destination/...
I think I found a really nice an concise solution by using GNU tar:
tar cf - -T files.txt | tar xf - -C ~/destination --strip-components=1
Note the --strip-components option that allows to remove an arbitrary number of path components from the beginning of the file name.
One minor problem though: It seems tar always "compresses" the whole content of folders mentioned in files.txt (at least I couldn't find an option to ignore folders), but that is most easily solved using grep:
cat files.txt | grep -v '/$' > files2.txt
This might not be the most graceful solution - but it works:
for file in $(cat files.txt); do
echo "checking for $file"
if [[ -f "$file" ]]; then
file_folder=$(dirname "$file")
destination_folder=/destination/${file_folder#/root/}
echo "copying file $file to $destination_folder"
mkdir -p "$destination_folder"
cp "$file" "$destination_folder"
fi
done
I had a look at cp and rsync, but it looks like they would benefit more if you to cd into /root first.
However, if you did cd to the correct directory before hand, you could always run it as a subshell so that you would be returned to your original location once the subshell has finished.
I have a folder structure like this:
A big parent folder named Photos. This folder contains 900+ subfolders named a_000, a_001, a_002 etc.
Each of those subfolders contain more subfolders, named dir_001, dir_002 etc. And each of those subfolders contain lots of pictures (with unique names).
I want to move all these pictures contained in the subdirectories of a_xxx inside a_xxx. (where xxx could be 001, 002 etc)
After looking in similar questions around, this is the closest solution I came up with:
for file in *; do
if [ -d $file ]; then
cd $file; mv * ./; cd ..;
fi
done
Another solution I got is doing a bash script:
#!/bin/bash
dir1="/path/to/photos/"
subs= `ls $dir1`
for i in $subs; do
mv $dir1/$i/*/* $dir1/$i/
done
Still, I'm missing something, can you help?
(Then it would be nice to discard the empty dir_yyy, but not much of a problem at the moment)
You could try the following bash script :
#!/bin/bash
#needed in case we have empty folders
shopt -s nullglob
#we must write the full path here (no ~ character)
target="/path/to/photos"
#we use a glob to list the folders. parsing the output of ls is baaaaaaaddd !!!!
#for every folder in our photo folder ...
for dir in "$target"/*/
do
#we list the subdirectories ...
for sub in "$dir"/*/
do
#and we move the content of the subdirectories to the parent
mv "$sub"/* "$dir"
#if you want to remove subdirectories once the copy is done, uncoment the next line
#rm -r "$sub"
done
done
Here is why you don't parse ls in bash
Make sure the directory where the files exist is correct (and complete) in the following script and try it:
#!/bin/bash
BigParentDir=Photos
for subdir in "$BigParentDir"/*/; do # Select the a_001, a_002 subdirs
for ssdir in "$subdir"/*/; do # Select dir_001, … sub-subdirs
for f in "$ssdir"/*; do # Select the files to move
if [[ -f $f ]]; do # if indeed are files
echo \
mv "$ssdir"/* "$subdir"/ # Move the files.
fi
done
done
done
No file will be moved, just printed. If you are sure the script does what you want, comment the echo line and run it "for real".
You can try this
#!/bin/bash
dir1="/path/to/photos/"
subs= `ls $dir1`
cp /dev/null /tmp/newscript.sh
for i in $subs; do
find $dir1/$i -type f -exec echo mv \'\{\}\' $dir1/$i \; >> /tmp/newscript.sh
done
then open /tmp/newscript.sh with an editor or less and see if looks like what you are trying to do.
if it does then execute it with sh -x /tmp/newscript.sh
Let's say I have two directories with the same structure and I want to set the timestamps of files, contained in the second to those of the first if and only if the content of the files is the same.
I give an answer here but if you guys have less clumsy and more efficient ways of achieving the goal, that would be perfect.
An even easier way is simply:
rsync -uav /path/to/dir1/ /path/to/dir2
(removing v suppresses --verbose output)
note: the trailing '/' following dir1. It tells rsync to take the contents of dir1 instead of dir1 itself.
One possible solution is this script:
#!/bin/bash
OLDDIR=$(readlink -f $1)
NEWDIR=$(readlink -f $2)
cd $NEWDIR
for file in $(find .); do
file2=$OLDDIR/$file
if test -e "$file2" && diff >/dev/null -q "$file" "$file2" ; then
touch -r "$file2" "$file"
fi
done
I'am trying to write simple script that will get files name from one folder and search them in another folder and remove if found them in that folder.
Got two folder like
/home/install/lib
/home/install/bin
/home/install/include
and
/usr/local/lib
/usr/local/bin
/usr/local/include
I want to remove all file's from /usr/local/lib{bin,include} that contains in /home/install/lib{bin,include}. For example having
/home/install/lib/test1
/usr/local/lib/test1
scritp will remove /usr/local/lib/test1. I tried to do it from each separate directory
/home/install/lib:ls -f -exec rm /usr/local/lib/{} \;
but nothing. Can you help me to manage with this simple script?
Create script rmcomm
#!/bin/bash
a="/home/install/$1"
b="/usr/local/$1"
comm -12 <(ls "$a") <(ls "$b") | while read file; do
rm "$b/$file"
done
Then call this script for every pair:
for dir in lib bin include; do rmcomm "$dir"; done
Here's something simple. Remove the echo from the line containing rm to run it after you've ensured it's doing what you want:
#!/bin/bash
dirs[0]=lib
dirs[1]=bin
dirs[2]=include
pushd /home/install
for dir in "${dirs[#]}"
do
for file in $(find $dir -type f)
do
# Remove 'echo' below once you're satisfied the correct files
# are being removed
echo rm /usr/local/$file
done
done
popd