I am sure this is possible with a bash script, but I want to make sure I'm not missing something obvious. Google hasn't been much help, so maybe you can be.
Assume a directory has the following files
dir/file1
dir/newfile2
dir/oldfile3
I would like to figure out the best solution for copying all files in the folder to a folder 2 levels deep based on the first two letters of the filename, so the result would be
dir/f/i/file1
dir/n/e/newfile2
dir/o/l/oldfile3
Something like this should do it:
cd dir
for file in *; do
newpath="${file:0:1}/${file:1:1}"
mkdir -p "$newpath"
cp "$file" "$newpath"
done
Be sure all your filenames are two chars or more, though.
${var:n:m} is simply Bash syntax for "substring of var starting at n of length m."
If there can also be arbitrary subdirectories, either add -r to the cp command if you want to copy recursively or add a test to ignore directories in the for loop:
cd dir
for file in *; do
if [ -f "$file" ]; then
newpath="${file:0:1}/${file:1:1}"
mkdir -p "$newpath"
cp "$file" "$newpath"
fi
done
Related
I have a txt file with a list of files (approximately 500) for example:
file_0_hard.msOut
file_1_hard.msOut
file_10_hard.msOut
.
.
.
file_1000_hard.msOut
I want to delete all those files whose name is not in the txt file. All of these files are in the same directory. How can I do this using bash where I read the text file and then delete all those files in the directory that are not in the text file. Help would be appreciated.
Along the lines of user1934428
There is something to say for this solution. But since we have linux at our disposal with a strong filesystem in use I hope. we can make hardlinks; The only requirement for that the destination is on the same filesystem.
So along those lines:
make a directory to store the files you want to keep.
hardlink (ln {file} {target}) ; as this does not cost extra disk space, it only stores the inode number in the new directory file.
remove all files
move the files back from their origin.
And actually this would be about the same as:
mv {files} {save spot}
remove all files
mv {save spot}/{files} back
Which does pretty much the same thing. Then again; it is a nice way to learn about the power of a hardlink.
you may try this :
cd path/dir
for f in *; do
if ! grep -Fxq "$f" pathToFile/file.txt; then
rm -r "$f"
else
printf "exists-- %s \n" ${f}
fi
done
In case you are wondering (as I did) what -Fxq means in plain English:
F: Affects how PATTERN is interpreted (fixed string instead of a regex)
x: Match whole line
q: Shhhhh... minimal printing
Assuming the directory in question is mydir
set -e
cd mydir
tmpdir=/tmp/x$$ # adapt this to your taste
mv $(<list.txt) $tmpdir
cd ..
rm -r mydir
mkdir mydir
mv $tmpdir/* mydir
rm -r $tmpdir
Basically, instead to delete those files you want to keep, you safe them, then delete everything, and then restore them. For your case, this is probably faster than doing the other way around.
UPDATE:
As Michiel commented, it is advisable that you place your tmpdir in the same file system as mydir.
Consider a list of files (e.g. files.txt) similar (but not limited) to
/root/
/root/lib/
/root/lib/dir1/
/root/lib/dir1/file1
/root/lib/dir1/file2
/root/lib/dir2/
...
How can I copy the specified files (not any other content from the folders which are also specified) to a location of my choice (e.g. ~/destination) with a) intact folder structure but b) N folder components (in the example just /root/) stripped from the path?
I already managed to use
cp --parents `cat files.txt` ~/destination
to copy the files with an intact folder structure, however this results in all files ending up in ~/destination/root/... when I'd like to have them in ~/destination/...
I think I found a really nice an concise solution by using GNU tar:
tar cf - -T files.txt | tar xf - -C ~/destination --strip-components=1
Note the --strip-components option that allows to remove an arbitrary number of path components from the beginning of the file name.
One minor problem though: It seems tar always "compresses" the whole content of folders mentioned in files.txt (at least I couldn't find an option to ignore folders), but that is most easily solved using grep:
cat files.txt | grep -v '/$' > files2.txt
This might not be the most graceful solution - but it works:
for file in $(cat files.txt); do
echo "checking for $file"
if [[ -f "$file" ]]; then
file_folder=$(dirname "$file")
destination_folder=/destination/${file_folder#/root/}
echo "copying file $file to $destination_folder"
mkdir -p "$destination_folder"
cp "$file" "$destination_folder"
fi
done
I had a look at cp and rsync, but it looks like they would benefit more if you to cd into /root first.
However, if you did cd to the correct directory before hand, you could always run it as a subshell so that you would be returned to your original location once the subshell has finished.
I have a folder structure like this:
A big parent folder named Photos. This folder contains 900+ subfolders named a_000, a_001, a_002 etc.
Each of those subfolders contain more subfolders, named dir_001, dir_002 etc. And each of those subfolders contain lots of pictures (with unique names).
I want to move all these pictures contained in the subdirectories of a_xxx inside a_xxx. (where xxx could be 001, 002 etc)
After looking in similar questions around, this is the closest solution I came up with:
for file in *; do
if [ -d $file ]; then
cd $file; mv * ./; cd ..;
fi
done
Another solution I got is doing a bash script:
#!/bin/bash
dir1="/path/to/photos/"
subs= `ls $dir1`
for i in $subs; do
mv $dir1/$i/*/* $dir1/$i/
done
Still, I'm missing something, can you help?
(Then it would be nice to discard the empty dir_yyy, but not much of a problem at the moment)
You could try the following bash script :
#!/bin/bash
#needed in case we have empty folders
shopt -s nullglob
#we must write the full path here (no ~ character)
target="/path/to/photos"
#we use a glob to list the folders. parsing the output of ls is baaaaaaaddd !!!!
#for every folder in our photo folder ...
for dir in "$target"/*/
do
#we list the subdirectories ...
for sub in "$dir"/*/
do
#and we move the content of the subdirectories to the parent
mv "$sub"/* "$dir"
#if you want to remove subdirectories once the copy is done, uncoment the next line
#rm -r "$sub"
done
done
Here is why you don't parse ls in bash
Make sure the directory where the files exist is correct (and complete) in the following script and try it:
#!/bin/bash
BigParentDir=Photos
for subdir in "$BigParentDir"/*/; do # Select the a_001, a_002 subdirs
for ssdir in "$subdir"/*/; do # Select dir_001, … sub-subdirs
for f in "$ssdir"/*; do # Select the files to move
if [[ -f $f ]]; do # if indeed are files
echo \
mv "$ssdir"/* "$subdir"/ # Move the files.
fi
done
done
done
No file will be moved, just printed. If you are sure the script does what you want, comment the echo line and run it "for real".
You can try this
#!/bin/bash
dir1="/path/to/photos/"
subs= `ls $dir1`
cp /dev/null /tmp/newscript.sh
for i in $subs; do
find $dir1/$i -type f -exec echo mv \'\{\}\' $dir1/$i \; >> /tmp/newscript.sh
done
then open /tmp/newscript.sh with an editor or less and see if looks like what you are trying to do.
if it does then execute it with sh -x /tmp/newscript.sh
I want to copy all files inside my current directory to a new subdirectory. What is the unix command for this?
To copy all 'files' you will need to exclude the copy of other subdirectories. You can do that with a short loop and compound-command, e.g.
for i in *; do [ -f "$i" ] && cp -a "$i" directoryname; done
There are several ways to do it. Let me know if that gives you any trouble.
And since you are new to shell programming, a compound-command is simply two commands (or more) tied together with && (which means execute the second if the first succeeds) and || (which means execute the second only if the first fails). So above, the loop simply does the following:
for i in *; do # for each linux-file in dir
[ -f "$i" ] && cp -a "$i" directoryname # test if plain file && copy
done
You can use cp -R command. With -R flag it copies directories recursively.
For example:
cp -R directory1/ directory2/
copies whole directory1 to directory2.
Replace the sub-directory-name:
$cp -R * sub-directory-name/
I want my script to copy the files in addresses.list (which contains the absolute path of those files) in my folder3 adding to the name of files the variable k (name of the lower folders they came from).
The script makes sense to me but it does not work, it states:"cp: cannot create regular file"; any suggestions?
#!/bin/bash
cd /folder1/
for k in $( cat lowerfolders_list ); do
for f in $( cat addresses.lst ); do
cp $f "/folder1/folder2/folder3/${k}_$f"
cd /folder1/
done
done
The exact error I get is this " cp: cannot create regular file `/A/B/C/D/E/folder1/folder2/folder3/name of my $k/path_to_my_file/myfile.sdf': No such file or directory "
EDIT
It would be ok even if I could copy those files naming them only after the $k but when I tried to do this my output is the last file in my addresses.lst multiple times with every name on my lowerfolders_list.
This is the script I used:
#!/bin/bash
cd /folder1/
for k in $( cat lowerfolders_list ); do
for f in $( cat addresses.lst ); do
cp "$f" "/folder1/folder2/folder3/${k}"
done
cd -
done
Any suggestions?
EDIT
Resolved
#!/bin/bash
cd /folder1/
for k in $( cat lowerfolders_list ); do
for f in $( cat addresses.lst ); do
myfile=$( basename "$f" )
cp "$f" "/folder1/folder2/folder3/${k}_${myfile}"
done
cd -
done
Thanks to all the people that contributed.
Sorry but keeping commands that worked for another purpose without understanding what do they do is always going to put you into trouble. In marketing it may appear to work but in computer science, doing something you don't understand breaks very fast. Just stop that practice early. Try to first understand what is the directory structure. Read man cd and man cp and man mkdir.
So the cd command is screwing your script because it changes directory and the list of files and directories you read initially become invalid paths. But maybe I have an idea what were you trying to do.
Second thing is that it's unsafe to read list of dirs and files like that. If they have spaces in the name, it will break bad. But lets leave that for the time being.
Then you're not creating the directory structure "/folder1/folder2/folder3". If that doesn't exist prior running your script, it will also break. Use mkdir -p.
So my best guess for what you're trying to do will be something like that:
for k in $( cat lowerfolders_list ); do
cd "${k}"
for f in $( cat addresses.lst ); do
cp "$f" "/folder1/folder2/folder3/${k}_$f"
done
cd -
done