I have a txt file with a list of files (approximately 500) for example:
file_0_hard.msOut
file_1_hard.msOut
file_10_hard.msOut
.
.
.
file_1000_hard.msOut
I want to delete all those files whose name is not in the txt file. All of these files are in the same directory. How can I do this using bash where I read the text file and then delete all those files in the directory that are not in the text file. Help would be appreciated.
Along the lines of user1934428
There is something to say for this solution. But since we have linux at our disposal with a strong filesystem in use I hope. we can make hardlinks; The only requirement for that the destination is on the same filesystem.
So along those lines:
make a directory to store the files you want to keep.
hardlink (ln {file} {target}) ; as this does not cost extra disk space, it only stores the inode number in the new directory file.
remove all files
move the files back from their origin.
And actually this would be about the same as:
mv {files} {save spot}
remove all files
mv {save spot}/{files} back
Which does pretty much the same thing. Then again; it is a nice way to learn about the power of a hardlink.
you may try this :
cd path/dir
for f in *; do
if ! grep -Fxq "$f" pathToFile/file.txt; then
rm -r "$f"
else
printf "exists-- %s \n" ${f}
fi
done
In case you are wondering (as I did) what -Fxq means in plain English:
F: Affects how PATTERN is interpreted (fixed string instead of a regex)
x: Match whole line
q: Shhhhh... minimal printing
Assuming the directory in question is mydir
set -e
cd mydir
tmpdir=/tmp/x$$ # adapt this to your taste
mv $(<list.txt) $tmpdir
cd ..
rm -r mydir
mkdir mydir
mv $tmpdir/* mydir
rm -r $tmpdir
Basically, instead to delete those files you want to keep, you safe them, then delete everything, and then restore them. For your case, this is probably faster than doing the other way around.
UPDATE:
As Michiel commented, it is advisable that you place your tmpdir in the same file system as mydir.
Related
I need to add prefix C_ and then move files from tmp location to target location.
Here is the script
I am not allowed to place script in current directory.
for tmpfile in /home/asmita/tmp
do
mv "$tmpfile" "C_${tmpfile}"
mv "C_${tmpfile}" /home/tgasmita
done
When I try moving prefixed files I get error C_/home/asmita/tmp/xyz.txt not found. as entire path is stored in tmpfile variable.
Change your code to use the basename and the dirname command to get the filename and the directory name. Use these to combine the values and get the new path.
for tmpfile in /home/asmita/tmp
do
DIRPATH=$(dirname "${tmpfile}")
FILENAME=$(basename "${tmpfile}")
mv "$tmpfile" "${DIRPATH}C_${FILENAME}"
mv "${DIRPATH}C_${FILENAME}" /home/tgasmita
done
There are many ways to do this. First let me fix your script.
1.In below code make sure that you are passing only filename not the full path. Below script is valid only if know the FILENAME
#!/bin/sh
for tmpfile in /home/asmita/tmp
do
mv "{tmpfile}/filename.txt" "/home/tgasmita/C_filename.txt"
done
2.If you do not know file name and you want to rename and move .txt files to another folder then you might like below script.
#!/bin/sh
lines=`find /home/asmita/tmp -name "*.txt" -printf "%f\n";`
for i in ${lines[#]}
do
mv "/home/asmita/tmp/${i}" "/home/tgasmita/C_${i}"
done
Please note that I m looking for only text file in source folder(/home/asmita/tmp
) You can change .txt to other extension according to requirement. If you want to move and rename all files from source folder then just replace "*.txt" to ".*" from second line.
I have a whole bunch of files in source folder need to be updated to destination folder. the layout in source folder is flat, no sub directory such as:
src\a.h
\a.c
\b.h
\b.c
\c.h
\c.c
The destination folder contains multiple layers deep sub directory such as:
dst\App\a.h
\App\a.c
\USBD\CDC\b.h
\USBD\CDC\b.c
\USBH\CORE\FUNCTION\c.h
\USBH\CORE\FUNCTION\c.h
I need to search all the files in the "dst" directory recursively, if any of them match any of the file in the "src" directory, then rename the one in "dst" as XX.sav (XX is the original name) and copy the one from "dst" to "src".
thanks
Assuming that you actually want to copy from src to dst, that src and dst are siblings, and that the command is run from the common parent directory, try:
find dst -type f -exec sh -c 'test -f src/${0##*/} &&
{ mv $0 $0.sav && cp src/${0##*/} $0; }' {} \;
If the final two assumptions are not correct, try (untested):
find /path/to/dst -type f -exec sh -c 'test -f $1/${0##*/} &&
{ mv $0 $0.sav && cp $1/${0##*/} $0; }' {} /path/to/src \;
You should be careful: if there are any existing .sav files in dst they may be overwritten, and if there are any .sav files in src you may wind up with .sav.sav files in dst that are duplicates of the related .sav file that this command is itself creating. Use mv -i to address these issues if they are relevant. If the number of existing files is high, mv -i may not be an appropriate solution.
There are several steps to finding a solution to your problem.
First of all, you need to iterate over all files in src folder to make sure that you know what to look for in dst. This can be done easily in bash like this:
for filename in src/*
do
echo "$filename" # simple sanity check
done
Now that you know what you're looking for, you can use command find to try to find the files in dst. Since this command enables recursive look-ups itself, you don't need to do anything special. However, you'll probably want to look for a name of a file, not the whole path also containing directory structure. Something like this should work.
for filename in src/*
do
pattern=$(basename $filename) # remove folder from filename
for another_filename in $(find dst -name "$pattern")
do
echo "$filename -> $another_filename" # another sanity check
done
done
Now that you have a file from src matched with a file from dst, you can simply use cp and mv to achieve what you need.
This is nowhere near the shortest possible solution, but it should be easy to understand and improve once you figure out the commands you need.
I have a few files with the format ReportsBackup-20140309-04-00 and I would like to send the files with same pattern to the files as the example to the 201403 file.
I can already create the files based on the filename; I would just like to move the files based on the name to their correct folder.
I use this to create the directories
old="directory where are the files" &&
year_month=`ls ${old} | cut -c 15-20`&&
for i in ${year_month}; do
if [ ! -d ${old}/$i ]
then
mkdir ${old}/$i
fi
done
you can use find
find /path/to/files -name "*201403*" -exec mv {} /path/to/destination/ \;
Here’s how I’d do it. It’s a little verbose, but hopefully it’s clear what the program is doing:
#!/bin/bash
SRCDIR=~/tmp
DSTDIR=~/backups
for bkfile in $SRCDIR/ReportsBackup*; do
# Get just the filename, and read the year/month variable
filename=$(basename $bkfile)
yearmonth=${filename:14:6}
# Create the folder for storing this year/month combination. The '-p' flag
# means that:
# 1) We create $DSTDIR if it doesn't already exist (this flag actually
# creates all intermediate directories).
# 2) If the folder already exists, continue silently.
mkdir -p $DSTDIR/$yearmonth
# Then we move the report backup to the directory. The '.' at the end of the
# mv command means that we keep the original filename
mv $bkfile $DSTDIR/$yearmonth/.
done
A few changes I’ve made to your original script:
I’m not trying to parse the output of ls. This is generally not a good idea. Parsing ls will make it difficult to get the individual files, which you need for copying them to their new directory.
I’ve simplified your if ... mkdir line: the -p flag is useful for “create this folder if it doesn’t exist, or carry on”.
I’ve slightly changed the slicing command which gets the year/month string from the filename.
I'm trying to make a simple script that copies all of my $HOME into another folder in $HOME called Backup/. This includes all hidden files and folders, and excludes Backup/ itself. What I have right now for the copying part is the following:
shopt -s dotglob
for file in $HOME/*
do
cp -r $file $HOME/Backup/
done
Bash tells me that it cannot copy Backup/ into itself. However, when I check the contents of $HOME/Backup/ I see that $HOME/Backup/Backup/ exists.
The copy of Backup/ in itself is useless. How can I get bash to copy over all the folders except Backup/. I tried using extglob and using cp -r $HOME/!(Backup)/ but it didn't copy over the hidden files that I need.
try rsync. you can exclude file/directories .
this is a good reference
http://www.maclife.com/article/columns/terminal_101_using_rsync_locally
Hugo,
A script like this is good, but you could try this:
cp -r * Backup/;
cp -r .* Backup/;
Another tool used with backups is tar. This compresses your backup to save disk space.
Also note, the * does not cover . hidden files.
I agree that using rsync would be a better solution, but there is an easy way to skip a directory in bash:
for file in "$HOME/"*
do
[[ $file = $HOME/Backup ]] && continue
cp -r "$file" "$HOME/Backup/"
done
This doesn't answer your question directly (the other answers already did that), but try cp -ua when you want to use cp to make a backup. This recurses directories, copies rather than follows links, preserves permissions and only copies a file if it is newer than the copy at the destination.
I am writing the following script to copy *.nzb files to a folder to queue them for Download.
I wrote the following script
#!/bin/bash
#This script copies NZB files from Downloads folder to HellaNZB queue folder.
${DOWN}="/home/user/Downloads/"
${QUEUE}="/home/user/.hellanzb/nzb/daemon.queue/"
for a in $(find ${DOWN} -name *.nzb)
do
cp ${a} ${QUEUE}
rm *.nzb
done
it gives me the following error saying:
HellaNZB.sh: line 5: =/home/user/Downloads/: No such file or directory
HellaNZB.sh: line 6: =/home/user/.hellanzb/nzb/daemon.queue/: No such file or directory
Thing is that those directories exsist, I do have right to access them.
Any help would be nice.
Please and thank you.
Variable names on the left side of an assignment should be bare.
foo="something"
echo "$foo"
Here are some more improvements to your script:
#!/bin/bash
#This script copies NZB files from Downloads folder to HellaNZB queue folder.
down="/home/myusuf3/Downloads/"
queue="/home/myusuf3/.hellanzb/nzb/daemon.queue/"
find "${down}" -name "*.nzb" | while read -r file
do
mv "${file}" "${queue}"
done
Using while instead of for and quoting variables that contain filenames protects against filenames that contain spaces from being interpreted as more than one filename. Removing the rm keeps it from repeatedly producing errors and failing to copy any but the first file. The file glob for -name needs to be quoted. Habitually using lowercase variable names reduces the chances of name collisions with shell variables.
If all your files are in one directory (and not in multiple subdirectories) your whole script could be reduced to the following, by the way:
mv /home/myusuf3/Downloads/*.nzb /home/myusuf3/.hellanzb/nzb/daemon.queue/
If you do have files in multiple subdirectories:
find /home/myusuf3/Downloads/ -name "*.nzb" -exec mv {} /home/myusuf3/.hellanzb/nzb/daemon.queue/ +
As you can see, there's no need for a loop.
The correct syntax is:
DOWN="/home/myusuf3/Downloads/"
QUEUE="/home/myusuf3/.hellanzb/nzb/daemon.queue/"
for a in $(find ${DOWN} -name *.nzb)
# escape the * or it will be expanded in the current directory
# let's just hope no file has blanks in its name
do
cp ${a} ${QUEUE} # ok, although I'd normally add a -p
rm *.nzb # again, this is expanded in the current directory
# when you fix that, it will remove ${a}s before they are copied
done
Why don't you just use rm $(a}?
Why use a combination of cp and rm anyway, instead of mv?
Do you realize all files will end up in the same directory, and files with the same name from different directories will overwrite each other?
What if the cp fails? You'll lose your file.