I have a text file that contains a list of files like this:
Contents of list.txt:
file1.txt
file2.txt
file3.txt
I need to search through a directory (and it's subdirectories) to find each file and copy it into another directory.
Current directory:
-dir
-subdir1
-subdir2
-subdir3
-subdir4
-outputdir
In this example, I might:
find file1.txt in subdir3
find file2.txt in subdir1
file3.txt might not exist
In this case, it would copy file1.txt & file2.txt into outputdir
I don't have much experience will doing this on the command line, but have 1200 files to move, so I can't do it manually. This is the closest thing I've gotten to the right thing, but it doesn't find any of the files because they are all in subdirectories:
xargs -a list.txt cp -t outputdir
An explanation of the command you give would also be immensely helpful. Searches around have also brought up "bash for loops", which I tried to use but couldn't figure out all the intricacies:
FOR /R "%~dp0" %%I IN (.) DO for /f "usebackq delims=" %%a in ("%~dp0list.txt") do echo d |xcopy "%%I\%%a" "outputdir" /e /i
If you have bash 4, use the following:
shopt -s globstar nullglob
while IFS= read -r fname; do
files=("$dir"/**/"$fname")
if (( ${#files[#]} > 0 )); then
cp "${files[#]}" "$outputdir"
fi
done < list.txt
The while loop reads from list.txt one line at a time. The ** pattern matches zero or more directories; files could contain 0 or more matching files. If the array is not empty, all file names are passed to cp for copying.
For older versions, use find:
while IFS= read -r fname; do
find "$dir" -name "$fname" -type f -exec cp {} "$outputdir" \;
done < list.txt
This simply locates all matching files under $dir and runs cp on each.
Related
What I want to achieve
I would like to copy the structure of an existing folder including folders and files but I would like the size of the files to be 0. So it would create an empty file, with the same name and extension.
Windows
I know how to do it by running : robocopy source destination /create /e /xc /xn /xo
Question
What is the equivalent command I could use on Mac ?
Here is a script that might do what you want.
#!/usr/bin/env bash
source=$1
dest=$2
while IFS= read -rd '' file; do
mkdir -p "$dest${file%/*}" || exit
touch -r "$file" "$dest$file"
done < <(find "$source" -type f -print0)
Howto use
./myscript source/ destination/
I am facing a problem with the following shell script:
#!/bin/bash
searchPattern=".*\/.*\.abc|.*\/.*\.xyz|.*\/.*\.[0-9]{3}"
subFolders=$(find -E * -type d -regex ".*201[0-4][0-1][0-9].*|.*20150[1-6].*" -maxdepth 0 | sed 's/.*/"&"/')
echo "subFolders: $subFolders"
# iterate through subfolders
for thisFolder in $subFolders
do
echo "The current subfolder is: $thisFolder"
find -E $thisFolder -type f -iregex $searchPattern -maxdepth 1 -print0 | xargs -0 7z a -mx=9 -uz1 -x!.DS_Store ${thisFolder}/${thisFolder}_data.7z
done
The idea behind it is to archive filetypes with the ending .abc, .xyz and .000-.999 in one 7z archive per subfolder. However, I can't manage to deal with folders including spaces. When I run the script as shown above I always get the following error:
find: "20130117_test": No such file or directory
If I run the script with the line
subFolders=$(find -E * -type d -regex ".*201[0-4][0-1][0-9].*|.*20150[1-6].*" -maxdepth 0 | sed 's/.*/"&"/')
changed to
subFolders=$(find -E * -type d -regex ".*201[0-4][0-1][0-9].*|.*20150[1-6].*" -maxdepth 0)
the script works like charm, but of course not for folders containing space.
Strangely enough, when I execute the following line directly in shell, it works as expected:
find -E "20130117_test" -type f -iregex ".*\/.*\.abc|.*\/.*\.xyz|.*\/.*\.[0-9]{3}" -maxdepth 1 -print0 | xargs -0 7z a -mx=9 -uz1 -x!.DS_Store "20130117_test"/"20130117_test"_data.7z
I know the issue is somehow related to the storing of a list of folders (in quotes) in the subFolders variable, but I simply cannot find a way to make it work properly.
I hope someone more advanced in shell can help me out here.
In general, you should not use find in an attempt to generate a list of file names. You especially cannot build a quoted list the way you are attempting; there is a difference between quotes in a parameter value and quotes around a parameter expansion. Here, especially, you can just use simple patterns:
shopt -s nullglob
subFolders=(
*201[0-4][0-1][0-9]*
*20150[1-6]*
)
for thisFolder in "${subFolders[#]}"; do
echo "The current subfolder is: $thisFolder"
to_archive=(
*/*.abc
*/*.xyz
*/*.[0-9][0-9][0-9]
)
7z a -mx9 -uz1 -x!.DS_Store "$thisFolder/$thisFolder_data.7z" "${to_archive[#]}"
done
Combining the input from gniourf_gniourf and chepner I was able to produce the following code, which does exactly what I want.
#!/bin/bash
shopt -s nullglob
find -E "$PWD" -type d -maxdepth 1 -regex ".*201[0-5][0-1][0-9].*" -print0 | while IFS="" read -r -d "" thisFolder ; do
echo "The current folder is: $thisFolder"
to_archive=( "$thisFolder"/*.[Aa][Bb][Cc] "$thisFolder"/*.[Xx][Yy][Zz] "$thisFolder"/*.[0-9][0-9][0-9] )
if [ ${#to_archive[#]} != 0 ]
then
7z a -mx=9 -uz1 -x!.DS_Store "$thisFolder"/"${thisFolder##*/}"_data.7z "${to_archive[#]}" && rm "${to_archive[#]}"
fi
done
shopt -s nullglob leads to ignorance towards non-matching characters
find... searches for directories matching the regex pattern and streams each matching folder to the while loop using the null separator.
inside the while loop I can safely quote the $thisFolder variable expansion and therefore deal with possible spaces.
using absolute paths instead of relative paths instructs 7z to create no folders inside the archive
I have two directories, say dir1 and dir2, that have exactly the same directory structure. How do I recursively copy all the *.txt files from dir1 to dir2?
Example:
I want to copy from
dir1/subdir1/file.txt
dir1/subdir2/someFile.txt
dir1/.../..../anotherFile.txt
to
dir2/subdir1/file.txt
dir2/subdir2/someFile.txt
dir2/.../..../anotherFile.txt
The .../... in the last file example means this could be any sub-directory, which can have sub-directories itself.
Again I want to do this programmatically. Here's the pseudo-code
SRC=dir1
DST=dir2
for f in `find ./$SRC "*.txt"`; do
# $f should now be dir1/subdir1/file.txt
# I want to copy it to dir2/subdir1/file.txt
# the next line coveys the idea, but does not work
# I'm attempting to substitute "dir1" with "dir2" in $f,
# and store the new path in tmp.txt
echo `sed -i "s/$SRC/$DST/" $f` > tmp.txt
# Do the copy
cp -f $f `cat tmp.txt`
done
You can simply use rsync. This answer is based from this thread.
rsync -av --include='*.txt' --include='*/' --exclude='*' dir1/ dir2/
If you only have .txt files in dir1, this would work:
cp -R dir1/* dir2/
But if you have other file extensions, it will copy them too. In this case, this will work:
cd /path/to/dir1
cp --parents `find . -name '*.txt'` path/to/dir2/
I'am trying to write simple script that will get files name from one folder and search them in another folder and remove if found them in that folder.
Got two folder like
/home/install/lib
/home/install/bin
/home/install/include
and
/usr/local/lib
/usr/local/bin
/usr/local/include
I want to remove all file's from /usr/local/lib{bin,include} that contains in /home/install/lib{bin,include}. For example having
/home/install/lib/test1
/usr/local/lib/test1
scritp will remove /usr/local/lib/test1. I tried to do it from each separate directory
/home/install/lib:ls -f -exec rm /usr/local/lib/{} \;
but nothing. Can you help me to manage with this simple script?
Create script rmcomm
#!/bin/bash
a="/home/install/$1"
b="/usr/local/$1"
comm -12 <(ls "$a") <(ls "$b") | while read file; do
rm "$b/$file"
done
Then call this script for every pair:
for dir in lib bin include; do rmcomm "$dir"; done
Here's something simple. Remove the echo from the line containing rm to run it after you've ensured it's doing what you want:
#!/bin/bash
dirs[0]=lib
dirs[1]=bin
dirs[2]=include
pushd /home/install
for dir in "${dirs[#]}"
do
for file in $(find $dir -type f)
do
# Remove 'echo' below once you're satisfied the correct files
# are being removed
echo rm /usr/local/$file
done
done
popd
Is there any way of batch renaming files in sub directories?
For example:
Rename *.html to *.htm in a folder which has directories and sub directories.
Windows command prompt: (If inside a batch file, change %x to %%x)
for /r %x in (*.html) do ren "%x" *.htm
This also works for renaming the middle of the files
for /r %x in (website*.html) do ren "%x" site*.htm
find . -regex ".*html$" | while read line;
do
A=`basename ${line} | sed 's/html$/htm/g'`;
B=`dirname ${line}`;
mv ${line} "${B}/${A}";
done
In python
import os
target_dir = "."
for path, dirs, files in os.walk(target_dir):
for file in files:
filename, ext = os.path.splitext(file)
new_file = filename + ".htm"
if ext == '.html':
old_filepath = os.path.join(path, file)
new_filepath = os.path.join(path, new_file)
os.rename(old_filepath, new_filepath)
If you have forfiles (it comes with Windows XP and 2003 and newer stuff I think) you can run:
forfiles /S /M *.HTM /C "cmd /c ren #file *.HTML"
In Bash, you could do the following:
for x in $(find . -name \*.html); do
mv $x $(echo "$x" | sed 's/\.html$/.htm/')
done
In bash use command rename :)
rename 's/\.htm$/.html/' *.htm
# or
find . -name '*.txt' -print0 | xargs -0 rename 's/.txt$/.xml/'
#Obs1: Above I use regex \. --> literal '.' and $ --> end of line
#Obs2: Use find -maxdepht 'value' for determine how recursive is
#Obs3: Use -print0 to avoid 'names spaces asdfa' crash!
I'm sure there's a more elegant way, but here's the first thing that popped in my head:
for f in $(find . -type f -name '*.html'); do
mv $f $(echo "$f" | sed 's/html$/htm/')
done
On Linux, you may use the 'rename' command to rename files in batch.
AWK on Linux. For the first directory this is your answer... Extrapolate by recursively calling awk on dir_path perhaps by writing another awk which writes this exact awk below... and so on.
ls dir_path/. | awk -F"." '{print "mv file_name/"$0" dir_path/"$1".new_extension"}' |csh
On Unix, you can use rnm:
rnm -rs '/\.html$/.htm/' -fo -dp -1 *
Or
rnm -ns '/n/.htm' -ss '\.html$' -fo -dp -1 *
Explanation:
-ns : name string (new name). /n/ is a name string rule that expands to the filename without the extension.
-ss : search string (regex). Searches for files with match.
-rs : replace string of the form /search_regex/replace_part/modifier
-fo : file only mode
-dp : depth of directory (-1 means unlimited).
there is pretty powerfull forfiles command:
forfiles /? gives u hint of what is possible with the command.
in this case it can be used like:
forfiles /S /M *.html /C "cmd /c rename #file #fname.htm"