touch source
$ echo dest.{000000..999999} | tr ' ' '\n' | while read dest ; do echo cp -v source $dest ; done
cp -v source dest.000000
cp -v source dest.000001
cp -v source dest.000002
cp -v source dest.000003
cp -v source dest.000004
cp -v source dest.000005
cp -v source dest.000006
cp -v source dest.000007
cp -v source dest.000008
cp -v source dest.000009
...
Well, this is gonna take forever, mainly because each copy invokes a new cp process.
Let's try with xargs:
$ echo dest.{000000..999999} | xargs -n 1000 cp source
cp: target 'dest.000999' is not a directory
Yeah, right, when giving multiple arguments, cp assumes that n-1 arguments are source files, and the nth argument is a destination directory.
I need a command that works differently:
mycp source dest1 dest2 dest3 ...
How could I achieve this, without invoking a new process for each copy?
(based on the suggestion by Cyrus)
This works:
function multi-cp () {
local source="$1"
shift
tee "${#}" < "$source" > /dev/null
}
echo dest.{000000..999999} | xargs -n 1000 | while read -r destinations ; do
multi-cp source $destinations
done
We use while because xargs can not call functions (there are ways around this, but they have other problems). We still use xargs to split the arguments in manageable chunks.
This assumes that the arguments have no spaces (which is the case, since we are in control).
Related
I have gotten into the habit of using the following command to quickly open relevant files in my editor for code reviews.
git diff --name-only master... | xargs realpath | someEditor
But recently been encountering a problem where the first or second file might have been deleted and the editor will produce an error message and not open the rest of the files forcing me to open them one by one.
Is there a similar command I can type that will skip the files that do not exist?
Narrowly Addressing The Problem: With Readlink
Since you're on Ubuntu (which has GNU readlink), you can use readlink -e, which has the exact behavior you're hoping that realpath will provide:
git diff --name-only master... \
| xargs -d $'\n' readlink -e -- \
| xargs -d $'\n' someEditor
Since you're on Ubuntu, which provides GNU xargs, this code will correctly handling filenames with spaces, literal quotes, or literal backslashes on account of using -d $'\n' on both xargs invocations.
Narrowly Addressing The Problem: With Realpath
If we want to stick with your existing tools (which is to say, realpath instead of readlink) and just add an existence test, that could look like:
git diff --name-only master... \
| xargs -d $'\n' sh -c 'for f; do [ -e "$f" ] && realpath "$f"; done' _ \
| xargs -d $'\n' someEditor
Implementation Advice
By the way -- consider encapsulating your desired implementation in a shell function in your ~/.bashrc. That might be something like...
forEachChangedFile {
local -a files
readarray -t files < <(
git diff --name-only "${branchExpr:-master...}" \
| xargs -d $'\n' readlink -e --
)
if (( $# )); then
"$#" "${files[#]}"
else
"${EDITOR:-someEditor}" "${files[#]}"
fi
}
...later used as:
forEachChangedClient vim
or
branchExpr=someBranch..someOtherBranch forEachChangedFile emacs-client
The advantage of not using xargs for the final invocation of the editor is that it leaves the editor's stdin clear, so you can use it for editors that communicate with the user over that channel.
I'm trying to create a folder using txt file and copy the file. I have two file types:
try.txt
Changes/EMAIL/header-20-percent-off.gif
Changes/EMAIL/header-50-percent-off.gif
demo of folder named zip2
zip2/EMAIL/header-20-percent-off.gif
zip2/EMAIL/header-50-percent-off.gif
Code:
mkdir -p dirname `xargs -a try.txt`
cp -R {Dont know how this will work :( }
Actual output:
Changes/EMAIL/header-20-percent-off.gif/
/header-50-percent-off.gif/
Expected output:
Changes/EMAIL/header-20-percent-off.gif
/header-50-percent-off.gif
As you can see for some reason it thinks header-20-percent-off.gif and header-50-percent-off.gif are directories.
Once Changes/Email/ is created I would like to copy the two gif files header-20-percent-off.gif and header-50-percent-off.gif there.
First create folders:
<try.txt xargs -d$'\n' dirname | xargs -d$'\n' mkdir -p
Then copy files. First prepare the stream with proper source and destination directories with sed and then pass to xargs:
sed 's#^Changes/\(.*\)#zip2/\1\n&#' try.txt |
xargs -d$'\n' -n2 cp
But if you are not proficient in bash, just read the stream line by line:
while IFS= read -r dest; do
dir=$(dirname "$dest")
mkdir -p "$dir"
src=$(sed 's#^Changes#zip2#' <<<"$dest")
cp "$src" "$dest"
done < try.txt
Don't use backticks `, they are highly discouraged. Use $(...) for command substitution instead.
Just doing xargs -a try.txt without a command makes little sense, just $(cat try.txt) or better $(<try.txt).
Use -t option with xargs to see what is it doing.
Explicitly specify the delimeter with xargs -d$'\n' - otherwise xargs will parse " ' and \ specially.
I believe with some luck and work you could just use rsync with something along rsync --include-from=try.txt changes/ zip2/.
I have a small bug and don't know how to solve it. I want to copy files from a big folder with many files, where the files contain a specific string. For this I use grep, ack or (in this example) ag. When I'm inside the folder it matches without problem, but when I want to do it with a loop over the files in the following script it doesn't loop over the matches. Here my script:
ag -l "${SEARCH_QUERY}" "${INPUT_DIR}" | while read -d $'\0' file; do
echo "$file"
cp "${file}" "${OUTPUT_DIR}/${file}"
done
SEARCH_QUERY holds the String I want to find inside the files, INPUT_DIR is the folder where the files are located, OUTPUT_DIR is the folder where the found files should be copied to. Is there something wrong with the while do?
EDIT:
Thanks for the suggestions! I took this one now, because it also looks for files in subfolders and saves a list with all the files.
ag -l "${SEARCH_QUERY}" "${INPUT_DIR}" > "output_list.txt"
while read file
do
echo "${file##*/}"
cp "${file}" "${OUTPUT_DIR}/${file##*/}"
done < "output_list.txt"
Better implement it like below with a find command:
find "${INPUT_DIR}" -name "*.*" | xargs grep -l "${SEARCH_QUERY}" > /tmp/file_list.txt
while read file
do
echo "$file"
cp "${file}" "${OUTPUT_DIR}/${file}"
done < /tmp/file_list.txt
rm /tmp/file_list.txt
or another option:
grep -l "${SEARCH_QUERY}" "${INPUT_DIR}/*.*" > /tmp/file_list.txt
while read file
do
echo "$file"
cp "${file}" "${OUTPUT_DIR}/${file}"
done < /tmp/file_list.txt
rm /tmp/file_list.txt
if you do not mind doing it in just one line, then
grep -lr 'ONE\|TWO\|THREE' | xargs -I xxx -P 0 cp xxx dist/
guide:
-l just print file name and nothing else
-r search recursively the CWD and all sub-directories
match these works alternatively: 'ONE' or 'TWO' or 'THREE'
| pipe the output of grep to xargs
-I xxx name of the files is saved in xxx it is just an alias
-P 0 run all the command (= cp) in parallel (= as fast as possible)
cp each file xxx to the dist directory
If i understand the behavior of ag correctly, then you have to
adjust the read delimiter to '\n' or
use ag -0 -l to force delimiting by '\0'
to solve the problem in your loop.
Alternatively, you can use the following script, that is based on find instead of ag.
while read file; do
echo "$file"
cp "$file" "$OUTPUT_DIR/$file"
done < <(find "$INPUT_DIR" -name "*$SEARCH_QUERY*" -print)
I need to rename files which are in this structure:
Dir1
--file1
--file2
-- ...
Dir2
--file1
--file2
-- ...
...
Dir62
--file1101
--file1102
-- ...
The new names would be 1_01,1_02 in 1 dir and 2_01,2_02 in 2nd dir and so on...
is there a way to do it in a single go...
Currently, I am using:
ls | cat -n | while read n f; do mv "$f" "10_$n.png"; done
Which work in 1 dir at a time...
Any better way, please?
If you run this command, it will use GNU Parallel to start a new bashshell in each of the directories in parallel, and run ls in each one in parallel independently:
parallel --dry-run -k 'cd {} && ls' ::: */
Sample Output
cd Dir01/ && ls
cd Dir02/ && ls
cd Dir78/ && ls
If you remove the --dry-run it will do it for real.
So, instead of running ls, let's now look at using the rename command in each of the directories. The following will rename all the files in a directory with sequentially increasing numbers ($N):
rename --dry-run '$_=$N' *
Sample Output
'file87' would be renamed to '1'
'file88' would be renamed to '2'
'file89' would be renamed to '3'
'fred' would be renamed to '4'
All the foregoing suggests the command you want would be:
parallel --dry-run -k 'cd {} && rename --dry-run "s/.*/{#}_\$N/" *' ::: */
You can run it as it is and it will just show you what it is going to do, without actually doing anything.
If you like the look of that, remove the first --dry-run and run it again and it will actually go into each subdirectory and do a dry-run of the rename, again without actually changing anything.
If you still like the look of the command, make a small copy of your files somewhere in a temporary directory and try removing both the --dry-run parameters ands if it lives up to your needs.
ls -1 -d ./*/ | cat -n | xargs -I % bash -c 'echo "%" | while read dirnum dirname; do { ls "${dirname}" | cat -n | while read filenum filename; do { mv -v "${dirname}${filename}" "${dirnum}_${filenum}.png"; }; done }; done'
We create a directory structure with mkdir and touch:
mkdir Dir{1,2,3,4,5}
touch Dir{1,2,3,4,5}/file{1,2,3}
Which gives the result:
1_1.png
1_2.png
1_3.png
2_1.png
2_2.png
2_3.png
3_1.png
3_2.png
3_3.png
4_1.png
4_2.png
4_3.png
5_1.png
5_2.png
5_3.png
I am trying to move all my video files that are in my pictures directory to my movies Directory. This is on a Mac by the way.
I thought I could simple Recurse through all my picture directories with an "ls -R"
Then I pipe that to grep -i ".avi" This give me all the movie files.
Now I pipe these values to "mv -n $1 ~/Movies" this I am hoping would move the files to the Movies folder.
I have a few Problems.
1. The "ls -R" does not list the path when listing the files. So I think I may fail to move the file.
2. I can not seem to get the file name to assign to the $1 in the mv command.
All together my command looks like this: Note I am running this from ~/Pictures
ls -R | grep -i ".avi" | mv -n $1 ~/Movies
So right now I am not sure which part is failing but I do get this error:
usage: mv [-f | -i | -n] [-v] source target
mv [-f | -i | -n] [-v] source ... directory
If I remove the 'mv' command I get a listing of avi files with out the path. Example Below:
4883.AVI
4884.AVI
4885.AVI
4886.AVI
4887.AVI
...
Any one have any ideas on how I can get the path in the 'ls' or how to pass a value in between the '|' commands.
Thanks.
It's better if you use the find command:
$ find -name "*.avi" -exec mv {} ~/Movies \;
you should create simple copy.sh like this
#!/bin/bash
cp $1 ~/Movies/
An run command ./copy.sh "$(ls | grep avi)"
The bash for loop can help you find all the avi files easily
shopt -s nullglob
for file in *.avi
do
mv "$file" "$file" ~/Movies/"$file"
done
you can achieve this in many ways, one of it in my openion:
ls -R | grep -i ".avi" | while read movie
do
echo " moving $movie"
mv $movie ~/Movies/
done
Use backticks
mv `ls *.avi` ~/Movies