How to make xargs append trailing arguments with -n? - xargs

xargs is good at inserting initial arguments:
seq 0 10 | xargs -n 3 echo foo
produces this output:
foo 0 1 2
foo 3 4 5
foo 6 7 8
foo 9 10
What about when I also want trailing arguments?
That is, what command:
seq 0 10 | xargs -n 3 <WHAT GOES HERE?>
will produce the following desired output:
foo 0 1 2 bar
foo 3 4 5 bar
foo 6 7 8 bar
foo 9 10 bar
I tried the following:
seq 0 10 | xargs -n 3 -I {} echo foo {} bar
which is almost right, except that it apparently forces 1 item per command line, which isn't what I want:
foo 0 bar
foo 1 bar
foo 2 bar
foo 3 bar
foo 4 bar
foo 5 bar
foo 6 bar
foo 7 bar
foo 8 bar
foo 9 bar
foo 10 bar

Use the -i parameter of xargs such as:
ls -1 | xargs -t -i echo "Found {} file"
Note that the -t is only there to show the command that will be issued. When running live, leave off the -t parameter.
The {} is replaced by the actual argument. Note, that this does mean each command is run only once per file. Not as a group of files. If you need to use a different replace string, specify it after the -i
ls -1 | xargs -t -i[] xargs echo "Found []{} file"
Would keep {} after the filename which is now replaced on the occurance of []
To achieve desired output
Create a script file called runme.sh
#!/bin/sh
echo "foo $# bar"
Make sure to chmod +X and then use
seq 0 10 | xargs -n 3 ./runme.sh
This would produce
foo tests tree.php user_admin.php bar
foo user_domains.php user_group_admin.php utilities.php bar
foo vdef.php bar

After figuring this out with a head-start from #netniV's answer,
I now see that the man page actually contains an example
showing how to do it:
xargs sh -c 'emacs "$#" < /dev/tty' emacs
Launches the minimum number of copies of Emacs needed, one after the
other, to edit the files listed on xargs' standard input. This example
achieves the same effect as BSD's -o option, but in a more flexible and portable way.
And the wikipedia page shows a similar technique, with an explanation of the dummy arg at the end:
Another way to achieve a similar effect is to use a shell as the launched command, and deal with the complexity in that shell, for
example:
$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 bash -c 'for filename; do cp -a "$filename" ~/backups; done' bash
The word bash at the end of the line is interpreted by bash -c as special parameter $0. If the word bash weren't present, the name of
the first matched file would be assigned to $0 and the file wouldn't
be copied to ~/backups. Any word can be used instead of bash, but
since $0 usually expands to the name of the shell or shell script
being executed, bash is a good choice.
So, here's how to do it:
seq 0 10 | xargs -n 3 sh -c 'echo foo "$#" bar' some_dummy_string
The output is as desired:
foo 0 1 2 bar
foo 3 4 5 bar
foo 6 7 8 bar
foo 9 10 bar

Related

Correct way of quoting command substitution

I have simple bash script which only outputs the filenames that are given to the script as positional arguments:
#!/usr/bin/env bash
for file; do
echo "$file"
done
Say I have files with spaces (say "f 1" and "f 2"). I can call the script with a wildcard and get the expected output:
$ ./script f*
> f 1
> f 2
But if I use command substitution it doesn't work:
$ ./script $(echo f*)
> f
> 1
> f
> 2
How can I get the quoting right when my command substition outputs multiple filenames with spaces?
Edit: What I ultimatively want is to pass filenames to a script (that is slightly more elaborate than just echoing their names) in a random order, e.g. something like that:
./script $(ls f* | shuf)
With GNU shuf and Bash 4.3+:
readarray -d '' files < <(shuf --zero-terminated --echo f*)
./script "${files[#]}"
where the --zero-terminated can handle any filenames, and readarray also uses the null byte as the delimiter.
With older Bash where readarray doesn't support the -d option:
while IFS= read -r -d '' f; do
files+=("$f")
done < <(shuf --zero-terminated --echo f*)
./script "${files[#]}"
In extreme cases with many files, this might run into command line length limitations; in that case,
shuf --zero-terminated --echo f*
could be replaced by
printf '%s\0' f* | shuf --zero-terminated
Hat tip to Socowi for pointing out --echo.
It's very difficult to get this completely correct. A simple attempt would be to use %q specifier to printf, but I believe that is a bashism. You still need to use eval, though. eg:
$ cat a.sh
#!/bin/sh
for x; do echo $((i++)): "$x"; done
$ ./a.sh *
0: a.sh
1: name
with
newlines
2: name with spaces
$ eval ./a.sh $(printf "%q " *)
0: a.sh
1: name
with
newlines
2: name with spaces
This feels like an XY Problem. Maybe you should explain the real problem, someone might have a much better solution.
Nonetheless, working with what you posted, I'd say read this page on why you shouldn't try to parse ls as it has relevant points; then I suggest an array.
lst=(f*)
./script "${lst[#]}"
This will still fail if you reparse it as the output of a subshell, though -
./script $( echo "${lst[#]}" ) # still fails same way
./script "$( echo "${lst[#]}" )" # *still* fails same way
Thinking about how we could make it work...
You can use xargs:
$ ls -l
total 4
-rw-r--r-- 1 root root 0 2021-08-13 00:23 ' file 1'
-rw-r--r-- 1 root root 0 2021-08-13 00:23 ' file 2'
-rw-r--r-- 1 root root 0 2021-08-13 00:23 ' file 3'
-rw-r--r-- 1 root root 0 2021-08-13 00:23 ' file 4'
-rwxr-xr-x 1 root root 35 2021-08-13 00:25 script
$ ./script *file*
file 1
file 2
file 3
file 4
$ ls *file* | shuf | xargs -d '\n' ./script
file 4
file 2
file 1
file 3
If your xargs does not support -d:
$ ls *file* | shuf | tr '\n' '\0' | xargs -0 ./script
file 3
file 1
file 4
file 2

how to group all arguments as position argument for `xargs`

I have a script which takes in only one positional parameter which is a list of values, and I'm trying to get the parameter from stdin with xargs.
However by default, xargs passes all the lists to my script as positional parameters, e.g. when doing:
echo 1 2 3 | xargs myScript, it will essentially be myScript 1 2 3, and what I'm looking for is myScript "1 2 3". What is the best way to achieve this?
Change the delimiter.
$ echo 1 2 3 | xargs -d '\n' printf '%s\n'
1 2 3
Not all xargs implementations have -d though.
And not sure if there is an actual use case for this but you can also resort to spawning another shell instance if you have to. Like
$ echo -e '1 2\n3' | xargs sh -c 'printf '\''%s\n'\'' "$*"' sh
1 2 3
If the input can be altered, you can do this. But not sure if this is what you wanted.
echo \"1 2 3\"|xargs ./myScript
Here is the example.
$ cat myScript
#!/bin/bash
echo $1; shift
echo $1; shift
echo $1;
$ echo \"1 2 3\"|xargs ./myScript
1 2 3
$ echo 1 2 3|xargs ./myScript
1
2
3

Get xargs to word-split placeholder {}

(Though word splitting has a specific definition in Bash, in this post it means to split on spaces or tabs.)
Demonstrating the question using this input to xargs,
$ cat input.txt
LineOneWithOneArg
LineTwo WithTwoArgs
LineThree WithThree Args
LineFour With Double Spaces
and this Bash command to echo the arguments passed to it,
$ bash -c 'IFS=,; echo "$*"' arg0 arg1 arg2 arg3
arg1,arg2,arg3
notice how xargs -L1 word-splits each line into multiple arguments.
$ xargs <input.txt -L1 bash -c 'IFS=,; echo "$*"' arg0
LineOneWithOneArg
LineTwo,WithTwoArgs
LineThree,WithThree,Args
LineFour,With,Double,Spaces
However, xargs -I{} expands the whole line into {} as a single argument.
$ xargs <input.txt -I{} bash -c 'IFS=,; echo "$*"' arg0 {}
LineOneWithOneArg
LineTwo WithTwoArgs
LineThree WithThree Args
LineFour With Double Spaces
While this is perfectly reasonable behavior for the vast majority of cases, there are times when word-splitting behavior (the first xargs example) is preferred.
And while xargs -L1 can be seen as a workaround, it can only be used to place arguments at the end of the command line, making it impossible to express
$ xargs -I{} command first-arg {} last-arg
with xargs -L1. (That is of course, unless command is able to accept arguments in a different order, as is the case with options.)
Is there any way to get xargs -I{} to word-split each line when expanding the {} placeholder?
Sort of.
echo -e "1\n2 3" | xargs sh -c 'echo a "$#" b' "$0"
Outputs:
a 1 2 3 b
ref: https://stackoverflow.com/a/35612138/1563960
Also:
echo -e "1\n2 3" | xargs -L1 sh -c 'echo a "$#" b' "$0"
Outputs:
a 1 b
a 2 3 b

Pipe stdout to command which itself needs to read from own stdin

I would like to get the stdout from a process into another process not using stdin, as that one is used for another purpose.
In short I want to accomplish something like that:
echo "a" >&4
cat | grep -f /dev/fd/4
I got it running using an file as source for file descriptor 4, but that is not what I want:
# Variant 1
cat file | grep -f /dev/fd/4 4<pattern
# Variant 2
exec 4<pattern
cat | grep -f /dev/fd/4
exec 4<&-
My best try is that, but I got the following error message:
# Variant 3
cat | (
echo "a" >&4
grep -f /dev/fd/4
) <&4
Error message:
test.sh: line 5: 4: Bad file descriptor
What is the best way to accomplish that?
You don't need to use multiple streams to do this:
$ printf foo > pattern
$ printf '%s\n' foo bar | grep -f pattern
foo
If instead of a static file you want to use the output of a command as the input to -f you can use a process substitution:
$ printf '%s\n' foo bar | grep -f <(echo foo)
foo
For POSIX shells that lack process substitution, (e.g. dash, ash, yash, etc.).
If the command allows string input, (grep allows it), and the input string containing search targets isn't especially large, (i.e. the string doesn't exceed the length limit for the command line), there's always command substitution:
$ printf '%s\n' foo bar baz | grep $(echo foo)
foo
Or if the input file is multi-line, separating quoted search items with '\n' works the same as grep OR \|:
$ printf '%s\n' foo bar baz | grep "$(printf "%s\n" foo bar)"
foo
bar

Use argument twice from standard output pipelining

I have a command line tool which receives two arguments:
TOOL arg1 -o arg2
I would like to invoke it with the same argument provided it for arg1 and arg2, and to make that easy for me, i thought i would do:
each <arg1_value> | TOOL $1 -o $1
but that doesn't work, $1 is not replaced, but is added once to the end of the commandline.
An explicit example, performing:
cp fileA fileA
returns an error fileA and fileA are identical (not copied)
While performing:
echo fileA | cp $1 $1
returns the following error:
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
any ideas?
If you want to use xargs, the [-I] option may help:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separa‐
tor is the newline character. Implies -x and -L 1.
Here is a simple example:
mkdir test && cd test && touch tmp
ls | xargs -I '{}' cp '{}' '{}'
Returns an Error cp: tmp and tmp are the same file
The xargs utility will duplicate its input stream to replace all placeholders in its argument if you use the -I flag:
$ echo hello | xargs -I XXX echo XXX XXX XXX
hello hello hello
The placeholder XXX (may be any string) is replaced with the entire line of input from the input stream to xargs, so if we give it two lines:
$ printf "hello\nworld\n" | xargs -I XXX echo XXX XXX XXX
hello hello hello
world world world
You may use this with your tool:
$ generate_args | xargs -I XXX TOOL XXX -o XXX
Where generate_args is a script, command or shell function that generates arguments for your tool.
The reason
each <arg1_value> | TOOL $1 -o $1
did not work, apart from each not being a command that I recognise, is that $1 expands to the first positional parameter of the current shell or function.
The following would have worked:
set - "arg1_value"
TOOL "$1" -o "$1"
because that sets the value of $1 before calling you tool.
You can re-run a shell to perform variable expansion, with sh -c. The -c takes an argument which is command to run in a shell, performing expansion. Next arguments of sh will be interpreted as $0, $1, and so on, to use in the -c. For example:
sh -c 'echo $1, i repeat: $1' foo bar baz will print execute echo $1, i repeat: $1 with $1 set to bar ($0 is set to foo and $2 to baz), finally printing bar, i repeat: bar
The $1,$2...$N are only visible to bash script to interpret arguments to those scripts and won't work the way you want them to. Piping redirects stdout to stdin and is not what you are looking for either.
If you just want a one-liner, use something like
ARG1=hello && tool $ARG1 $ARG1
Using GNU parallel to use STDIN four times, to print a multiplication table:
seq 5 | parallel 'echo {} \* {} = $(( {} * {} ))'
Output:
1 * 1 = 1
2 * 2 = 4
3 * 3 = 9
4 * 4 = 16
5 * 5 = 25
One could encapsulate the tool using awk:
$ echo arg1 arg2 | awk '{ system("echo TOOL " $1 " -o " $2) }'
TOOL arg1 -o arg2
Remove the echo within the system() call and TOOL should be executed in accordance with requirements:
echo arg1 arg2 | awk '{ system("TOOL " $1 " -o " $2) }'
Double up the data from a pipe, and feed it to a command two at a time, using sed and xargs:
seq 5 | sed p | xargs -L 2 echo
Output:
1 1
2 2
3 3
4 4
5 5

Resources