When invoking xargs with only -n1, xargs executes a separate echo command for every item:
$ echo 1 2 | xargs -n1
1
2
But when using -n1 with the -I option, which passes the string to be replaced to xargs, it passes all the arguments to a single echo command, effectively ignoring-n1:
$ echo 1 2 | xargs -n1 -I% echo %
1 2
My goal is to execute an arbitrary command with different arguments:
$ echo 1 2 | xargs -n1 -I% mycommand %
# What I want to achieve
mycommand 1
mycommand 2
but I'm quite baffled by the behavior I'm seeing, so:
Why xargs seemingly ignore -n1?
What is the correct way to do what I am trying to? Note that I don't want to deal with any files while doing so.
From xargs(1):
-I replace-str
Replace occurrences of replace-str in the initial-arguments with
names read from standard input. Also, unquoted blanks do not
terminate input items; instead the separator is the newline
character. Implies -x and -L 1.
$ echo $'1\n2' | xargs -n1 -I% echo %
1
2
$ echo $'1\n2' | xargs -n1 -I% echo '*' %
* 1
* 2
Related
I'm trying to pipe a list of values through xargs. Here's a simple example:
echo "Hello Hola Bonjour" | xargs -I _ echo _ Landon
I would expect this to output the following:
Hello Landon
Hola Landon
Bonjour Landon
Instead, the command outputs this:
Hello Hola Bonjour Landon
What am I missing?
Under -I, man xargs says
unquoted blanks do not terminate input items; instead the
separator is the newline character
You can specify a different delimiter (at least in GNU xargs):
printf 'Hello Hola Bonjour' | xargs -d' ' -I _ echo _ Landon
More portably, use \0 as the delimiter and -0 to use it:
printf '%s\0' Hello Hola Bonjour | xargs -0 -I _ echo _ Landon
The delimiter needs to be changed, and you also have to specify the -L option. So either change the delimiter via -d like the other answer suggested, or pipe to sed and replace space with linefeed
echo "Hello Hola Bonjour" | sed -e 's/ /\n/g' | xargs -L 1 -I _ echo _ Landon
Results in
Hello Landon
Hola Landon
Bonjour Landon
Sometimes changing the delimiters is not enough. xargs will sometimes take all the input arguments and pass it all at once. There is no splitting of the arguments.
e.g.
seq 1 7 | xargs echo
results in
1
2
3
4
5
6
7
being passed to xargs, so the output would be
1 2 3 4 5 6 7
If you add a -L 1 shown in the xargs man page
-L max-lines
Use at most max-lines nonblank input lines per command line. Trailing blanks cause an input line to be
logically continued on the next input line. Implies -x.
seq 1 7 | xargs -L 1 echo
then you will see
1
2
3
4
5
6
7
You can also covert it manually to a for loop which lets you setup multi line statements more easily.
# be absolutely sure the values the for loop iterates over is well sanitized to avoid glob expansion of the *.
for i in Hello Hola Bonjour
do
if [ "$i" = "Hello" ]
then
echo "$i Landon, language detected as English!"
else
echo "$i Landon, language detected as non English."
fi
done
The -I flag changes the delimiter to newline.
Unquoted blanks do not terminate input items; instead the separator is the newline character.
You can read about it here.
You have to manually specify the delimiter to be a space. Echo also inserts a newline by default, which messes up xargs. Use the -n flag to remove the newline.
Here is the fixed command:
echo -n "Hello Hola Bonjour" | xargs -d' ' -I _ echo _ Landon
I have a black list to save tag id list, e.g. 1-3,7-9, actually it represents 1,2,3,7,8,9. And could expand it by below shell
for i in {1..3,7..9}; do for j in {$i}; do echo -n "$j,"; done; done
1,2,3,7,8,9
but first I should convert - to ..
echo -n "1-3,7-9" | sed 's/-/../g'
1..3,7..9
then put it into for expression as a parameter
echo -n "1-3,7-9" | sed 's/-/../g' | xargs -I # for i in {#}; do for j in {$i}; do echo -n "$j,"; done; done
zsh: parse error near `do'
echo -n "1-3,7-9" | sed 's/-/../g' | xargs -I # echo #
1..3,7..9
but for expression cannot parse it correctly, why is so?
Because you didn't do anything to stop the outermost shell from picking up the special keywords and characters ( do, for, $, etc ) that you mean to be run by xargs.
xargs isn't a shell built-in; it gets the command line you want it to run for each element on stdin, from its arguments. just like any other program, if you want ; or any other sequence special to be bash in an argument, you need to somehow escape it.
It seems like what you really want here, in my mind, is to invoke in a subshell a command ( your nested for loops ) for each input element.
I've come up with this; it seems to to the job:
echo -n "1-3,7-9" \
| sed 's/-/../g' \
| xargs -I # \
bash -c "for i in {#}; do for j in {\$i}; do echo -n \"\$j,\"; done; done;"
which gives:
{1..3},{7..9},
Could use below shell to achieve this
# Mac newline need special treatment
echo "1-3,7-9" | sed -e 's/-/../g' -e $'s/,/\\\n/g' | xargs -I# echo 'for i in {#}; do echo -n "$i,"; done' | bash
1,2,3,7,8,9,%
#Linux
echo "1-3,7-9" | sed -e 's/-/../g' -e 's/,/\n/g' | xargs -I# echo 'for i in {#}; do echo -n "$i,"; done' | bash
1,2,3,7,8,9,
but use this way is a little complicated maybe awk is more intuitive
# awk
echo "1-3,7-9,11,13-17" | awk '{n=split($0,a,","); for(i=1;i<=n;i++){m=split(a[i],a2,"-");for(j=a2[1];j<=a2[m];j++){print j}}}' | tr '\n' ','
1,2,3,7,8,9,11,13,14,15,16,17,%
echo -n "1-3,7-9" | perl -ne 's/-/../g;$,=",";print eval $_'
I'm trying to get the count of a matching pattern from a variable to check the count of it, but it's only returning 1 as the results, here is what I'm trying to do:
x="HELLO|THIS|IS|TEST"
echo $x | grep -c "|"
Expected result: 3
Actual Result: 1
Do you know why is returning 1 instead of 3?
Thanks.
grep -c counts lines not matches within a line.
You can use awk to get a count:
x="HELLO|THIS|IS|TEST"
echo "$x" | awk -F '|' '{print NF-1}'
3
Alternatively you can use tr and wc:
echo "$x" | tr -dc '|' | wc -c
3
$ echo "$x" | grep -o '|' | grep -c .
3
grep -c does not count the number of matches. It counts the number of lines that match. By using grep -o, we put the matches on separate lines.
This approach works just as well with multiple lines:
$ cat file
hello|this|is
a|test
$ grep -o '|' file | grep -c .
3
The grep manual says:
grep, egrep, fgrep - print lines matching a pattern
and for the -c flag:
instead print a count of matching lines for each input file
and there is just one line that match
You don't need grep for this.
pipe_only=${x//[^|]} # remove everything except | from the value of x
echo "${#pipe_only}" # output the length of pipe_only
Try this :
$ x="HELLO|THIS|IS|TEST"; echo -n "$x" | sed 's/[^|]//g' | wc -c
3
With only one pipe with perl:
echo "$x" |
perl -lne 'print scalar(() = /\|/g)'
I have a command line tool which receives two arguments:
TOOL arg1 -o arg2
I would like to invoke it with the same argument provided it for arg1 and arg2, and to make that easy for me, i thought i would do:
each <arg1_value> | TOOL $1 -o $1
but that doesn't work, $1 is not replaced, but is added once to the end of the commandline.
An explicit example, performing:
cp fileA fileA
returns an error fileA and fileA are identical (not copied)
While performing:
echo fileA | cp $1 $1
returns the following error:
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
any ideas?
If you want to use xargs, the [-I] option may help:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separa‐
tor is the newline character. Implies -x and -L 1.
Here is a simple example:
mkdir test && cd test && touch tmp
ls | xargs -I '{}' cp '{}' '{}'
Returns an Error cp: tmp and tmp are the same file
The xargs utility will duplicate its input stream to replace all placeholders in its argument if you use the -I flag:
$ echo hello | xargs -I XXX echo XXX XXX XXX
hello hello hello
The placeholder XXX (may be any string) is replaced with the entire line of input from the input stream to xargs, so if we give it two lines:
$ printf "hello\nworld\n" | xargs -I XXX echo XXX XXX XXX
hello hello hello
world world world
You may use this with your tool:
$ generate_args | xargs -I XXX TOOL XXX -o XXX
Where generate_args is a script, command or shell function that generates arguments for your tool.
The reason
each <arg1_value> | TOOL $1 -o $1
did not work, apart from each not being a command that I recognise, is that $1 expands to the first positional parameter of the current shell or function.
The following would have worked:
set - "arg1_value"
TOOL "$1" -o "$1"
because that sets the value of $1 before calling you tool.
You can re-run a shell to perform variable expansion, with sh -c. The -c takes an argument which is command to run in a shell, performing expansion. Next arguments of sh will be interpreted as $0, $1, and so on, to use in the -c. For example:
sh -c 'echo $1, i repeat: $1' foo bar baz will print execute echo $1, i repeat: $1 with $1 set to bar ($0 is set to foo and $2 to baz), finally printing bar, i repeat: bar
The $1,$2...$N are only visible to bash script to interpret arguments to those scripts and won't work the way you want them to. Piping redirects stdout to stdin and is not what you are looking for either.
If you just want a one-liner, use something like
ARG1=hello && tool $ARG1 $ARG1
Using GNU parallel to use STDIN four times, to print a multiplication table:
seq 5 | parallel 'echo {} \* {} = $(( {} * {} ))'
Output:
1 * 1 = 1
2 * 2 = 4
3 * 3 = 9
4 * 4 = 16
5 * 5 = 25
One could encapsulate the tool using awk:
$ echo arg1 arg2 | awk '{ system("echo TOOL " $1 " -o " $2) }'
TOOL arg1 -o arg2
Remove the echo within the system() call and TOOL should be executed in accordance with requirements:
echo arg1 arg2 | awk '{ system("TOOL " $1 " -o " $2) }'
Double up the data from a pipe, and feed it to a command two at a time, using sed and xargs:
seq 5 | sed p | xargs -L 2 echo
Output:
1 1
2 2
3 3
4 4
5 5
I try to use xargs with both -n 1 and -I {} at the same time, but with no success...
Let's say you want to move 1.txt 2.txt 3.txt to other names:
for i in 1 2 3; do
mv ${i}.txt ${i}_ren.txt
done
now I try to pass the 1 2 3 values to xargs via echo, but it does not appear to work.
I use:
echo 1 2 3 | xargs -n 1 -I {} mv {}.txt {}_ren.txt
What am I doing wrong?
What am I doing wrong?
You are doing wrong because -I is for each input line.
You can either replace space with new line by:
echo 1 2 3 | tr ' ' '\n' | xargs -I {} -P0 mv {}.txt {}_ren.txt
or use seq instead of echo by:
seq 3 | xargs -I {} -P0 mv {}.txt {}_ren.txt
or without -I by
echo 1 2 3 |
tr ' ' '\n' |
sed 's/.*/&.txt &_ren.txt/' |
xargs -n2 -P0 mv
It seems impossible to combine xargs options -n and -I ,
see e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=31858>
However it is possible to use only one pipe instead of
echo 2 3 4 | xargs -n 1 | xargs -I {} mv {}.txt {}_ren.txt
to achieve the same effect.
For this the newline at the end of the echo output has to be disabled (system-dependent);
using space as delimeter for xargs:
echo -n 2 3 4 | xargs -d " " -I {} mv {}.txt {}_ren.txt
Unfortunately this solution is probably not less intricated.