I'm trying to do the following with xargs
pacman -Q | grep xf86-video | awk '{print $1}' | xargs pacman -R to remove all xf86-video-* driver on my machine. To make the question more clear, here is the output of pacman -Q | grep xf86-video | awk '{print $1}':
xf86-video-ark
xf86-video-ati
xf86-video-dummy
xf86-video-fbdev
xf86-video-glint
xf86-video-i128
xf86-video-intel
xf86-video-mach64
xf86-video-neomagic
xf86-video-nouveau
....
when I redirect the result to xargs, the output looks like this:
The point is, the command which xargs is about to execute need user to do some additional input(as you can see it needs a Yes/No), but xargs automatically add a unknown symbol #, and exit, which causes my purpose UNACHIEVED.
WHY xargs would do this or, what can I do to use xargs for command with prompt?
You can use
xargs -a <(pacman -Q | awk '/xf86-video/{print $1}') pacman -R
Explanation:
Without further arguments xargs does not work with interactive (command line) applications.
The reason for that is, that by defaultxargs gets its input from stdin but interactive applications also expect input from stdin. To prevent the applications from grabbing input that is intended for xargs, xargs redirects stdin from /dev/null for the applications it runs. This leads to the application just receiving an EOF. (Running just pacman -R SOMEPACKAGE and pressing Ctrl+D has the same effect).
To get xargs to work with interactive commands you have to use the --arg-file=FILE argument (short -a FILE). This tells xargs to get the arguments from FILE. This also leaves stdin unchanged.
So you could either put your package list into a temporary file
pacman -Q | awk '/xf86-video/{print $1}' > /tmp/packagelist
xargs -a /tmp/packagelist pacman -R
rm /tmp/packagelist
or you can use zsh's process substitution mechanism <(list). When executing a line with <(list), <(list) is replaced by a filename from where the output of list can be read.
xargs -a <(pacman -Q | awk '/xf86-video/{print $1}') pacman -R
The single # you get is not from xargs but from zsh itself. If the shell options PROMPT_CR and PROMPT_SP are set (which both are by default) zsh tries to preserve partial lines, that is lines that did not end with a newline. To signify that such a line has been preserved zsh prints an inverse+bold character at the end of that line, by default % for normal users and # for root.
You can eliminate the use of xargs with a single hyphen
Additionally, if stdin is not from a terminal and a single hyphen (-) is passed as an argument, targets will be read from stdin.
— https://archlinux.org/pacman/pacman.8.html
pacman -Q | awk '/xf86-video/{print $1}' | pacman -R -
You can use xargs, though ...
If you wish to use xargs, use it's -o, --open-tty option:
--open-tty
-o
Reopen stdin as /dev/tty in the child process before executing the command, thus allowing that command to be associated to the terminal while xargs reads from a different stream, e.g. from a pipe. This is useful if you want xargs to run an interactive application.
grep -lz PATTERN * | xargs -0o vi ①
— https://www.gnu.org/software/findutils/manual/html_node/find_html/xargs-options.html
① That should be an uppercase Z grep option (bug filed).
For your particular case:
pacman -Q | awk '/xf86-video/{print $1}' | xargs -o pacman -R
You need to run pacman the second time with the --noconfirm option:
pacman -Q | grep xf86-video | awk '{print $1}' | xargs pacman -R --noconfirm
This will disable 'are you sure' messages, and do things without requiring input.
Related
cat a.txt | xargs -I % echo %
In the example above, xargs takes echo % as the command argument. But in some cases, I need multiple commands to process the argument instead of one. For example:
cat a.txt | xargs -I % {command1; command2; ... }
But xargs doesn't accept this form. One solution I know is that I can define a function to wrap the commands, but I want to avoid that because it is complex. Is there a better solution?
cat a.txt | xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _
...or, without a Useless Use Of cat:
<a.txt xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _
To explain some of the finer points:
The use of "$arg" instead of % (and the absence of -I in the xargs command line) is for security reasons: Passing data on sh's command-line argument list instead of substituting it into code prevents content that data might contain (such as $(rm -rf ~), to take a particularly malicious example) from being executed as code.
Similarly, the use of -d $'\n' is a GNU extension which causes xargs to treat each line of the input file as a separate data item. Either this or -0 (which expects NULs instead of newlines) is necessary to prevent xargs from trying to apply shell-like (but not quite shell-compatible) parsing to the stream it reads. (If you don't have GNU xargs, you can use tr '\n' '\0' <a.txt | xargs -0 ... to get line-oriented reading without -d).
The _ is a placeholder for $0, such that other data values added by xargs become $1 and onward, which happens to be the default set of values a for loop iterates over.
You can use
cat file.txt | xargs -i sh -c 'command {} | command2 {} && command3 {}'
{} = variable for each line on the text file
With GNU Parallel you can do:
cat a.txt | parallel 'command1 {}; command2 {}; ...; '
For security reasons it is recommended you use your package manager to
install. But if you cannot do that then you can use this 10 seconds
installation.
The 10 seconds installation will try to do a full installation; if
that fails, a personal installation; if that fails, a minimal
installation.
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
I prefer style which allows dry run mode (without | sh) :
cat a.txt | xargs -I % echo "command1; command2; ... " | sh
Works with pipes too:
cat a.txt | xargs -I % echo "echo % | cat " | sh
This is just another approach without xargs nor cat:
while read stuff; do
command1 "$stuff"
command2 "$stuff"
...
done < a.txt
This seems to be the safest version.
tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'command1 "$#"; command2 "$#";' ''
(-0 can be removed and the tr replaced with a redirect (or the file can be replaced with a null separated file instead). It is mainly in there since I mainly use xargs with find with -print0 output) (This might also be relevant on xargs versions without the -0 extension)
It is safe, since args will pass the parameters to the shell as an array when executing it. The shell (at least bash) would then pass them as an unaltered array to the other processes when all are obtained using ["$#"][1]
If you use ...| xargs -r0 -I{} bash -c 'f="{}"; command "$f";' '', the assignment will fail if the string contains double quotes. This is true for every variant using -i or -I. (Due to it being replaced into a string, you can always inject commands by inserting unexpected characters (like quotes, backticks or dollar signs) into the input data)
If the commands can only take one parameter at a time:
tr '[\n]' '[\0]' < a.txt | xargs -r0 -n1 /bin/bash -c 'command1 "$#"; command2 "$#";' ''
Or with somewhat less processes:
tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'for f in "$#"; do command1 "$f"; command2 "$f"; done;' ''
If you have GNU xargs or another with the -P extension and you want to run 32 processes in parallel, each with not more than 10 parameters for each command:
tr '[\n]' '[\0]' < a.txt | xargs -r0 -n10 -P32 /bin/bash -c 'command1 "$#"; command2 "$#";' ''
This should be robust against any special characters in the input. (If the input is null separated.) The tr version will get some invalid input if some of the lines contain newlines, but that is unavoidable with a newline separated file.
The blank first parameter for bash -c is due to this: (From the bash man page) (Thanks #clacke)
-c If the -c option is present, then commands are read from the first non-option argument com‐
mand_string. If there are arguments after the command_string, the first argument is assigned to $0
and any remaining arguments are assigned to the positional parameters. The assignment to $0 sets
the name of the shell, which is used in warning and error messages.
One thing I do is to add to .bashrc/.profile this function:
function each() {
while read line; do
for f in "$#"; do
$f $line
done
done
}
then you can do things like
... | each command1 command2 "command3 has spaces"
which is less verbose than xargs or -exec. You could also modify the function to insert the value from the read at an arbitrary location in the commands to each, if you needed that behavior also.
Another possible solution that works for me is something like -
cat a.txt | xargs bash -c 'command1 $#; command2 $#' bash
Note the 'bash' at the end - I assume it is passed as argv[0] to bash. Without it in this syntax the first parameter to each command is lost. It may be any word.
Example:
cat a.txt | xargs -n 5 bash -c 'echo -n `date +%Y%m%d-%H%M%S:` ; echo " data: " $#; echo "data again: " $#' bash
My current BKM for this is
... | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
It is unfortunate that this uses perl, which is less likely to be installed than bash; but it handles more input that the accepted answer. (I welcome a ubiquitous version that does not rely on perl.)
#KeithThompson's suggestion of
... | xargs -I % sh -c 'command1; command2; ...'
is great - unless you have the shell comment character # in your input, in which case part of the first command and all of the second command will be truncated.
Hashes # can be quite common, if the input is derived from a filesystem listing, such as ls or find, and your editor creates temporary files with # in their name.
Example of the problem:
$ bash 1366 $> /bin/ls | cat
#Makefile#
#README#
Makefile
README
Oops, here is the problem:
$ bash 1367 $> ls | xargs -n1 -I % sh -i -c 'echo 1 %; echo 2 %'
1
1
1
1 Makefile
2 Makefile
1 README
2 README
Ahh, that's better:
$ bash 1368 $> ls | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
1 #Makefile#
2 #Makefile#
1 #README#
2 #README#
1 Makefile
2 Makefile
1 README
2 README
$ bash 1369 $>
Try this:
git config --global alias.all '!f() { find . -d -name ".git" | sed s/\\/\.git//g | xargs -P10 -I{} git --git-dir={}/.git --work-tree={} $1; }; f'
It runs ten threads in parallel and does what ever git command you want to all repos in the folder structure. No matter if the repo is one or n levels deep.
E.g: git all pull
I have good idea to solve the problem.
Only write a comman mcmd, then you can do
find . -type f | xargs -i mcmd echo {} ## cat {} #pipe sed -n '1,3p'
The mcmd content as follows:
echo $* | sed -e 's/##/\n/g' -e 's/#pipe/|/g' | csh
I am trying to speed up the processing of a database. I migrated towards xargs. But I'm seriously stuck. Piping a list of arguments to xargs does not work if the command invoked by xargs isn't a built in. I can't figure out why. Here is my code:
#!/bin/bash
list='foo
bar'
test(){
echo "$1"
}
echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' 'test' {}
So there is no output at all. And test function never gets executed. But if I replace "test" in the "xargs" command with "echo" or "printf" it works fine.
You can't pass a shell function to xargs directly, but you can invoke a shell.
printf 'foo\0bar\0' |
xargs -r -0 sh -c 'for f; do echo "$f"; done' _
The stuff inside sh -c '...' can be arbitrarily complex; if you really wanted to, you could declare and then use your function. But since it's simple and nonrecursive, I just inlined the functionality.
The dummy underscore parameter is because the first argument after sh -c 'script' is used to populate $0.
Because your question seems to be about optimization, I imagine you don't want to spawn a separate shell for every item passed to xargs -- if you did, nothing would get faster. So I put in the for loop and took out the -I etc arguments to xargs.
xargs takes an executable as an argument (including custom scripts) rather than a function defined in the environment.
Either move your code to a script or use xargs to pass arguments to an external command.
Change from:
echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' 'test' {}
To:
export -f test
echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' sh -c 'test {}'
I've seen a solution from 'jac' on the bbs.archlinux.org web-site that uses a primary and secondary (slave) pair of scripts that are very efficients. Instead of an internal 'function' that normally would accept a single $1 parameter, the primary sends a list of parameters to its secondary where a while-loop handles each member of the list as consecutive $1 values. Here's a sample pair I'm using to apply the 'file' command to a bunch of executables, which in my case all begin with "em" in the filename. Make changes as necessary:
#!/bin/bash
# primary: showfil
ls -l em* | grep '^-rwx' | awk '{$1=$2=$3=$4=$5=$6=$7=$8=""; print $0}' | xargs -I% ~/showfilf "%"
~/showfilf fixmstr spisort trc
exit 0
#!/bin/bash
# secondary: showfilf
myarch=$(uname -s | grep 'arwin')
while [[ -n "$1" ]]; do
if [ -x "$1" ]; then
if [ -n "$myarch" ]; then
file "./$1"
else
myfile=$(file "./$1" | awk '{print $1" "$3" "$10" "$11" "$12}')
myfile=${myfile%(uses}
myfile=${myfile%for}
echo "$myfile"
fi
fi
shift
done
exit 0
This code works on Darwin (Mac) and Linux, and probably other systems. The 'grep' in the primary retains only executable files, not directories or symlinks. The 'awk' eliminates the first eight fields of 'ls' and retains just the filename,which is passed to 'xargs', which builds a list of quoted filenames to send to 'showfilf'. There's a separate invocation of 'showfilf' with three other filenames in the list. 'showfilf' has a while-loop which processes the list. Note that there is system-dependent code here, determined by 'uname -s' and 'grep'. Lastly, make these scripts executable, and place them on your $PATH, such as $HOME. If your $PATH doesn't include your $HOME, I recommend you modify it in your .bashrc or .bash_login something like this: export PATH=$PATH:$HOME
I can extract the contents of all the \include statements from a latex file (and append ".tex" to each one) with
grep -P "\\\\include{" Thesis_master.tex |sed -n -e"s/\\\\include{/$1/" -e" s/}.*$/.tex/p"
(I couldn't get lookbehinds working in grep, hence the pipe through sed. That gives me a list of filenames, one per line. I'd now like to pass those files to aspell. But aspell only accepts one filename as an argument, so I can't just tack |xargs aspell -c on the end.
I've read this related question but that reads from a file line by line through xargs. So how do I get it to read from the output of sed line by line?
I think xargs -L 1 should do what you need:
grep -P "\\\\include{" Thesis_master.tex | \
sed -n -e"s/\\\\include{/$1/" -e" s/}.*$/.tex/p" | \
xargs -L 1 aspell -c
(Backslash line continuation added for readability)
This will cause xargs to call aspell exactly once per line from the sed pipe.
Since your aspell commands appear to exit with a 255 code, this causes xargs to stop. You could trick xargs into not exiting by doing something like:
grep -P "\\\\include{" Thesis_master.tex | \
sed -n -e"s/\\\\include{/$1/" -e" s/}.*$/.tex/p" | \
xargs -L 1 -I % bash -c "aspell -c %; true"
This will run aspell in a subshell, followed by the true command which always return a 0 exit code to xargs.
The grep recipe is:
grep -oP '\\include{\K.+?(?=})' latex.file | xargs aspell ...
I have a bash script, replace.sh with the following contents:
ack-grep -a -l -i --print0 --text "$1" | xargs -0 -n 1 sed -i -e 's/$1/$2/g'
When I try and run it as, eg:
replace.sh something somethingnew
The prompt returns without errors but no changes have been made to any files.
If I manually type:
ack-grep -a -l -i --print0 --text "something" | xargs -0 -n 1 sed -i -e 's/something/somethingelse/g'
The files get changed as expected.
Ths $1 syntax seems to work for other scripts I've written. I'm guessing I'm missing something to do with escaping the args or something?
Thanks!
Ludo.
Variable substitutions aren't done in single quotes, try:
ack-grep -a -l -i --print0 --text "$1" | xargs -0 -n 1 sed -i -e "s/$1/$2/g"
See the bash man page section on QUOTING.
Use "" instead of '' in the sed expression. It will not prevent the variablename-resolving. What you are actually doing now is replacing $1 to $2. You can test in console (without writing a script) like this:
$ a=something
$ b=somethingelse
$ sed 's/$a/$b/g' testfile
$ sed "s/$a/$b/g" testfile
This isn't related to your question, but some help on using ack.
The -a and --text conflict with each other. -a will give you a superset of --text. Use one or the other.
Also, it looks like you might as well use grep -Z instead of ack since you're not using any of ack's functionality that is a superset of grep.
In general, if you're using ack in a pipeline, you should probably be using good ol' grep instead.
I want to count the number of lines output from a command in a bash script. i.e.
COUNT=ls | wc -l
But I also want the script to output the original output from ls. How to get this done? (My actual command is not ls and it has side effects. So I can't run it twice.)
The tee(1) utility may be helpful:
$ ls | tee /dev/tty | wc -l
CHANGES
qpi.doc
qpi.lib
qpi.s
4
info coreutils "tee invocation" includes this following example, which might be more instructive of tee(1)'s power:
wget -O - http://example.com/dvd.iso \
| tee >(sha1sum > dvd.sha1) \
>(md5sum > dvd.md5) \
> dvd.iso
That downloads the file once, sends output through two child processes (as started via bash(1) process substitution) and also tee(1)'s stdout, which is redirected to a file.
ls | tee tmpfile | first command
cat tmpfile | second command
Tee is a good way to do that, but you can make something simpler:
ls > __tmpfile
cat __tmpfile | wc -l
cat __tmpfile
rm __tmpfile