I have very simple bash script:
#!/bin/bash
echo -n "A: ";
grep -v ">" | grep -o "A" $1 | wc -l;
I type
./script.sh 1.fasta
I got
A: 131
But the curcor is still blicking and my script is not finishing. What's wrong here?
Thank you.
This is the problem command:
grep -v ">" | grep -o "A" $1 | wc -l;
Since first command grep -v ">" is waiting for the input from STDIN as you haven't supplied any file to be searched by grep.
PS: Even grep -o "A" $1 is also problem since piped command will take input from output of the previous command in chain.
Probably you meant:
grep -v ">" "$1" | grep -o "A" | wc -l
Your first grep does not have a file argument so it will read from standard input:
grep -v ">" | grep -o "A" $1 | wc -l;
(read stdin) (read $1)
The reason why you get the 131 is because your second grep does have a file argument so it's getting all lines in $1 that have an A. However it's still waiting around for the end of the first grep to finish (which you can do with CTRL-D).
What you probably wanted to do is this:
grep -v ">" "$1" | grep -o "A" | wc -l
This will find all lines in $1 without a >, then all occurrences of A in that, counting them.
Related
I'm running this series of commands
passwd=`wc -l /etc/passwd`
echo $passwd
Returns:
34 /etc/passwd
What do I need to do to this so that it will only show the output of wc -l?
Just read from standard input instead of giving wc a file name:
$ passwd=`wc -l < /etc/passwd`
$ echo "$passwd"
86
wc still outputs quite a bit of padding, but the file name is omitted (because wc has no idea what file the data comes from).
Using awk perhaps ?
$ passwd=$(wc -l /etc/passwd | awk '{print $1}')
$ echo $passwd
32
Using cut, from cut (GNU coreutils)
$ passwd=$(wc -l /etc/passwd | cut -d" " -f1)
$ echo $passwd
32
That's the default behaviour of wc:
» wc -l /etc/passwd
28 /etc/passwd
There is no way to tell wc not to output the filename.
wc returns also the filename, but there are other ways to do it. Some examples:
passwd=`wc -l /etc/passwd | grep -o [1-9]\*`
or
passwd=`wc -l /etc/passwd | cut -f1 -d' '`
(answer from this question: get just the integer from wc in bash)
cat a.txt | xargs -I % echo %
In the example above, xargs takes echo % as the command argument. But in some cases, I need multiple commands to process the argument instead of one. For example:
cat a.txt | xargs -I % {command1; command2; ... }
But xargs doesn't accept this form. One solution I know is that I can define a function to wrap the commands, but I want to avoid that because it is complex. Is there a better solution?
cat a.txt | xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _
...or, without a Useless Use Of cat:
<a.txt xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _
To explain some of the finer points:
The use of "$arg" instead of % (and the absence of -I in the xargs command line) is for security reasons: Passing data on sh's command-line argument list instead of substituting it into code prevents content that data might contain (such as $(rm -rf ~), to take a particularly malicious example) from being executed as code.
Similarly, the use of -d $'\n' is a GNU extension which causes xargs to treat each line of the input file as a separate data item. Either this or -0 (which expects NULs instead of newlines) is necessary to prevent xargs from trying to apply shell-like (but not quite shell-compatible) parsing to the stream it reads. (If you don't have GNU xargs, you can use tr '\n' '\0' <a.txt | xargs -0 ... to get line-oriented reading without -d).
The _ is a placeholder for $0, such that other data values added by xargs become $1 and onward, which happens to be the default set of values a for loop iterates over.
You can use
cat file.txt | xargs -i sh -c 'command {} | command2 {} && command3 {}'
{} = variable for each line on the text file
With GNU Parallel you can do:
cat a.txt | parallel 'command1 {}; command2 {}; ...; '
For security reasons it is recommended you use your package manager to
install. But if you cannot do that then you can use this 10 seconds
installation.
The 10 seconds installation will try to do a full installation; if
that fails, a personal installation; if that fails, a minimal
installation.
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
I prefer style which allows dry run mode (without | sh) :
cat a.txt | xargs -I % echo "command1; command2; ... " | sh
Works with pipes too:
cat a.txt | xargs -I % echo "echo % | cat " | sh
This is just another approach without xargs nor cat:
while read stuff; do
command1 "$stuff"
command2 "$stuff"
...
done < a.txt
This seems to be the safest version.
tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'command1 "$#"; command2 "$#";' ''
(-0 can be removed and the tr replaced with a redirect (or the file can be replaced with a null separated file instead). It is mainly in there since I mainly use xargs with find with -print0 output) (This might also be relevant on xargs versions without the -0 extension)
It is safe, since args will pass the parameters to the shell as an array when executing it. The shell (at least bash) would then pass them as an unaltered array to the other processes when all are obtained using ["$#"][1]
If you use ...| xargs -r0 -I{} bash -c 'f="{}"; command "$f";' '', the assignment will fail if the string contains double quotes. This is true for every variant using -i or -I. (Due to it being replaced into a string, you can always inject commands by inserting unexpected characters (like quotes, backticks or dollar signs) into the input data)
If the commands can only take one parameter at a time:
tr '[\n]' '[\0]' < a.txt | xargs -r0 -n1 /bin/bash -c 'command1 "$#"; command2 "$#";' ''
Or with somewhat less processes:
tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'for f in "$#"; do command1 "$f"; command2 "$f"; done;' ''
If you have GNU xargs or another with the -P extension and you want to run 32 processes in parallel, each with not more than 10 parameters for each command:
tr '[\n]' '[\0]' < a.txt | xargs -r0 -n10 -P32 /bin/bash -c 'command1 "$#"; command2 "$#";' ''
This should be robust against any special characters in the input. (If the input is null separated.) The tr version will get some invalid input if some of the lines contain newlines, but that is unavoidable with a newline separated file.
The blank first parameter for bash -c is due to this: (From the bash man page) (Thanks #clacke)
-c If the -c option is present, then commands are read from the first non-option argument com‐
mand_string. If there are arguments after the command_string, the first argument is assigned to $0
and any remaining arguments are assigned to the positional parameters. The assignment to $0 sets
the name of the shell, which is used in warning and error messages.
One thing I do is to add to .bashrc/.profile this function:
function each() {
while read line; do
for f in "$#"; do
$f $line
done
done
}
then you can do things like
... | each command1 command2 "command3 has spaces"
which is less verbose than xargs or -exec. You could also modify the function to insert the value from the read at an arbitrary location in the commands to each, if you needed that behavior also.
Another possible solution that works for me is something like -
cat a.txt | xargs bash -c 'command1 $#; command2 $#' bash
Note the 'bash' at the end - I assume it is passed as argv[0] to bash. Without it in this syntax the first parameter to each command is lost. It may be any word.
Example:
cat a.txt | xargs -n 5 bash -c 'echo -n `date +%Y%m%d-%H%M%S:` ; echo " data: " $#; echo "data again: " $#' bash
My current BKM for this is
... | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
It is unfortunate that this uses perl, which is less likely to be installed than bash; but it handles more input that the accepted answer. (I welcome a ubiquitous version that does not rely on perl.)
#KeithThompson's suggestion of
... | xargs -I % sh -c 'command1; command2; ...'
is great - unless you have the shell comment character # in your input, in which case part of the first command and all of the second command will be truncated.
Hashes # can be quite common, if the input is derived from a filesystem listing, such as ls or find, and your editor creates temporary files with # in their name.
Example of the problem:
$ bash 1366 $> /bin/ls | cat
#Makefile#
#README#
Makefile
README
Oops, here is the problem:
$ bash 1367 $> ls | xargs -n1 -I % sh -i -c 'echo 1 %; echo 2 %'
1
1
1
1 Makefile
2 Makefile
1 README
2 README
Ahh, that's better:
$ bash 1368 $> ls | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
1 #Makefile#
2 #Makefile#
1 #README#
2 #README#
1 Makefile
2 Makefile
1 README
2 README
$ bash 1369 $>
Try this:
git config --global alias.all '!f() { find . -d -name ".git" | sed s/\\/\.git//g | xargs -P10 -I{} git --git-dir={}/.git --work-tree={} $1; }; f'
It runs ten threads in parallel and does what ever git command you want to all repos in the folder structure. No matter if the repo is one or n levels deep.
E.g: git all pull
I have good idea to solve the problem.
Only write a comman mcmd, then you can do
find . -type f | xargs -i mcmd echo {} ## cat {} #pipe sed -n '1,3p'
The mcmd content as follows:
echo $* | sed -e 's/##/\n/g' -e 's/#pipe/|/g' | csh
I am trying to write a script which extracts data from the file "nohup.out" using tail -f and executing dig command on condition.
#!/bin/bash
nohup proxychains firefox
tail -f nohup.out | xargs if [[ {} == *"denied"* ]]
then
dig -x `cut -d '-' -f 6 {} | cut -d ':' -f 1`&;
fi
Output of nohup.out is
|S-chain|-<>-10.1.1.16:80-<><>-93.184.220.29:80-<--denied
|S-chain|-<>-10.1.1.16:80-<><>-93.184.220.29:80-<--denied
|S-chain|-<>-10.1.1.16:80-<><>-216.58.209.77:443-<><>-OK
|S-chain|-<>-10.1.1.16:80-<><>-46.28.247.89:443-<><>-OK
With the below command I am able to extract the IP for reverse DNS lookup.
cut -d '-' -f 6 | cut -d ':' -f 1
I am not able to find a way to pass the argument to cut and if command.
What you need is to convert the if statement into an argument to bash. Doing a simplistic transform, assuming that the code in the question has a chance of working, you get:
tail -f nohup.out |
xargs -I '{}' bash -c "if [[ {} == *"denied"* ]]; then dig -x $(cut -d '-' -f 6 {} | cut -d ':' -f 1) & fi"
This is exactly the same basic treatment as was needed for a for loop being executed by nohup — you need a shell to run the built-in command. See Why can't I use Unix nohup with Bash for loop? for an exactly analogous situation.
However, on further reflection, you want to cut the string which is the IP address, not the file with that as a name, so the command needs to echo the string into the cut commands. You also have to tear your hair getting the sub-commands executed correctly; you need a backslash before the $ of $(…), or before each of the back-ticks if you insist on using `…` notation, as well as using backslash-double-quote to protect the angle-brackets in the string.
tail -f nohup.out |
xargs -I '{}' bash -c "if [[ '{}' != *denied* ]]; then echo dig -x \"\$(echo '{}' | cut -d '-' -f 6 | cut -d ':' -f 1)\" & fi"
Now we need to debate the use of the condition and two cut commands (and the general hair loss). You could use:
tail -f nohup.out |
grep -v denied |
xargs -I '{}' bash -c "echo dig -x \$(echo '{}' | cut -d '-' -f 6 | cut -d ':' -f 1) &"
or, more sensibly:
tail -f nohup.out |
awk -F '[-:]' '/denied/ { next } { print "dig -x " $7 " &" }' |
sh -x
or any of a myriad other ways to do it.
awk -F- '!/denied/ {print $6}'
splits each input line in fields separated by -, ignores the lines matching denied and extracts the 6th field of the remaining lines. With you example it outputs:
216.58.209.77:443
46.28.247.89:443
Following is my script, every time I run this it goes into else part. when I run the TEST2EVAL command it gives me 1
#!/bin/sh
TEST2EVAL='ps auxf | grep some.jar | grep -v grep | wc -l'
if [ "$TEST2EVAL" = 1 ]
then
java -jar /path/to/jar &
else
echo "Running"
fi
Assuming you are trying to find out if any processes are running with some.jar on their command lines you probably want:
if pgrep -f some.jar; then
echo running;
else
echo not running;
fi
In in order save the output of a command in a variable, you have to enclose the command in backticks (`), not single quotes ('). Thus, change the second line of your script to:
TEST2EVAL=`ps auxf | grep some.jar | grep -v grep | wc -l`
You are using the wrong quotes for command substitution: not single quotes:
TEST2EVAL='ps auxf | grep some.jar | grep -v grep | wc -l'
but backquotes:
TEST2EVAL=`ps auxf | grep some.jar | grep -v grep | wc -l`
Better yet, use TEST2EVAL=$(ps auxf | grep some.jar | grep -v grep | wc -l) instead. It's much clearer, supported by all POSIX-compatible shells, and can be nested more easily when necessary.
I am writing a script to kill all instances of the same process. As it is going to be used on Linux, AIX, HP-UX and Solaris, I need to use only built-in bash (sh) functions. That's why killall, pkill, etc. don't work for me.
Once there is only one instance of a process it should be just killed in traditional way:
kill -TERM `ps -ef | grep -v grep | grep $process | awk '{print $2}'`
However sometimes the program has extra instances running and that's why ps -ef | … returns more than one PID. That needs to be reported.
example:
bash-3.2$ ps -ef | grep -v grep | grep perl | awk '{print $2}'
5267
5268
5269
5270
5271
My thought was to store those values in a temporary variable and then send kill signal to each in a for loop.
bash-3.2$ tmp=`ps -ef | grep -v grep | grep perl | awk '{print $2}'`
bash-3.2$ echo $tmp
5267 5268 5269 5270 5271
However I still need the information if such a case occurred (how many instances were present).
It seems I need to check the whole string stored in the tmp variable and maybe count spaces?
Anyway the questions reduces to how to check how many values the $tmp variable stores?
For maximum portability and reliability, use -A (POSIX synonym of -e) and a custom format with -o rather than -f.
Your filtering of the output of ps is brittle: it may match other processes. You've had to exclude the grep process, and you may need to exclude your script as well, and there may be other completely innocent processes caught in the fray (such as your script itself) because their command line happens to contain $process as a substring. Make your filtering as strict as possible. With ps -o pid= -o comm=, you get just two columns (PID and command without arguments) with no header.
You don't need to use a loop to do the killing, kill accepts multiple arguments. For the counting, let the shell do it: you have a whitespace-separated list of numbers, to let the shell do the word splitting (with $(…) outside quotes) and count the number of resulting words ($#).
count_and_kill_processes () {
set -- $(ps -A -o pid= -o comm= |
awk -v "name=$process" '$2 == name {print $1}')
count=$#
if [ $# -ne 0 ]; then kill "$#"; fi
}
count_and_kill_processes foo
# now the number of killed processes is in $count
If your shell is bash or ksh on all machines, you can use an array.
pids=($(ps -A -o pid= -o comm= |
awk -v "name=$process" '$2 == name {print $1}') )
if [[ $# -ne 0 ]]; then kill "$#"; fi
# the number of killed processes is ${#pids}
use xargs:
ps aux | grep -ie perl | awk '{print $2}' | xargs kill -9
You can use a loop which should work in both cases:
for pid in $(ps -ef | grep -v grep | grep $process | awk '{print $2}'); do
echo $pid
done
Or count the number of matches:
if [ $(ps -ef | grep -v grep | grep $process | awk '{print $2}' | wc -l) -gt 1 ]
then
# more than one
fi