Scenario
I'm trying add the following flag before the execution of main process in shell script to make sure that this script is running only once at a time by checking the number of pids of same-name script:
this_bash=$(basename $0)
this_pid=${$}
is_running="$(pidof -x $this_bash -o $this_pid | wc -l )"
I found it always returns 1, even when there is no other script with the same name running.
Investigation
For further information I tried this:
z=$(pidof -x $this_bash -o $this_pid)
echo "[$z]"
echo "[$(pidof -x $this_bash -o $this_pid)]"
echo "[$($z | wc -l )]"
echo "[$(pidof -x $this_bash -o $this_pid | wc -l )]"
the square brackets are to make sure there are no hidden white space chars.
The result was:
[]
[]
[0]
[1]
Question
I don't understand why storing pidof as variable returns the expected result, while piping the commands directly does not.
Related
I have a bash/zsh command with multiple pipes | that fails when using set -o pipefail. For simplicity assume the command is
set -o pipefail; echo "123456" | head -c2 | grep 5 | cat
How do I quickly find out which command is the first to fail and why? I know I can check the exit code, but that doesn't show which part of the pipeline failed first.
Is there something simpler than the rather verbose check of building up the pipeline one by one and checking for the first failing exit code?
Edit: I removed the contrived code example I made up as it confused people about my purpose of asking. The actual command that prompted this question was:
zstdcat metadata.tsv.zst | \
tsv-summarize -H --group-by Nextclade_pango --count | \
tsv-filter -H --ge 'count:2' | \
tsv-select -H -f1 >open_lineages.txt
In bash, use echo "${PIPESTATUS[#]}" right after the command to get the exit status for each component in a space separated list:
#!/bin/bash
$ set -o pipefail; echo "123456" | head -c2 | grep 5 | cat
$ echo ${PIPESTATUS[#]}
0 0 1 0
Beware zsh users, you need to use the lower case pipestatus instead:
#!/bin/zsh
$ set -o pipefail; echo "123456" | head -c2 | grep 5 | cat
$ echo $pipestatus
0 0 1 0
In fish you can also simply use echo $pipestatus for the same output.
${PIPESTATUS[#]} right after is the answer you were looking for. However, I want to advise on the first example. It's a good habit to anticipate error, so instead of testing after you should have check the path prior everything.
if [ -d "/nonexistent_directory" ]; then
# here pipe shouldn't fail to grep
# ...unless there's something wrong with "foo"
# ...but "grep" may be a failure if the pattern isn't found
if ! ls -1 "/nonexistent_directory" | grep 'foo' ; then
echo "The command 'grep foo' failed."
# else
# echo "The pipeline succeeded."
fi
else
echo "The command 'ls /nonexistent_directory' failed."
fi
Whenever possible, avoid greping ls output in script, that' fragile...
I have the below code to find out the number of instances of current script running that is running with same arg1. But looks like the script creates a subshell and executes this command which also shows up in output. What would be the better approach to find the number of instances of running script ?
$cat test.sh
#!/bin/bash
num_inst=`ps -ef | grep $0 | grep $1 | wc -l`
echo $num_inst
$ps aux | grep test.sh | grep arg1 | grep -v grep | wc -l
0
$./test.sh arg1 arg2
3
$
I am looking for a solution that matches all running instance of ./test.sh arg1 arg2 not the one with ./test.sh arg10 arg20
The reason this creates a subshell is that there's a pipeline inside the command substitution. If you run ps -ef alone in a command substitution, and then separately process the output from that, you can avoid this problem:
#!/bin/bash
all_processes=$(ps -ef)
num_inst=$(echo "$all_processes" | grep "$0" | grep -c "$1")
echo "$num_inst"
I also did a bit of cleanup on the script: double-quote all variable references to avoid weird parsing, used $() instead of backticks, and replaced grep ... | wc -l with grep -c.
You might also replace the echo "$all_processes" | ... with ... <<<"$all_processes" and maybe the two greps with a single grep -c "$0 $1":
...
num_inst=$(grep -c "$0 $1" <<<"$all_processes")
...
Modify your script like this:
#!/bin/bash
ps -ef | grep $0 | wc -l
No need to store the value in a variable, the result is printed to standard out anyway.
Now why do you get 3?
When you run a command within back ticks (fyi you should use syntax num_inst=$( COMMAND ) and not back ticks), it creates a new sub-shell to run COMMAND, then assigns the stdout text to the variable. So if you remove the use of $(), you will get your expected value of 2.
To convince yourself of that, remove the | wc -l, you will see that num_inst has 3 processes, not 2. The third one exists only for the execution of COMMAND.
I try to constantly display the size of the current folder using watch , the below command does not work however, what do I do wrong ? I use zsh shell
$ watch ls -a | wc -l
what do I do wrong ?
Shell parses | as a pipe. So when shell sees:
watch ls -a | wc -l
It parses it as two command with one command output redirected to the other:
( watch ls -a ) | ( wc -l )
It runs the command watch with two arguments ls and -a and a command wc with single argument -l. Because watch ls -a never ends and wc -l only outputs when the input ends, you don't see anything printed out. wc -l waits until all input lines are printed, which never happens.
Because watch internally calls the shell, you can:
watch 'ls -a | wc -l'
This runs a single command watch with one argument ls -a | wc -l. watch internally spawns a shell and passes the string ls -a | wc -l to it. Then this internal shell spawns two new processes ls -a and wc -l with input/output connected.
I have a shell script that outputs information about successes to stdout, and also does a grep looking for errors in logs
inner.sh:
# do some things
echo success
# do other things
echo success
grep 'error' logs/*
I have another shell script that calls this one, counts up the successes and compares them to an expected number of successes:
outer.sh:
bash ./inner.sh | grep success | wc -l # I compare this number to the expected number
What I can't figure out how to do is have the output of grep go to stderr, so its not counted by the wc -l in outer.sh, but rather makes it around the wc to the terminal for the operator to see.
So I want a command like stdin_to_stderr that I can pipe the grep to, that would make any results it finds leave inner.sh on its stderr.
Is there already such a thing? Or do I just need to write the tiny script that would do this? Or am I thinking about this wrong?
bash ./inner.sh >&2 output_log_file
grep -c success output_log_file -- Count of success
grep -v -c success output_log_file -- Count of not "success"
Example :
echo -e "success.\nerror.\nsuccess.\nError.\nsuccess.\nerror" | grep -c success
echo -e "success.\nerror.\nsuccess.\nError.\nsuccess." | grep -v -c success
output :
3
2
I have some code that I would like to have the $? variable of.
VARIABLE=`grep "searched_string" test.log | sed 's/searched/found/'`
Is there any way to test if this entire line (rather than just the sed command) was completed successfully? If I try the following code right after it:
if [ "$?" -ne 0 ]
then
echo 1
exit
fi
it doesn't run even if the grep part of the statement fails.
Could someone show how to resolve this issue?
Use the
echo ${PIPESTATUS[#]}
will print out the array of exit-statuses of all commands.
$ ls | grep . | wc -l
28
$ echo ${PIPESTATUS[#]}
0 0 0
but
$ ls | grep nonexistentfilename | wc -l
0
$ echo ${PIPESTATUS[#]}
0 1 0 #the grep returns 1 - pattern not found
or
$ ls nonexistentfilename | grep somegibberish | wc -l
ls: nonexistentfilename: No such file or directory
0
$ echo ${PIPESTATUS[#]}
1 1 0 #ls and grep fails
for exact command status
echo ${PIPESTATUS[1]} #for the grep
also here is the
set -o pipefail
from the docs
pipefail
If set, the return value of a pipeline is the value of the
last (rightmost) command to exit with a non-zero status, or zero if
all commands in the pipeline exit successfully. This option is
disabled by default.
$ ls nonexistentfile | wc -c
ls: nonexistentfile: No such file or directory
0
$ echo $?
0
$ set -o pipefail
$ ls nonexistentfile | wc -c
ls: nonexistentfile: No such file or directory
0
$ echo $?
1
EDIT based on the comment
Youre probably tried the next:
VARIABLE=$(grep "searched_string" test.log | sed 's/searched/found/')
echo "${PIPESTATUS[#]}"
Of course, this can't work because the whole $(...) part runs in the subshell (another process) and therefore any variable what is created is lost when the subshell exits. (at the ))
You should put the whole PIPESTATUS mechanism into $(...) like next:
variable=$(
grep "searched_string" test.log | sed 's/searched/found/'
# do something with PIPESTATUS
# you should not echo anythig to stdout (because will be captured into $variable)
# you can echo on stderr - e.g.
echo "=${PIPESTATUS[#]}=" >&2
)
Also, the second line of the comment is an solution, eg:
var_with_status=$(command | commmand2 ; echo ":DELIMITER:${PIPESTATUS[#]}")
now, the $var_with_status will contain not only the result of the command | command2 but the PIPESTATUS too, delimited with some unique delimiter, so you can extract it...
Also, the set -o pipefail will indicate the result - if you don't need exact place of the fail.
Also you can write the PIPESTATUS in some temp-file (in the subshell) and the parent can read it and delete the temp-file...
Also is possible print the PIPESTATUS into different file-descriptors in the subshell and read this descriptor in the parent shell, but....
... beware do not fall into the XY problem, where you will make extremelly complicated script, only because you don't want change the logic of the processing.
e.g. you can always break you script into safe parts, like:
var1=$(grep 'str' test.log)
#check the `$var1` and do something with the error indicated with `$?`
var2=(sed '....' <<<"$var1")
#check the `$var2` and do something with the error indicated with `$?`
#and so on
simple enough?
So, ask yourself - do you really need mungling with how to get the PIPESTATUS form an subshell?
Ps: don't use uppercase variable names. could interfere with some environment variables and causes hard-to-debug problems..