I'd like to return the results of a script that also kicks off a background task. The command substitution operator waits for the background task, making the call slow. I created the following example to illustrate the problem:
function answer {
sleep 5 &
echo string
}
echo $(answer)
Is there a way to call a command without waiting on any background jobs it creates?
Thanks,
Mark
The problem is that sleep inherits stdout and keeps it open. You can simply redirect stdout:
answer() {
sleep 5 > /dev/null &
echo "string"
}
echo "$(answer)"
If you are intending for the program to continue merrily along in the mean time while the function works, you can just call the function to run in the background.
function answer {
sleep 5
echo Second
}
echo $(answer) &
echo First
The output of which will be
First
Second
Related
In a bash script lets take the extreme below examples where the call/start of the myFn() is 5 minutes before the echo of $inVar -> $myvar happens. During this time between the function start and the interaction with the $myvar, it is updated.
myvar=$(... //some json
alpha=hello )
myFn(){
local -n inVar=$1
//wait for 5 mins .... :)
echo $inVar
}
myFn "myvar"
if [ -z $var ]
then
//wait 5 mins
//but life goes on and this is non blocking
//so other parts of the script are running
echo $myvar
fi
myvar=$(... // after 2 mins update alpha
alpha=world
)
As the $myvar is passed to myFn(), when is $myvar actually read,
at myFn call time (when the function is called/starts)
at the reference copy time inVar=$1
when the echo $inVar occurs
and is this the same for other types of processes such as while, if etc?
You're setting inVar as a nameref, so the value is not known until the variable is expanded at the echo statement
HOWEVER
In your scenario, myFn is "non blocking", meaning you launch it in the background. In this case, the subshell gets a copy of the current value of myVar -- if myVar gets updated subsequently, that update is happening in the current shell, not the background shell.
To demonstrate:
$ bash -c '
fn() { local -n y=$1; sleep 2; echo "in background function, y=$y"; }
x=5
fn x &
x=10
wait
'
in background function, y=5
TL;DR: namerefs and background processes don't mix well.
This question already has answers here:
Is it possible to make a bash shell script interact with another command line program?
(6 answers)
Closed 1 year ago.
I have a program JLinkExe which opens it's own prompt uponn execution. So I normally run it like this:
JLinkExe
and then type the commands at it's prompt that appears:
J-Link>
There are many applications with it's own prompt and I am interested in a general method that would enable me to send commands to any kind of application that has it's own prompt.
I already tried two methods. They both try to send commands in this order:
connect
Enter
Enter
Enter
Enter
erase
loadbin program.bin , 0x0
r
q
but both fail. Here is the first method:
{ echo 'connect';
echo '';
echo '';
echo '';
echo '';
echo 'erase';
echo 'loadbin program.bin , 0x0';
echo 'r';
echo 'q'; } | JLinkExe
And the second method (source):
JLinkExe <<EOF
connect
erase
loadbin program.bin , 0x0
r
q
EOF
I found these method on the internet but I don't understand why they fail. Especially the first one that worked in the past...
Can anyone propose any better / working / universally applicable method?
I think it might be because here-docs do not wait for output. Unfortunately for you I switched company, thus can't test my code below.
#! /bin/bash
expect <<-EOF
set timeout -1
spawn JLinkExe
expect "J-Link> " { send "connect\r" }
expect "J-Link> " { send "\r" }
expect "J-Link> " { send "\r" }
expect "J-Link> " { send "\r" }
expect "J-Link> " { send "\r" }
expect "J-Link> " { send "erase\r" }
expect "J-Link> " { send "loadbin program.bin , 0x0\r" }
expect "J-Link> " { send "r\r" }
expect "J-Link> " { send "q\r" }
expect eof
catch wait result
exit [lindex \$result 3]
EOF
exit $?
Except waits until J-Link> turns up and then sends the command through the connection.
If it doesn't work please notify me. I'll try to help you after the weekend :-)
EDIT:
A: Why did you wrap everything in expect 2>&1 <<-EOF and EOF?
You can add expect in the shebang, but I often use it as part of my Bash scripts. My knowledge of Bash is better.
B: Why a -EOF instead of EOF?
That's because <<-EOF allows leading tabs when you want to end the here-doc. You can indent it in functions for instance.
C: Why did you redirect stderr to stdout (2>&1)?
In your case I should've removed this. I took the code from one of my other answer about expect and tailored it to your needs.
D: What does catch wait result and exit [lindex \$result 3] do after we catch the eof?
Nice question, I had to look this one up a little myself:
lindex takes 4rd argument in \$result and exits the here-doc (0 is arg 1).
\$result is set by catch wait result.
Catch takes the output of wait and puts that into result.
Wait returns four integers:
First: pid of process that's being waited on.
Second: spawn ID.
Third: -1 for errors, 0 otherwise.
Forth: Exit status of the program as set by the OS.
Sources:
https://linux.die.net/man/1/expect
https://www.tcl.tk/man/tcl/TclCmd/catch.html
https://www.tcl.tk/man/tcl/TclCmd/lindex.html
Note that you have to escape the $ in the here-doc, otherwise Bash tries to process it. Hence \$result.
E: Why you exit with exit $?
Bash exits a script with the last known error code. Although you can leave it implicitly, I like to add it anyhow. It keeps the script more readable for beginners.
I have a long (~2,000 lines) script that I'm trying to log for future debugging. Right now I have:
function log_with_time()
{
while read a; do
echo `date +'%H:%M:%S.%4N '` " $a" >> $LOGFILE
done
}
exec 7> >(log_with_time)
BASH_XTRACEFD=7
PS4=' exit($?)ln:$LINENO: '
set -x
echo "helloWorld 1"
which gives me very nice logging for any and all commands that are run:
15:18:03.6359 exit(0)ln:28: echo 'helloWorld 1'
The issue that I'm running into is that xtrace seems to be asynchronous. With longer scripts, the log times fall behind the actual time the commands are called, and the exit code doesn't match the logged command.
There has to be a better way to do this but I'd be happy if I could just synchronize xtrace.
...
tldr: How can I generally log the time, command and exit code for all commands in a script?
...
(First time posting, feedback appreciated)
UPDATE:
exec {BASH_XTRACEFD}>>$LOGFILE
PS4=' time:$(date +%H:%M:%S.%4N) ln:$LINENO: '
set -x
fail()
{
echo "fail" >> $LOGFILE
return 1
}
trap 'echo exit:$? >> $LOGFILE' DEBUG
fail
solves all of my synchronization issues. exit codes and timestamps are working beautifully. My only issue now is one of formatting: the trap itself is getting reported by xtrace.
time:18:30:07.6080 ln:27: fail
time:18:30:07.6089 ln:12: echo fail
fail
time:18:30:07.6126 ln:13: return 1
time:18:30:07.6134 ln:28: echo exit:1
exit:1
I've tried setting +x in the trap but then set +x gets logged. If I could find a way to omit one line from xtrace, this log would be perfect.
The async behavior is coming from the process substitution -- anything in >(...) is running in its own subshell on the other end of a FIFO. Since it's a separate process, it's inherently unsynchronized.
You don't need log_with_time here at all, though, and so you don't need BASH_XTRACEFD redirecting to a process substitution in the first place. Consider:
# aside: $(date ...) has a *huge* amount of performance overhead here. Personally, I'd
# advise against using it, unless you really need all that precision; $SECONDS will
# be orders-of-magnitude cheaper.
PS4=' prior-exit:$? time:$(date +%H:%M:%S.%4N) ln:$LINENO: '
...thereafter:
$ true
prior-exit:0 time:16:01:17.2509 ln:28: true
$ false
prior-exit:0 time:16:01:18.4242 ln:29: false
$ false
prior-exit:1 time:16:01:19.2963 ln:30: false
$ true
prior-exit:1 time:16:01:20.2159 ln:31: true
$ true
prior-exit:0 time:16:01:20.8650 ln:32: true
Per conversation with Charles Duffy in the comments to whom all credit is given:
Process substitution >(...) is asynchronous, allowing the log writing to fall behind and out of sync with the xtrace.
Instead use:
exec {BASH_XTRACEFD}>>$LOGFILE
PS4=' time:$(date +%H:%M:%S.%4N) ln:$LINENO: '
for synchronously logging the time and line.
Furthermore, xtrace is triggered before running the command, making it a bad candidate for capturing exit codes. Instead use:
trap 'echo exit:$? >> $LOGFILE' DEBUG
to log the exit codes of each command since trap triggers on command completion. Note that this won't report on every step in a function call like xtrace will. (could use some help with the phrasing here)
No solution yet for omitting the trap from xtrace, but it's good enough:
LOGFILE="SomeFile.log"
exec {BASH_XTRACEFD}>>$LOGFILE
PS4=' time:$(date +%H:%M:%S.%4N) ln:$LINENO: '
set -x
fail() # test function that returns 1
{
echo "fail" >> $LOGFILE
return 1
}
success() # test function that returns 0
{
echo "success" >> $LOGFILE
return 0
}
trap 'echo $? >> $LOGFILE' DEBUG
fail
success
echo "complete"
yields:
time:14:10:22.2686 ln:21: trap 'echo $? >> $LOGFILE' DEBUG
time:14:10:22.2693 ln:23: echo 0
0
time:14:10:22.2736 ln:23: fail
time:14:10:22.2741 ln:12: echo fail
fail
time:14:10:22.2775 ln:13: return 1
time:14:10:22.2782 ln:24: echo 1
1
time:14:10:22.2830 ln:24: success
time:14:10:22.2836 ln:17: echo success
success
time:14:10:22.2873 ln:18: return 0
time:14:10:22.2881 ln:26: echo 0
0
time:14:10:22.2912 ln:26: echo complete
Using ksh93 i'm attempting to wait for a background process ,run_cataloguer(), to finish, from within a separate background process ,send_mail(), using the script below:
#!/usr/bin/env ksh
function run_cataloguer
{
echo "In run_cataloguer()"
sleep 2
echo "leaving run_cataloguer()"
}
function send_mail
{
echo "In send_mail()"
#jobs
wait_for_cataloguer
sleep 1
echo "Leaving send_mail() "
}
function wait_for_cataloguer
{
echo "In wait_for_cataloguer() PID_CAT = $PID_CAT"
wait $PID_CAT
waitRet=$?
echo "waitRet = $waitRet"
}
run_cataloguer &
PID_CAT=$!
echo "PID_CAT = $PID_CAT"
send_mail &
wait # Wait for all
echo "Finished main"
The following output is seen:
PID_CAT = 1265
In run_cataloguer()
In send_mail()
In wait_for_cataloguer() PID_CAT = 1265
waitRet = 127 # THIS SHOULD be 0
Leaving send_mail()
leaving run_cataloguer()
Finished main
The problem is
waitRet = 127
which means the wait command can't see $PID_CAT, so it doesn't wait for run_cataloguer() to finish and
"leaving send_mail()"
is printed before
"leaving run_cataloguer()"
If I run send_mail in the foreground then waitRet = 0, which is correct.
So, it appears that you cannot wait for a background process from within a separate background process.
Also, if I uncomment the jobs command, nothing is returned , which appears to confirm the previous statement.
If anyone has a solution ,apart form using flag files, :), it would be much appreciated.
It looks like this cannot be done. The solution I used was from Parvinder here:
wait child process but get error: 'pid is not a child of this shell'
In shell script I am trying to wait for non-child process. I got reference on how to do it from:
WAIT for "any process" to finish
My shell script structure is:
Main.sh
func1(){
return 1
}
func2(){
# Wait for func1 to finish
while kill -0 "$pid_func1"; do
sleep 0.5
done
}
# Call function 1 in background
func1 &
pid_func1=$!
func2 &
In this case how do I receive the return value of func1 inside function func2?
You generally cannot capture the exit status of non-child processes. You may be able to work something involving logging the exit codes to status files and then reading the values, but otherwise you're not going to be able to capture the values
I used anothe shell variable to store the return status in this case and checked value of this shell variable whereever required. Find a sample shell script below to simulate the scenario.
#!/bin/bash
func1(){
retvalue=23 # return value which needs to be returned
status_func1=$retvalue # store this value in shell variable
echo "func1 executing"
return $retvalue
}
func2(){
# Not possible to use wait command for pid of func1 as it is not a child of func2
#wait $pid_func1
#ret_func1=$?
while kill -0 "$pid_func1"; do
echo "func1 is still executing"
sleep 0.5
done
echo "func2 executing"
#echo "func1 ret: $ret_func1"
echo "func1 ret: $status_func1"
}
# Main shell script starts here
func1 &
pid_func1=$!
func2 &
Hope its useful for others who are facing the same issue.