handling tcl expect application crash - bash

Question

Is it possible to spawn an application, send commands, expect results, but also 'respawn' the application in case it crashed and continue from the last point? I tried creating procedures for spawning, but I am not able to catch user shell prompt once the application gets closed

Answer 1

So it sounds like you are doing something like telneting or sshing
into a shell then running an application. The application dies or
hangs so the prompt does not return so expect does not return and so
you are stuck. You will need to use timeout to detect a hang and restart
your entire process ( including the spawn). Below is the boiler plate I start with when writing
expect scripts. The critical thing to help you resolve your problem
is to realize that spawn not only sets the spawn_id but also returns
the pid of the spawned process. You can use that pid to kill the
spawned process if you get an eof and/or a timeout. My boiler plate
kills the spawn process on timeout. The timeout does not bail from the
expect loop but waits for the eof before exiting. it also accumulates
the output of expect command so on timeout you may be able to see
where it dies. The boilerplate is in a proc called run. Spawn the process
and pass the pid and the spawn id . Reuse the boiler plate to define other
procs that will be the steps. put the procs in a script with a counter between
them as shown and repeat. The other answerer is correct that the unless the app
restarts where you left off you can need to start from scratch. If it does start from
where you left off. make the steps granular enough that you know what command
to repeat. and step to start from.
BoilerPlate
proc run { pid spawn_id buf } {
upvar $buf buffer; #buffer to accumulate output of expect
set bad 0;
set done 0;
exp_internal 0; # set to one for extensive debug
log_user 0; # set to one to watch action
expect {
-i $spawn_id
-re {} {
append buffer $expect_out(buffer); # accumultate expect output
exp_continue;
}
timeout {
send_user "timeout\n"
append buffer $expect_out(buffer); # accumultate expect output
exec kill -9 $pid
set bad 1
exp_continue;
}
fullbuffer {
send_user " buffer is full\n"
append buffer $expect_out(buffer); # accumultate expect output
exp_continue;
}
eof {
send_user "Eof detected\n"
append buffer $expect_out(buffer); # accumultate expect output
set done 1 ;
}
}
set exitstatus [ exp_wait -i $spawn_id ];
catch { exp_close -i $spawn_id };
if { $bad } {
if { $done } {
throw EXP_TIMEOUT "Application timeout"
}
throw BAD_ERROR "unexpected failure "
}
return $exitstatus
}
set count 0
set attempts 0 ; # try 4 times
while { $count == 0 && $attempts < 4 } {
set buff ""
set pid [spawn -noecho ssh user#host ]
try {
run $pid $::spawn_id buff
incr count
run2 $pid $::spawn_id buff
incr count
run3 $pid $::spawn_id buff
incr count
run4 $pid $::spawn_id buff
incr count
} trap EXP_TIMEOUT { a b } {
puts "$a $b"
puts " program failed at step $count"
} on error { a b } {
puts "$a $b"
puts " program failed at step $count"
} finally {
if { $count == 4 } {
puts "success"
} else {
set count 0
incr attempts
puts "$buff"
puts "restarting\n"
}
}
}