Bash: read input available (was any key hit?) - bash

Question
I am using Bash 5 and have a long-running loop which needs to check occasionally for various keys the user may have hit. I know how to do this using stty — see my answer below — but it's more ugly than ought.
Essentially, I'm looking for a clean way to do this:
keyhit-p() {
if "read -n1 would block"; then
return false
else
return true
fi
}
Non-solution: read -t 0
I have read the bash manual and know about read -t 0. That does not do what I want, which is to detect if any input is available. Instead, it only returns true if the user hits ENTER (a complete line of input).
For example:
while true; do
if read -n1 -t0; then
echo "This only works if you hit enter"
break
fi
done

A working answer, albeit ugly
While the following works, I am hoping someone has a better answer.
#!/bin/bash
# Reset terminal's stty to previous values on exit.
trap 'stty $(stty --save)' EXIT
keyhit-p() {
# Return true if input is available on stdin (any key has been hit).
local sttysave=$(stty --save)
stty -icanon min 1 time 0 # ⎫
read -t0 # ⎬ Ugly: This ought to be atomic so the
local status=$? # ⎪ terminal's stty is always restored.
stty $sttysave # ⎭
return $status
}
while true; do
echo -n .
if ! keyhit-p; then
continue
else
while keyhit-p; do
read -n1
echo Key: $REPLY
done
break
fi
done
This alters the user's terminal settings (stty) before the read and attempts to write them back afterward, but does so non-atomically. It's possible for the script to get interrupted and leave the user's terminal in an incorrect state. I'd like to see an answer which solves that problem, ideally using only the tools built in to bash.
A faster, even uglier answer
Another flaw in the above routine is that it takes a lot of CPU time trying to get everything right. It requires calling an external program (stty) three times just to check that nothing has happened. Forks can be expensive in loops. If we dispense with correctness, we can get a routine that runs two orders of magnitude (256×) faster.
#!/bin/bash
# Reset terminal's stty to previous values on exit.
trap 'stty $(stty --save)' EXIT
# Set one character at a time input for the whole script.
stty -icanon min 1 time 0
while true; do
echo -n .
# We save time by presuming `read -t0` no longer waits for lines.
# This may cause problems and can be wrong, for example, with ^Z.
if ! read -t0; then
continue
else
while read -t0; do
read -n1
echo Key: $REPLY
done
break
fi
done
Instead of changing to non-canonical mode only during the read test, this script sets it once at the beginning and uses an exception handler when the script exits to undo it.
While I like that the code looks cleaner, the atomicity flaw of the original version is exacerbated because the SUSPEND signal isn't handled. If the user's shell is bash, icanon is enabled when the process is suspended, but NOT disabled when the process is foregrounded. That makes read -t0 return FALSE even when keys (other than Enter) are hit. Other user shells may not enable icanon on ^Z as bash does, but that's even worse as entering commands will no longer work as usual.
Additionally, requiring non-canonical mode to be left on all the time may cause other problems as the script gets longer than this trivial example. It is not documented how non-canonical mode is supposed to affect read and other bash built-ins. It seems to work in my tests, but will it always? Chances of running into problems would multiply when calling — or being called by — external programs. Maybe there would be no issues, but it would require tedious testing.

Related

How to handle interrupt signal when reading from stdin in bash [duplicate]

This question already has answers here:
SIGINT to cancel read in bash script?
(2 answers)
Closed 2 years ago.
I'm playing around with bash read functionality. I like what I have so far as a simple layer on top of my current shell. The read -e does tab-complete and previous commands, and sending EOF with ctrl+d gets me back to my original shell. Here's my reference:
Bash (or other shell): wrap all commands with function/script
I'd like some help handling SIGINT, ctrl+c. In a normal shell, if you start typing and hit ^C halfway, it immediately ends the line. For this simple example, after ^C, I still have to hit return before it's registered.
How do I keep the nice things that readline does, but still handle SIGINT correctly? Ideally, it would send a continue statement to the while read loop, or somehow send a \n to STDIN where my read is waiting.
Example code:
#!/bin/bash
# Emulate bash shell
gtg=1
function handleCtrl-C {
# What do I do here?
gtg=0
return
}
trap handleCtrl-C INT
while read -e -p "> " line
do
if [[ $gtg == 1 ]] ; then
eval "$line"
fi
gtg=1
done
I think I came up with something finally I liked. See SIGINT to cancel read in bash script? for that answer.
Reading man 7 signal tells that some system calls have a restartable flag set as a result will return back to the command
For some system calls, if a signal is caught while the call is
executing and the call is prematurely terminated, the call is
auto-matically restarted. Any handler installed with signal(3) will
have the SA_RESTART flag set, meaning that any restartable system call
will not return on receipt of a signal. The affected system calls
include read(2), write(2), sendto(2), recvfrom(2),sendmsg(2), and
recvmsg(2) on a communications channel or a low speed device and
during a ioctl(2) or wait(2). However, calls that have already
committed are not restarted, but instead return a partial success (for
example, a short read count). These semantics could be changed with
siginterrupt(3).
You can try printing the value input to line and verify that the read is resumed after CtrlC return until new line is hit. Type in something like "exit", followed by Ctrl-C and then "exit" the output comes out as "exitexit". Make the following change and run for the above test case
echo ">$line<"
if [ $gtg == 1 ] ; then
You'll the output as
You can verify this with a C program as well.

Bash script to start Solr deltaimporthandler

I am after a bash script which I can use to trigger a delta import of XML files via CRON. After a bit of digging and modification I have this:
#!/bin/bash
# Bash to initiate Solr Delta Import Handler
# Setup Variables
urlCmd='http://localhost:8080/solr/dataimport?command=delta-import&clean=false'
statusCmd='http://localhost:8080/solr/dataimport?command=status'
outputDir=.
# Operations
wget -O $outputDir/check_status_update_index.txt ${statusCmd}
2>/dev/null
status=`fgrep idle $outputDir/check_status_update_index.txt`
if [[ ${status} == *idle* ]]
then
wget -O $outputDir/status_update_index.txt ${urlCmd}
2>/dev/null
fi
Can I get any feedback on this? Is there a better way of doing it? Any optimisations or improvements would be most welcome.
This certainly looks usable. Just to confirm, you intend to run this ever X minutes from your crontab? That seems reasonsable.
The only major quibble (IMHO) is discarding STDERR information with 2>/dev/null. Of course it depends on what are your expectations for this system. If this is for a paying customer or employer, do you want to have to explain to the boss, "gosh, I didn't know I was getting error message 'Cant connect to host X' for the last 3 months because we redirect STDERR to /dev/null"! If this is for your own project, and your monitoring the work via other channels, then not so terrible, but why not capture STDERR to file, and if check that there are no errors. as a general idea ....
myStdErrLog=/tmp/myProject/myProg.stderr.$(/bin/date +%Y%m%d.%H%M)
wget -O $outputDir/check_status_update_index.txt ${statusCmd} 2> ${myStdErrLog}
if [[ ! -s ${myStdErrLog} ]] ; then
mail -s "error on myProg" me#myself.org < ${myStdErrLog}
fi
rm ${myStdErrLog}
Depending on what curl includes in its STDERR output, you may need filter what is in the StdErrLog to see if there are "real" error messages that you need to have sent to you.
A medium quibble is your use backticks for command substitution, if you're using dbl-sqr-brackets for evaluations, then why not embrace complete ksh93/bash semantics. The only reason to use backticks is if you think you need to be ultra-backwards compatible and that you'll be running this script under the bourne shell (or possibly one of the stripped down shells like dash).Backticks have been deprecated in ksh since at least 1993. Try
status=$(fgrep idle $outputDir/check_status_update_index.txt)
The $( ... ) form of command substitution makes it very easy to nest multiple cmd-subtitutions, i.e. echo $(echo one $(echo two ) ). (Bad example, as the need to nest cmd-sub is pretty rare, I can't think of a better example right now).
Depending on your situation, but in a large production environement, where new software is installed to version numbered directories, you might want to construct your paths from variables, i.e.
hostName=localhost
portNum=8080
SOLRPATH=/solr
SOLRCMD='delta-import&clean=false"
urlCmd='http://${hostName}:${portNum}${SOLRPATH}/dataimport?command=${SOLRCMD}"
The final, minor quibble ;-). Are you sure ${status} == *idle* does what you want?
Try using something like
case "${status}" in
*idle* ) .... ;;
* ) echo "unknown status = ${status} or similar" 1>&2 ;;
esac
Yes, your if ... fi certainly works, but if you want to start doing more refined processing of infomation that you put in your ${status} variable, then case ... esac is the way to go.
EDIT
I agree with #alinsoar that 2>/dev/null on a line by itself will be a no-op. I assumed that it was a formatting issue, but looking in edit mode at your code I see that it appears to be on its own line. If you really want to discard STDERR messages, then you need cmd ... 2>/dev/null all on one line OR as alinsoar advocates, the shell will accept redirections at the front of the line, but again, all on one line ;-!.
IHTH

Concurrent logging in bash scripts

I am currently trying to figure out why a shell script fails at concurrent logging every once in a while.
I have a shell function like the following:
log()
{
local l_text=$1
local l_file="/path/to/logs/$(date +%Y%m%d)_script.log"
local l_line="$(date +'%Y-%m-%d %H:%M:%S') $(hostname -s) ${l_text}"
echo ${l_line} >> ${l_file}
}
Now every once in a while this fails with a syntax error:
/path/to/script.sh: command substitution: line 163: syntax error near unexpected token `)'
/path/to/script.sh: command substitution: line 163: `hostname -s) ${l_text}'
The problem is, that I have multiple sub-processes, which each want to log as well as send traps (during which logging is performed as well). I haved debugged the problem and found out, that this happens, when the function is entered three times simultaneously. First the main process enters, then the child. After the date part of l_text is executed, main get's interrupted by a trap which is caused by child and in this trap tries to log something. The child and the trap finish their logging nicely, but then main is resumed after the trap and tries to execute the hostname part (presumedly) and fails with this error.
So it seems like main does not like being put to sleep while it is producing the $(date +'%Y-%m-%d %H:%M:%S') $(hostname -s) ${l_text} part of the log statement and cannot resume nicely. I was assuming this should work fine, because I am just using local variables and thread safe output methods.
Is this a general concurrency problem I am running into here? Or is this very specific for the trap mechanism in bash scripts? I know about the commodities of SIGNAL handling in C, so I am aware that only certain operations are allowed in SIGNAL handlers. However I am not aware if the same precautions also apply when handling SIGNALs in a bash script. I tried to find documentation on this, but none of the documents I could find gave any indications of problems with SIGNAL handling in scripts.
EDIT:
Here is an actuall simple script that can be used to replicate the problem:
#!/bin/bash
log() {
local text="$(date +'%Y-%m-%d %H:%M:%S') $(hostname -s) $1"
echo $text >> /dev/null
}
sub_process() {
while true; do
log "Thread is running"
kill -ALRM $$
sleep 1
done
}
trap "log 'received ALRM'" ALRM
sub_process &
sub_process_pid=$!
trap "kill ${sub_process_pid}; exit 0" INT TERM
while true; do
log "Main is running"
sleep 1
done
Every once in a while this script will get killed because of a syntax error in line 5. Line 5 is echo $text >> /dev/null, but since the syntax error also mentiones the hostname command, similar to the one I posted above, I am assuming there is an of-by-one error as well and the actual error is in line 4, which is local text="$(date +'%Y-%m-%d %H:%M:%S') $(hostname -s) $1".
Does anybody know what to do with the above script to correct it? I alread tried moving out the construction of the string into some temporary variables:
log() {
local thedate=$(date +'%Y-%m-%d %H:%M:%S')
local thehostname=$(hostname -s)
local text="${thedate} ${thehostname} $1"
echo $text >> /dev/null
}
This way the error appears less frequently, but it still is present, so this is not a real fix.
I would say that this is definitely a bug in bash and I would encourage you to report it to the bash developers. At the very least, you should never get a syntax error for what is syntactically correct code.
For the record, I get the same results as you with GNU bash, version 4.2.10(1)-release (x86_64-pc-linux-gnu).
I found that you can workaround the problem by not calling a function in your trap handler. E.g. replacing
trap "log 'received ALRM'" ALRM
with
trap "echo $(date +'%Y-%m-%d %H:%M:%S') $(hostname -s) received ALRM" ALRM
makes the script stable for me.
I know about the commodities of SIGNAL handling in C, so I am aware
that only certain operations are allowed in SIGNAL handlers. However I
am not aware if the same precautions also apply when handling SIGNALs
in a bash script.
I guess you shouldn't have to take special precautions but apparently in practice you do. Given that the problem seem to go away without the function call, I'm guessing that something in bash either isn't re-entrant where it should be or fails prevent re-entry in the first place.

How can I have output from one named pipe fed back into another named pipe?

I'm adding some custom logging functionality to a bash script, and can't figure out why it won't take the output from one named pipe and feed it back into another named pipe.
Here is a basic version of the script (http://pastebin.com/RMt1FYPc):
#!/bin/bash
PROGNAME=$(basename $(readlink -f $0))
LOG="$PROGNAME.log"
PIPE_LOG="$PROGNAME-$$-log"
PIPE_ECHO="$PROGNAME-$$-echo"
# program output to log file and optionally echo to screen (if $1 is "-e")
log () {
if [ "$1" = '-e' ]; then
shift
$# > $PIPE_ECHO 2>&1
else
$# > $PIPE_LOG 2>&1
fi
}
# create named pipes if not exist
if [[ ! -p $PIPE_LOG ]]; then
mkfifo -m 600 $PIPE_LOG
fi
if [[ ! -p $PIPE_ECHO ]]; then
mkfifo -m 600 $PIPE_ECHO
fi
# cat pipe data to log file
while read data; do
echo -e "$PROGNAME: $data" >> $LOG
done < $PIPE_LOG &
# cat pipe data to log file & echo output to screen
while read data; do
echo -e "$PROGNAME: $data"
log echo $data # this doesn't work
echo -e $data > $PIPE_LOG 2>&1 # and neither does this
echo -e "$PROGNAME: $data" >> $LOG # so I have to do this
done < $PIPE_ECHO &
# clean up temp files & pipes
clean_up () {
# remove named pipes
rm -f $PIPE_LOG
rm -f $PIPE_ECHO
}
#execute "clean_up" on exit
trap "clean_up" EXIT
log echo "Log File Only"
log -e echo "Echo & Log File"
I thought the commands on line 34 & 35 would take the $data from $PIPE_ECHO and output it to the $PIPE_LOG. But, it doesn't work. Instead I have to send that output directly to the log file, without going through the $PIPE_LOG.
Why is this not working as I expect?
EDIT: I changed the shebang to "bash". The problem is the same, though.
SOLUTION: A.H.'s answer helped me understand that I wasn't using named pipes correctly. I have since solved my problem by not even using named pipes. That solution is here: http://pastebin.com/VFLjZpC3
it seems to me, you do not understand what a named pipe really is. A named pipe is not one stream like normal pipes. It is a series of normal pipes, because a named pipe can be closed and a close on the producer side is might be shown as a close on the consumer side.
The might be part is this: The consumer will read data until there is no more data. No more data means, that at the time of the read call no producer has the named pipe open. This means that multiple producer can feed one consumer only when there is no point in time without at least one producer. Think of it of door which closes automatically: If there is a steady stream of people keeping the door always open either by handing the doorknob to the next one or by squeezing multiple people through it at the same time, the door is open. But once the door is closed it stays closed.
A little demonstration should make the difference a little clearer:
Open three shells. First shell:
1> mkfifo xxx
1> cat xxx
no output is shown because cat has opened the named pipe and is waiting for data.
Second shell:
2> cat > xxx
no output, because this cat is a producer which keeps the named pipe open until we tell him to close it explicitly.
Third shell:
3> echo Hello > xxx
3>
This producer immediately returns.
First shell:
Hello
The consumer received data, wrote it and - since one more consumer keeps the door open, continues to wait.
Third shell
3> echo World > xxx
3>
First shell:
World
The consumer received data, wrote it and - since one more consumer keeps the door open, continues to wait.
Second Shell: write into the cat > xxx window:
And good bye!
(control-d key)
2>
First shell
And good bye!
1>
The ^D key closed the last producer, the cat > xxx, and hence the consumer exits also.
In your case which means:
Your log function will try to open and close the pipes multiple times. Not a good idea.
Both your while loops exit earlier than you think. (check this with (while ... done < $PIPE_X; echo FINISHED; ) &
Depending on the scheduling of your various producers and consumers the door might by slam shut sometimes and sometimes not - you have a race condition built in. (For testing you can add a sleep 1 at the end of the log function.)
You "testcases" only tries each possibility once - try to use them multiple times (you will block, especially with the sleeps ), because your producer might not find any consumer.
So I can explain the problems in your code but I cannot tell you a solution because it is unclear what the edges of your requirements are.
It seems the problem is in the "cat pipe data to log file" part.
Let's see: you use a "&" to put the loop in the background, I guess you mean it must run in parallel with the second loop.
But the problem is you don't even need the "&", because as soon as no more data is available in the fifo, the while..read stops. (still you've got to have some at first for the first read to work). The next read doesn't hang if no more data is available (which would pose another problem: how does your program stops ?).
I guess the while read checks if more data is available in the file before doing the read and stops if it's not the case.
You can check with this sample:
mkfifo foo
while read data; do echo $data; done < foo
This script will hang, until you write anything from another shell (or bg the first one). But it ends as soon as a read works.
Edit:
I've tested on RHEL 6.2 and it works as you say (eg : bad!).
The problem is that, after running the script (let's say script "a"), you've got an "a" process remaining. So, yes, in some way the script hangs as I wrote before (not that stupid answer as I thought then :) ). Except if you write only one log (be it log file only or echo,in this case it works).
(It's the read loop from PIPE_ECHO that hangs when writing to PIPE_LOG and leaves a process running each time).
I've added a few debug messages, and here is what I see:
only one line is read from PIPE_LOG and after that, the loop ends
then a second message is sent to the PIPE_LOG (after been received from the PIPE_ECHO), but the process no longer reads from PIPE_LOG => the write hangs.
When you ls -l /proc/[pid]/fd, you can see that the fifo is still open (but deleted).
If fact, the script exits and removes the fifos, but there is still one process using it.
If you don't remove the log fifo at the cleanup and cat it, it will free the hanging process.
Hope it will help...

How to extend bash shell?

would like to add new functionality to the bash shell. I need to have a queue for executions.
What is the easy way to add new functionality to the bash shell keeping all native functions?
I would like to process the command line, then let the bash to execute them. For users it should be transparent.
Thanks Arman
EDIT
I just discovered prll.sourceforge.net it does exactly what I need.
Its easier than it seems:
#!/bin/sh
yourfunctiona(){ ...; }
...
yourfunctionz(){ ...; }
. /path/to/file/with/more/functions
while read COMMANDS; do
eval "$COMMANDS"
done
you can use read -p if you need a prompt or -t if you want it to timeout ... or if you wanted you could even use your favorite dialog program in place of read and pipe the output to a tailbox
touch /tmp/mycmdline
Xdialog --tailbox /tmp/mycmdline 0 0 &
COMMANDS="echo "
while ([ "$COMMANDS" != "" ]); do
COMMANDS=`Xdialog --stdout --inputbox "Text here" 0 0`
eval "$COMMANDS"
done >>/tmp/mycmdline &
To execute commands in threads you can use the following in place of eval $COMMANDS
#this will need to be before the loope
NUMCORES=$(awk '/cpu cores/{sum += $4}END{print sum}' /proc/cpuinfo)
for i in {1..$NUMCORES};do
if [ $i -eq $NUMCORES ] && #see comments below
if [ -d /proc/$threadarray[$i] ]; then #this core already has a thread
#note: each process gets a directory named /proc/<its_pid> - hacky, but works
continue
else #this core is free
$COMMAND &
threadarray[$i]=$!
break
fi
done
Then there is the case where you fill up all threads.
You can either put the whole thing in a while loop and add continues and breaks,
or you can pick a core to wait for (probably the last) and wait for it
to wait for a single thread to complete use:
wait $threadarray[$i]
to wait for all threads to complete use:
wait
#I ended up using this to keep my load from getting to high for too long
another note: you may find that some commands don't like to be threaded, if so you can put the whole thing in a case statement
I'll try to do some cleanup on this soon to put all of the little blocks together (sorry, I'm cobbling this together from random notes that I used to implement this exact thing, but can't seem to find)

Resources