How can I make the following commands exit immediately after the first line is matched? I understand that SIGPIPE is not sent to cat until it tries to write next time (tail bug report), but I don't understand how to solve this issue.
cat <( echo -ne "asdf1\nzxcv1\nasdf2\n"; sleep 5; echo -ne "zxcv2\nasdf3\n" ) | grep --line-buffered zxcv | head --lines=1
cat <( echo -ne "asdf1\nzxcv1\nasdf2\n"; sleep 5; echo -ne "zxcv2\nasdf3\n" ) | grep --max-count=1 zxcv
NB: I actually had tail --follow before the pipesign, but replaced it with catand sleep to simplify testing. The shell in question is GNU bash 4.4.12(1)-release, and I'm running MINGW that came with Git-for-Windows 2.12.2.2.
CLARIFICATION: I have a jboss server which is started in a docker container and which outputs couple thousand lines of text within three minutes to a log file. My goal is to watch this file until a status line is printed, analyze line contents and return it to a human or Jenkins user. Of course, I can grep whole file and sleep for a second in a loop, but I'd rather avoid this if at all possible. Furthermore, this looping would interfere with my usage of timeout routine to limit maximum execution time. So, is it possible to listen for a pipe until a certain line appears and stop as soon as that happens?
Related question: Why does bash ignore SIGINT if its currently running child handles it?
Interesting question! I've verified that head dies after printing the first line (removed background job noise):
$ (printf '%s\n' a b a; sleep 5; printf '%s\n' a) | grep --line-buffered a | head --lines=1 & sleep 1; pstree $$
a
bash─┬─bash───sleep
├─grep
└─pstree
At first glance, it appears head doesn't send SIGPIPE, but I get conflicting information from running strace grep:
$ (printf '%s\n' a b a; sleep 10; printf '%s\n' a) | strace grep --line-buffered a | head --lines=1
…
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=21950, si_uid=1000} ---
+++ killed by SIGPIPE +++
… and killing grep:
$ (printf '%s\n' a b a; sleep 10; printf '%s\n' a) | grep --line-buffered a | head --lines=1 & sleep 1; kill -PIPE $(pgrep grep); sleep 5; pstree $$
a
bash─┬─bash───sleep
└─pstree
Killing grep and then sleep fixes the issue:
$ (printf '%s\n' a b a; sleep 10; printf '%s\n' a) | grep --line-buffered a | head --lines=1 & sleep 1; kill -PIPE $(pgrep grep); sleep 1; kill -PIPE $(pgrep sleep); sleep 5; pstree $$
a
bash───pstree
Conclusion: WTF?
I've ended up doing following to be able to break following log both on a matching line and after a timeout.
#!/bin/sh
TOP_PID=$$
container_id="$1"
LOG_PATH=/opt/jboss/jboss-eap-6.2/standalone/log/server.log
await_startup () {
status=$(check_status)
follow_log --timeout $timeout &
local bgjob_pid; local bgjob_status;
bgjob_pid=$(jobs -p)
test -n "$bgjob_pid" || die "Could not start background job to follow log."
bgjob_status=true
while [ "$status" = "not started" ] && $bgjob_status; do
sleep 1s
status=$(check_status)
if kill -0 $bgjob_pid 2>/dev/null; then
bgjob_status=true
else
bgjob_status=false
fi
done
kill -KILL $bgjob_pid 2>/dev/null
}
follow_log () {
# argument parsing skipped...
docker exec $container_id timeout $timeout tail --follow=name ---disable-inotify --max-unchanged-stats=2 /$LOG_PATH
}
check_status () {
local line;
line=$(docker exec $container_id grep --extended-regexp --only-matching 'JBoss EAP .+ started.+in' /$LOG_PATH | tail --lines=1)
if [ -z "$line" ]; then
printf "not started"
elif printf "%s" "$line" | grep --quiet "with errors"; then
printf "started and unhealthy"
else
printf "healthy"
fi
}
die () {
test -n "$1" && printf "%s\n" "$1"
kill -s TERM $TOP_PID
return 1
} 1>&2
Related
I have an executable that accepts queries from stdin and responds to them, reading until EOF. Additionally I have an input file and a special command, let's call those EXEC, FILE and CMD respectively.
What I need to do is:
Pass FILE to EXEC as input.
Disregard all the output corresponding to commands read from FILE (/dev/null/).
Pass CMD as the last command.
Fetch output for the last command and save it in a variable.
EXEC's output can be multiline for each query.
I know how to pass FILE + CMD into the EXEC:
echo ${CMD} | cat ${FILE} - | ${EXEC}
but I have no idea how to fetch only output resulting from CMD.
Is there a magical one-liner that does this?
After looking around I've found the following partial solution:
mkfifo mypipe
(tail -f mypipe) | ${EXEC} &
cat ${FILE} | while read line; do
echo ${line} > mypipe
done
echo ${CMD} > mypipe
This allows me to redirect my input, but now the output gets printed to screen. I want to ignore all the output produced by EXEC in the while loop and get only what it prints for the last line.
I tried what first came into my mind, which is:
(tail -f mypipe) | ${EXEC} > somefile &
But it didn't work, the file was empty.
This is race-prone -- I'd suggest putting in a delay after the kill, or using an explicit sigil to determine when it's been received. That said:
#!/usr/bin/env bash
# route FD 4 to your output routine
exec 4> >(
output=; trap 'output=1' USR1
while IFS= read -r line; do
[[ $output ]] && printf '%s\n' "$line"
done
); out_pid=$!
# Capture the PID for the process substitution above; note that this requires a very
# new version of bash (4.4?)
[[ $out_pid ]] || { echo "ERROR: Your bash version is too old" >&2; exit 1; }
# Run your program in another process substitution, and close the parent's handle on FD 4
exec 3> >("$EXEC" >&4) 4>&-
# cat your file to FD 3...
cat "$file" >&3
# UGLY HACK: Wait to let your program finish flushing output from those commands
sleep 0.1
# notify the subshell writing output to disk that the ignored input is done...
kill -USR1 "$out_pid"
# UGLY HACK: Wait to let the subprocess actually receive the signal and set output=1
sleep 0.1
# ...and then write the command for which you actually want content logged.
echo "command" >&3
In validating this answer, I'm doing the following:
EXEC=stub_function
stub_function() {
local count line
count=0
while IFS= read -r line; do
(( ++count ))
printf '%s: %s\n' "$count" "$line"
done
}
cat >file <<EOF
do-not-log-my-output-1
do-not-log-my-output-2
do-not-log-my-output-3
EOF
file=file
export -f stub_function
export file EXEC
Output is only:
4: command
You could pipe it into a sed:
var=$(YOUR COMMAND | sed '$!d')
This will put only the last line into the variable
I think, that your proram EXEC does something special (open connection or remember state). When that is not the case, you can use
${EXEC} < ${FILE} > /dev/null
myvar=$(echo ${CMD} | ${EXEC})
Or with normal commands:
# Do not use (printf "==%s==\n" 1 2 3 ; printf "oo%soo\n" 4 5 6) | cat
printf "==%s==\n" 1 2 3 | cat > /dev/null
myvar=$(printf "oo%soo\n" 4 5 6 | cat)
When you need to give all input to one process, perhaps you can think of a marker that you can filter on:
(printf "==%s==\n" 1 2 3 ; printf "%s\n" "marker"; printf "oo%soo\n" 4 5 6) | cat | sed '1,/marker/ d'
You should examine your EXEC what could be used. When it is running SQL, you might use something like
(cat ${FILE}; echo 'select "DamonMarker" from dual;' ; echo ${CMD} ) |
${EXEC} | sed '1,/DamonMarker/ d'
and write this in a var with
myvar=$( (cat ${FILE}; echo 'select "DamonMarker" from dual;' ; echo ${CMD} ) |
${EXEC} | sed '1,/DamonMarker/ d' )
how can i check if a process is running, if it is running then echo "process is running" and keep them from using that process til its finished. i have this piece of code but i cant get it to not allow them to use that process after it echos:
#!/bin/bash
SERVICE=EXAMPLE
if ps ax | grep -v grep | grep -v grep | grep $SERVICE > /dev/null
then
echo -e ""
echo -e "${LIGHTRED}[!] ${WHITE}Please wait till process is finished."
fi
It seems you want to write a loop, not a single if statement.
And you probably want to sleep a bit between checking the condition.
#!/bin/bash
SERVICE=EXAMPLE
while ps ax | grep -v grep | grep "$SERVICE" > /dev/null
do
echo
echo -e "${LIGHTRED}[!] ${WHITE}Please wait till process is finished."
sleep 60
fi
The condition can be simpler if you have pgrep:
while pgrep "$SERVICE" >/dev/null
(Or the simpler while pgrep -q "$SERVICE" if your implementation of pgrep supports it.)
When there is no matching process (already finished or not started yet),
then the script will not produce any output.
If you want to get some output in that case,
then you can rework like this:
while true
do
if pgrep "$SERVICE" > /dev/null; then
echo
echo -e "${LIGHTRED}[!] ${WHITE}Please wait till process is finished."
sleep 60
else
echo "Process '$SERVICE' not running"
break
fi
fi
To print the message only once and just wait until the process is no longer running:
is_running() {
pgrep "$SERVICE" > /dev/null
}
if is_running; then
echo -e "${LIGHTRED}[!] ${WHITE}Please wait till process is finished."
while true; do
sleep 60
is_running || break
done
else
echo "Process '$SERVICE' not running"
fi
Another solution
#!/bin/bash
tput civis # hide cursor
until ! ps -ef | grep "$SERVICE" | grep -v "grep" > /dev/null; do
while ps -ef | grep "$SERVICE" | grep -v "grep" > /dev/null; do
echo -en "\r${LIGHTRED}[!] ${WHITE}Please wait till process is finished."
done
printf "\r%80s" "" # clear line
echo -en "\rProcess '$SERVICE' is completed!\n"
done
tput cnorm # show cursor again
This solution is useful also if you have multiple instance of the service together.
The problem is that tail -f doesn't exits by itself and my script stalls.
I am using sleep 15 because after triggering the shutdown command string 'org.apache.coyote.AbstractProtocol' appears after 15 sec in logs. In this way tail -100 works and gives the desired result.
Below is the snippet of code which I have written and its working fine, but I want to check it dynamically by tail -f. Please guide me how can I do it by tail -f and exiting the tail by some way.
ALFRESCO_LOGS="$CATALINA_BASE/logs"
printf "Stopping alfresco instance.\n\n"
$CATALINA_HOME/bin/shutdown.sh &> /dev/null
sleep 15
CHECK_ALFRESCO_LOGS=$(tail -100 $ALFRESCO_LOGS/catalina.out | grep --line-buffered 'org.apache.coyote.AbstractProtocol' | awk 'NR==1{print $NF}')
echo "$CHECK_ALFRESCO_LOGS"
if [[ "$CHECK_ALFRESCO_LOGS" == "stop" ]];
then
echo "Alfresco instance has been stopped."
fi
If you're OK with tail -f receiving SIGPIPE, this is a straightforward way:
while read CHECK_ALFRESCO_LOGS; do
echo "$CHECK_ALFRESCO_LOGS"
if [[ "$CHECK_ALFRESCO_LOGS" == "stop" ]]; then
echo "Alfresco instance has been stopped."
break
fi
done < <( tail -f $ALFRESCO_LOGS/catalina.out | awk '/org.apache.coyote.AbstractProtocol/{print $NF}')
When the condition matches, the while loop ends. That closes the input redirection, and thus the tail process receives SIGPIPE, which closes it.
Welcome, I have a short script to kill processes which works longer than specified time for UIDs bigger than. How to exclude for example mc command from killing?
#!/bin/bash
#
#Put the minimum(!) UID to kill processes
UID_KILL=500
#Put the time in seconds which the process is allowed to run below
KILL_TIME=300
KILL_LIST=`{
ps -eo uid,pid,lstart | tail -n+2 |
while read PROC_UID PROC_PID PROC_LSTART; do
SECONDS=$[$(date +%s) - $(date -d"$PROC_LSTART" +%s)]
if [ $PROC_UID -ge $UID_KILL -a $SECONDS -gt $KILL_TIME ]; then
echo -n "$PROC_PID "
fi
done
}`
#KILLING LOOP
while sleep 1
do
if [[ -n $KILL_LIST ]]
then
kill $KILL_LIST
fi
done
change inner command like this :
ps -eo comm,uid,pid,lstart | tail -n+2 | grep -v '^your_command' | ...
this will exclude 'your_command' from the list.
see STANDARD FORMAT SPECIFIERS in man ps for more about ps -o.
Hello I created a script to kill processes ordered by age however everytime the PIDs are changing... how can I solve this
here is my script
#!/bin/bash
#Argument = -c check -k kill -l list
usage()
{
cat << EOF
usage: $0 options
This script kills all the processes running and leaves the last one sorted by age running.
OPTIONS:
-c checks how many proccess are runnig it needs string argument
-k Kill all the processes and leaves just the last sorted by age running
-l Show the list of procesess to be killed.
EOF
}
CHECK=
KILL=
LIST=
while getopts "hc:k:l:" OPTION
do
case $OPTION in
h)
usage
exit 1
;;
c)
CHECK=$OPTARG
ps -ef | grep -i $CHECK | wc -l
;;
k)
KILL=$OPTARG
T2=$(ps -ef | grep -i "$KILL" | awk '{print $3,$5}' | sort -r +1 | sed 1d |awk '{print $1}')
for f in $T2; do
echo "killing $f"
kill $f
done
;;
l)
LIST=$OPTARG
T2=$(ps -ef | grep -i "$LIST" | awk '{print $3,$5}' | sort -r +1 | sed 1d |awk '{print $1}')
for f in $T2; do
echo "PID $f"
done
;;
?)
usage
exit
;;
esac
done
if [[ -z KILL ]] || [[ -z LIST ]] || [[ -z CHECK ]]
then
usage
exit 1
fi
and also I don't understand why when I call the script with no arguments the help doesn't show up
The PIDs will change if another program is restarting it when it's killed. This is actually pretty common with daemons.
usage is never called because you're checking whether the strings KILL etc. are empty, not the variables. Just add a dollar sign in front of them.