referred
Tried the following, to execute the below nohup commands inside a container,
nohup sh $KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone-mongo.properties $KAFKA_HOME/config/connect-mongo-sink.properties &
nohup /bin/bash $KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone-mongo.properties $KAFKA_HOME/config/connect-mongo-sink.properties &
And tried the below as well,
declare -a parameters=( "/opt/kafka/config/connect-standalone-mongo.properties" "/opt/kafka/config/connect-mongo-sink.properties")
for parameter in "${parameters[#]}"
do
echo "$1" > "/tmp/test02.txt"
nohup /bin/sh $KAFKA_HOME/bin/connect-standalone.sh -p $parameter 1> ${parameter}_nohup.out 2> ${parameter}_nohup.err &
#message received
echo "$1" > "/tmp/test03.txt"
done[referred the following link as well][1]
when I tried the below I am getting and empty out file,
$(nohup .$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone-mongo.properties $KAFKA_HOME/config/connect-mongo-sink.properties > out.txt &)
The below way of scripting works for me!!!!!!!!!
ex=' /opt/kafka/bin/connect-standalone.sh /opt/kafka/config/connect-standalone-mongo.properties /opt/kafka/config/connect-mongo-sink.properties > /opt/kafka/kafka-mongo-info.out 2> /opt/kafka/kafka-mongo-error.err &'
while true
do
# nohup $ex 1> nohup_kms.out 2> nohup_kms.err | tee -a /var/log/my_uber_script.log
# nohup $ex 1> nohup_kms.out 2> nohup_kms.err
done
you can try any one inside the loop
Related
I would like to run a bunch of nohup commands in succession and store the output in different files.
I came up with this:
$ nohup python3 foo.py 0 &> nohup-0-foo.out && \
nohup python3 foo.py 1 &> nohup-1-foo.out && \
nohup python3 foo.py 2 &> nohup-2-foo.out && \
nohup python3 foo.py 3 &> nohup-3-foo.out && \
nohup python3 foo.py 4 &> nohup-4-foo.out &
But this seems not to work (python3 foo.py 0 is executed, but it stop after this).
Q: How can I make the commands run in succession?
Couple ways.
One, throw them all in a file -
$: cat script
#!/bin/bash
for i in {0..4}; do python3 foo.py $i &> $i.out || break; done
And run that.
nohup ./script > script.out &
Two, if you just want to to it on the command line:
{ trap '' HUP; for i in {0..4}; do python3 foo.py $i &> 0.out || break; done; } &
or
{ trap '' HUP;
python3 foo.py 0 &> 0.out &&
python3 foo.py 1 &> 1.out &&
python3 foo.py 2 &> 2.out &&
python3 foo.py 3 &> 3.out &&
python3 foo.py 4 &> 4.out & # why running *this one* in background?
}
Basically what nohup does is trap hangup signals and make sure you have all your output logged, so there should be little difference.
I try to use following command to sent out 4 spark jobs in parallel, and wait for all of them finished before starting new step. However I notice the $cmd_trainSparkModelx commands are empty inside 'xargs'. How to pass them into xargs?
eval $cmd_prepare_step
xargs -P 4 -I {} sh -c 'eval "$1"' - {} <<'EOF'
#eval "$cmd_trainSparkModel1"
#eval "$cmd_trainSparkModel2"
#eval "$cmd_trainSparkModel3"
#eval "$cmd_trainSparkModel4"
echo "$cmd_trainSparkModel1"
echo "$cmd_trainSparkModel2"
echo "$cmd_trainSparkModel3"
echo "$cmd_trainSparkModel4"
EOF
echo "finished training"
eval $cmd_postTraining_step
following command works. But would still like to see how to make xargs work, as it can also control how many jobs to run at the same time.
echo "$cmd_trainSparkModel1" &
p1=&!
echo "$cmd_trainSparkModel2" &
p2=&!
echo "$cmd_trainSparkModel3" &
p3=&!
echo "$cmd_trainSparkModel4" &
p4=&!
wait $p1 $p2 $p3 $p4
I have this long process in a bash script (one of many) and so I'm experimenting with sending all output to a separate window to monitor, with any errors logged to errorlog.txt.
For example:
rsync -vahPz foo#bar:/bigfolder/ ./ >> /dev/pts/4 2>> errorlog.txt
Trouble is that above doesn't display any errors on the separate window.
Is there a way to redirect errors to both my separate window at /dev/pts/4 and errorlog.txt, while still redirecting normal output to /dev/pts/4 too?
Something like:
rsyncblah >> /dev/pts/4 2>> errorlog.txt && /dev/pts/4
You can use tee with process substitution, like this:
your_cmd 1> >(tee -a /dev/pts/2 >> out.log) 2> >(tee -a /dev/pts/2 >> err.log)
Alternatively, you can use the process substitution only for stderr - because it is needed - and redirect stdout, just as normal, through a pipe:
your_cmd 2> >(tee -a /dev/pts/2 >> err.log) | tee -a /dev/pts/2 >> out.log
bash Open new window for input/output
1. Create a new window and trapping his pts id
exec {BACKFD}<> <(:)
xterm -e bash -c "exec {CLOSEFD}<> <(:);
echo >/proc/$$/fd/$BACKFD \$\$ \$CLOSEFD \$(tty);
read -u \$CLOSEFD" &
read -u $BACKFD bpid bclose btty
same for using gnome-terminal:
exec {BACKFD}<> <(:)
gnome-terminal -- bash -c "exec {CLOSEFD}<> <(:);
echo >/proc/$$/fd/$BACKFD \$\$ \$CLOSEFD \$(tty);
read -u \$CLOSEFD" &
read -u $BACKFD bpid bclose btty
xterm -e is to be replaced by:
gnome-terminal -- # for gnome
mate-terminal -- # for mate
konsole -e # for KDE
and so on...
Note read -u \$CLOSEFD will ensure new window will stay open until we close them.
2. playing with new window
printf >$btty '%s\n' "Hello world!"
sleep 2
dialog --gauge >$btty <$btty 'Fill the tank' 20 60 < <(
for i in {0..100};do echo $i;sleep .02;done)
clear <$btty >$btty
if dialog --yesno >$btty <$btty 'Do this match?' 20 60
then echo yes
else echo no
fi
clear <$btty >$btty
answer=$(
dialog --menu 'Choose on of:' 20 60 12 a foo b bar c baz 2>&1 >$btty <$btty
)
echo You choose: $answer
clear <$btty >$btty
3. As requested, sharing output of a command in a log file AND new window:
exec {SHAREDOUT}> >(tee $btty >>/path/to/logfile)
exec {SHAREDERR}> >(tee $btty >>/path/to/errfile)
ls -ld /t{mp,nt} 2>&${SHAREDERR} >&${SHAREDOUT}
rsync -vahPz --log-file=/proc/self/fd/$SHAREDOUT 2>&${SHAREDERR} \
foo#bar:/bigfolder/. /destpath/.
4. Closing window:
echo >/proc/$bpid/fd/$bclose
I am trying to send the following line :
trap 'test -n "$SSH_AUTH_SOCK" && eval `/usr/bin/ssh-agent -k`' 0
to a file from Windows to WSL.
Here is what I've got so far :
bash -c "echo -e 'trap test -n \"\$SSH_AUTH_SOCK\" && eval \`/usr/bin/ssh-agent -k\` 0' >> $HOME/test"
I can't find a way to quote the trap argument. Thanks in advence for your help.
A colleague of mine found a solution : use a fonction
bash -c "echo -e 'onexit() { test -n \"\$SSH_AUTH_SOCK\" && eval \`/usr/bin/ssh-agent -k\`; }' > $HOME/testA"
bash -c "echo -e trap onexit 0 >> $HOME/testA"
I have a GNU screen named demo, I want to send commands to it. How do I do this?
screen -S demo -X /home/aa/scripts/outputs.sh
yeilds No screen session found.
and doing screen -ls shows that it isn't running.
If the Screen session isn't running, you won't be able to send things to it. Start it first.
Once you've got a session, you need to distinguish between Screen commands and keyboard input. screen -X expects a Screen command. The stuff command sends input, and if you want to run that program from a shell prompt, you'll have to pass a newline as well.
screen -S demo -X stuff '/home/aa/scripts/outputs.sh
'
Note that this may be the wrong approach. Are you sure you want to type into whatever is active in that session? To direct the input at a particular window, use
screen -S demo -p 1 -X stuff '/home/aa/scripts/outputs.sh
'
where 1 is the window number (you can use its title instead).
To start a new window in that session, use the screen command instead. (That's the screen Screen command, not the screen shell command.)
screen -S demo -p 1 -X screen '/home/aa/scripts/outputs.sh'
I put this together to capture the output from the commands. It also handles stdin if you want to pipe some input.
function xscreen {
# Usage: xscreen <screen-name> command...
local SCREEN_NAME=$1
shift
# Create screen if it doesn't exist
if ! screen -list | grep $SCREEN_NAME >/dev/null ; then
screen -dmS $SCREEN_NAME
fi
# Create I/O pipes
local DIR=$( mktemp -d )
local STDIN=$DIR/stdin
local STDOUT=$DIR/stdout
local STDERR=$DIR/stderr
mkfifo $STDIN $STDOUT $STDERR
trap 'rm -f $STDIN $STDOUT $STDERR; rmdir $DIR' RETURN
# Print output and kill stdin when both pipes are closed
{ cat $STDERR >&2 & cat $STDOUT & wait ; fuser -s -PIPE -k -w $STDIN ; } &
# Start the command (Clear line ^A^K, enter command with redirects, run with ^O)
screen -S $SCREEN_NAME -p0 -X stuff "$(echo -ne '\001\013') { $* ; } <$STDIN 1> >(tee $STDOUT) 2> >(tee $STDERR >&2)$(echo -ne '\015')"
# Forward stdin
cat > $STDIN
# Just in case stdin is closed
wait
}
Taking it a step further, it can be useful to call this function over ssh:
ssh user#host -n xscreen somename 'echo hello world'
Maybe combine it with something like ssh user#host "$(typeset -f xscreen); xscreen ..." so you don't have to have the function already defined on the remote host.
A longer version in a bash script that handles the return status and syntax errors:
#!/bin/bash
function usage {
echo "$(basename $0) [[user#]server:[port]] <screen-name> command..." >&2
exit 1
}
[[ $# -ge 2 ]] || usage
SERVER=
SERVERPORT="-p 22"
SERVERPAT='^(([a-z]+#)?([A-Za-z0-9.]+)):([0-9]+)?$'
if [[ "$1" =~ $SERVERPAT ]]; then
SERVER="${BASH_REMATCH[1]}"
[[ -n "${BASH_REMATCH[4]}" ]] && SERVERPORT="-p ${BASH_REMATCH[4]}"
shift
fi
function xscreen {
# Usage: xscreen <screen-name> command...
local SCREEN_NAME=$1
shift
if ! screen -list | grep $SCREEN_NAME >/dev/null ; then
echo "Screen $SCREEN_NAME not found." >&2
return 124
# Create screen if it doesn't exist
#screen -dmS $SCREEN_NAME
fi
# Create I/O pipes
local DIR=$( mktemp -d )
mkfifo $DIR/stdin $DIR/stdout $DIR/stderr
echo 123 > $DIR/status
trap 'rm -f $DIR/{stdin,stdout,stderr,status}; rmdir $DIR' RETURN
# Forward ^C to screen
trap "screen -S $SCREEN_NAME -p0 -X stuff $'\003'" INT
# Print output and kill stdin when both pipes are closed
{
cat $DIR/stderr >&2 &
cat $DIR/stdout &
wait
[[ -e $DIR/stdin ]] && fuser -s -PIPE -k -w $DIR/stdin
} &
READER_PID=$!
# Close all the pipes if the command fails to start (e.g. syntax error)
{
# Kill the sleep when this subshell is killed. Ugh.. bash.
trap 'kill $(jobs -p)' EXIT
# Try to write nothing to stdin. This will block until something reads.
echo -n > $DIR/stdin &
TEST_PID=$!
sleep 2.0
# If the write failed and we're not killed, it probably didn't start
if [[ -e $DIR/stdin ]] && kill $TEST_PID 2>/dev/null; then
echo 'xscreen timeout' >&2
wait $TEST_PID 2>/dev/null
# Send ^C to clear any half-written command (e.g. no closing braces)
screen -S $SCREEN_NAME -p0 -X stuff $'\003'
# Write nothing to output, triggers SIGPIPE
echo -n 1> $DIR/stdout 2> $DIR/stderr
# Stop stdin by creating a fake reader and sending SIGPIPE
cat $DIR/stdin >/dev/null &
fuser -s -PIPE -k -w $DIR/stdin
fi
} &
CHECKER_PID=$!
# Start the command (Clear line ^A^K, enter command with redirects, run with ^O)
screen -S $SCREEN_NAME -p0 -X stuff "$(echo -ne '\001\013') { $* ; echo \$? > $DIR/status ; } <$DIR/stdin 1> >(tee $DIR/stdout) 2> >(tee $DIR/stderr >&2)$(echo -ne '\015')"
# Forward stdin
cat > $DIR/stdin
kill $CHECKER_PID 2>/dev/null && wait $CHECKER_PID 2>/dev/null
# Just in case stdin is closed early, wait for output to finish
wait $READER_PID 2>/dev/null
trap - INT
return $(cat $DIR/status)
}
if [[ -n $SERVER ]]; then
ssh $SERVER $SERVERPORT "$(typeset -f xscreen); xscreen $#"
RET=$?
if [[ $RET == 124 ]]; then
echo "To start screen: ssh $SERVER $SERVERPORT \"screen -dmS $1\"" >&2
fi
exit $RET
else
xscreen "$1" "${#:2}"
fi