Pipe stdout and stderr through ssh - bash

Consider the following example:
{ echo 1 | tee /dev/stderr 2> >(sed -e 's|1|err|' >&2) 1> >(sed -e 's|1|out|') ; }
which prints
out
err
Note that out out is printed on stdout and err on stderr.
Question: How to do this remotely via ssh?
More precisely, how to run
ssh host 'echo 1 | tee /dev/stderr SOME_MAGIC_HERE'
st. again out/err pops up on stdout/stderr (for an appropriate bash magic SOME_MAGIC_HERE).
Clearly, the following works:
ssh host 'echo 1 | tee /dev/stderr' 2> >(sed -e 's|1|err|' >&2) 1> >(sed -e 's|1|out|')
But that executes sed locally, and I'd rather want to do that remotely on host.

after the update:
ssh host 'echo 1 | tee >(cat - | sed -e "s|1|err|" >&2) | sed -e "s|1|out|"'
out
err
The idea is to use <pipes> | for processing /dev/stdout and use process substitution in combination with tee to create the /dev/stderr part.
Now it works as expected:
$ ssh host 'echo 1 | tee >(cat - | sed -e "s|1|err|" >&2) | sed -e "s|1|out|"' > /dev/null
err
$ ssh host 'echo 1 | tee >(cat - | sed -e "s|1|err|" >&2) | sed -e "s|1|out|"' 2> /dev/null
out
original answer:
The following command executes by changing your <single quotes> into <double quotes> :
ssh host 'echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|")'
but this has everything in /dev/stdout. Example:
$ ssh host 'echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|")' > /dev/null
$ ssh host 'echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|")' 2> /dev/null
out
err
and this is exactly what your original command does on the host system:
{ echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|") ; } >/dev/null
{ echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|") ; } 2>/dev/null
out
err
The ssh program normally handles the passing of /dev/stdout,/dev/stderr and /dev/stdin correctly:
$ ssh host "echo 1; echo 2 > /dev/stderr" > /dev/null
2
$ ssh host "echo 1; echo 2 > /dev/stderr" 2> /dev/null
1

Related

How to use nohup with curly braces?

I try to run the following command (ref.) using nohup, which basically separates stdout and stderr into two processes.
{ foo 2>&1 1>&3 3>&- | sed -u 's/^/err: /'; } 3>&1 1>&2 | sed -u 's/^/out: /'
The foo script is like below.
#!/bin/bash
while true; do
echo a
echo b >&2
sleep 1
done
This is the test result.
$ nohup { foo 2>&1 1>&3 3>&- | sed -u 's/^/err: /'; } 3>&1 1>&2 | sed -u 's/^/out: /' >/dev/null 2>&1 &
-bash: syntax error near unexpected token `}'
That's syntatically impossible. But you can wrap your {} in a sh -c cmd:
nohup sh -c 'foo 2>&1 1>&3 3>&- | sed -u "s/^/err: /"'
Notice I change the single quote for sed to double quote.

shell script to create text using data

I have one file that contains data like this
User= yojpackco
Domains=yojpack.com
User= yugmaimpre
Domains=yugmaimpressions.com
User= yvmarathej
Domains=yvmarathejewellers.com
User= zawargauge
Domains=zawargauges.com
User= zealservi
Domains=zeal-services.com
User= zenithwor
Domains=zenith-worldwide.com
I want to create a txt like this using the first data where domains will replace actual URL and user will replace User FROM ABOVE
echo 'Sitemap: https://Domains/sitemap.xml' | tee -a /home/User/public_html/robots.txt >/dev/null
echo 'Sitemap: https://yojpack.com/sitemap.xml' | tee -a /home/yojpackco/public_html/robots.txt >/dev/null
echo 'Sitemap: https://yugmaimpressions.com/sitemap.xml' | tee -a /home/yugmaimpre/public_html/robots.txt >/dev/null
echo 'Sitemap: https://yvmarathejewellers.com/sitemap.xml' | tee -a /home/zawargauge/public_html/robots.txt >/dev/null
echo 'Sitemap: https://zawargauges.com/sitemap.xml' | tee -a /home/zealservi/public_html/robots.txt >/dev/null
echo 'Sitemap: https://zenith-worldwide.com/sitemap.xml' | tee -a /home/zenithwor/public_html/robots.txt >/dev/null
while read -r user domain; do
echo "echo "Sitemap: https://${domain}/sitemap.xml" | tee -a /home/"${user}"/public_html/robots.txt"
done < <(sed -E 's/^.*=[ ]?//;/^ *$/d' INPUT_FILE|paste - -)
# echo output
echo Sitemap: https://yojpack.com/sitemap.xml | tee -a /home/yojpackco/public_html/robots.txt
echo Sitemap: https://yugmaimpressions.com/sitemap.xml | tee -a /home/yugmaimpre/public_html/robots.txt
echo Sitemap: https://yvmarathejewellers.com/sitemap.xml | tee -a /home/yvmarathej/public_html/robots.txt
echo Sitemap: https://zawargauges.com/sitemap.xml | tee -a /home/zawargauge/public_html/robots.txt
echo Sitemap: https://zeal-services.com/sitemap.xml | tee -a /home/zealservi/public_html/robots.txt
echo Sitemap: https://zenith-worldwide.com/sitemap.xml | tee -a /home/zenithwor/public_html/robots.txt

Redirect output to mulitple files with tee and grep

I spent a lot of hours to get this to run:
redirecting output from a script STDOUT + STDERR toLogfile 1 and a grep to Logfile 2
The First logfile should contain the complete output, and the second logfile only Start and End-Lines (grep).
I tried different syntax, but nothing works.
./run.sh 2>&1 | tee -a /var/log/log1.log | (grep 'START|END') > /var/log/myscripts.log
./run.sh 2>&1 | tee -a /var/log/log1.log | grep 'Start' > /var/log/myscripts.log
./run.sh 2>&1 | tee -a /var/log/log1.log | egrep 'Start' > /var/log/myscripts.log
./run.sh 2>&1 | tee -a /var/log/log1.log | grep -E 'Start' > /var/log/myscripts.log
the output will be redirected only to the first log. The second log is empty.
I don't know why; do you have any ideas?
Example-Lines from Output
this should be complete in the log1.log
(the script is java startet via shell script)
26.09.2014 20:38:51 | start script > load_stats.sh
26.09.2014 20:38:51 | [DB DATA]
26.09.2014 20:38:51 | Host > locahost
26.09.2014 20:38:51 | User > leroy
... more ...
26.09.2014 20:39:23 | fin script > load_stats.sh
I want to grep this in myscripts.log
26.09.2014 20:38:51 | start script > load_stats.sh
26.09.2014 20:39:23 | fin script > load_stats.sh
I think the problem is the format, timestamp, whitespaces.
I thought grep 'word' will catch me this both lines, but it doesn't.
Stupid.
./run.sh 2>&1 | tee -a /var/log/log1.log | sed -nE '/(start script|end script)/p' >> /var/log/myscripts.log
did not work, log1 ok, mysrctips.log empty
tail -f -n 500 /var/log/log1.log | sed -nE '/(start script|end script)/p'
works well in the shell. but in combination of all it doesn't.
execute a script > redirect to log 1 > redirect and filter(grep,egrep,sed,..) to log 2
This works fine for me:
$ cat <<_DATA | tee out1 | grep -E 'START|END' > out2
hello
START1
foo
END2
bar
_DATA
$ cat out1
hello
START1
foo
END2
bar
$ cat out2
START1
END2
This should work
./run_test.sh 2>&1 | tee -a /var/log/log1.log | grep -E 'start script|fin Main-Job' > /var/log/myscripts.log
# ^^ append ^^^^^^^ - same as egrep ^overwrite/create
prints
26.09.2014 20:38:51 | start script > load_stats.sh
26.09.2014 20:39:23 | fin Main-Job > load_stats.sh
if want
overwrite/create the log1 - delete the -a
append to myscripts.log use >>
the egrep is deprecated - therefore is better to use grep -E
Also, instead of the grep you can use sed too, like:
sed -nE '/(start script|fin Main-Job)/p'

Bash - redirect stdout to log and screen with stderr only to log

I would like to do the following;
Redirect a copy of stdout to logfile and keep stdout on the screen.
Redirect stderr to the same logfile and not display on the screen.
Code without stdout to screen:
#!/bin/bash
exec 1> >(sed -u 's/^/INF: /' >> common.log)
exec 2> >(sed -u 's/^/ERR: /' >> common.log)
echo "some txt"
echo "an error" >&2
echo "some more txt"
echo "one more error" >&2
Log:
INF: some txt
INF: some more txt
ERR: an error
ERR: one more error
The first issue is buffering which I tried to negate with sed '-u' for unbuffered.
Code with stdout to screen:
#!/bin/bash
exec 1> >(sed -u 's/^/INF: /' | tee -a common.log)
exec 2> >(sed -u 's/^/ERR: /' >> common.log)
echo "some txt"
echo "an error" >&2
echo "some more txt"
echo "one more error" >&2
Results in the screen hanging (had to Ctrl-C) and log still buffered. Suggestions?
Does this work for you?
command 2> >(sed -u 's/^/ERR: /' >> common.log) | sed -u 's/^/INF: /' | tee -a common.log
Where command is your command.

tee and pipelines inside a bash script

i need to redirect stout and stderr in bash each to separate file.
well i completed this command:
((/usr/bin/java -jar /opt/SEOC2/seoc2.jar 2>&1 1>&3 | tee --append /opt/SEOC2/log/err.log) 3>&1 1>&2 | tee --append /opt/SEOC2/log/app.log) >> /opt/SEOC2/log/combined.log 2>&1 &
which works fine running from a command line.
trying to put the very same command into bash script
...
12 cmd="(($run -jar $cmd 2>&1 1>&3 | tee --append $err) 3>&1 1>&2 | tee --append $log) >> $combined 2>&1"
...
30 echo -e "Starting servis..."
31 $cmd &
32 pid=`ps -eo pid,args | grep seoc2.jar | grep -v grep | cut -c1-6`
33 if [ ! -z $pid ]; then
...
leads to error like this:
root#operator:/opt/SEOC2# seoc2 start
Starting servis...
/usr/local/bin/seoc2: line 31: ((/usr/bin/java: dir or file doesn't exist
tried to cover this command by $( ), ` ` etc but with no effect at all :(
any suggestion or advice would be very appreciated, playing around for hours already :/
thanx a lot
Rene
If you store the whole command line in a variable you have to use eval to execute it:
cmd="(($run -jar $cmd 2>&1 1>&3 | tee --append $err) 3>&1 1>&2 | tee --append $log) >> $combined 2>&1"
...
eval $cmd &

Resources