I am running the following (backup) code in a Ruby script:
for directory in directories
print `s3sync.rb --exclude="#{EXCLUDE_REGEXP}" --delete --progress -r #{directory} my.amazon.backup.bucket:#{directory}`
end
I would like the output of the executed subprocess to be echoed to the terminal in real time (as opposed to having to wait until the subprocess returns). How do I do that?
IO.popen creates a process and returns an IO object for stdin and stdout of this process.
IO.popen("s3sync.rb …").each do |line|
print line
end
If you don't need for your code to see stdout, and it's sufficient that a human sees it, than system is fine. If you need your code to see it, there are numerous solutions, popen being the simplest, giving your code access to stdout, and Open3 giving your code access to both stdout and stderr. See: Ruby Process Management
Oops, figured it out right away. I had to use exec() instead of ``
This does what I want:
for directory in directories
exec `s3sync.rb --exclude="#{EXCLUDE_REGEXP}" --delete --progress -r #{directory} my.amazon.backup.bucket:#{directory}`
end
Related
I have a shell script which writes all output to logfile
and terminal, this part works fine, but if I execute the script
a new shell prompt only appear if I press enter. Why is that and how do I fix it?
#!/bin/bash
exec > >(tee logfile)
echo "output"
First, when I'm testing this, there always is a new shell prompt, it's just that sometimes the string output comes after it, so the prompt isn't last. Did you happen to overlook it? If so, there seems to be a race where the shell prints the prompt before the tee in the background completes.
Unfortunately, that cannot fixed by waiting in the shell for tee, see this question on unix.stackexchange. Fragile workarounds aside, the easiest way to solve this that I see is to put your whole script inside a list:
{
your-code-here
} | tee logfile
If I run the following script (suppressing the newline from the echo), I see the prompt, but not "output". The string is still written to the file.
#!/bin/bash
exec > >(tee logfile)
echo -n "output"
What I suspect is this: you have three different file descriptors trying to write to the same file (that is, the terminal): standard output of the shell, standard error of the shell, and the standard output of tee. The shell writes synchronously: first the echo to standard output, then the prompt to standard error, so the terminal is able to sequence them correctly. However, the third file descriptor is written to asynchronously by tee, so there is a race condition. I don't quite understand how my modification affects the race, but it appears to upset some balance, allowing the prompt to be written at a different time and appear on the screen. (I expect output buffering to play a part in this).
You might also try running your script after running the script command, which will log everything written to the terminal; if you wade through all the control characters in the file, you may notice the prompt in the file just prior to the output written by tee. In support of my race condition theory, I'll note that after running the script a few times, it was no longer displaying "abnormal" behavior; my shell prompt was displayed as expected after the string "output", so there is definitely some non-deterministic element to this situation.
#chepner's answer provides great background information.
Here's a workaround - works on Ubuntu 12.04 (Linux 3.2.0) and on OS X 10.9.1:
#!/bin/bash
exec > >(tee logfile)
echo "output"
# WORKAROUND - place LAST in your script.
# Execute an executable (as opposed to a builtin) that outputs *something*
# to make the prompt reappear normally.
# In this case we use the printf *executable* to output an *empty string*.
# Use of `$ec` is to ensure that the script's actual exit code is passed through.
ec=$?; $(which printf) ''; exit $ec
Alternatives:
#user2719058's answer shows a simple alternative: wrapping the entire script body in a group command ({ ... }) and piping it to tee logfile.
An external solution, as #chepner has already hinted at, is to use the script utility to create a "transcript" of your script's output in addition to displaying it:
script -qc yourScript /dev/null > logfile # Linux syntax
This, however, will also capture stderr output; if you wanted to avoid that, use:
script -qc 'yourScript 2>/dev/null' /dev/null > logfile
Note, however, that this will suppress stderr output altogether.
As others have noted, it's not that there's no prompt printed -- it's that the last of the output written by tee can come after the prompt, making the prompt no longer visible.
If you have bash 4.4 or newer, you can wait for your tee process to exit, like so:
#!/usr/bin/env bash
case $BASH_VERSION in ''|[0-3].*|4.[0-3]) echo "ERROR: Bash 4.4+ needed" >&2; exit 1;; esac
exec {orig_stdout}>&1 {orig_stderr}>&2 # make a backup of original stdout
exec > >(tee -a "_install_log"); tee_pid=$! # track PID of tee after starting it
cleanup() { # define a function we'll call during shutdown
retval=$?
exec >&$orig_stdout # Copy your original stdout back to FD 1, overwriting the pipe to tee
exec 2>&$orig_stderr # If something overwrites stderr to also go through tee, fix that too
wait "$tee_pid" # Now, wait until tee exits
exit "$retval" # and complete exit with our original exit status
}
trap cleanup EXIT # configure the function above to be called during cleanup
echo "Writing something to stdout here"
So I want to put a file onto s3. Here is the cmd:
/usr/bin/s3cmd --rr --access_key="$access_key" --secret_key="$secret_key" put "$FILEPATH/$ZIPPED_FILE" "$s3_path/$ZIPPED_FILE"
And this works perfectly except in my bash shell, it prints out this message: upload:: command not found. Anyone encounter this?
This almost certainly means you're running the stdout of s3cmd as a command itself. For instance, that could happen if you were to run:
# BAD: runs the command, then runs its output as another command
`/usr/bin/s3cmd --rr --access_key="$access_key" --secret_key="$secret_key" put "$FILEPATH/$ZIPPED_FILE" "$s3_path/$ZIPPED_FILE"`
To fix that, just take the backticks out, and write:
# GOOD: just invokes your command, with its output written to stdout
/usr/bin/s3cmd --rr --access_key="$access_key" --secret_key="$secret_key" put "$FILEPATH/$ZIPPED_FILE" "$s3_path/$ZIPPED_FILE"
I have a script which needs to run in screen so I included
#!/usr/bin/screen /bin/bash
as the hash bang and it works great. The only problem is that when the script crashes I don't know what happened, the output is lost and all I know is that screen terminated.
My script is interactive so I need to see stdout and stderr in the terminal and I also want stdout and stderr logged in case it crashed.
I tried to run the script like
./test-screen-in-bash.sh 2>&1|tee test1.log
which results in an empty test1.log file
Can somebody please explain what am I doing wrong.
Thanks to #JID comments I was able to find what i was looking for.
I removed the screen from hash bang and used the method from the link provided by #JID here in the first answer.
I ended up with
#!/bin/bash
if [ -z "$STY" ]; then exec screen -L /bin/bash "$0"; fi
./myscript.sh
Now when I run the above, myscript.sh runs in screen and the whole output from the session is dumped to screenlog.n files.
Similar to redirect COPY of stdout to log file from within bash script itself, but I'd also like to preserve stdout as a TTY device.
For example, I have the following scripts:
/tmp/teed-off$ cat some-script
#!/usr/bin/env ruby
if $stdout.tty?
puts "stdout is a TTY"
else
puts "stdout is NOT a TTY"
end
/tmp/teed-off$ cat wrapper
#!/usr/bin/env bash
exec > >(tee some-script.log)
./some-script
When I run them, the wrapper eats stdout as a TTY device:
/tmp/teed-off$ ./some-script
stdout is a TTY
/tmp/teed-off$ ./wrapper
stdout is NOT a TTY
How can I flip that behavior around so that the script believes that its in a TTY even when executed via the wrapper?
It won't be trivial, but I think you can do it via pseudo-ttys. I'm not sure that there's any standard tool, other than perhaps expect, that would do it for you.
It takes a bit of thinking about. You'd have a control program that would open the pseudo-tty master, then the slave. The slave would be connected to the output of ./some-script. The master would be read by the control program, which would copy the data it reads from the master to the file and to standard output.
I've not tried coding that up. I'm not sure whether you could do it with standard shell commands; I can't think of any way. So, I think there will be some C coding to be done.
look for dup2 it duplicates a file descriptor
int dup2(int oldfd, int newfd);
Pretty new to Jenkins and I have simple yet annoying problem. When I run job (Build) on Jenkins I am triggering ruby command to execute my test script.
Problem is Jenkins is not displaying output in real time from console. Here is trigger log.
Building in workspace /var/lib/jenkins/workspace/foo_bar
No emails were triggered.
[foo_bar] $ /bin/sh -xe /tmp/hudson4042436272524123595.sh
+ ruby /var/lib/jenkins/test-script.rb
Basically it hangs on this output until build is complete than it just shows full output. Funny thing is this is not consistent behavior, sometimes it works as it should. But most of the time there is no real time console output.
Jenkins version: 1.461
To clarify some of the answers.
ruby or python or any sensible scripting language will buffer the output; this is in order to minimize the IO; writing to disk is slow, writing to a console is slow...
usually the data gets flush()'ed automatically after you have enough data in the buffer with special handling for newlines. e.g. writing a string without newline then sleep() would not write anything until after the sleep() is complete (I'm only using sleep as an example, feel free to substitute with any other expensive system call).
e.g. this would wait 8 seconds, print one line, wait 5 more seconds, print a second line.
from time import sleep
def test():
print "ok",
time.sleep(3)
print "now",
time.sleep(5)
print "done"
time.sleep(5)
print "again"
test()
for ruby, STDOUT.sync = true, turns the autoflush on; all writes to STDOUT are followed by flush(). This would solve your problem but result in more IO.
STDOUT.sync = true
for python, you can use python -u or the environment variable PYTHONUNBUFFERED to make stdin/stdout/stout not buffered, but there are other solutions that do not change stdin or stderr
export PYTHONUNBUFFERED=1
for perl, you have autoflush
autoflush STDOUT 1;
Make sure your script is flushing its stdout and stderr.
In my case I had a buffering issue similar to what you describe but I was using python.
The following python code fixed it for me:
import sys
sys.stdout.flush()
I'm not a Ruby coder, but Google reveals the following:
$stdout.flush
It seems to me that python -u works as well.
E.g. In batch command
python -u foo.py
Easiest solution here is to turn on syncing buffer to output. Something that #Craig wrote about in his answer but one line solution that will cover whole script, and not require you to flush buffer many times.
Just write
STDOUT.sync = true
Logic behind is simple, to avoid using IO operations many times output is buffered. To disable this use
STDOUT.sync = false
This is Ruby solution ofc.
Each of the other answers is specific to one program or another, but I found a more general solution here:
https://unix.stackexchange.com/a/25378
You can use stdbuf to alter the buffering behavior of any program.
In my case, I was piping output from a shell script through tee and grep to split lines into either the console or a file based on content. The console was hanging as described by OP. This solved it:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | stdbuf -oL -eL grep LOG:
Eventually I discovered I could just pass --line-buffered to grep for the same result:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | grep --line-buffered LOG:
The other answers are correct in saying that you need to ensure standard output is not buffered.
The other thing to be aware of is that Jenkins itself does line by line buffering. If you have a slow-running process that emits single characters (for example, an nunit test suite summary that prints a . for a successful test and an E for an error) you will not see anything until the end of line.
[True for my Jenkins 1.572 running on a Windows box.]
For some commands, including tee a the best choice for unbuffering is a program called unbuffer from expect package.
Usage example:
instead of
somecommand | tee /some/path
do
somecommand | unbuffer -p tee /some/path
Sources and more info:
https://stackoverflow.com/a/11337310/2693875
https://unix.stackexchange.com/a/25375/53245
The Operating-System is buffering output-data by nature, to save CPU, and so does Jenkins.
Looks like you are using a shell-command to run your Ruby script -
I suggest running your Ruby script directly via the dedicated plugin:
Jenkins Ruby Plugin
(may need to install it)
Python buffered its output traces and print it at the end of script to minimize writing on console as writing to console is slow.
You can use following command after your traces. It will flush all traces to console, which are queued before that command.
sys.stdout.flush()