I used the method system to start a process. the pid of that process is being stored in a file worker.pid
however I need to generate the log of this process, how can I store the output of this process?
the process is being created with this command:
system "bundle exec rake resque:work >> ./resque.log QUEUE=* PIDFILE=#{pid_file} &"
P.S.: I am using ruby 1.8, BACKGROUND=yes won`t work.
P.S.2: platform linux
Maybe what you're looking for is IO.popen
This lets you fork off a subprocess and access it's output via an IO object
# fork off a one-off task
# and return the output as a string
ls = IO.popen("ls")
ls.read
# or return an array of lines
IO.popen("ls").readlines
# or create a continuing task
tail = IO.popen("tail -f /some/log/file.log")
loop do
puts tail.gets
end
I suggest you read the documentation,
but you can also write to the stream, and do all sorts of clever stuff.
If I'm understanding what you are trying to achieve correctly, you are looking for the Open3 class. http://www.ruby-doc.org/stdlib-1.8.7/libdoc/open3/rdoc/Open3.html
Related
I'm scanning a folder for audio files and converting them to mp3.
Works great in RUBY.
However, once the first transcoding is done, it stops the whole loop
Here's a breakdown of my code process.
def scanFolder
# lots of code above to get folder list, check for incorrect files etc..
audioFileList.each {
|getFile|
exec_command = "ffmpeg #{getFile} #{newFileName}"
exec exec_command
}
end
What's happening is that it's transcoding the first file it finds, then it stops the whole function. Is there a way to force it to continue?
The ffmpeg does run and finish correctly at the moment, so it's not breaking anything
exec replaces the current process by running the given command. Example:
2.0.0-p598 :001 > exec 'echo "hello"'
hello
shivam#bluegene:$
You can see how exec replaces the irb with system echo which then exits automatically.
Therefore try using system instead. Here the same example using system:
2.0.0-p598 :003 > system 'echo "hello"'
hello
=> true
2.0.0-p598 :004 >
You can see I am still in irb and its not exited after executing the command.
This makes your code as follows:
def scanFolder
# lots of code above to get folder list, check for incorrect files etc..
audioFileList.each {
|getFile|
exec_command = "ffmpeg #{getFile} #{newFileName}"
system exec_command
}
end
Along with shivam's answer about using system, the spawn method may also be useful here:
http://ruby-doc.org//core-2.1.5/Process.html#method-c-spawn
I have a shell script named test.sh. How can I trigger the test.sh from Ruby?
I want test.sh to run as a background process, what means in Ruby it is a ansync call.
STDERR and STDOUT also need to be written to a specific file.
Any ideas?
#TanzeebKhalili's answer works, but you might consider Kernel.spawn(), which doesn't wait for the process to return:
pid = spawn("./test.sh")
Process.detach(pid)
Note that, according to the documentation, whether you use spawn() or manually fork() and system(), you should grab the PID and either Process.detach() or Process.wait() before exiting.
Regarding redirecting standard error and output, that's easy with spawn():
pid = spawn("./test.sh", :out => "test.out", :err => "test.err")
Process.detach(pid)
Try this:
Process.fork { system "./test.sh" }
Won't work on windows, for which you can use threading.
I think IO.popopen also deserves a mention here. Something like this would append the output from multiple runs of the command. to stdout.log and stderr.log
open('stdout.log', 'a') { |file|
file.puts(
IO.popen(["./test.sh"], :err => ["stderr.log", "a"]) { |result|
result.read
}
)
end
Pretty new to Jenkins and I have simple yet annoying problem. When I run job (Build) on Jenkins I am triggering ruby command to execute my test script.
Problem is Jenkins is not displaying output in real time from console. Here is trigger log.
Building in workspace /var/lib/jenkins/workspace/foo_bar
No emails were triggered.
[foo_bar] $ /bin/sh -xe /tmp/hudson4042436272524123595.sh
+ ruby /var/lib/jenkins/test-script.rb
Basically it hangs on this output until build is complete than it just shows full output. Funny thing is this is not consistent behavior, sometimes it works as it should. But most of the time there is no real time console output.
Jenkins version: 1.461
To clarify some of the answers.
ruby or python or any sensible scripting language will buffer the output; this is in order to minimize the IO; writing to disk is slow, writing to a console is slow...
usually the data gets flush()'ed automatically after you have enough data in the buffer with special handling for newlines. e.g. writing a string without newline then sleep() would not write anything until after the sleep() is complete (I'm only using sleep as an example, feel free to substitute with any other expensive system call).
e.g. this would wait 8 seconds, print one line, wait 5 more seconds, print a second line.
from time import sleep
def test():
print "ok",
time.sleep(3)
print "now",
time.sleep(5)
print "done"
time.sleep(5)
print "again"
test()
for ruby, STDOUT.sync = true, turns the autoflush on; all writes to STDOUT are followed by flush(). This would solve your problem but result in more IO.
STDOUT.sync = true
for python, you can use python -u or the environment variable PYTHONUNBUFFERED to make stdin/stdout/stout not buffered, but there are other solutions that do not change stdin or stderr
export PYTHONUNBUFFERED=1
for perl, you have autoflush
autoflush STDOUT 1;
Make sure your script is flushing its stdout and stderr.
In my case I had a buffering issue similar to what you describe but I was using python.
The following python code fixed it for me:
import sys
sys.stdout.flush()
I'm not a Ruby coder, but Google reveals the following:
$stdout.flush
It seems to me that python -u works as well.
E.g. In batch command
python -u foo.py
Easiest solution here is to turn on syncing buffer to output. Something that #Craig wrote about in his answer but one line solution that will cover whole script, and not require you to flush buffer many times.
Just write
STDOUT.sync = true
Logic behind is simple, to avoid using IO operations many times output is buffered. To disable this use
STDOUT.sync = false
This is Ruby solution ofc.
Each of the other answers is specific to one program or another, but I found a more general solution here:
https://unix.stackexchange.com/a/25378
You can use stdbuf to alter the buffering behavior of any program.
In my case, I was piping output from a shell script through tee and grep to split lines into either the console or a file based on content. The console was hanging as described by OP. This solved it:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | stdbuf -oL -eL grep LOG:
Eventually I discovered I could just pass --line-buffered to grep for the same result:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | grep --line-buffered LOG:
The other answers are correct in saying that you need to ensure standard output is not buffered.
The other thing to be aware of is that Jenkins itself does line by line buffering. If you have a slow-running process that emits single characters (for example, an nunit test suite summary that prints a . for a successful test and an E for an error) you will not see anything until the end of line.
[True for my Jenkins 1.572 running on a Windows box.]
For some commands, including tee a the best choice for unbuffering is a program called unbuffer from expect package.
Usage example:
instead of
somecommand | tee /some/path
do
somecommand | unbuffer -p tee /some/path
Sources and more info:
https://stackoverflow.com/a/11337310/2693875
https://unix.stackexchange.com/a/25375/53245
The Operating-System is buffering output-data by nature, to save CPU, and so does Jenkins.
Looks like you are using a shell-command to run your Ruby script -
I suggest running your Ruby script directly via the dedicated plugin:
Jenkins Ruby Plugin
(may need to install it)
Python buffered its output traces and print it at the end of script to minimize writing on console as writing to console is slow.
You can use following command after your traces. It will flush all traces to console, which are queued before that command.
sys.stdout.flush()
I have one simple script:
fork do
STDOUT.reopen(File.open('/tmp/log', 'w+'))
STDOUT.sync = true
exec 'bundle exec ruby script.rb'
end
script.rb:
loop do
sleep 1
puts "MESSAGE"
end
When it work, all outputs is buffering(?) and writes to /tmp/log by big pices.
It works only if I modify script:
$stdout.puts "MESSAGE"
$stdout.flush
How can I do the same without modifying script.rb ?
Thanks.
When you call exec, you create a new process, and although this process inherits the file you set as standard out, it doesn't inherit the other settings, in particular the sync setting.
In order to get unbuffered output in the new process, you need to set it in that process. If you don't want to modify script.rb one workaround could be to create another file, named somethig like sync.rb containing just:
STDOUT.sync = true
which you can then require when running your command:
exec 'bundle exec ruby -r./sync script.rb'
The new Ruby process will now require sync.rb, which simply sets sync mode on STDOUT to true before executing your script.
I have a bunch of Ruby scripts which are long running, I'd like to ensure that every 30 seconds or so these are up.
I normally start the commands by simply ruby script-name.rb
How can I configure monit to look after these scripts?
Update: I tried to follow this method to create a wrapper script which would then launch the ruby process but it did not seem to create the .pid file and typing './wrapper-script stop' did nothing :/
Should I write the pid inside ruby or use a wrapper script to create the pid necessary for monit?
The Monit Wiki has a lot of configuration examples:
http://mmonit.com/wiki/Monit/ConfigurationExamples
Just pick a simple one and modify it according to your needs.
Update: the wrapper script should create the pid for you in this line:
echo $$ > /var/run/xyz.pid;
Did you adapt the script to your needs? Is it executable (chmod +x)? Does it have write permissions for the destination? Maybe post the wrapper you are trying to use so I can help you more.
You don't need to write a wrapper script or try any kind of black magic, just use the Daemons library and you're done.
Imagine that you have a class Worker that has a method "run" that enters an infinite loop reading from a socket or anything like that, here's how you'd write your Daemons wrapper:
# this is file my_worker_control.rb
require 'rubygems'
require 'daemons'
require 'worker'
Daemons.run_proc(:app_name => 'my_worker', :dir_mode => :system, :log_output => true ) do
Worker.run
end
Once the script is done, just call it from your command line or an init.d script:
my_worker_control.rb run|start|stop|restart
This config will generate a "my_worker.pid" file under "/var/run" and you can use monit to watch over the process by using this file.
Modify the file :
/etc/init.d/skeleton
You will need to slightly modify it, and then :
chmod +x /etc/init.d/process_name
sudo update-rc.d process_name defaults
sudo /etc/init.d/process_name (start| stop| reload )
Now just use Monit with the pid at /var/run/process.pid
start location : sudo /etc/init.d/process start
stop location : sudo /etc/init.d/process stop
Cheers
Writing the pid file in your ruby script may be easiest for you (just open a file and write $$ in it). That said, the wrapper script approach should work fine. Does your script have permission to write to a file in /var/run (or wherever you are putting the pidfile)?
As an alternative (to monit), have a look at bluepill.
(Surely out of subject but) as it is about ruby, why don't you use : http://god.rubyforge.org/ ?
Add this line to your ruby script yourapp.rb, that creates a pid file named yourapp.pid
File.open('/somepath/yourapp.pid', 'w') {|f| f.write Process.pid }
Configure Monit to check for the pid in /etc/monit/conf.d/yourapp
check process yourapp with pidfile /somepath/yourapp.pid