I am writing an application, which shall work with networks.
As a GUI I am using rubyqt.
To determine if a Server is up I have to ping it (with net/ping).
But I ran in to a problem. If the server is down,
the GUI freezes for the timeout, even if I put the code in a Thread or IO.popen loop eg.
Thread.new('switch') do
if Net::PingExternal.new("195.168.255.244",timeout=0.9).ping then
down = false
else
down = true
end
end
will freeze for 0.9 seconds. As the QtThreads are not yet working with rubyqt,
does somebody have an idea to make the GUI don't freeze (apart from reducing the timeout)?
I was thinking about putting the pinging-part in an external program, which writes the status (up/down) in a file, which the actual program then reads, but this solution seems to be a bit clumsy.
Have you considered abstracting that operation from the request altogether? If you move the costly operation to an external library you could easily queue it up and execute it using something like delayed_job (http://github.com/tobi/delayed_job/tree/master) which would remove the risk of it halting the request at all.
Maybe this is what you are looking for...?
Related
Say I want to write a tail like application for Windows to monitor a bunch of files. Such an application should report when some of the monitored files is updated by any other application.
It can be assumed that the files being monitored are being constantly appended by other processes, but not modified in any other way. Before implementing some pooling solution (that is, iterate through the files to be monitored, seek to the end of each one, record this pointer, compare to previous end etc.) I would appreciate if someone more experienced with the Overlapped IO could tell me if I can make use of it.
For instance, is it possible to write the monitoring application in such a way that it opens all the files that need to be monitored, seek to the end of them, and try to read one byte with ReadFileEx() registering a callback.
Is there a way to make this work so that when another process write to some of the files the proper callback is invoked? Or necessarily the monitoring application will always get an EOF for such a call?
Is this approach a sensible one? Or is it a bad idea?
I have a problem which I think might be solvable with a batch file, but I've only used batch files once or twice and don't know enough to try and solve this on my own. For context I'm running Windows 10 Home Edition, and have some programming experience, though it is primarily mathematical, i.e. R and MATLAB.
The problem is this: I have two programs, in this case Spotify and Toastify, which run together, with Toastify running in the background. I'll refer to them as S and T, respectively. If I run T, S runs as well, but if I close S, T remains running in the background. For reasons of convenience, I would rather that closing S also close T, so that when I want to use them again later I need only reopen T rather than checking if it's still running in the background, because T doesn't let you run multiple instances.
I'm wondering if there is an easy way to write a batch file (or something else if this isn't a good approach) that will open T (and so also S), and then 'listen' for S to close, at which point it closes T as well.
You need to use a Job Object. See Working example of CreateJobObject/SetInformationJobObject pinvoke in .net? and Kill child process when parent process is killed
Do not try 'monitoring for one process', that leaves zombies when the monitoring crashes.
I am performing very rapid file access in ruby (2.0.0 p39474), and keep getting the exception Too many open files
Having looked at this thread, here, and various other sources, I'm well aware of the OS limits (set to 1024 on my system).
The part of my code that performs this file access is mutexed, and takes the form:
File.open( filename, 'w'){|f| Marshal.dump(value, f) }
where filename is subject to rapid change, depending on the thread calling the section. It's my understanding that this form relinquishes its file handle after the block.
I can verify the number of File objects that are open using ObjectSpace.each_object(File). This reports that there are up to 100 resident in memory, but only one is ever open, as expected.
Further, the exception itself is thrown at a time when there are only 10-40 File objects reported by ObjectSpace. Further, manually garbage collecting fails to improve any of these counts, as does slowing down my script by inserting sleep calls.
My question is, therefore:
Am I fundamentally misunderstanding the nature of the OS limit---does it cover the whole lifetime of a process?
If so, how do web servers avoid crashing out after accessing over ulimit -n files?
Is ruby retaining its file handles outside of its object system, or is the kernel simply very slow at counting 'concurrent' access?
Edit 20130417:
strace indicates that ruby doesn't write all of its data to the file, returning and releasing the mutex before doing so. As such, the file handles stack up until the OS limit.
In an attempt to fix this, I have used syswrite/sysread, synchronous mode, and called flush before close. None of these methods worked.
My question is thus revised to:
Why is ruby failing to close its file handles, and how can I force it to do so?
Use dtrace or strace or whatever equivalent is on your system, and find out exactly what files are being opened.
Note that these could be sockets.
I agree that the code you have pasted does not seem to be capable of causing this problem, at least, not without a rather strange concurrency bug as well.
So let's say pretty often a script runs that opens a browser and does web things:
require 'watir-webdriver'
$browser = Watir::Browser.new(:firefox, :profile => "botmode")
=> #<Watir::Browser:0x7fc97b06f558 url="about:blank" title="about:blank">
It could end gracefully with a browser.close, or it could crash sooner and leave behind the memory-hungry Firefox process, unnoticed until they accumulate and slow the server to a crawl.
My question is twofold:
What is a good practice to ensure that even in case of script failure anywhere leading to immediate error exit, the subprocess will always get cleaned up (I already have lots of short begin-rescue-end blocks peppered for other unrelated small tests)
More importantly, can I simply remember this Watir::Browser:0x7fc97b06f558 object address or PID somehow and re-assign it to another $browser variable in a whole new Ruby process, for example irb? I.e. can an orphaned browser on webdriver be re-attached later in another program using watir-webdriver on the same machine? From irb I could then get in and re-attach to the browser left behind by the crashed Ruby script, to examine the website it was on, check what went wrong, what elements are different than expected, etc.
Another hugely advantageous use of the latter would be to avoid the overhead of potentially hundreds of browser startups and shutdowns per day...best to keep one alive as sort of a daemon. The first run would attempt to reuse a previous browser object using my specially prepared botmode profile, otherwise create one. Then I would deliberately not call $browser.close at the end of my script. If nothing else I run an at job to kill the Xvfb :99 display FF runs inside of at the end of the day anyway (giving FF no choice but to die with it, if still running). Yes I am aware of Selenium standalone jar, but trying to avoid that java service footprint too.
Apologies if this is more a basic Ruby question. I just wasn't sure how to phrase it and keep getting irrelevant search results.
I guess, U cant just remember the variable from another process. But the solution might be creating a master process and process your script in loop in thread, periodically checking the browser running state. I'm using some thing similar in my acceptance tests on Cucumber + watir. So it will be some thing like that:
require 'rubygems'
require 'firewatir' # or watir
#browser = FireWatir::Firefox.new
t = Thread.new do
#browser.goto "http://google.com"
#call more browser actions here
end
while not_exit?
if t.stop?
# error occurred in thread, restart or exit
end
if browser_live?
# browser was killed for a some reason
# restart or exit
end
end
#browser.close
not_exit? - can be over TRAP for the ctrl+C
browser_live? - you can check if firefox browser running with processes listings
It is quite tricky but might work for you
You can use DRb like this:
browsers pool:
require 'drb'
require 'watir'
browser = Watir::Browser.new :chrome
DRb.start_service 'druby://127.0.0.1:9395', browser
gets
and then from test script use this browser:
require 'drb'
browser = DRbObject.new_with_uri 'druby://127.0.0.1:9395'
browser.goto 'stackoverflow.com'
I'm pretty sure that at the point ruby exits, any handles or pointers to something like a browser object would become invalid. So re-using something in a later ruby process is likely not a good approach. In addition I might be wrong on this, but it does seem that webdriver is not very good at connecting to a running browser process. So for your approach to work it would really all need to be wrapped by some master process that was calling all the tests etc.. and hey wait a sec, that's starting to sound like a framework, which you might already (or perhaps should be) using in the first place.
So a better solution is probably to look at whatever framework you are using to run your tests and investigate any capability for 'setup/teardown' actions (which can go by different names) which are run before and after either each test, groups of tests, or all tests. Going this way is good since most frameworks are designed to allow you to run any single test, or set of tests that you want to. And if your tests are well designed they can be run singly without having to expect the system was left in some perfect state by a prior test. Thus these sorts of setup/teardown actions are designed to work that way as well.
As an example Cucumber has this at the feature level, with the idea of a 'background' which is basically intended as a way to dry out scenarios by defining common steps to run before each scenario in a feature file. (such as navigating to and logging into your site) This could include a call to a series of steps that would look to see if a browser object existed, and if not create one. However you'd need to put that in every feature file which starts to become rather non dry.
Fortunately cucumber also allows a way to do this in one place via the use of Hooks. You can define hooks to run before steps, in the event of specific conditions, 'before' and 'after' each scenario, as well as code that runs once before any scenarios, and code defined to run 'at_exit' where you could close the browser after all scenarios have run.
If I was using cucumber I'd look at the idea of a some code in env.rb that would run at the start to create a browser, complemented by at_exit code to close the browser. Then perhaps also code in a before hook which could check to see that the browser is still there and re-create it if needed, and maybe logout actions in a after hook. Leave stuff like logging in for the individual scenarios, or a background block if all scenarios in a feature login with the same sort of user.
Not so much a solution but a workaround for part 1 of my question, using pkill. Posting here since it turned out to be a lot less trivial than I had hoped.
After the ruby script exits, its spawned processes (which may not at all belong in the same PID tree anymore, like firefox-bin) have a predictable "session leader" which turned out to be the parent of the bash shell calling rubyprogram.rb in my case. Available as $PPID in Bash, for when you have to go higher than $$.
Thus to really clean up unwanted heavyweight processes eg. after a ruby crash:
#!/bin/bash
# This is the script that wraps on top of Ruby scripts
./ruby_program_using_watirwebdriver_browser.rb myparams & # spawn ruby in background but keep going below:
sleep 11 # give Ruby a chance to launch its web browser
pstree -panu $$ # prints out a process tree starting under Bash, the parent of Ruby. Firefox may not show!
wait # now wait for Ruby to exit or crash
pkill -s $PPID firefox-bin # should only kill firefox-bin's caused above, not elsewhere on the system
# Another way without pkill, will also print out what's getting killed if anything:
awk '$7=="firefox-bin" && $3=="'$PPID'" {print $1}' <(ps x -o pid,pgid,sess,ppid,tty,time,comm) | xargs -rt kill
OPTIONAL
And since I use a dedicated Xvfb Xwindows server just for webdriving on DISPLAY :99, I can also count on xkill:
timeout 1s xwininfo -display :99 -root -all |awk '/("Navigator" "Firefox")/ {print $1}' |xargs -rt xkill -display :99 -id
# the timeout is in case xkill decides to wait for user action, when window id was missing
Just an update on part 2 of my question.
It seems one CAN serialize a Watir:Browser object with YAML, and because it's text-based the contents were quite interesting to me (e.g. some things I've only dreamed of tweaking hidden inside private elements of private classes...but that's a separate topic)
Deserializing from YAML is still trouble. While I haven't tested beyond the first try it gives me some kind of reg exp parse error...not sure what that's about.
(more on that at at how to serialize an object using TCPServer inside? )
Meanwhile, even attempting to serialize with Marshal, which is also built-in to Ruby but stores in binary format, results in a very reasonable-sounding error about not being able to dump a TCPServer object (apparently contained within my Watir:Browser pointed to by $browser)
All in all I'm not surprised at these results, but still pretty confident there is a way, until Watir arrives at something more native (like PersistentWebdriver or how it used to be in the days of jssh when you could simply attach to an already running browser with the right extension)
Until then, if serialization + deserialization to a working object gets too thorny I'll resort to daemonizing a portion of my Ruby to keep objects persistent and spare the frequent and costly setup/teardowns. And I did take a gander at some established (unit testing) frameworks but none seem to fit well yet within my overall software structure--I'm not web testing after all.
I want to increase the throughput of a script which does net I/O (a scraper). Instead of making it multithreaded in ruby (I use the default 1.9.1 interpreter), I want to launch multiple processes. So, is there a system for doing this to where I can track when one finishes to re-launch it again so that I have X number running at any time. ALso some will run with different command args. I was thinking of writing a bash script but it sounds like a potentially bad idea if there already exists a method for doing something like this on linux.
I would recommend not forking but instead that you use EventMachine (and the excellent em-http-request if you're doing HTTP). Managing multiple processes can be a bit of a handful, even more so than handling multiple threads, but going down the evented path is, in comparison, much simpler. Since you want to do mostly network IO, which consist mostly of waiting, I think that an evented approach would scale as well, or better than forking or threading. And most importantly: it will require much less code, and it will be more readable.
Even if you decide on running separate processes for each task, EventMachine can help you write the code that manages the subprocesses using, for example, EventMachine.popen.
And finally, if you want to do it without EventMachine, read the docs for IO.popen, Open3.popen and Open4.popen. All do more or less the same thing but give you access to the stdin, stdout, stderr (Open3, Open4), and pid (Open4) of the subprocess.
You can try fork http://ruby-doc.org/core/classes/Process.html#M003148
You can get the PID in return and see if this process run again or not.
If you want manage IO concurrency. I suggest you to use EventMachine.
You can either
implement (or find an equivalent gem) a ThreadPool (ProcessPool, in your case), or
prepare an array of all, let's say 1000 tasks to be processed, split it into, say 10 chunks of 100 tasks (10 being the number of parallel processes you want to launch), and launch 10 processes, of which each process right away receives 100 tasks to process. That way you don't need to launch 1000 processes and control that not more than 10 of them work at the same time.