In my ruby on rails project, I have to take pull from sql-server to my mysql database.
When I run my project on port 3000, it makes system busy when I want to take pull.
I want such method or way which system can detect, how many ports are running for ruby application and how to close if it is not in use ?
Thanks in advance.
Hard to understand exactly what you're asking for, but I'm assuming that when you are synchronizing databases, the system becomes busy and you can't serve any pages. This is a perfect example for the use of a background job that allows you to do tasks like this without affecting the rails application. The two gems that come to mind that will allow you to do this is Delayed_job and Resque. An outstanding screencast for doing this type of stuff is listed below as well.
http://github.com/collectiveidea/delayed_job
https://github.com/defunkt/resque/
http://railscasts.com/episodes/171-delayed-job
http://railscasts.com/episodes/271-resque
Related
I am trying to create a Ruby daemon process which clients will be able to connect to.
I need to ensure that the remote Ruby process always remains up and available for connection, so I need to detect network outages or unreachable errors.
I was thinking of having a heartbeat mechanism at the application level between clients and the server, and a timeout in the client if the connection fails.
I was told the select method in Ruby could be of help as well but not sure.
Can anyone share any good links/resources or impart some general wisdom to create reliable and fast daemon processes in Ruby?
I think a lot of people would use eventmachine for this type of application. At its core, it uses epoll (which is similar to select) to decide which socket to deal with next. There are lots of gems that build on eventmachine to allow you to run different types of servers. One example is em-websocket.
I have just written my first EventMachine application. In development, to start the server, all I do is:
ruby myapp.rb
Which runs my application until I kill it with control+C. In production, this doesn't seem like the right way to do it.
How would I go about running this on my production server?
Check out daemons: http://daemons.rubyforge.org/ - a simple gem written for precisely this use case.
At PostRank we always used God to start/restart our production EventMachine APIs.
I prefer to have a completely external process handling my daemons rather than using something like the daemons library but that's a personnal preference.
You have many solutions out there, here those I know of, all of them will restart your application when it crash more or les quickly and some offer a management interface whether it is a cli or a web interface:
supervisord (http://supervisord.org/): he one I prefer so far
daemontools (http://cr.yp.to/daemontools.html): works well but can be anoying to configure
god as mentionned (http://god.rubyforge.org/): Never used it most for this horrible and cryptic config file syntax
And the last one is whatever comes with your linux distrib, init can run an application and restart it when it dies, you have neary no control over it but it can do the job.
You can type "man inittab" to learn more.
We have a requirement to build a small Sinatra app which will capture events from an external API and add them to a queue for processing by a Rails application. We could be receiving hundreds of thousands of events per day.
Given that resque rules itself out by not being able to guarantee that jobs won't get lost, what other options are out there. We've looked at delayed_job and that doesn't play well with Sinatra, so what other alternatives are there for something fast, reliable and scalable.
Have you looked at Beanstalk?
http://kr.github.com/beanstalkd/
http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/
There's an example Sinatra/Beanstalk app on GitHub:
https://github.com/adamwiggins/clockwork-sinatra-beanstalk
Alternatively you might want to check out RabbitMQ with ruby-amqp, but I think I'd first try the Beanstalk approach (it handles the workload you describe in your post for us):
https://github.com/ruby-amqp/amqp
I came across God which seems good but I am wondering if anyone knows of other process monitoring and control frameworks that I can compare god with.
God has the following features:
Config file is written in Ruby
Easily write your own custom conditions in Ruby
Supports both poll and event based conditions
Different poll conditions can have different intervals
Integrated notification system (write your own too!)
Easily control non-daemonizing scripts
The last one is what i am having difficulty with.
Have a look at Ubic (CPAN page here but do read installation details on the github project page).
Ubic isn't a monitoring framework par se but a LSB compliant extensible Service Manager.
Its written and configurable all in Perl. A simple example would be:
# /etc/ubic/services/test
use Ubic::Service::SimpleDaemon;
return Ubic::Service::SimpleDaemon->new({ bin => "sleep 1000" });
To start above is: ubic start test. To check its running or not: ubic status test. To stop service (suprisingly!) is: ubic stop test.
Ubic keeps an eye on all its services so when test service stops after 1000 seconds then Ubic will automatically restart it again.
Some more links:
Mailing list
Ubic - how to implement your first service (blog post)
Ubic - code reuse and customizations (blog post)
/I3az/
I am a big fan of Monit. It's written in C, but does everything you want.
I particularly liked that I was able to compile a thin version that worked beautifully on an ARM based system with only 64 MB of RAM.
You might want to read God vs Monit on SO to get a comparison.
Bluepill is a great process monitoring/administration framework.
It's written in Ruby but it can monitor anything, I use it to monitor Unicorn processes.
It even runs on 1.9.2.
Doesn't leak memory.
Has support for demonizing processes that don't demonize themselves.
All around easy, even with RVM!
I'm working on a consumer web app that needs to do a long running background process that is tied to each customer request. By long running, I mean anywhere between 1 and 3 minutes.
Here is an example flow. The object/widget doesn't really matter.
Customer comes to the site and specifies object/widget they are looking for.
We search/clean/filter for widgets matching some initial criteria. <-- long running process
Customer further configures more detail about the widget they are looking for.
When the long running process is complete the customer is able to complete the last few steps before conversion.
Steps 3 and 4 aren't really important. I just mention them because we can buy some time while we are doing the long running process.
The environment we are working in is a LAMP stack-- currently using PHP. It doesn't seem like a good design to have the long running process take up an apache thread in mod_php (or fastcgi process). The apache layer of our app should be focused on serving up content and not data processing IMO.
A few questions:
Is our thinking right in that we should separate this "long running" part out of the apache/web app layer?
Is there a standard/typical way to break this out under Linux/Apache/MySQL/PHP (we're open to using a different language for the processing if appropriate)?
Any suggestions on how to go about breaking it out? E.g. do we create a deamon that churns through a FIFO queue?
Edit: Just to clarify, only about 1/4 of the long running process is database centric. We're working on optimizing that part. There is some work that we could potentially do, but we are limited in the amount we can do right now.
Thanks!
Consider providing the search results via AJAX from a web service instead of your application. Presumably you could offload this to another server and let you web application deal with the content as you desire.
Just curious: 1-3 minutes seems like a long time for a lookup query. Have you looked at indexes on the columns you are querying to improve the speed? Or do you need to do some algorithmic process -- perhaps you could perform some of this offline and prepopulate some common searches with hints?
As Jonnii suggested, you can start a child process to carry out background processing. However, this needs to be done with some care:
Make sure that any parameters passed through are escaped correctly
Ensure that more than one copy of the process does not run at once
If several copies of the process run, there's nothing stopping a (not even malicious, just impatient) user from hitting reload on the page which kicks it off, eventually starting so many copies that the machine runs out of ram and grinds to a halt.
So you can use a subprocess, but do it carefully, in a controlled manner, and test it properly.
Another option is to have a daemon permanently running waiting for requests, which processes them and then records the results somewhere (perhaps in a database)
This is the poor man's solution:
exec ("/usr/bin/php long_running_process.php > /dev/null &");
Alternatively you could:
Insert a row into your database with details of the background request, which a daemon can then read and process.
Write a message to a message queue which a daemon then read and processed.
Here's some discussion on the Java version of this problem.
See java: what are the best techniques for communicating with a batch server
Two important things you might do:
Switch to Java and use JMS.
Read up on JMS but use another queue manager. Unix named pipes, for instance, might be an acceptable implementation.
Java servlets can do background processing. You could do something similar to this technology in a web technology with threading support. I don't know about PHP though.
Not a complete answer but I would think using AJAX and passing the 2nd step to something thats faster then PHP (C, C++, C#) then a PHP function pick the results off of some stack most likely just a database.