I'm working on a Ruby web Application that uses the Net::FTP library. One part of it allows users to interact with an FTP site via AJAX. When the user does something, and AJAX call is made, and then Ruby reconnects to the FTP server, performs an action, and outputs information.
Every time the AJAX call is made, Ruby has to reconnect to the FTP server, and that's slow. Is there a way I could cache this FTP connection? I've tried caching in the session hash, but "We're sorry, but something went wrong" is displayed, and a TCP dump is outputted in my logs whenever I attempt to store it in the session hash. I haven't tried memcache yet.
Any suggestions?
What you are trying to do is possible, but far from trivial, and Rails doesn't offer any built-in support for it. In fact you will need to descend to the OS level to get this done, and if you have more than one physical server then it will get even more complicated.
First, you can't store a connection in the session. In fact you don't want to store any Ruby object in the session for many reasons, including but not limited to:
Some types of objects have trouble being marshalled/unmarshalled
Deploying could break stuff if the model changes and people have outdates stuff serialized in their session
If you are using the cookie session store then you only have 4k
So in general, you only ever want to put primitives like strings, numbers and booleans into the session.
Now as far as an FTP connection is concerned, this falls into the category of things that can't be serialized/unserialized reliably. The reason is because it's not just a Ruby object, but also has a socket open which is going to be closed as soon as the original object is garbage collected.
So, in order to keep a FTP connection persistent, it can't be stored in a controller instance variable because the controller instance is per-request. You could try to instantiate it it somewhere outside the controller instance, but that has the potential for memory leaks if you are not very careful to clean up the connections, and besides, if you have more than one app server instance then you would also need to find a way to guarantee that the user talks to the same app server instance on each request, or it wouldn't be able to find the hook. So all in all, keeping the session open in the Ruby process is a non-starter.
What you need to do is open the connection in a separate process that any of the ruby processes can talk to. There's really no established and standard way to do that, you'll have to roll your own. You could look into DRb to provide some of the primitives you will need.
AJAX can't directly talk to FTP. It's designed for HTTP. That doesn't stop you from writing something that does cache the FTP server though. You probably should profile it to find out what's really slow. My guess is that the FTP access is just slow. Caching it may be a mixed blessing though. How do you know when the content of the ftp site changes?
Related
I've been building some apps that connect to a SQL backend. I use ajax calls to hit WebMethods, a WebAPI, etc.
I notice that the first initial call to the SQL backend retrieves the data fairly slow. I can only assume that this is because it must first negotiate credentials first before retrieving the data. It probably caches this somewhere, and thus, any calls made afterwards come back very fast.
I'm wondering if there's an ideal, or optimal way, to initialize this connection.
My thought was to make a simple GET call right when the page loads (grabbing something very small, like a single entry). I probably wouldn't be using the returned data in any useful way, other than to ensure that any calls afterwards come back faster.
Is this an okay way to approach fixing the initial delay? I'd love to hear how others handle this.
Cheers!
There are a number of reasons that your first call could be slower than subsequent ones
Depending on your server platform, code may be compiled when first executed
You may not have an active DB connection in your connection pool
The database may not have cached indices or data on the first call
Some VM platforms may take a while to allocate sufficient resources to your server if it has been idle for a while.
One way I deal with those types of issues on the server side is to add startup code to my web service that fetches data likely to be used by many callers when the service first initializes (e.g. lookup tables, user credential tables, etc).
If you only control the client, consider that you may well wish to monitor server health (I use the open source monitoring platform Zabbix. There are also many commercial web-based monitoring solutions). Exercising the server outside of end-user code is probably better than making an extra GET call from a page that an end user has loaded.
my rails application generates lots of small sqlite databases using DataMapper. After data saved, .sqlite-file must be uploaded on a remote server and destroyed locally.
My question is how to make DataMapper close .sqlite db connection and free repo's memory? Application should generate many databases, so it's important to save server resources.
Only way I googled is
DataObjects::Pooling.pools.each do {|pool| pool.dispose}
which is totally unacceptable for me I think because it seems to be closing all DataMapper connections, however few databases can be generated in parallel threads and I want to destroy DataMapper's repository too.
Sorry for my English.
Also, DataMapper::Repository.adapters is a hash of current Repository objects. You may be able to dig around in there to get at the connections.
I'm not aware of any nice way of doing this. This discussion is pertinent, however:
http://www.mail-archive.com/datamapper#googlegroups.com/msg02894.html
Apparently, it is possible to reopen a connection using DataMapper.setup(), but it seems that the closing of connections is handled automatically.
However, maybe these observations will help:
It's possible to store a reference to the Adapter, e.g.
a = DataMapper.setup(:default, "sqlite:db/development.sqlite3")
Viewing this object shows that the path is stored, implying that it's for that particular connection, rather than a SQLite adapter in general, or such:
p a
#<DataMapper::Adapters::SqliteAdapter:0x00000001aa9258 #name=:default, #options={"scheme"=>"sqlite", "user"=>nil, "password"=>nil, "host"=>nil, "port"=>nil, "query"=>nil, "fragment"=>nil, "adapter"=>"sqlite3", "path"=>"db/development.sqlite3"}, #resource_naming_convention=DataMapper::NamingConventions::Resource::UnderscoredAndPluralized, #field_naming_convention=DataMapper::NamingConventions::Field::Underscored, #normalized_uri=sqlite3:db/development.sqlite3?scheme=sqlite&user=&password=&host=&port=&query=&fragment=&adapter=sqlite3&path=db/development.sqlite3>
Presumably, this can somehow be marked for garbage collection or something (will simply setting it to nil work?).
There is also a close_connection() method in DataMapper::Adapters::DataObjectsAdapter, but it's protected, and I'm not sure whether or how this could be used.
Hope this provides some pointers!
Firstly, I am fairly new to server side scripting so I don't know if this question makes sense.
Suppose a page has stored sessions for PHP, Perl and ASP. Is there a quick and easy way to emulate the browser being closed and re-opened to reset/destroy/clear all the sessions at once? Or would one have to go through the session clearing for every server language you used individually?
If you mean close the browser, then session_destroy sounds like your ticket.
If you mean destroy all sessions it's a little different; closing the browser wouldn't destroy all sessions unless all users closed their browsers. Generally speaking, every browser/user has one session.
Doing this depends on how your sessions are implemented. Typically PHP stored session info in files... so if you figured out which directory contained the session files, and just cleared it with a rm -R * or something of the like... that should do the trick. Be sure to check that there are only PHP session files in that directory before you do it though ;)
You could also set up a PHP script to use sessions in a MySQL database (or any other medium you want) using session_set_save_handler and other methods like it. If you did something like this, you could have more control over clearing all sessions (and it would help you deal with exposed session data)
In ASP.NET you have Session.Abandon which destroys all the objects stored in a Session object and releases their resources.
I'm trying to create a small and basic "ajax" based multiplayer game. Coordinates of objects are being given by a PHP "handler". This handler.php file is being polled every 200MS, by using ajax.
Since there is no need to poll when nothing happens, I wonder, is there something that could do the same thing without frequent polling? Eg. Comet, though I heard that you need to configure server side applications for Comet. It's a shared webserver, so I can't do that.
Maybe prevent the handler.php file from even returning a response if nothing has to be changed at the client, is that possible? Then again you'd still have the client uselessly asking for a response even though something hasn't changed yet. Basically, it should only use bandwidth and sever resources if something needs to be told to the client, eg. the change of an object's coordinates.
Comet is generally used for this kind of thing, and it can be a fragile setup as it's not a particularly common technology so it can be easy not to "get it right." That said, there are more resources available now than when I last tried it ~2 years ago.
I don't think you can do what you're thinking and have handler.php simply not return anything and stop execution: The web server will keep the connection open and prevent any further polling until handler.php does something (terminates or provides output). When it does, you're still handling a response.
You can try a long polling technique, where your AJAX allows a very large timeout (e.g. 30 seconds), and handler.php spins without responding until it has something to report, then returns. (You'll want to make sure the spinning is not resource-intensive). If handler.php "expires" and nothing happens, have it exit and let AJAX poll again. Since it only happens every 30 seconds, it will be a huge improvement over ~5 times a second. That would keep your polling to a minimum.
But that's the sort of thing Comet is designed for.
As Ajax only offers you a client server request model (normally termed pull, rather than push), the only way to get data from the server is via requests. However a common technique to get around this is for the server to only respond when it has new data. So the client makes a request, the server hangs on to that request until something happens and then replies. This gets around the need for frequent polling even when the data hasn't changed as you only need the client send a new request after it gets a response.
Since you are using PHP, one simple method might be to have the PHP code call the sleep command for 200ms at a time between checks for data changes and then return the data to the client when it does change.
EDIT: I would also recommend having a timeout on the request. So if nothing happens for say 2 seconds, a "no change" message is sent back. That way the client knows the server is still alive and processing its request.
Since this is tagged “html5”: HTML5 has <eventsource> and WebSocket, but the implementation side is still in the future tense in practice.
Opera implemented an old version of <eventsource> called <event-source>.
Here's a solution - use a SaaS comet provider, such as WebSync On-Demand. No server resources to worry about, shared hosting or not, since it's all offloaded, and you can push out the information as needed.
Since it's SaaS, it'll work with any server language. For PHP, there's already a publisher written and ready to go.
The server must take part in this. Check with the hosting provider what modules are available. Or try to convince them to support Comet.
Maybe you should consider a small Virtual Private Server (VPS) for this.
One thing to add on the long polling suggestions: If you're on a shared server, this solution will have limited scalability, as each active long poll will keep a connection (and a server-side process to service that connection) active. Your provider most likely has limits (either policy-defined or de facto) on the number of connections you can have open at a time, so you'll hit a wall if you have more sessions/windows than that playing concurrently.
Can I store a database connection in the Session object?
It is generally not recommended to do so, a connection string in the Application variable, with a nice helper function/class is a much preferred method. Here is some reference. (Dead link removed because it now leads to a phishy site)
I seem to recall doing so will have the effect of single threading your application which would be a bad thing.
In general, I wouldn't store any objects in Application variables (and certainly not in session variables).
When it comes to database connections, it's a definite no-no; besides, there is absolutely no need.
If you are use ADO to communicate with the database, if you use the same connection string (yes, by all means, store this in an Application variable) for all your database connections, 'connection pooling' will be implemented behind the scenes. This means that when you release a connection, it isn't actually destroyed - it is put to one side for the next guys who wants the same connection. So next time you request the same connection, it is pulled 'off the shelf' rather than having to be explicitly created and instantiated - which is a quite a nice efficiency improvement.
From this link http://support.microsoft.com/default.aspx/kb/243543
You shouldnt store database connection in Session.
From what I understand, if you do then subsequent ASP requests for the same user must use the same thread.
Therefore if you have a busy site its likely that 'your' thread will already be being used by someone else, so you will have to wait for it to become available.
Multiply this up by lots more users and you will get everyone waiting for everyone elses thread and a not very responsive site.
As said by CJM, there is no need to store a connection in a Session object : connection pooling is much better.