my rails application generates lots of small sqlite databases using DataMapper. After data saved, .sqlite-file must be uploaded on a remote server and destroyed locally.
My question is how to make DataMapper close .sqlite db connection and free repo's memory? Application should generate many databases, so it's important to save server resources.
Only way I googled is
DataObjects::Pooling.pools.each do {|pool| pool.dispose}
which is totally unacceptable for me I think because it seems to be closing all DataMapper connections, however few databases can be generated in parallel threads and I want to destroy DataMapper's repository too.
Sorry for my English.
Also, DataMapper::Repository.adapters is a hash of current Repository objects. You may be able to dig around in there to get at the connections.
I'm not aware of any nice way of doing this. This discussion is pertinent, however:
http://www.mail-archive.com/datamapper#googlegroups.com/msg02894.html
Apparently, it is possible to reopen a connection using DataMapper.setup(), but it seems that the closing of connections is handled automatically.
However, maybe these observations will help:
It's possible to store a reference to the Adapter, e.g.
a = DataMapper.setup(:default, "sqlite:db/development.sqlite3")
Viewing this object shows that the path is stored, implying that it's for that particular connection, rather than a SQLite adapter in general, or such:
p a
#<DataMapper::Adapters::SqliteAdapter:0x00000001aa9258 #name=:default, #options={"scheme"=>"sqlite", "user"=>nil, "password"=>nil, "host"=>nil, "port"=>nil, "query"=>nil, "fragment"=>nil, "adapter"=>"sqlite3", "path"=>"db/development.sqlite3"}, #resource_naming_convention=DataMapper::NamingConventions::Resource::UnderscoredAndPluralized, #field_naming_convention=DataMapper::NamingConventions::Field::Underscored, #normalized_uri=sqlite3:db/development.sqlite3?scheme=sqlite&user=&password=&host=&port=&query=&fragment=&adapter=sqlite3&path=db/development.sqlite3>
Presumably, this can somehow be marked for garbage collection or something (will simply setting it to nil work?).
There is also a close_connection() method in DataMapper::Adapters::DataObjectsAdapter, but it's protected, and I'm not sure whether or how this could be used.
Hope this provides some pointers!
Related
My Java application is processing a lot of information parsed from XML. For some methods, I need to validate some info in an SQL Database on another machine. I am using JDBC, currently for each validation, I call a DB Handling method that opens a connection and returns the result set to validate. I am not sure if I am taking the best design option. This seems very dummy and expensive to me. I am wondering if there are better practices.
Is there is a proper way to have the connection open through the whole application run time so no time is wasted (opening connection) for every validation iteration (I might have hundreds of thousands).
Should I build another application that retrieves all needed tables and convert them to XMLs that are saved on my machine. Later, my application parses them normally and have better access and performance
I am open to any better suggestion
Of course you can use one db connection for whole life of application. Use Singleton pattern. If your program work long and do not use database for a longer time it may loose db connection (some kind of timeout on network equipment etc.). For such cases you can use db pool. Such pool should manage longer time of inactivity or give you separate db connections for separate threads.
I think your solution with converting data into XML is not good. Why do you want to convert it into XML? How often is this data changed? How big is the database you want to copy? How do you want to synchronize your local copy with database? I think that local copy in XML files adds too many problems. If data is small than maybe you can read it at start of your program, save it in some data structures and use to verify other data? It can even be SQLite or other small database. But I would go this way only if singleton or db pool performance is really weak.
Using Hibernate and Spring, my app needs to get the opportunity to manipulate the connection at the start and end of every database update. Our first guess was to override the transaction manager, but that is seeming to have some side effects from an intermittent "Pre-bound JDBC Connection found!" error to a few other harder to describe symptoms.
What is the best/easiest way to get this oppostunity. I saw someone suggest overriding the data source and then wrapping the connection, but is that really the best idea? Wrapping the connection seems dangerous, espceially since Websphere has it's own version (WSconnection).
Ideas?
Manipulation of the connection is done the easiest when a DataSource.getConnection is done, that way you know for certain that something is going to happen. So you could simply create a DataSourceProxy/-Wrapper (give it a name) which does this. There is one thing you have to take care of and that is, probably on close of the connection, that you undo the manipulation (if that is needed that is).
Instead of creating a proxy you could use some AOP to execute the code (like in this answer.
My application have performance issues, so i started to investigate this from the root: "The connection with the database".
The best practices says: "Open a connection, use it and close is as soon as possible", but i dont know the overhead that this causes, so the question is:
1 -"Open, Use, Close connections as soon as possible is the best aproach using ODP.NET?"
2 - Is there a way and how to use connection pooling with ODP.NET?
I thinking about to create a List to store some connections strings and create a logic to choose the "best" connection every time i need. Is this the best way to do it?
Here is a slide deck containing Oracle's recommended best practices:
http://www.oracle.com/technetwork/topics/dotnet/ow2011-bp-performance-deploy-dotnet-518050.pdf
You automatically get a connection pool when you create an OracleConnection. For most middle tier applications you will want to take advantage of that. You will also want to tune your pool for a realistic workload by turning on Performance Counters in the registry.
Please see the ODP.NET online help for details on connection pooling. Pool settings are added to the connection string.
Another issue people run into a lot with OracleConnections is that the garbage collector does not realize how truly resource intensive they are and does not clean them up promptly. This is compounded by the fact that ODP.NET is not fully managed and so some resources are hidden from the garbage collector. Hence the best practice is to Close() AND Dispose() all Oracle ODP.NET objects (including OracleConnection) to force them to be cleaned up.
This particular issue will be mitigated in Oracle's fully managed provider (a beta will be out shortly)
(EDIT: ODP.NET, Managed Driver is now available.)
Christian Shay
Oracle
The ODP.NET is a data provider for ADO.NET.
The best practice for ADO.Net is Open, Get Data (to memory), close, use in memory data.
For example using a OracleDataReader to load data in a DataTable in memory and close connection.
[]'s
For a single transaction this is best but for multiple transaction where you commit at the end this might not be the best solution. You need to keep the connection open until the transaction either committed or rolled back. How do you manage that and also how do you check the connection still exist in that case?(ie network failure) There is ConnectionState.Broken property which does not work at this point.
I'm working on a Ruby web Application that uses the Net::FTP library. One part of it allows users to interact with an FTP site via AJAX. When the user does something, and AJAX call is made, and then Ruby reconnects to the FTP server, performs an action, and outputs information.
Every time the AJAX call is made, Ruby has to reconnect to the FTP server, and that's slow. Is there a way I could cache this FTP connection? I've tried caching in the session hash, but "We're sorry, but something went wrong" is displayed, and a TCP dump is outputted in my logs whenever I attempt to store it in the session hash. I haven't tried memcache yet.
Any suggestions?
What you are trying to do is possible, but far from trivial, and Rails doesn't offer any built-in support for it. In fact you will need to descend to the OS level to get this done, and if you have more than one physical server then it will get even more complicated.
First, you can't store a connection in the session. In fact you don't want to store any Ruby object in the session for many reasons, including but not limited to:
Some types of objects have trouble being marshalled/unmarshalled
Deploying could break stuff if the model changes and people have outdates stuff serialized in their session
If you are using the cookie session store then you only have 4k
So in general, you only ever want to put primitives like strings, numbers and booleans into the session.
Now as far as an FTP connection is concerned, this falls into the category of things that can't be serialized/unserialized reliably. The reason is because it's not just a Ruby object, but also has a socket open which is going to be closed as soon as the original object is garbage collected.
So, in order to keep a FTP connection persistent, it can't be stored in a controller instance variable because the controller instance is per-request. You could try to instantiate it it somewhere outside the controller instance, but that has the potential for memory leaks if you are not very careful to clean up the connections, and besides, if you have more than one app server instance then you would also need to find a way to guarantee that the user talks to the same app server instance on each request, or it wouldn't be able to find the hook. So all in all, keeping the session open in the Ruby process is a non-starter.
What you need to do is open the connection in a separate process that any of the ruby processes can talk to. There's really no established and standard way to do that, you'll have to roll your own. You could look into DRb to provide some of the primitives you will need.
AJAX can't directly talk to FTP. It's designed for HTTP. That doesn't stop you from writing something that does cache the FTP server though. You probably should profile it to find out what's really slow. My guess is that the FTP access is just slow. Caching it may be a mixed blessing though. How do you know when the content of the ftp site changes?
Can I store a database connection in the Session object?
It is generally not recommended to do so, a connection string in the Application variable, with a nice helper function/class is a much preferred method. Here is some reference. (Dead link removed because it now leads to a phishy site)
I seem to recall doing so will have the effect of single threading your application which would be a bad thing.
In general, I wouldn't store any objects in Application variables (and certainly not in session variables).
When it comes to database connections, it's a definite no-no; besides, there is absolutely no need.
If you are use ADO to communicate with the database, if you use the same connection string (yes, by all means, store this in an Application variable) for all your database connections, 'connection pooling' will be implemented behind the scenes. This means that when you release a connection, it isn't actually destroyed - it is put to one side for the next guys who wants the same connection. So next time you request the same connection, it is pulled 'off the shelf' rather than having to be explicitly created and instantiated - which is a quite a nice efficiency improvement.
From this link http://support.microsoft.com/default.aspx/kb/243543
You shouldnt store database connection in Session.
From what I understand, if you do then subsequent ASP requests for the same user must use the same thread.
Therefore if you have a busy site its likely that 'your' thread will already be being used by someone else, so you will have to wait for it to become available.
Multiply this up by lots more users and you will get everyone waiting for everyone elses thread and a not very responsive site.
As said by CJM, there is no need to store a connection in a Session object : connection pooling is much better.