I am getting this error:
'could not obtain a database connection within 5 seconds (waited 5.001017 seconds). The max pool size is currently 16; consider increasing it.'
First I got this error, I bumped up the count from 5 to 16. But it's still happening and i am the only one test out the database. Why is this happening when I am the only user?
I am not on rails btw. I am using:
ActiveRecord::Base.establish_connection ({
:adapter => 'mysql2',
:database => 'ck',
:host => 'localhost',
:username => 'root',
:password => '',
:pool => 16,
})
and using Sinatra.
Thanks
As Frederick pointed out you need to return opened ActiveRecord connections to the connection pool.
If you're using the Thin server, in threaded mode, then you need to add this to your Sinatra app:
after do
ActiveRecord::Base.connection.close
end
...instead of using the ConnectionManagement suggestion. The reason is that Thin splits the request processing over 2 threads and the thread that is closing the ActiveRecord connection is not the same as the thread that opened it. As ActiveRecord tracks connections by thread ID it gets confused and won't return connections correctly.
It sounds like you are not returning connections to the pool at the end of the request. If not then each request that uses the db will consume 1 connection and eventually you'll exhaust the pool and start getting the error messages you describe
Active Record provides a rack middleware to handle this ActiveRecord::ConnectionAdapters::ConnectionManagement, which should take care of things as long as its earlier in the middleware chain than anything that will access active record.
You can also take care of the connection management yourself. The docs have more details but one way of doing it is sticking all of your db accesses in a block like this
ActiveRecord::Base.connection_pool.with_connection do
...
end
Which checks out a connection at the start of the block and checks it back in afterwards.
It's better to use the middleware provided with ActiveRecord:
use ActiveRecord::ConnectionAdapters::ConnectionManagement
As Frederick pointed out you need to return opened ActiveRecord connections to the connection pool.
As kuwerty suggests, when you are using Thin, ConnectionManagement will not return connections to the pool. I suggest that instead of closing the current connection like kuwerty says, you return the connection to the pool like so.
after do
ActiveRecord::Base.clear_active_connections!
end
For those who want to reproduce the problem, try this example.
EDIT:
I made an explanation of why using middleware ActiveRecord::Connectionadapters::ConnectionManagement won't work with Thin in threaded mode, which you can find here.
Related
I am building a service in Ruby 2.4.4, with Sinatra 2.0.5, ActiveRecord 5.2.2, Puma 3.12.0. (I'm not using rails.)
My code looks like this. I have an endpoint which opens a DB connection (to a Postgres DB) and runs some DB queries, like this:
POST '/endpoint' do
# open a connection
ActiveRecord::Base.establish_connection(##db_configuration)
# run some queries
db_value = TableModel.find_by(xx: yy)
return whatever
end
after do
# after the endpoint finishes, close all open connections
ActiveRecord::Base.clear_all_connections!
end
When I get two parallel requests to this endpoint, one of them fails with this error:
2019-01-12 00:22:07 - ActiveRecord::ConnectionNotEstablished - No connection pool with 'primary' found.:
C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:1009:in `retrieve_connection'
C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/activerecord-5.2.2/lib/active_record/connection_handling.rb:118:in `retrieve_connection'
C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/activerecord-5.2.2/lib/active_record/connection_handling.rb:90:in `connection'
C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/activerecord-5.2.2/lib/active_record/core.rb:207:in `find_by'
...
My discovery process went this way so far.
I looked at the connection usage in Postgres, thinking I might leak connections - no, I didn't seem to.
Just in case, I increased the connection pool to 16 (corresponding to 16 Puma threads) - didn't help.
Then I looked into the ActiveRecord sources. Here I realized why 2) didn't help. The problem is not that I can't get a connection, but I can't get a connection pool (yes, yes, it says that in the exception). The #owner_to_pool map variable, from which a connection pool is obtained, stores the process_id as key, and as values - connection pools (actually, the value is also a map, where the key is a connection specification and the value, I presume, is an actual pool instance). In my case, I have only one connection spec to my only db.
But Puma is a multithreaded webserver. It runs all requests in the same process but in different threads.
Because of that, I think, the following happens:
The first request, starting in process_id=X, thread=Y, "checks out" the connection pool in establish_connection, based on process_id=X, - and "takes" it. Now it's not present in the #owner_to_pool.
The second request, starting in the same process_id=X, but different thread=Z, tries to do the same - but the connection pool for process_id=X is not present in owner_to_pool. So the second request doesn't get a connection pool and fails with that exception.
The first request finished successfully and puts the connection pool for process_id=X back in place by calling clear_all_connections.
Another request, starting after all that, and not having any parallel requests in parallel threads, will succeed, because it will pick up the connection pool and put it back again with no problems.
Although I am not sure I understand everything 100% correctly, but it seems to me that something like this happens.
Now, my question is: what do I do with all this? How do I make the multithreaded Puma webserver work correctly with ActiveRecord's connection pool?
Thanks a lot in advance!
This question seems similar, but unfortunately it doesn't have an answer, and I don't have enough reputation to comment on it and ask the author if they solved it.
So, basically, I didn't realize I was establish_connection is creating a connection pool. (Yes, yes, I said so myself in the question. Still, didn't quite realize it.)
What I ended up doing, is this:
require ....
# create the connection pool with the required configuration - once; it'll belong to the process
ActiveRecord::Base.establish_connection(db_configuration)
at_exit {
# close all connections on app exit
ActiveRecord::Base.clear_all_connections!
}
class SomeClass < Sinatra::Base
POST '/endpoint' do
# run some queries - they'll automatically use a connection from the pool
db_value = TableModel.find_by(xx: yy)
return whatever
end
end
I'm using ActiveRecord in a Ruby script, without using rails, just running the Ruby script. The script kicks off threads which access the database. Eventually, I get:
could not obtain a database connection within 20.000 seconds
database configuration:
The pool is set for 10 connection. The timeout is set for 20 seconds.
I tried without using connection poll calls directly, but I'm not clear on how that is supposed to be managed. So I put cxn = ActiveRecord::Base.connection_pool.checkout and ActiveRecord::Base.connection_pool.checkin(cxn) around the database access portions of the code. I still get the error.
I put some debug in the script, and I can see the checkout/checkin calls happening. There are 7 successful checkout/checkins. There are 13 total checkouts, so 6 left open.
I'm also seeing:
undefined method `run_callbacks'
when the timeout occurs.
Thank you,
Jerome
You need to explicitly release connections back to the ActiveRecord connection pool. You either need to explicitly call ActiveRecord::Base.clear_active_connections! or else define
a helper method which does the following:
def with_connection(&block)
ActiveRecord::Base.connection_pool.with_connection do
yield block
end
end
which you would use as follows:
def my_method
with_connection do
User.where(:id => 1)
end
end
which should release your connection when the block exits.
Be warned that ActiveRecord uses the current Thread ID to manage connections, so if you are spawning off threads each thread needs to release connections once done with them. Your Threads are most likely not clearing connections properly, or you are spawning more threads than you have connections available in your connection pool.
I have a ruby script that needs to run continually on the server. I've daemonized it using the daemon gem, and in my script I have it running in an infinite loop, since the daemon gem handles starting and stopping of the process that kicks off my script. In my script, I start out by setting up my DB instance using the Sequel gem and tiny_tds. Like so:
DB = Sequel.connect(adapter: 'tinytds', host: MSSQLHost, database: MSSQLDatabase, user: MSSQLUser, password: MSSQLPassword)
Then I have a loop do that is my infinite loop. Inside that, I test to see if I have a connection using DB.test_connection and then I query the DB every second or so to check if there is new content using a query such as:
DB['SELECT * FROM dbo.[MyTable]'].all do |row|
# MY logic here
# As part of my logic I test to see if I need to delete this row in the table and if so I use
DB.run('DELETE FROM dbo.[MyTable] WHERE some condition')
end
Then at the end of my logic, just before I loop again, I do:
sleep 1
DB.disconnect
All of this works great for about an hour to an hour and a half with everything checking the table, doing the logic, deleting rows, etc., then it dies and gives me this error message TinyTds::Error: Adaptive Server connection timed out
My question, why is that happening? Do I need to reformat my code in a different way? Why doesn't the DB.test_connection do what it is advertised to do? The documentation on that says it checks for a connection in the connection pool, and uses it if it finds it, and creates a new one otherwise.
Any help would be much appreciated
DB.test_connection just acquires a connection from the connection pool, it doesn't check that the connection is still valid (it must have been valid at one point or it wouldn't be in the pool). There's no way that a connection is still valid without actually sending a query. You can use the connection_validator extension that ships with Sequel if you want to do that automatically.
If you are loading Sequel before forking, you need to make sure you call DB.disconnect before forking, otherwise you can end up with multiple forked processes sharing the same connection, which can cause many different issues.
I finally ended up just putting a rescue statement in there that caught this, and re-ran my line of code to create the DB instance, yes, it puts a warning in my log about already setting that instance, but I guess I could just make that not a contstant an that would go away. Anyway, it appears to be working now, and the times it does timeout, I'm recovering gracefully from those. I just wish I could have figured out why it was/is disconnecting like it is.
I have an application that I am coding to have the logging info be sent over tcpsocket to a server and have a monitor client connect to the server to view the logging data . So far i am able to get to the stage where the info is sent to the server however I need some thoughts on how to go about the next stage. Using Ruby tcpsever what methodologies can I use to have the server resend the incoming data to a client? How can I have data stored across threads?
require "socket"
server_socket = TCPServer.new('localhost', 2200)
loop do
# Create a new thread for each connection.
Thread.start(server_socket.accept) do |session|
# check if received is viewer request
line = session.gets
if line =~ /viewer/
#filter = line[/\:(.*?)\:/]
session.puts "Listining for #{filter}"
loop do
if (log = ### need input here from logging app ###)
# Show if filter is set for all or cli matches to filter
if #filter == ':all:' || log =~ /\:(.*?)\:/
session.puts log
end
# Read trace_viewer input.
if session.gets =~ /quit/
# close the connections
session.puts "Closing connection. Bye!"
session.close
break
end
end
end
end
end
end
With clients connecting to the server it sounds like a typical client/server configuration, with some clients sending data, and others requesting it. Is there any reason you don't use a standard HTTP client/server, i.e., a web server, instead of reinventing the wheel? Use Sinatra or Padrino, or even Rails and you'll be mostly finished.
How can I have data stored across threads?
Ruby's Thread module includes Queue, which is good for moving data around between threads. The document page has an example which should help.
The same ideas for using a queue would apply to using tables. I'd recommend using a database to act as a queue for your use, rather than do it in memory. A power outage, or app crash will lose all the data if it's an in-memory queue and the clients haven't retrieved everything. Writing and reading a database means the data would survive such problems.
Keep the schema simple, provide a reasonable index on it, and it should be fast enough for most uses. You'll need some housekeeping code to keep the database clean, but that's easy using SQL, or an ORM like Sequel or ActiveRecord.
I mean the one which was previously established as
DB = Sequel.sqlite('my_blog.db')
or
DB = Sequel.connect('postgres://user:password#localhost/my_db')
or
DB = Sequel.postgres('my_db', :user => 'user', :password => 'password', :host => 'localhost')
or etcetera.
The Sequel::Database class has no public instance method called "disconnect" or so though it has "connect" one.
Maybe somebody already faced that problem. I would appreciate any idea.
As Mladen Jablanović points out, you can just do:
DB.disconnect
Which will disconnect all of the available connections in that Sequel::Database instance's connection pool. You can't choose a specific connection to disconnect, and it wouldn't make sense to. The sharded connection pools do support disconnecting all connections for a specific shard, though.