I have been using oci8 for over a year now for several batch processes. There I used to make oracle calls based on a particular frequency without any high number of parallel requests. Recently I started using this driver to process multiple number of user requests in parallel using go routines. The connections go through 90% of the times but for remaining 10% I see an error driver: bad connection being thrown from this driver. This is generally happening in two situations:
When the connection was left idle for too long(happens for few requests).
When there is a spike in number of connections.
Actions taken:
Already checked with my oracle DB for connection/session limits. There is no such limit on it.
Tried forking the branch and adding error logs which didn't seem to compile.
Most of the people who have faced this issue mentioned wrong handling of multiple connections at the same time. For me that is something done by oci8.
Please help!
Related
How would Trains go about sending any missing data to the server in the following scenarios?
Internet connection breaks temporarily while running an experiment
Internet connection breaks and doesn't come back before the experiment ends (any manual way to send all the data that was missed?)
The machine running Trains server resets in the middle of an experiment
Disclaimer: I'm part of the allegro.ai Trains team
Trains will auto retry to send logs, basically forever. The logs/metrics are sent in a background thread so it should not interfere with execution. You can set the backoff parameter, to control the retry frequency, by adjusting the sdk.network.iteration.retry_backoff_factor_sec parameter in your ~/trains.conf file, see example here
The experiment will try to flush all metrics to the backend when the experiment ends, i.e. the process will wait at_exit until all metrics are sent. This means if the connection was dropped, it will retry until it is up again. If the experiment was aborted manually, there is no way to capture/resend those lost metric reports. That said with the new 0.16 version, offline mode was introduced. This way one can run the entire experiment offline, then later report all logs/metrics/artifacts.
The Trains-Server machine is fully stateless (the states themselves are stored in the databases on the machine) this means that from the experiment perspective, the connection was dropped for a few minutes and then it's available again. To your question, if the Trains-Server restarted, it is transparent to all experiments and they continue as usual, no reports will be lost.
Reading this article: http://go-database-sql.org/accessing.html
It says that the sql.DB object is designed to be long-lived and that we should not Open() and Close() databases frequently. But what should I do if I have 10 different MySQL servers and I have sharded them in a way that I have 511 databases in each server for example the way Pinterest shards their data with MySQL?
https://medium.com/#Pinterest_Engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f
Then would I not need to constantly access new nodes with new databases all the time? As I understand then I have to Open and Close the database connection all the time depending on which node and database I have to access.
It also says that:
If you don’t treat the sql.DB as a long-lived object, you could
experience problems such as poor reuse and sharing of connections,
running out of available network resources, or sporadic failures due
to a lot of TCP connections remaining in TIME_WAIT status. Such
problems are signs that you’re not using database/sql as it was
designed.
Will this be a problem? How should I solve this issue then?
I am also interested in the question. I guess there could be such solution:
Minimize number of idle connection in pool db.SerMaxIdleConns(N)
Make map[serverID]*sql.DB. When you have no such connection - add it to map.
Make Dara more local - so backends usually go to “their” databases. However Pinterest seems not to use it.
Increase number of sockets and files on backend machines so they can keep more open connections.
Provide some reasonable idle timeout so very old unused connections could be closed.
I recently migrated my Postgres database from Windows to CentOS 6.7.
On Windows the database never used much CPU, but on Linux I see it using a constant ~30% CPU (using top). (4 core on machine)
Anyone know if this is normally, or why it would be doing this?
The application seems to run fine, and as fast or faster than Windows.
Note, it is a big database, 100gb+ data, 1000+ databases.
I tried using Pgadmin to monitor the server status, but the server status hangs, and fails to run, error "the log_filename parameter must be equal"
With 1000 databases I expect vacuum workers and stats collector to spend a lot of time checking about what needs maintenance.
I suggest you to do two things
raise the autovacuum_naptime parameter to reduce the frequency of checks
put the stats_temp_directory on a ramdisk
You probably also set a high max_connections limit to allow your clients to use those high number of databases and this is another probable source of CPU load, due to the high number of 'slots' to be checked every time a backend has to synchronize with the others.
There could be multiple reasons for increasing server loads.
If you are looking for query level loads on server then you should match a specific Postgres backend ID to a system process ID using the pg_stat_activity system table.
SELECT pid, datname, usename, query FROM pg_stat_activity;
Once you know what queries are running you can investigate further (EXPLAIN/EXPLAIN ANALYZE; check locks, etc.)
You may have lock contention issues, probably due to very high max_connections. Consider lowering max_connections and using a connection pooler if this is the case. But that can increase turn around time for clients connections.
Might be Windows System blocking connections and not allowing to use system. And now Linus allowing its connections to use CPU and perform faster. :P
Also worth read:
How to monitor PostgreSQL
Monitoring CPU and memory usage from Postgres
I'm getting ActiveRecord::ConnectionTimeoutError in a daemon that runs independently from the rails app. I'm using Passenger with Apache and MySQL as the database.
Passenger's default pool size is 6 (at least that's what the documentation tells me), so it shouldn't use more than 6 connections.
I've set ActiveRecord's pool size to 10, even though I thought that my daemon should only need one connection. My daemon is one process with multiple threads that calls ActiveRecord here and there to save stuff to the database that it shares with the rails app.
What I need to figure out is whether the threads simply can't share one connection or if they just keep asking for new connections without releasing their old connections. I know I could just increase the pool size and postpone the problem, but the daemon can have hundreds of threads and sooner or later the pool will run out of connections.
The first thing I would like to know is that Passenger is indeed just using 6 connections and that the problem lies with the daemon. How do I test that?
Second I would like to figure out if every thread need their own connection or if they just need to be told to reuse the connection they already have. If they do need their own connections, maybe they just need to be told to not hold on to them when they're not using them? The threads are after all sleeping most of the time.
You can get to the connection pools that ActiveRecord is using through ActiveRecord::Base.connection_handler.connection_pools it should be an array of connection pools. You probably will only have one in there and it has a connections method on it. To get an array of connections it knows about.
You can also do a ActiveRecord::Base.connection_handler.connection_pools.each(&:clear_stale_cached_connections!) and it will checkin any checked out connections which thread is no longer alive.
Don't know if that helps or confuses more
As of February 2019, clear_state_cached_connections has been deprecated and moved to reap
Commit
Previous accepted answer updated:
ActiveRecord::Base.connection_handler.connection_pools.each(&:reap)
I am using JDBC with proxool connection pool to connect to mysql DB.
I am selecting large number of rows from multiple threads and after some time i get an error saying communication link failure, Last packet sent to the server was ...ago.
I am closing connection,statement,resultSet in every thread.
The fetching time increases gradually and the exception occurs after 5-10 minutes.
I doubt it is a memory leak, but cant find any clue.
Please let me know the possible reasons.
Thanks,
Kaka
it may related on your Connection Timeout, try to increase it.
con.setConnectionTimeout(X);