So, I have a Java app on Heroku that uses RedisCloud addon.
The addon clearly states that the free version comes with a maximum of 30 Connections:
The problem is that Im getting this error:
ERR max number of clients reached
So the first thing I did obviously was check the RedisCloud monitor and to my surprise, It establishes a limit of 10 Connections:
The question:
Why are we getting a connection limit of 10 on RedisCloud when the limit on the Heroku addon says it should be 30?
It appears that your add-on is using an old version of the plan from before we launched our Bigger and Imporved XXXL Free plan earlier this year.
The easiest way to resolve that is to use the Heroku toolkit belt and run the command:
heroku addons:upgrade rediscloud:30 -a <your app's name>
Related
I was running some tests to understand the MaxMemory-Reserved & MaxMemory-Policy and we faced “Server Closed the connection” error few times when Redis DB was almost full. Here are the details:
1) Created the Redis Cache with Standard C1(1 GB) tier and chose “allkeys-lru” and max-memory-reserved as 50 MB
2) Ran the Redis Benchmark tool to add the Keys in Redis DB to make sure Redis DB is almost full.
3) As soon as DB reached around ~960-980 MB, again ran Benchmark tool to add some more keys and got following error. In which all scenarios this error can occur?
Note: The Connected_Clients value was 0 when we ran the info command just before we encountered this error.
4) At same time ran the info command on Azure Portal Console and got the output as “Error”.
5) This error lasted approximately for 2-3 Mins and we were able to add keys after that. And once we ran the info command again, we got following stat. Here we see that difference between used_memory and used_memory_rss is around 76 MB. Do you think the above error could be because of this?
info
Server redis_version:3.2.3
redis_mode:standalone
os:Windows
arch_bits:64
multiplexing_api:winsock_IOCP
hz:10
Clients
connected_clients:2
client_longest_output_list:0
client_biggest_input_buf:0
client_total_writes_outstanding:0
client_total_sent_bytes_outstanding:0
blocked_clients:0
Memory
used_memory:968991592
used_memory_human:924.10M
used_memory_rss:1049776128
used_memory_rss_human:1001.14M
used_memory_peak:1070912296
used_memory_peak_human:1021.30M
used_memory_lua:37888
maxmemory:1100000000
maxmemory_human:1.02G
maxmemory_policy:allkeys-lru
mem_allocator:jemalloc-3.6.0 #
Most likely you are running into scenario of high un-authenticated connections. Redis-benchmark first creates all the client connections (in your case -c 400 connections) and then authenticates them. The delay in auth causes high number of unauthenticated connections from a single IP and Azure Redis Cache closes them for DOS protection. Hence, the error “Server closed the connection”
You can try the redis-benchmark from here, which I have modified to authenticate as soon as a connection has been made and should solve this issue.
I am running a basic Jhipster generated app - see bottom for setup details - that is to be run on a single basic Heroku dyno for the time being. I use the embedded tomcat approach and all is working out fine apart from the start-up time on the production profile.
Running the server locally on my machine with Foreman I get the following results:
On the dev profile I get start-up times of <30sec for both the local and the remote Heroku database.
On the prod profile this goes up to >100sec for both cases.
This causes Heroku to terminate the instance before start-up completes due to them requiring the server to bind to their designated port within 60 seconds.
Thus my question is how/if I could reduce that time. I am aware of Heroku offering to increase the timeout interval to 120sec on a per-application basis. I would appreciate a more elegant approach, though, especially since I am running on a basic dyno so that timeout extension might not even be sufficient.
I know of Jetty/WebappRunner, too, but would prefer to stick to the simpler embedded tomcat setup if at all possible.
Finally, I have seen that for Rails there is the possibility of employing a proxy during start-up to bind to Heroku's port right away - I did not find a Spring equivalent, though.
Thanks in advance!
Setup
jhipster v017.2
authenticationType": "token"
hibernateCache": "no"
clusteredHttpSession": "no"
websocket": "no"
databaseType": "sql"
devDatabaseType": "postgresql"
prodDatabaseType": "postgresql"
useCompass": true
buildTool": "maven",
frontendBuilder": "grunt"
javaVersion": "7"
connecting to a Heroku Postgres database using Dbeaver client to after enabling ssl and using org.postgresql.ssl.NonValidatingFactory takes about 4-5 minutes to connect to the database.
Is this behavior normal?
Just uncheck "Show non-default databases".
It crashes / freezes because it tries to load a huge amount of databases in that host
No, this is not normal at all. Connecting should be quite instantaneous. A few seconds is already way too long, and minutes is completely off, so something else is going on.
If you try connecting with some other tool (like psql directly), do you have the same problem? I'd check there to make sure your code or some dependency is not doing something odd.
My Heroku app is www.inflationtrends.com.
Usually, when I run "pg:info" in Git Bash to see how many connections there are, that number is zero.
Recently, I've seen a spike in traffic -- not much, only a little over 1,000 in the past 48 hours -- and when I ran "pg:info" this morning (around 11 a.m. Eastern time), the result shows that there are 4 or 5 open connections.
My app is run using the Ruby gem Sinatra. In the Sinatra file, I have the following code:
after do
DB.disconnect
end
The "after do" loop disconnects from the PostgreSQL database after a page is loaded.
The variable "DB" has the connection info for my PostgreSQL database (username, password, host, port number, SSL mode requirement):
DB = Sequel.postgres(
db_name,
:user=>user,
:password=>password,
:host=>host,
:port=>port,
:sslmode=>sslmode
)
Is there some reason that there are open connections? Are there ways to close these connections? Are there more efficient ways to handle this situation?
An alternate way to check the number of open connections on Heroku is to type this into your console/terminal and replace "myapp" with your app's name:
heroku pg:info -a myapp
Have you considered that perhaps your site is getting traffic? When people visit your site and use your application connections will be opened.
Try adding some tracking code (such as Google Analytics) to your web pages, then check if the number of recorded visitors matches the number of open connections.
It is also possible that the database has connections opened by various maintenance tasks, such as backing up.
I grabbed the following toolbelt add-on which worked perfectly.
https://github.com/heroku/heroku-pg-extras#usage
heroku pg:killall --app xyz
I'm trying to get the production logs for the past few weeks off of heroku but when I do heroku logs, it just returns a few lines showing the production log for today.
Any way to get heroku logs for the past few weeks?
Thanks.
Any number up to 500 lines can now be retrieved using the -n flag.
heroku logs -n 420
As well you can also run:
heroku logs -t
And let that run for a while.
EDIT: And you can use third party tools like papertrail. See : papertrail link
(Correcting my own old response) Previously, Heroku only provided you with access to the last 100 lines. Now this limit apppears to have been raised.
There's also this pretty cool sounding logentries addon, with generous free offerings.
Not sure about going back given their limitations - but going forward you can forward your logs to an external syslog server.
Syslog drains - Premium Add-on