I have a Play application hosted by Heroku with 5 dynos. Its seems like my dynos were been restarted randomly in different time period. For example, three of them were restarted by itself 22 hours ago, and two of them were restarted 10 hours ago (not sure if this time was triggered by clearing the cache). It seems like that cached data isn't persistent between dynos. My problem is when I sent same request to my Heroku application multiple times, I get different cached response, in the response, some are most up to date data, others were old data. I assume this is because my request was processed by different dyno. After restart all my dyno fixed the problem(I assume this also clear cache in all dynos).
So I want to know what triggered the random dyno restart, and why it does that?
How to solve cached data inconsistency in this case?
Thanks
I think you should use mutualised cache in order to avoid this kind of problem when you scale horizontally.
Couchbase is a good solution to do it. We use this internally at Clever Cloud (http://www.clever-cloud.com/en/), that is the reason why we released a Couchbase as a service.
As for dyno restarts, did you try the documentation? Dyno's are cycled at least once per day
Related
Newbie in Newrelic here. I have an API service hosted on Heroku and being monitored at Newrelic.
While I was studying how to use newrelic. I found out my 2 workers are being underutilised with very low RPM and low transaction time. So I decided to cut down to one worker which saves me $36 a month. =]
Shortly after that I received tonnes of logEntries emails stating request timeouts of one of my web dynos. Looking into Newrelic. I found out that one of my actions are being called suspciously high number of times for 2-3 minutes.
The action being V1::CarsController#Index, which basically shows a collection of cars.
While I was not sure whether the deletion of one worker dyno has caused memcached to do something, I also suspect that may be someone is trying scrap the data off the database. I am not too sure how to further investigate into the issue. I wonder if I can track down the request IP and see it is the same? or how can I further investigate?
If further information is needed I am happy to provide in Edits!
Thanks
Is there any article on how synchronized are the times across multiple dynos in heroku, and how reliable they are?
I am running a node.js application on Heroku and need to know if the time that I get using Date.now() or new Date() is synchronized across multiple dynos?
Using cluster module, it seems that all the processes running on the same dyno have the same time. Do they use the local machine time or a clock server?
I tested with two dynos in the same app to print the timestamp as soon as they start and they seemed to have the same time. Is this reliable?
I did not test with dynos on different apps. Are they likely to have the same time or in a reliable difference range?
By the same time, i mean the time differences were less than 1000ms, which can be due to slightly difference process staring times.
I don't think so and I don't think it's reliable.
The post How do I establish clock synchronization in the cloud (AWS, heroku, etc) across many nodes? have some good general information both in the question and answer about NTP, virtual machines and the cloud.
This is a little late, but YES, Heroku dynos have synchronized time.
Screenshot of the Support Ticket:
NTP is pretty accurate too, and 1000ms is definitely within NTP's range.
I want to host a static website on Heroku, but not sure how many dynos to start off with.
It says on this page: https://devcenter.heroku.com/articles/dyno-requests that the number of requests a dyno can serve, depends on the language and framework used. But I have also read somewhere that 1 dyno only handles one request at a time.
A little confused here, should 1 web dyno be enough to host a static website with very small traffic (<1000 views/month, <10/hour)? And how would you go about estimating additional use of dynos as traffic starts to increase?
Hope I worded my question correctly. Would really appreciate your input, thanks in advance!
A little miffed since I had a perfectly valid answer deleted but here's another attempt.
Heroku dynos are single threaded, so they are capable of dealing with a single request at a time. If you had a dynamic page (php, ruby etc) then you would look at how long a page takes to respond at the server, say it took 250ms to respond then a single dyno could deal with 4 requests a second. Adding more dynos increases concurrency NOT performance. So if you had 2 dynos, in this scenario you be able to deal with 8 requests per second.
Since you're only talking static pages, their response time should be much faster than this. Your best way to identify if you need more is to look at your heroku log output and see if you have sustained levels of the 'queue' value; this means that the dynos are unable to keep up and requests are being queued for processing.
Since most HTTP 1.1 clients will create two TCP connections to the webserver when requesting resources, I have a hunch you'll see better performance on single clients if you start two dynos, so the client's pipelined resources requests can be handled pipelined as well.
You'll have to decide if it is worth the extra money for the (potentially slight) improved performance of a single client.
If you ever anticipate having multiple clients requesting information at once, then you'll probably want more than two dynos, just to make sure at least one is readily available for additional clients.
In this situation, if you stay with one dyno. The first one is free, the second one puts you over the monthly minimum and starts to trigger costs.
But, you should also realize with one dyno on Heroku, the app will go to sleep if it hasn't been accessed recently (I think this is around 30 minutes). In that case, it can take 5-10 seconds to wake up again and can give your users a very slow initial experience.
There are web services that will ping your site, testing for it's response and keeping it awake. http://www.wekkars.com/ for example.
Can anyone have an estimate of how much data can be inserted in a 5MB database?
Also, would 1 Dyno handle a slashdot,hackernews, etc frontpage?
Thanks.
Quiet a surprising amount... well enough to get you started thats for sure.
I use 1 dyno all the time for low traffic (like my personal website and a few xml servers, but obviously the great thing is if you start getting loads of visitors to your site and are having performance issues all it takes is one little click to add extra dynos.
You should also consider the worker hours, not sure about what you are doing, but a lot of apps could need background process.
This service could be interesting if you are using Ruby, it scales dynos on Heroku:
http://hirefireapp.com
I created two very simple Heroku apps to test out the service, but it's often taking several seconds to load the page when I first visit them:
Cropify - Basic Sinatra App (on github)
Textile2HTML - Even more basic Sinatra App (on github)
All I did was create a simple Sinatra app and deploy it. I haven't done anything to mess with or test the Heroku servers. What can I do to improve response time? It's very slow right now and I'm not sure where to start. The code for the projects are on github if that helps.
If your application is unused for a while it gets unloaded (from the server memory).
On the first hit it gets loaded and stays loaded until some time passes without anyone accessing it.
This is done to save server resources. If no one uses your app why keep resources busy and not let someone who really needs use them ?
If your app has a lot of continous traffic it will never be unloaded.
There is an official note about this.
You might also want to investigate the caching options you have on Heroku w/ Varnish and Memcached. These are persisted independent of the dynos.
For example, if you have an unchanging homepage, you can cache that for extended periods in Varnish by adding Cache-Control headers to the response. Then your users won't experience the load hit until they want to "do something" rather than when they arrive.
You should check out Tom Robinson's answer to "Scalability: How Does Heroku Work?" on Quora: http://www.quora.com/Scalability/How-does-Heroku-work
Heroku divides up server resources among many different customers/applications. Your app is allotted blocks of computing power. Heroku partitions based on resource demand. When you have a popular application that demands more power, you can pay for more 'dynos' (application containers) and then get a larger chunk of the pie in return.
In your case though, you are running a free app that few people--if any outside of you--are visiting/using. Therefore, Heroku cuts down on the resources you're getting by unloading your app--putting it in hibernation essentially--until there is a request made to your address. When that happens, and your app has been idling for a long time, it takes time to reload.
Add 1 extra dyno to keep your app from falling asleep, if that reload time is important.
I am having the same problem. I deployed a Rails 3 (1.9.2) app last night and it's slow. I know that 1.9.2/Rails 3 is in BETA on Heroku but the support ticket said it should be fine using some instructions they sent me.
I understand that the first request after a long time takes the longest. Makes sense. But simply loading pages that don't even connect to a DB taking 10 seconds sometimes is pretty bad.
Anyway, you might want to try what I'm going to do. That is profile my app and see how long it takes locally. If it's taking 400ms then something is wrong. But if I get 50ms locally and it still takes 10 seconds on Heroku then something is definitely wrong.
Obviously, caching helps but you only get 5MB for free and once again, with ONE person using the site, it shouldn't be that slow.
I had the same problem with every app I have put on via heroku's free account. Now there are options of adding dynos so that your app does not get offloaded while it is not being used for a while, you can also try using redis or memcached for caching. But I used a hacky solution for my small scale project, I basically built web scraper put it inside an infinite loop with sleep and tada the website actually had much better response times(I guess it was not getting offloaded because of the script).