I have an app deployed on meteor.com, which after 2-5 days of working perfectly suddenly becomes unresponsive to requests. Restarting the app by deploying it again causes it to work properly.
What's the best way to debug this? I don't know how to reproduce the issue on my local server since it only happens intermittently.
Nothing seemed off in meteor logs other than this line:
spiderable: phantomjs failed: { [Error: Command failed: ] killed: true, code: null, signal: 'SIGTERM' } stderr:
I tried removing the spiderable smart package and the issue still occurs.
Meteor.com's hosting via meteor deploy will kill your app if its not visited for somewhere near 2-4 hours.
This doesn't mean that its dead though, if you load it up in your web browser it'll take a couple of seconds extra to get running again. It might take longer if you have a heavy startup load, though.
Meteor hasn't brought up details of its paid solutions yet, but if you want to keep your site up for longer it should automatically if it has a visit every hour or two. I'm not suggesting that its the best way to do it, but you could remotely ping it too.
Related
I am using Memurai 2.0.2 for cache in my distributed application. It runs different services on different machines and all services have Memurai details with them.
The problem that happens is, that sometimes Memurai process just hangs. The Memurai process keeps on running but no queries are served. I am not able to create a connection to it. It's log file consists of an error:
Error trying to rename the existing AOF to old tempfile: Broken pipe
This generally occurs when I restart the Memurai service. Although I am not sure what is the reason for it. Memurai works fine if I restart its service once.
What can be the issue here? What steps can I take to avoid/ minimize its occurrence?
Memurai 2.0.2 is fairly outdated now. Perhaps get the latest version (3.1.4 at the time of this response) at https://www.memurai.com/get-memurai
For whoever looking for an answer, this happened because another service restarted Memurai service when background rewriting of AOF was in progress. Due to this, some zombie processes were getting created and when Memurai started again, this error was coming up.
The solution that we did was to check if any background rewriting is happening by using settings aof_rewrite_scheduled and aof_rewrite_in_progress from Persistence info. If any of these flags is true then don't stop the service.
Disclaimer: Please tell me if the question is too broad, and I will do my best to narrow it down.
We have an Heroku app which is running 2 web 1X dynos. This infrastructure has been running for the last 9 months.
However, in the last few weeks, we had several episodes where the app would see its response times skyrocketing for about an hour, before returning to normal without us doing anything about it.
On the pictures below, you can find an extract of Heroku Metrics during one of these "episodes", which happened yesterday afternoon.
As you can see, response time is going up and eventually, almost any request made to the server gets a timeout. During the event, it was barely possible to even load the home page of our website, hosted on this app. Most of the times, we would get the "Application Error" Heroku page.
What I see is:
The amount of requests to the server (failed or not) was not crazily high (less than 1000 every 10 minutes). For this reason, I think a DDOS attack is out of the picture.
Everything that is shown by Heroku Logs is that the failed request get a 503 (Service Unavailable) error, which would make me think about an overload.
The dynos do not seem overloaded. The memory usage is low, and the dyno load is reasonable, nothing unusual.
Heroku reported no issue during our crash event, as https://status.heroku.com/ states (last incident was on the 1st of July).
Restarting the dynos through several methods (from the interface, a command line or triggering an automatic deployment via our Gitlab repository) had no effect.
I am quite unsure as to how to interpretate these metrics, and what would be the solution to ensure this kind of episode does not happen again.
So my question is: where should I look? Is there some kind of documentation about how to investigate crashes on Heroku apps?
My app sometimes crashes and I would like to know if there is a Add-on or Dyno that I can deploy which will detect when my app is crashed and restart is automatically.
There's no add-on which will auto-deploy your crashed app on heroku, but here's a work around.
Use a log-management add-on(like papertrail)
Watch out for log messages you get when your heroku app crashes.
Add custom web-hooks that gets triggered whenever you app gives one of those error-logs that correspond to an app crash.
Make these web-hooks to call a custom service that redeploys your application. https://help.papertrailapp.com/kb/how-it-works/web-hooks/
When you encounter an "app crash" situation, it usually means only the "web dyno" is crashed. Your service workers might still be running. Though this approach answers your query, I would recommend not to use it in practice because you would want to manually see you logs, find the reason for app-crash and fix it so you don't have to face it in future.
I've got a phantomjs app (http://css.benjaminbenben.com) running on heroku - it works well for some time but then I have to run heroku restart because it requests start timing out.
I'm looking for a stop-gap solution (I've gone from around 6 to 4500 daily visitors over the last week), and I was considering exiting the process after it had served a set number of requests to fire a restart.
Will this work? And would this be considered bad practice?
(in case you're interested, the app source is here - https://github.com/benfoxall/wtcss)
It'd work, as long as you don't crash within I think 10 minutes of the last crash. If it's too frequent the process will stay down.
It's not bad practice, but it's not great practice. You should figure out what is causing your server to hang, of course.
I am using Meteor on Heroku (free tier) with MongoHQ. My app is very simple right now, it loads 3-4 entries from a Collection, but when I deploy it to Heroku, I am seeing ridiculous load times (1-2 minutes). The HTML is rendered immediately. When I deploy to Meteor.com's free server, load times are a lot lower but still about 15 seconds for 4 tiny pieces of data. I'm not seeing this whatsoever when I deploy locally, app pulls data from the DB right away.
It is worth noting that I don't think it's an "idling" issue for Heroku. Even if I already have one browser window with the app just opened, if I use a different browser and try again I still get 1-2 minute load times. Once the data is loaded, however, performance goes back to being great, I can read and write with no problems.
What am I missing? I'm not seeing any errors in the console, mongo shows several queries in the logs and shows that it is responding quickly with 4 documents, but apparently somewhere in the middle there's a traffic jam. Any help with this is greatly appreciated, if I can't get past this Meteor is useless for my needs right now.
UPDATE: I've been watching it closely in Firebug, and it looks like the performance is largely inconsistent. Sometimes a simple refresh will take 1 minute, sometimes it will take 10 seconds. But what I've noticed is that the times when its slow, it GETs the sockjs/info file, then right after that the sockjs POST is aborted (sometimes multiple times). When it runs fast, the POST and subsequent POSTs run smoothly
Slow:
GET http://pocleaderboard.herokuapp.com/sockjs/info 200 OK 22ms
POST http://pocleaderboard.herokuapp.com/sockjs/029/su0d77fb/xhr Aborted
GET http://pocleaderboard.herokuapp.com/sockjs/info 200 OK 27ms
POST http://pocleaderboard.herokuapp.com/sockjs/132/uljqusxd/xhr Aborted
GET http://pocleaderboard.herokuapp.com/sockjs/info 200 OK 28ms
POST http://pocleaderboard.herokuapp.com/sockjs/154/kcbr6a5p/xhr Aborted
Fast(er):
GET http://pocleaderboard.herokuapp.com/sockjs/info 200 OK 1.08s
POST http://pocleaderboard.herokuapp.com/sockjs/755/xiggb555/xhr 200 OK 1.02s
Meteor gets loaded that fast locally, because it doesn't depend on your internet connection and the files can just be read from your harddrive and don't need to be downloaded.
And once the data is loaded it's the same everywhere you host, because the client (you) perform all actions on your cached mongo database and then just wait for the server to say if the action was alright or not.
But for the Heroku loading times, I have no idea, Sorry!
UPDATE:
These are the long-pulls from SockJS that is used by Meteor.
Normally these pulls only get Aborted on a hot code push (when a file is added/changed/removed).
Either you or Heroku seem to write or change something in the directory.
Because then a hot code push may be initiated by Meteor.
Heroku may not support web-sockets, which means you're stuck with the slower polling approach. See this:
https://devcenter.heroku.com/articles/using-socket-io-with-node-js-on-heroku