The following error is being returned after a system restart (there was a lack of memory issue) before the restart.
The Phusion Passenger application server tried to start the web application by communicating with a helper process that we call a "preloader". However, this helper process sent a response that looks like gibberish.
The response is as follows:
The response is a blank. Although it is suggested to fill out a bug report, I do not suspect this to be a bug, as everything was working previously. the only diagnostics provided indicate:
PID 1554
x Receive and process preloader response — 0.4s
What can be done about this?
A server re-start fixes the issue. Satisfactory outcome, unsatisfactory motive...
Related
I am using Memurai 2.0.2 for cache in my distributed application. It runs different services on different machines and all services have Memurai details with them.
The problem that happens is, that sometimes Memurai process just hangs. The Memurai process keeps on running but no queries are served. I am not able to create a connection to it. It's log file consists of an error:
Error trying to rename the existing AOF to old tempfile: Broken pipe
This generally occurs when I restart the Memurai service. Although I am not sure what is the reason for it. Memurai works fine if I restart its service once.
What can be the issue here? What steps can I take to avoid/ minimize its occurrence?
Memurai 2.0.2 is fairly outdated now. Perhaps get the latest version (3.1.4 at the time of this response) at https://www.memurai.com/get-memurai
For whoever looking for an answer, this happened because another service restarted Memurai service when background rewriting of AOF was in progress. Due to this, some zombie processes were getting created and when Memurai started again, this error was coming up.
The solution that we did was to check if any background rewriting is happening by using settings aof_rewrite_scheduled and aof_rewrite_in_progress from Persistence info. If any of these flags is true then don't stop the service.
My app runs on Heroku with unicorn and uses sucker_punch to send a small quantity of emails in the background without slowing the web UI. This has been working pretty well for a few weeks.
I changed the unicorn config to the Heroku recommended config. The recommended config
includes an option for the number of unicorn processes and I upped the number of processes from 2 to 3.
Apparently that was too much. The sucker_punch jobs stopped running. I have log messages that indicate when they are queued and I have messages that indicate when they start processing. The log shows them being queued but the processing never starts.
My theory is that I exceeded memory by going from 2 to 3 unicorns.
I did not find a message anywhere indicating a problem.
Q1: should I expect to find a failure messsage somewhere? Something like "attempting to start sucker_punch -- oops, not enough memory"?
Q2: Any suggestions on how I can be notified of a failure like this in the future.
Thanks.
If you are indeed exceeding dyno memory, you should find R14 or R15 errors in your logs. See https://devcenter.heroku.com/articles/error-codes#r14-memory-quota-exceeded
A more likely problem, though, given that you haven't found these errors, is that something within the perform method of your sucker punch worker is throwing an exception. I've found sucker punch tasks to be a pain to debug because it appears the lib swallows all exceptions silently. Try instantiating your task and calling perform on it from a rails console to make sure that it behaves as you expect.
For example, you should be able to do this without causing an exception:
task = YourTask.new
task.perform :something, 55
I'm trying to debug a very weird issue that is happening on our environments.
I have a Rails app running on ree-1.8.7-2011.12 that is exhibiting an strange behavior when the ElasticSearch server it is connected to is suspended for some reason (to suspend it, start the ES server in foreground then CTRL-Z it), the Ruby process hangs forever and doesn't timeout or anything like that. It doesn't happen if the ES server is suspended before the rails app connects for the first time, it only happens when the Rails app runs some queries and then the ES server decides to stop answering requests.
Since I'm not that good with native code, i don't even know where to start trying to figure this out (and yeah, unfortunately we can't upgrade our Ruby for many reasons right now). Going in the activity monitor and getting a sample of the process I get this thread dump:
So it seems that it's locked at this curl_wait_for_resolv call.
I'm using the latest ElasticSearch and the Rubberband ES adapter for Ruby. This is reproductible on Mac OS and Linux.
Any hints on how to debug this or correctly timeout the server?
And the issue was at Rubberband, I have filed a pull request for it.
I am trying to find out if any http requests are made during installation of an msi package. It appears to me that the process under which the http request is made shares no lineage with the process under which the installer executes.
For example, I install an app that makes http calls during installation. Using SysInternals process monitor, I see the process created when the install kicks off. Using MS NetworkMonitor I can see the process used to generate the http request. Filtering in Process Monitor after the fact shows that there is no relationship between the http process, and the install process.
I am thinking that somehow the OS says to use a new process whenever an http request is made. My most important requirement is that I be able to relate one to the other, in order to definitively say "This app installation called these http resources during install". So I don't have to have a perfect understanding of how it all works under the covers, but, I am at a standstill right now. I've concluded that there is no way to relate the two. Am I wrong?
Okay, let's assume that msiexec.exe invokes a helper and that helper invokes whatever is causing the HTTP traffic. Right after the first helper spawns the child it kills itself. This process is too short-lived to normally see the relationships here.
Enter the "Process Tree" feature of Process Monitor. Keep Process Monitor running without any filters on process events. After you are done you can then press Ctrl+T to see the Process Tree (see below).
The grayed icons tell you the process is not active anymore. Furthermore the last column is the end time of the process. But best of all you can see which process created which other process from this, even for very very short-lived processes.
Mark Russinovich, author of Process Monitor and its predecessors, demonstrated this at TechEd about a month ago.
Although this may not answer the question entirely, it should get you going in the right direction. After all Process Monitor also includes network activity filtering (albeit crude, compared to Network Monitor and Wireshark :)).
btw: the green bar in the above screenshot is the "timeline" where you can see the runtime of the process in relation to other processes. Very nifty.
Are you using a custom action for making web requests? Windows Installer Service Process runs custom actions in a separate instance of MSIEXEC than your main set-up instance.
More information here:
http://blogs.msdn.com/b/astebner/archive/2005/03/02/384088.aspx
I have a problem concerning a windows service that I have implemented. The service does what it has to and logs various information but when I try to stop it, it doesn't seem to stop and keeps logging.
I am not very familiar with services, so if anyone knows why this is happening please tell me.
Edit: I have tested the service on two servers and the weird thing is that on one server it stops normally, but on the other it doesn't. That is what I don't understand.
IN case you have any threads ensure that you exit those threads in your stop routine. Some threads may still be lingering which is why it does not stop.
One possible reason could be because of some remaining thread (as mentioned by 'ckv').
Have you set proper handler using RegisterServiceCtrlHandler function?
Also make sure that the SERVICE_STATUS structure that you use has its dwControlsAccepted set to accept shutdown or stop (by setting SERVICE_ACCEPT_SHUTDOWN & SERVICE_ACCEPT_STOP). and don't forget to update the service status (using SetServiceStatus function) once the service is up and running.