I tried to follow this:
https://www.vultr.com/docs/how-to-install-redmine-on-centos-7
I think I had it working on my own little centos7 box here. Though it seems it never comes up and always exits in the /var/log/httpd/error_log but don't see why. I do see two warnings but to me those shouldn't effect it starting up? Does any extra set of eyes see anything in this log?
[Wed May 08 15:12:53.300528 2019] [suexec:notice] [pid 26924] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[ N 2019-05-08 15:12:53.3486 26926/T1 age/Wat/WatchdogMain.cpp:1366 ]: Starting Passenger watchdog...
[ N 2019-05-08 15:12:53.3860 26929/T1 age/Cor/CoreMain.cpp:1339 ]: Starting Passenger core...
[ N 2019-05-08 15:12:53.3863 26929/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ W 2019-05-08 15:12:53.4070 26929/T1 age/Cor/CoreMain.cpp:1006 ]: WARNING: potential privilege escalation vulnerability detected. Phusion Passenger is running as root, and part(s) of the Passenger root path (/usr/local/rvm/gems/ruby-2.6.3/gems/passenger-6.0.2) can be changed by non-root user(s):
- /usr/local/rvm/gems/ruby-2.6.3/gems/passenger-6.0.2 is not secure: it can be modified by group rvm
- /usr/local/rvm/gems/ruby-2.6.3/gems is not secure: it can be modified by group rvm
- /usr/local/rvm/gems/ruby-2.6.3 is not secure: it can be modified by group rvm
- /usr/local/rvm/gems is not secure: it can be modified by group rvm
- /usr/local/rvm is not secure: it can be modified by group rvm
Please either fix up the permissions for the insecure paths, or install Passenger in a different location that can only be modified by root.
[ N 2019-05-08 15:12:53.4071 26929/T1 age/Cor/CoreMain.cpp:1014 ]: Passenger core online, PID 26929
[ N 2019-05-08 15:12:53.6931 26929/T8 age/Cor/CoreMain.cpp:671 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2019-05-08 15:12:53.6931 26929/T1 age/Cor/CoreMain.cpp:1245 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2019-05-08 15:12:53.6932 26929/Ta Ser/Server.h:902 ]: [ServerThr.2] Freed 0 spare client objects
[ N 2019-05-08 15:12:53.6932 26929/Ta Ser/Server.h:558 ]: [ServerThr.2] Shutdown finished
[ N 2019-05-08 15:12:53.6933 26929/T8 Ser/Server.h:902 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2019-05-08 15:12:53.6933 26929/T8 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2019-05-08 15:12:53.6933 26929/Tc Ser/Server.h:902 ]: [ServerThr.3] Freed 0 spare client objects
[ N 2019-05-08 15:12:53.6933 26929/Te Ser/Server.h:902 ]: [ServerThr.4] Freed 0 spare client objects
[ N 2019-05-08 15:12:53.6933 26929/Tc Ser/Server.h:558 ]: [ServerThr.3] Shutdown finished
[ N 2019-05-08 15:12:53.6933 26929/Te Ser/Server.h:558 ]: [ServerThr.4] Shutdown finished
[ N 2019-05-08 15:12:53.6938 26929/Tg Ser/Server.h:902 ]: [ApiServer] Freed 0 spare client objects
[ N 2019-05-08 15:12:53.6938 26929/Tg Ser/Server.h:558 ]: [ApiServer] Shutdown finished
WARNING: PassengerRoot cannot occur within <VirtualHost> section
[Wed May 08 15:12:53.732238 2019] [auth_digest:notice] [pid 26924] AH01757: generating secret for digest authentication ...
[ N 2019-05-08 15:12:53.7823 26956/T1 age/Wat/WatchdogMain.cpp:1366 ]: Starting Passenger watchdog...
[ N 2019-05-08 15:12:53.8208 26959/T1 age/Cor/CoreMain.cpp:1339 ]: Starting Passenger core...
[ N 2019-05-08 15:12:53.8211 26959/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ W 2019-05-08 15:12:53.8418 26959/T1 age/Cor/CoreMain.cpp:1006 ]: WARNING: potential privilege escalation vulnerability detected. Phusion Passenger is running as root, and part(s) of the Passenger root path (/usr/local/rvm/gems/ruby-2.6.3/gems/passenger-6.0.2) can be changed by non-root user(s):
- /usr/local/rvm/gems/ruby-2.6.3/gems/passenger-6.0.2 is not secure: it can be modified by group rvm
- /usr/local/rvm/gems/ruby-2.6.3/gems is not secure: it can be modified by group rvm
- /usr/local/rvm/gems/ruby-2.6.3 is not secure: it can be modified by group rvm
- /usr/local/rvm/gems is not secure: it can be modified by group rvm
- /usr/local/rvm is not secure: it can be modified by group rvm
Please either fix up the permissions for the insecure paths, or install Passenger in a different location that can only be modified by root.
[ N 2019-05-08 15:12:53.8418 26959/T1 age/Cor/CoreMain.cpp:1014 ]: Passenger core online, PID 26959
[Wed May 08 15:12:53.842702 2019] [lbmethod_heartbeat:notice] [pid 26924] AH02282: No slotmem from mod_heartmonitor
[Wed May 08 15:12:53.924503 2019] [mpm_prefork:notice] [pid 26924] AH00163: Apache/2.4.6 (CentOS) Phusion_Passenger/6.0.2 mod_auth_gssapi/1.5.1 mod_nss/1.0.14 NSS/3.28.4 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Wed May 08 15:12:53.924564 2019] [core:notice] [pid 26924] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[ N 2019-05-08 15:12:54.4980 26929/T1 age/Cor/TelemetryCollector.h:531 ]: Message from Phusion: End time can not be before or equal to begin time
[ N 2019-05-08 15:12:54.5497 26929/T1 age/Cor/CoreMain.cpp:1324 ]: Passenger core shutdown finished
[ N 2019-05-08 15:12:57.5521 26959/T5 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
I edited the 00-base.conf as indicated in the link, and the redmine.conf same, with the right paths that were displayed by the
passenger-install-apache2-module
command as described on the site too.
not sure what else to check or do, I did disable SELinux too.
Problem: I am not able start the ruby application with passenger + apache2
Ruby Version: ruby 2.1.2p95 (2014-05-08 revision 45877)
Phusion Passenger 5.0.29
OS: Debian 8 jessie
Apache/2.4.10
When starting apache with following command then getting error in the logs.
sudo /etc/init.d/apache2 start
tail -f /var/log/fk-portkey/production.log
Initializing Phusion Passenger...
[ 2019-03-22 10:31:03.1041 14486/7fc3bde5e740 agents/Base.cpp:1591 ]: Random seed: 1553230863
[ 2019-03-22 10:31:03.1042 14486/7fc3bde5e740 agents/Watchdog/Main.cpp:698 ]: Starting Watchdog...
[ 2019-03-22 10:31:03.1043 14486/7fc3bde5e740 agents/Watchdog/Main.cpp:538 ]: Options: { 'analytics_log_user' => 'nobody', 'default_group' => 'nogroup', 'default_python' => 'python', 'default_ruby' => '/usr/lib/ruby-flo/bin/ruby', 'default_user' => 'nobody', 'log_level' => '3', 'max_pool_size' => '6', 'passenger_root' => '/usr/lib/ruby-flo/lib/ruby/gems/2.1.0/gems/passenger-4.0.58', 'passenger_version' => '4.0.58', 'pool_idle_time' => '300', 'temp_dir' => '/tmp', 'union_station_gateway_address' => 'gateway.unionstationapp.com', 'union_station_gateway_port' => '443', 'user_switching' => 'true', 'web_server_passenger_version' => '4.0.58', 'web_server_pid' => '14485', 'web_server_type' => 'apache', 'web_server_worker_gid' => '33', 'web_server_worker_uid' => '33' }
[ 2019-03-22 10:31:03.1109 14489/7fd1250f7740 agents/Base.cpp:1591 ]: Random seed: 1553230863
[ 2019-03-22 10:31:03.1111 14489/7fd1250f7740 agents/HelperAgent/Main.cpp:643 ]: Starting PassengerHelperAgent...
[ 2019-03-22 10:31:03.1137 14489/7fd1250f7740 agents/HelperAgent/Main.cpp:650 ]: PassengerHelperAgent online, listening at unix:/tmp/passenger.1.0.14485/generation-0/request
[ 2019-03-22 10:31:03.1265 14495/7ff2ad310780 agents/Base.cpp:1591 ]: Random seed: 1553230863
[ 2019-03-22 10:31:03.1306 14495/7ff2ad310780 agents/LoggingAgent/Main.cpp:333 ]: Starting PassengerLoggingAgent...
[ 2019-03-22 10:31:03.1319 14495/7ff2ad310780 agents/LoggingAgent/Main.cpp:321 ]: PassengerLoggingAgent online, listening at unix:/tmp/passenger.1.0.14485/generation-0/logging
[ 2019-03-22 10:31:03.1325 14486/7fc3bde5e740 agents/Watchdog/Main.cpp:728 ]: All Phusion Passenger agents started!
[ 2019-03-22 10:31:03.1350 14485/7f8602b75780 apache2/Hooks.cpp:1583 ]: Shutting down Phusion Passenger...
[ 2019-03-22 10:31:03.1362 14489/7fd1250f7740 agents/HelperAgent/Main.cpp:599 ]: Received command to exit gracefully. Waiting until 5 seconds after all clients have disconnected...
[ 2019-03-22 10:31:03.1368 14486/7fc3bde5e740 agents/Watchdog/Main.cpp:741 ]: Web server exited gracefully; gracefully shutting down all agents...
When I start the application with the following command then getting error in the logs.
sudo /etc/init.d/fk-portkey start
tail -f /var/log/fk-portkey/production.log
Error: https://pastebin.com/gDJ9W8jt
Content of /tmp/passenger-error-Xjl3B1.html
When reloading apache getting following logs
/etc/init.d/apache2 reload
tail -f /var/log/fk-portkey/production.log
[ 2019-03-25 14:57:07.4026 11018/7fa77d79a700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2019-03-25 14:57:07.4027 11027/7f22a3ff5700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished
[ 2019-03-25 14:57:07.4027 11018/7fa782ca1780 age/Cor/CoreMain.cpp:901 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2019-03-25 14:57:07.4027 11027/7f22a37f4700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects
[ 2019-03-25 14:57:07.4028 11027/7f22a37f4700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished
[ 2019-03-25 14:57:07.4031 11018/7fa75dffb700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects
[ 2019-03-25 14:57:07.4032 11018/7fa75dffb700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished
[ 2019-03-25 14:57:07.4034 11018/7fa75effd700 Ser/Server.h:817 ]: [ServerThr.5] Freed 128 spare client objects
[ 2019-03-25 14:57:07.4035 11018/7fa75effd700 Ser/Server.h:464 ]: [ServerThr.5] Shutdown finished
[ 2019-03-25 14:57:07.4035 11018/7fa75f7fe700 Ser/Server.h:817 ]: [ServerThr.4] Freed 128 spare client objects
[ 2019-03-25 14:57:07.4036 11018/7fa75f7fe700 Ser/Server.h:464 ]: [ServerThr.4] Shutdown finished
[ 2019-03-25 14:57:07.4037 11027/7f22a9286780 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished
[ 2019-03-25 14:57:07.4041 11018/7fa77d79a700 Ser/Server.h:817 ]: [ServerThr.1] Freed 128 spare client objects
[ 2019-03-25 14:57:07.4042 11018/7fa75e7fc700 Ser/Server.h:817 ]: [ServerThr.6] Freed 128 spare client objects
[ 2019-03-25 14:57:07.4043 11018/7fa75e7fc700 Ser/Server.h:464 ]: [ServerThr.6] Shutdown finished
[ 2019-03-25 14:57:07.4043 11018/7fa75ffff700 Ser/Server.h:817 ]: [ServerThr.3] Freed 128 spare client objects
[ 2019-03-25 14:57:07.4043 11018/7fa75ffff700 Ser/Server.h:464 ]: [ServerThr.3] Shutdown finished
[ 2019-03-25 14:57:07.4042 11018/7fa77d79a700 Ser/Server.h:464 ]: [ServerThr.1] Shutdown finished
[ 2019-03-25 14:57:07.4053 11018/7fa77cf18700 Ser/Server.h:817 ]: [ServerThr.2] Freed 128 spare client objects
[ 2019-03-25 14:57:07.4054 11018/7fa77cf18700 Ser/Server.h:464 ]: [ServerThr.2] Shutdown finished
[ 2019-03-25 14:57:07.4402 11054/7f9245b52780 age/Wat/WatchdogMain.cpp:1291 ]: Starting Passenger watchdog...
[ 2019-03-25 14:57:07.4574 11058/7f82fd844780 age/Cor/CoreMain.cpp:982 ]: Starting Passenger core...
[ 2019-03-25 14:57:07.4578 11058/7f82fd844780 age/Cor/CoreMain.cpp:235 ]: Passenger core running in multi-application mode.
[ 2019-03-25 14:57:07.4632 11058/7f82fd844780 age/Cor/CoreMain.cpp:732 ]: Passenger core online, PID 11058
[ 2019-03-25 14:57:07.4768 11067/7f04580e0780 age/Ust/UstRouterMain.cpp:529 ]: Starting Passenger UstRouter...
[ 2019-03-25 14:57:07.4781 11067/7f04580e0780 age/Ust/UstRouterMain.cpp:342 ]: Passenger UstRouter online, PID 11067
[ 2019-03-25 14:57:07.4904 11018/7fa782ca1780 age/Cor/CoreMain.cpp:967 ]: Passenger core shutdown finished
App 11170 stdout:
I have set-up a ELK but I see elasticsearch not creating the Index and unable to upload the data, Service Elasticsearch and Logstash both are running..
Below is the details.. However I do not see anything on he logs.
Elastic config:
[root#aruba-elk2 rm_logs]# cat /etc/elasticsearch/elasticsearch.yml
# Elasticserach config
#########################
cluster.name: log-cohort-test
node.name: aruba-elk2
node.master: true
path:
data: /elk/lib/elasticsearch
logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
bootstrap.system_call_filter: False
[root#aruba-elk2 rm_logs]#
[root#aruba-elk2 rm_logs]#
LOGSTASH COnfig:
[root#aruba-elk2 rm_logs]# cat /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
[root#aruba-elk2 rm_logs]# cat /etc/logstash/conf.d/logstash-syslog.conf
input {
file {
path => [ "/elk/rm_logs/*.txt" ]
type => "rmlog"
}
}
filter {
if [type] == "rmlog" {
grok {
match => { "message" => "%{HOSTNAME:hostname},%{DATE:date},%{HOUR:hour1}:%{MINUTE:minute1},%{NUMBER}-%{WORD},%{USER:user},%{USER:user2} %{NUMBER:pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:number1} %{NUMBER:number2} %{DATA} %{HOUR:hour2}:%{MINUTE:minute2} %{HOUR:hour3}:%{MINUTE:minute3} %{GREEDYDATA:command},%{PATH:path}" }
add_field => [ "received_at", "%{#timestamp}" ]
}
}
}
output {
if [type] == "rmlog" {
elasticsearch {
hosts => ["aruba-elk2:9200"]
manage_template => false
index => "rmlog-%{+YYYY.MM.dd}"
#document_type => "messages"
}
}
}
Input data Source:
[root#aruba-elk2 rm_logs]# cd /elk/rm_logs/
[root#aruba-elk2 rm_logs]# ls -ltrh | head
total 2.6M
-rw-r--r-- 1 root root 558 Jan 11 11:27 dbxchw092.txt
-rw-r--r-- 1 root root 405 Jan 11 11:27 dbxtx220.txt
-rw-r--r-- 1 root root 241 Jan 11 11:27 dbxcvm139.txt
-rw-r--r-- 1 root root 455 Jan 11 11:27 dbxcnl038.txt
-rw-r--r-- 1 root root 230 Jan 11 11:27 dbxchw052.txt
-rw-r--r-- 1 root root 143 Jan 11 11:27 dbxtx222.txt
-rw-r--r-- 1 root root 577 Jan 11 11:27 dbxtx224.txt
-rw-r--r-- 1 root root 274 Jan 11 11:27 dbxcvm082.txt
-rw-r--r-- 1 root root 281 Jan 11 11:27 dbxcsb003.txt
Sample of above data file:
testhost-in2,19/01/11,06:34,04-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /test/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-
testhost-in2,19/01/11,06:40,09-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-\
testhost-in2,19/01/11,06:45,14-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:01 rm -rf /
LOGS:
Logstash logs:
[root#aruba-elk2 logstash]# cat logstash-plain.log
[2019-01-12T23:48:31,653][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-12T23:48:34,959][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>48, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-01-12T23:48:35,374][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://aruba-elk2:9200/]}}
[2019-01-12T23:48:35,588][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://aruba-elk2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://aruba-elk2:9200/][Manticore::SocketException] Connection refused"}
[2019-01-12T23:48:35,608][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//aruba-elk2:9200"]}
[2019-01-12T23:48:36,063][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_076330d5fd2c2b811bc1960a3d0547be", :path=>["/elk/rm_logs/*.txt"]}
[2019-01-12T23:48:36,095][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x424bb675 run>"}
[2019-01-12T23:48:36,155][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-01-12T23:48:36,156][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-01-12T23:48:36,542][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-01-12T23:48:40,796][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://aruba-elk2:9200/"}
[2019-01-12T23:48:40,855][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-01-12T23:48:40,859][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Elasticsearch LOGS:
[root#aruba-elk2 elasticsearch]# cat gc.log.0.current| tail
2019-01-13T00:13:29.280+0530: 1237.781: Total time for which application threads were stopped: 0.0002681 seconds, Stopping threads took: 0.0000316 seconds
2019-01-13T00:13:31.281+0530: 1239.782: Total time for which application threads were stopped: 0.0003670 seconds, Stopping threads took: 0.0000586 seconds
2019-01-13T00:13:32.281+0530: 1240.782: Total time for which application threads were stopped: 0.0003134 seconds, Stopping threads took: 0.0000708 seconds
2019-01-13T00:13:37.282+0530: 1245.783: Total time for which application threads were stopped: 0.0004663 seconds, Stopping threads took: 0.0001315 seconds
2019-01-13T00:13:51.284+0530: 1259.785: Total time for which application threads were stopped: 0.0004230 seconds, Stopping threads took: 0.0000691 seconds
2019-01-13T00:13:57.286+0530: 1265.787: Total time for which application threads were stopped: 0.0008421 seconds, Stopping threads took: 0.0002697 seconds
2019-01-13T00:13:58.287+0530: 1266.787: Total time for which application threads were stopped: 0.0004467 seconds, Stopping threads took: 0.0000706 seconds
2019-01-13T00:14:11.288+0530: 1279.789: Total time for which application threads were stopped: 0.0004702 seconds, Stopping threads took: 0.0001105 seconds
2019-01-13T00:14:18.289+0530: 1286.790: Total time for which application threads were stopped: 0.0004123 seconds, Stopping threads took: 0.0000750 seconds
Any help will be appreciated..
this is my first question.
I'm confusing on my IntelliJ Idea hanging 1 minute when starting and debugging with network on. Could anyone help me? Thanks.
I have tried IntelliJC and IntelliJU which are both 15.0.4. When I cut down the network, It works with no problem. But, with network on, it becomes hanging when starting and debugging.
Through logs, I found that there's almost 1 minute between I starting IntelliJ and its log beginning.
I even captured its network packets, found nothing, there was no connection.
I'm using Macbook, and I found the system log:
Mar 11 07:49:36 Coderec's MBP idea[1867]: Value of IDEA_JDK: (null)
Mar 11 07:49:36 Coderec's MBP idea[1867]: fullFileName is: /Applications/IntelliJ IDEA 15 CE.app/Contents/bin/idea.vmoptions
Mar 11 07:49:36 Coderec's MBP idea[1867]: fullFileName exists: /Applications/IntelliJ IDEA 15 CE.app/Contents/bin/idea.vmoptions
Mar 11 07:49:36 Coderec's MBP idea[1867]: Value of IDEA_VM_OPTIONS is (null)
Mar 11 07:49:36 Coderec's MBP idea[1867]: Processing VMOptions file at /Applications/IntelliJ IDEA 15 CE.app/Contents/bin/idea.vmoptions
Mar 11 07:49:36 Coderec's MBP idea[1867]: Done
and the IntelliJ log:
2016-03-11 07:50:37,529 [ 60385] INFO - #com.intellij.idea.Main - ------------------------------------------------------ IDE STARTED ------------------------------------------------------
2016-03-11 07:50:37,559 [ 60415] INFO - #com.intellij.idea.Main - IDE: IntelliJ IDEA (build #IC-143.2287.1, 23 Feb 2016 00:00)
2016-03-11 07:50:37,559 [ 60415] INFO - #com.intellij.idea.Main - OS: Mac OS X (10.11.3, x86_64)
2016-03-11 07:50:37,559 [ 60415] INFO - #com.intellij.idea.Main - JRE: 1.8.0_40-release-b132 (JetBrains s.r.o)
2016-03-11 07:50:37,559 [ 60415] INFO - #com.intellij.idea.Main - JVM: 25.40-b25 (OpenJDK 64-Bit Server VM)
2016-03-11 07:50:37,560 [ 60416] INFO - #com.intellij.idea.Main - JVM Args: -Dfile.encoding=UTF-8 -XX:+UseConcMarkSweepGC -XX:SoftRefLRUPolicyMSPerMB=50 -ea -Dsun.io.useCanonCaches=false -Djava.net.preferIPv4Stack=true -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Xverify:none -Xbootclasspath/a:../lib/boot.jar -Xms128m -Xmx750m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=240m -XX:+UseCompressedOops -Djb.vmOptionsFile=/Applications/IntelliJ IDEA 15 CE.app/Contents/bin/idea.vmoptions -Didea.java.redist=custom-jdk-bundled -Didea.home.path=/Applications/IntelliJ IDEA 15 CE.app/Contents -Didea.executable=idea -Didea.paths.selector=IdeaIC15
2016-03-11 07:50:37,561 [ 60417] INFO - #com.intellij.idea.Main - ext: /Applications/IntelliJ IDEA 15 CE.app/Contents/jre/jdk/Contents/Home/jre/lib/ext: [cldrdata.jar, dnsns.jar, jaccess.jar, jfxrt.jar, localedata.jar, meta-index, nashorn.jar, sunec.jar, sunjce_provider.jar, sunpkcs11.jar, zipfs.jar]
2016-03-11 07:50:37,561 [ 60417] INFO - #com.intellij.idea.Main - ext: /System/Library/Java/Extensions: [MRJToolkit.jar]
2016-03-11 07:50:37,581 [ 60437] INFO - #com.intellij.idea.Main - JNA library loaded (64-bit) in 20 ms
2016-03-11 07:50:37,584 [ 60440] INFO - #com.intellij.idea.Main - initializing environment
2016-03-11 07:50:37,585 [ 60441] INFO - .intellij.util.EnvironmentUtil - loading shell env: /bin/zsh -l -i -c '/Applications/IntelliJ IDEA 15 CE.app/Contents/bin/printenv.py' '/private/var/folders/w9/rkmh4kqs2l3bd5nf5mlp7k100000gq/T/intellij-shell-env.0.tmp'
2016-03-11 07:50:38,546 [ 61402] INFO - .intellij.util.EnvironmentUtil - shell environment loaded (19 vars)
2016-03-11 07:50:38,784 [ 61640] INFO - llij.ide.plugins.PluginManager - Cannot find optional descriptor duplicates-groovy.xml
2016-03-11 07:50:39,050 [ 61906] INFO - llij.ide.plugins.PluginManager - 31 plugins initialized in 436 ms
2016-03-11 07:50:39,052 [ 61908] INFO - llij.ide.plugins.PluginManager - Loaded bundled plugins: Android Support (10.1.3.0), Ant Support (1.0), Bytecode Viewer (0.1), CVS Integration (11), Copyright (8.1), Coverage (143.SNAPSHOT), Eclipse Integration (3.0), EditorConfig (1.0), Git Integration (8.1), GitHub (143.2287.1), Gradle (143.2287.1), Groovy (9.0), I18n for Java (143.2287.1), IDEA CORE (143.SNAPSHOT), IntelliLang (8.0), JUnit (1.0), Java Bytecode Decompiler (0.1), JavaFX (1.0), Kotlin (1.0.0-release-IJ143-75), Maven Integration (143.2287.1), Plugin DevKit (1.0), Properties Support (143.SNAPSHOT), Settings Repository (143.2287.1), Subversion Integration (1.1), Task Management (1.0), Terminal (0.1), TestNG-J (8.0), UI Designer (143.2287.1), XPathView + XSLT Support (4), XSLT-Debugger (1.4), hg4idea (10.0)
2016-03-11 07:50:39,384 [ 62240] INFO - ellij.util.io.PagedFileStorage - lower=100; upper=500; buffer=10; max=739246080
2016-03-11 07:50:39,415 [ 62271] INFO - pl.local.NativeFileWatcherImpl - Starting file watcher: /Applications/IntelliJ IDEA 15 CE.app/Contents/bin/fsnotifier
2016-03-11 07:50:39,423 [ 62279] INFO - pl.local.NativeFileWatcherImpl - Native file watcher is operational.
2016-03-11 07:50:40,025 [ 62881] INFO - rains.ide.BuiltInServerManager - built-in server started, port 63342
2016-03-11 07:50:40,564 [ 63420] INFO - plication.impl.ApplicationImpl - 84 application components initialized in 1951 ms
2016-03-11 07:50:40,581 [ 63437] INFO - .intellij.idea.IdeaApplication - App initialization took 63756 ms
2016-03-11 07:50:40,588 [ 63444] INFO - ntellij.util.proxy.CommonProxy - <html>You have JVM property "socksProxyHost" set to "127.0.0.1".<br>This may lead to incorrect behaviour. Proxy should be set in Settings | HTTP Proxy<br>This JVM property is old and its usage is not recommended by Oracle.<br>(Note: It could have been assigned by some code dynamically.)
2016-03-11 07:50:40,647 [ 63503] INFO - ntellij.util.proxy.CommonProxy - <html>You have JVM property "socksProxyHost" set to "127.0.0.1".<br>This may lead to incorrect behaviour. Proxy should be set in Settings | HTTP Proxy<br>This JVM property is old and its usage is not recommended by Oracle.<br>(Note: It could have been assigned by some code dynamically.)
2016-03-11 07:50:41,176 [ 64032] INFO - CompilerWorkspaceConfiguration - Available processors: 8
2016-03-11 07:50:41,575 [ 64431] INFO - ellij.project.impl.ProjectImpl - 148 project components initialized in 746 ms
2016-03-11 07:50:41,723 [ 64579] INFO - le.impl.ModuleManagerComponent - 2 module(s) loaded in 147 ms
2016-03-11 07:50:43,450 [ 66306] INFO - tartup.impl.StartupManagerImpl - /Users/coderec/IdeaProjects/pttms-server/pttms-server/.idea/misc.xml case-sensitivity: false
2016-03-11 07:50:44,351 [ 67207] INFO - .diagnostic.PerformanceWatcher - Pushing properties took 22ms; general responsiveness: ok; EDT responsiveness: ok
2016-03-11 07:50:44,934 [ 67790] INFO - tor.impl.FileEditorManagerImpl - Project opening took 4133 ms
2016-03-11 07:50:45,903 [ 68759] INFO - .diagnostic.PerformanceWatcher - Indexable file iteration took 1551ms; general responsiveness: ok; EDT responsiveness: ok
2016-03-11 07:50:46,373 [ 69229] INFO - lij.tasks.impl.TaskManagerImpl - Updating issues cache (every 20 min)
2016-03-11 07:50:47,309 [ 70165] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: Build process started. Classpath: /Applications/IntelliJ IDEA 15 CE.app/Contents/lib/jps-launcher.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/lib/tools.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/optimizedFileManager.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/ecj-4.4.jar
2016-03-11 07:50:47,678 [ 70534] INFO - ution.rmi.RemoteProcessSupport - "/Applications/IntelliJ IDEA 15 CE.app/Contents/jre/jdk/Contents/Home/jre/bin/java" -d64 -Djava.awt.headless=true -Didea.version==15.0.4 -Xmx512m -Didea.maven.embedder.version=3.0.5 -Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/resources_en.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/log4j.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/slf4j-api-1.7.10.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/slf4j-log4j12-1.7.10.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/jna-platform.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/snappy-in-java-0.3.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/picocontainer.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/jna.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/oromatcher.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/annotations.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/trove4j.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/jdom.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/util.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/lucene-core-2.4.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven-server-api.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3-server-common.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3-server-lib/archetype-catalog-2.2.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3-server-lib/archetype-common-2.2.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3-server-lib/maven-dependency-tree-1.2.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3-server-lib/nexus-indexer-3.0.4.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3-server-lib/nexus-indexer-artifact-1.0.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven30-server-impl.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/aether-api-1.13.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/aether-connector-wagon-1.13.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/aether-impl-1.13.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/aether-spi-1.13.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/aether-util-1.13.1.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/commons-cli-1.2.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/commons-io-2.2.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/commons-lang-2.6.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-aether-provider-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-artifact-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-compat-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-core-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-embedder-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-model-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-model-builder-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-plugin-api-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-repository-metadata-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-settings-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/maven-settings-builder-3.0.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/plexus-cipher-1.7.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/plexus-component-annotations-1.5.5.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/plexus-interpolation-1.14.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/plexus-sec-dispatcher-1.3.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/plexus-utils-2.0.6.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/sisu-guava-0.9.9.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/sisu-guice-3.1.0-no_aop.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/sisu-inject-bean-2.3.0.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/sisu-inject-plexus-2.3.0.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/wagon-file-2.8.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/wagon-http-2.8-shaded.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/wagon-http-shared-2.8.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/lib/wagon-provider-api-2.8.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/boot/plexus-classworlds-2.4.jar" org.jetbrains.idea.maven.server.RemoteMavenServer
2016-03-11 07:50:47,917 [ 70773] INFO - ution.rmi.RemoteProcessSupport - Port/ID: 4823/Maven30ServerImplf51fe04a
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 0 [RMI TCP Connection(3)-127.0.0.1] DEBUG Sisu - Add publisher: com.google.inject.internal.InjectorImpl#5fd5172c
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport -
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - -----[explicit bindings]-------------------------------------------------------
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 0. ProviderInstanceBinding{key=Key[type=com.google.inject.Injector, annotation=[none]], source=[unknown source], scope=Scopes.NO_SCOPE, provider=Provider<Injector>}
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 1. ProviderInstanceBinding{key=Key[type=java.util.logging.Logger, annotation=[none]], source=[unknown source], scope=Scopes.NO_SCOPE, provider=Provider<Logger>}
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 2. ProviderInstanceBinding{key=Key[type=org.slf4j.Logger, annotation=[none]], source=[unknown source], scope=Scopes.NO_SCOPE, provider=Provider<org.slf4j.Logger>}
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 3. InstanceBinding{key=Key[type=com.google.inject.Stage, annotation=[none]], source=[unknown source], instance=DEVELOPMENT}
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 4. InstanceBinding{key=Key[type=org.codehaus.plexus.context.Context, annotation=[none]], source=org.codehaus.plexus.DefaultPlexusContainer$ContainerModule.configure(DefaultPlexusContainer.java:800), instance={plexus=org.codehaus.plexus.DefaultPlexusContainer#290238e6}}
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 5. InstanceBinding{key=Key[type=java.util.Map, annotation=#org.sonatype.inject.Parameters], source=org.codehaus.plexus.DefaultPlexusContainer$ContainerModule.configure(DefaultPlexusContainer.java:801), instance=null}
2016-03-11 07:50:48,891 [ 71747] INFO - ution.rmi.RemoteProcessSupport - 6. InstanceBinding{key=Key[type=org.sonatype.guice.bean.locators.MutableBeanLocator, annotation=[none]], source=org.codehaus.plexus.DefaultPlexusContainer$ContainerModule.configure(DefaultPlexusContainer.java:803), instance=org.sonatype.guice.bean.locators.DefaultBeanLocator#47715323[Locked by thread RMI TCP Connection(3)-127.0.0.1]}
6~235, omitted
2016-03-11 07:50:48,904 [ 71760] INFO - ution.rmi.RemoteProcessSupport - 235. ConstructorBinding{key=Key[type=org.apache.maven.lifecycle.internal.LifecyclePluginResolver, annotation=[none]], source=ClassRealm[plexus.core, parent: null], scope=Scopes.SINGLETON}
2016-03-11 07:50:48,904 [ 71760] INFO - ution.rmi.RemoteProcessSupport - 236. ConstructorBinding{key=Key[type=org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder, annotation=[none]], source=ClassRealm[plexus.core, parent: null], scope=Scopes.SINGLETON}
2016-03-11 07:50:48,904 [ 71760] INFO - ution.rmi.RemoteProcessSupport - 237. ConstructorBinding{key=Key[type=org.apache.maven.lifecycle.internal.MojoDescriptorCreator, annotation=[none]], source=ClassRealm[plexus.core, parent: null], scope=Scopes.SINGLETON}
2016-03-11 07:50:48,904 [ 71760] INFO - ution.rmi.RemoteProcessSupport - -----[implicit bindings]-------------------------------------------------------
2016-03-11 07:50:48,904 [ 71760] INFO - ution.rmi.RemoteProcessSupport - -------------------------------------------------------------------------------
2016-03-11 07:50:48,904 [ 71760] INFO - ution.rmi.RemoteProcessSupport -
2016-03-11 07:50:49,935 [ 72791] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: Build process started. Classpath: /Applications/IntelliJ IDEA 15 CE.app/Contents/lib/jps-launcher.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/lib/tools.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/optimizedFileManager.jar:/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/ecj-4.4.jar
I finally solved this problem with resetting the host name of my mac, which was set with blank like "Coderec's MBP".
The illegal host name caused this confusing problem.
I have a problem with a process in an app of mine hosted on Heroku (2xweb + 1x worder dynos). The process ends with quite a heavy email being created and sent via SendGrid. This took some time, causing web worker timeouts and bad usability, so it was refactored into a worker, which I thought would solve the problem, but I'm getting situations like this:
Apr 10 17:12:48 wc heroku/web.2: Processing by DealsController#show as */*
[request is processed]
Apr 10 17:12:50 wc app/worker.1: [worker sending emails]
[a lot of lines with debug data cut]
Apr 10 17:12:53 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#load_avg_1m=0.29 sample#load_avg_5m=0.07 sample#load_avg_15m=0.02
Apr 10 17:12:53 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#memory_total=240.45MB sample#memory_rss=240.34MB sample#memory_cache=0.11MB sample#memory_swap=0.00MB sample#memory_pgpgin=85990pages sample#memory_pgpgout=24436pages
Apr 10 17:12:53 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#load_avg_1m=0.00 sample#load_avg_5m=0.01 sample#load_avg_15m=0.01
Apr 10 17:12:54 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#memory_total=844.16MB sample#memory_rss=511.82MB sample#memory_cache=0.00MB sample#memory_swap=332.34MB sample#memory_pgpgin=223581pages sample#memory_pgpgout=92554pages
Apr 10 17:12:54 wc heroku/web.2: Process running mem=844M(164.9%)
Apr 10 17:12:54 wc heroku/web.2: Error R14 (Memory quota exceeded)
Apr 10 17:12:54 wc app/web.2: ** [NewRelic][04/10/14 15:12:54 +0000 879182ef-13f0-4908-bf35-c487ccab6153 (468)] INFO : Starting Agent shutdown
Apr 10 17:12:55 wc app/worker.1: ** [Bugsnag] Bugsnag exception handler 1.6.2 ready, api_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[8 new relic lines cut]
Apr 10 17:12:56 wc app/heroku-postgres: source=HEROKU_POSTGRESQL_WHITE sample#current_transaction=144542 sample#db_size=189720760bytes sample#tables=92 sample#active-connections=12 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99981 sample#table-cache-hit-rate=0.99868 sample#load-avg-1m=0.36 sample#load-avg-5m=0.3 sample#load-avg-15m=0.285 sample#read-iops=38.367 sample#write-iops=13.221 sample#memory-total=7629464kB sample#memory-free=187884kB sample#memory-cached=6599816kB sample#memory-postgres=689216kB
Apr 10 17:12:56 wc app/worker.1: ** [NewRelic][04/10/14 15:12:55 +0000 858a3455-0b9f-4f75-9052-b419d4653703 (99)] INFO : Installing DelayedJob instrumentation hooks
[11 new relic lines cut]
Apr 10 17:12:59 wc app/worker.1: Delayed::Backend::ActiveRecord::Job Load (1.6ms) UPDATE "delayed_jobs" SET locked_at = '2014-04-10 15:12:59.482734', locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2014-04-10 15:12:59.481969' AND (locked_at IS NULL OR locked_at < '2014-04-10 11:12:59.482002') OR locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Apr 10 17:12:59 wc app/worker.1: ** [NewRelic][04/10/14 15:12:59 +0000 858a3455-0b9f-4f75-9052-b419d4653703 (99)] INFO : Starting Agent shutdown
Apr 10 17:13:09 wc app/worker.1: Delayed::Backend::ActiveRecord::Job Load (1.6ms) UPDATE "delayed_jobs" SET locked_at = '2014-04-10 15:13:09.486147', locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2014-04-10 15:13:09.485453' AND (locked_at IS NULL OR locked_at < '2014-04-10 11:13:09.485486') OR locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Apr 10 17:13:13 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#load_avg_1m=0.00 sample#load_avg_5m=0.01 sample#load_avg_15m=0.01
Apr 10 17:13:14 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#memory_total=463.75MB sample#memory_rss=153.29MB sample#memory_cache=0.00MB sample#memory_swap=310.46MB sample#memory_pgpgin=245188pages sample#memory_pgpgout=205946pages
Apr 10 17:13:14 wc app/web.2: Started GET "/user" for [IP.IP.IP.IP] at 2014-04-10 15:13:13 +0000
Apr 10 17:13:14 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#load_avg_1m=0.21 sample#load_avg_5m=0.06 sample#load_avg_15m=0.02
Apr 10 17:13:14 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#memory_total=157.25MB sample#memory_rss=157.14MB sample#memory_cache=0.11MB sample#memory_swap=0.00MB sample#memory_pgpgin=103179pages sample#memory_pgpgout=62923pages
Apr 10 17:13:16 wc app/web.2: Started GET "/user" for [IP.IP.IP.IP] at 2014-04-10 15:13:16 +0000
Apr 10 17:13:17 wc heroku/router: at=error code=H12 desc="Request timeout" method=GET path=/pages/planning host=www.cool-app.com request_id=c62a7ee5-11d8-4286-846a-a55861cc6a0e fwd="[IP.IP.IP.IP]" dyno=web.2 connect=2ms service=30000ms status=503 bytes=0
Apr 10 17:13:19 wc app/web.2: E, [2014-04-10T15:13:18.948990 #2] ERROR -- : worker=1 PID:12 timeout (31s > 30s), killing
Apr 10 17:13:19 wc app/worker.1: Delayed::Backend::ActiveRecord::Job Load (1.5ms) UPDATE "delayed_jobs" SET locked_at = '2014-04-10 15:13:19.489494', locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2014-04-10 15:13:19.488845' AND (locked_at IS NULL OR locked_at < '2014-04-10 11:13:19.488874') OR locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Apr 10 17:13:20 wc app/web.2: E, [2014-04-10T15:13:19.689336 #2] ERROR -- : reaped #<Process::Status: pid 12 SIGKILL (signal 9)> worker=1
Apr 10 17:13:21 wc app/web.2: Disconnected from ActiveRecord
Apr 10 17:13:27 wc app/web.2: Processing by UsersController#show as HTML
[web worker 2 works properly from now on]
The web.2 worker still crashes resulting in a ~20 second hang for the user and and "Application Error" screen being shown. What's strange is that this happens on different pages and seems to be linked to the worker in the background.
The line that especially confuses me (and the one probably symptomatic of the crash) is:
Apr 10 17:13:19 wc app/web.2: E, [2014-04-10T15:13:18.948990 #2] ERROR -- : worker=1 PID:12 timeout (31s > 30s), killing
What does it mean? It seems to me that the web.2 dyno is being killed because the worker=1 had a timout, which seems a bit crazy.
The configuration for the dynos is:
Dynos:
1x - 2 - web bundle exec unicorn -p $PORT -c ./config/unicorn.rb
1x - 1 - worker bundle exec rake jobs:work
Any ideas?
Solution: switch from delayed_job to sidekiq
To whom it may concern - despite several hours of debugging, I was never really able to get to the bottom of this. Decided to try switching this one job over to sidekiq. This solved the issue. Probably just a bug with the particular delayed_job -> heroku interaction. Maybe this would be working today...