I am attempting to migrate from Heroku to AWS, but my Sidekiq jobs keep failing with the following error:
Errno::EPIPE: Broken pipe # io_write - <STDOUT>
I can successfully run jobs from the console using perform_now, and everything works just fine in Heroku, so I am presuming the issue lies somewhere with my AWS setup. I have seen references to improper daemonization around Stack Overflow and Github but not sure how to solve the problem.
Right now I am launching my processes with the following command:
foreman start -f Procfile -p 3000 -e $VAR_FILES &
and I have tried the command both with and without the & at the end.
My Procfile looks like this:
web: bundle exec puma -t 1:2 -p ${PORT:-3000} -e ${RACK_ENV:-production}
worker: bundle exec sidekiq -C config/sidekiq.yml
log: tail -f log/production.log
and I have also tried it like this, following the instructions here (https://github.com/mperham/sidekiq/wiki/Logging#syslog):
worker: bundle exec sidekiq -C config/sidekiq.yml 2>&1 | logger -t sidekiq
My sidekiq.yml has logfile set to ./log/sidekiq.log, which I believe is supposed to redirecting logs away from STDOUT anyway.
I have seen the discussion here (https://github.com/mperham/sidekiq/issues/3188) and can verify that the rails12factor gem is not in my Gemfile.
But still the error persists... Can anyone lend a hand?
UPDATE: I can finally get a stack trace and see it is coming from a puts statement inside of the Neo4j.rb gem:
2017-04-07T15:46:53.553Z 697 TID-12a6r4 WARN: Errno::EPIPE: Broken pipe # io_write - <STDOUT>
2017-04-07T15:46:53.553Z 697 TID-12a6r4 WARN: /var/lib/gems/2.3.0/bundler/gems/neo4j-c804cb33bef8/lib/neo4j/session_manager.rb:60:in `write'
/var/lib/gems/2.3.0/bundler/gems/neo4j-c804cb33bef8/lib/neo4j/session_manager.rb:60:in `puts'
/var/lib/gems/2.3.0/bundler/gems/neo4j-c804cb33bef8/lib/neo4j/session_manager.rb:60:in `puts'
But still not sure how I can mitigate the issue. I have tried with RAILS_LOG_TO_STDOUT=enabled both set and unset.
I spoke to the gem maintainers and they removed the puts statements in v 8.0.13. It fixed the problem for me!
Related
TL;DR: update your bin/qgtunnel.
I've recently noticed an increase in my web dyno's memory usage. After digging a bit, I could see that the LD_PRELOAD variable that should be set with heroku-buildpack-jemalloc was not set correctly. I used a tiny script (bin/show_preload) that helped me debug that and trace which program was overriding LD_PRELOAD.
#!/usr/bin/env bash
echo "buildpack=foo preload='$LD_PRELOAD' at=start-app cmd='$#'"
$#
I introduced that in our Procfile:
web: bin/show_preload bin/qgtunnel bin/show_preload bin/start-nginx bin/show_preload bin/start-pgbouncer bin/show_preload bundle exec puma -C config/puma.rb
And when lauching on heroku I can see that bin/qgtunnel overrides our LD_PRELOAD configuration.
I created a tiny helper for the time being which makes sure I keep original value as well as what is added by bin/qgtunnel:
#!/usr/bin/env bash
after_qgtunnel_script=$(mktemp)
echo <<-BASH > $after_qgtunnel_script
# Retrieve previous LD_PRELOAD value
export LD_PRELOAD="\$LD_PRELOAD $LD_PRELOAD"
# Clean after usage
rm $after_qgtunnel_script
# Start following commands
$#
BASH
chmod +x $after_qgtunnel_script
bin/qgtunnel $after_qgtunnel_script $#
If you ever need this script use it in place of bin/qgtunnel
After reaching out to Quotaguard, they patched the qgtunnel binary and there is no error anymore:
curl https://quotaguard.s3.amazonaws.com/qgtunnel-2.4.1.tar.gz | tar xz
git add bin/qgtunnel vendor/nss_wrapper/libnss_wrapper.so
git commit -m "Update qgtunnel to fix LD_PRELOAD"
NOTE: new versions may occur since that one, see the related documentation
Why is command "bundle" not found when using sudo:
[root#desktop gitlab]# sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production
sudo: bundle: command not found
[root#desktop gitlab]#
but does exist when not using sudo:
[root#desktop gitlab]# bundle exec rake gitlab:setup RAILS_ENV=production
Warning
You are running as user root, we hope you know what you are doing.
Things may work/fail for the wrong reasons.
For correct results you should run this as user git.
This will create the necessary database tables and seed the database.
You will lose any previous data stored in the database.
Do you want to continue (yes/no)? no
Quitting...
[root#desktop gitlab]#
The reason I ask is I am following https://github.com/gitlabhq/gitlab-recipes/tree/master/install/centos, and it states to use sudo.
I've tried adding a -i flag as described by Using $ sudo bundle exec ... raises 'bundle: command not found' error, but get "This account is currently not available.".
Check if the PATH has the same values both with and without sudo. Apparently it cannot find bundle just because it is not listed in PATH
You can compare the outputs of following two lines
$ echo 'echo $PATH' | sh
$ echo 'echo $PATH' | sudo sh
Ideally sudo is supposed to leave PATH untouched. But this might be a side issue of your hosting distribution.
Edit by original poster. Output is:
[root#desktop etc]# echo 'echo $PATH' | sh
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
[root#desktop etc]# echo 'echo $PATH' | sudo sh
/sbin:/bin:/usr/sbin:/usr/bin:/user/local/bin
[root#desktop etc]#
The user was created without a bash login shell. Change this in centos using system-config-users. Then su git into /home/git and move to gitlab directory. Execute the bundle commands without the sudo tag. The next error you will encounter is the missing database.yml in the config dir. fix this with the correct password (i.e. copy the mysql or postgres sample and edit).
I had this issue I thought that my gitlab installed from source and I got same error. but after try Omnibus method for backup my issue solved
with this command:
sudo gitlab-rake gitlab:backup:create
Try :
sudo -u git -H env PATH=$PATH && bundle exec rake gitlab:check RAILS_ENV=production
to use the same PATH than current user.
How does one restart the Ruby clockwork gem?
After reading the Wiki, it seems you can only start it, not stop or restart it.
I don't want to manually kill the process and run it again.
The modern syntax to restart clockworkd is:
bin/clockworkd -c periodic-jobs.rb reload
If you've bundled your gems, as you should:
bundle exec bin/clockworkd -c periodic-jobs.rb reload
…where periodic-jobs.rb is your clockwork jobs config file.
Full options:
bin/clockworkd help
Usage: clockworkd -c FILE [options] start|stop|restart|run
--pid-dir=DIR Alternate directory in which to store the process ids. Default is /Users/jm3/Code/soakcity/tmp.
-i, --identifier=STR An identifier for the process. Default is clock file name.
-l, --log Redirect both STDOUT and STDERR to a logfile named clockworkd[.<identifier>].output in the pid-file directory.
--log-dir=DIR A specific directory to put the log files into (default location is pid directory).
-m, --monitor Start monitor process.
-c, --clock=FILE Clock .rb file. Default is /Users/jm3/Code/soakcity/clock.rb.
-d, --dir=DIR Directory to change to once the process starts
-h, --help Show this message
Learn more in the Demonization section of the clockwork source on GitHub. Hope that helps!
Assuming you started it as a daemon, then 'clockworkd -c YOUR_CLOCK.rb stop' should do the trick.
I'm attempting to write a bash script in ruby that will start a Resque worker for one of my apps.
The command that I generate from the params given in the console looks like this...
command = "ssh user##{#ip} 'cd /path/to/app; bundle exec rake resque:work QUEUE=#{#queue}&'"
`command`
The command is interpolated correctly and everything looks great. I'm asked to input the password for the ssh command and then nothing happens. I'm pretty sure my syntax is correct for making an ssh connection and running a line of code within that connection. ssh user#host 'execute command'
I've done a simpler command that only runs the mac say terminal command and that worked fine
command = "ssh user##{#ip} 'say #{#queue}'"
`command`
I'm running the rake task in the background because I have used that line once inside ssh and it will only keep the worker alive if you run the process in the background.
Any thoughts? Thanks!
I figured it out.
It was an rvm thing. I need to include . .bash_profile at the beginning of the scripts I wanted to run.
So...
"ssh -f hostname '. .bash_profile && cd /path/to/app && bundle exec rake resque:work QUEUE=queue'" is what I needed to make it work.
Thanks for the help #Casper
Ssh won't exit the session until all processes that were launched by the command argument have finished. It doesn't matter if you run them in the background with &.
To get around this problem just use the -f switch:
-f Requests ssh to go to background just before command execution. This is
useful if ssh is going to ask for passwords or passphrases, but the user
wants it in the background. This implies -n. The recommended way to start
X11 programs at a remote site is with something like ssh -f host xterm.
I.e.
"ssh -f user##{#ip} '... bundle exec rake resque:work QUEUE=#{#queue}'"
EDIT
In fact looking more closely at the problem it seems ssh is just waiting for the remote side to close stdin and stdout. You can test it easily like this:
This hangs:
ssh localhost 'sleep 10 &'
This does not hang:
ssh localhost 'sleep 10 </dev/null >/dev/null &'
So I assume the last version is actually pretty closely equivalent to running with -f.
I have a Procfile setup that is running a number of processes successfully:
# /Procfile
redis: bundle exec redis-server
sidekiq: bundle exec sidekiq -v -C ./config.yml
forward: forward 4567 mock-api
I need to add one more process - a Sinatra app that lives in a different directory on my machine. If I cd to the directory, I can start it from the Terminal with:
$ rackup -p 4567
And I can start it from a different directory using the Terminal with:
$ sh -c 'cd /Path/to/project/ && exec rackup -p 4567'
But how should I do this using foreman. I have tried adding the following, but it fails silently:
mock-api: sh -c 'cd /Path/to/project/ && exec rackup -p 4567'
Is this even possible? And if so, how?
Of all the stupid things ...
It was failing because of the hyphen in the process name.