How do I get Redmine to start fast on Linux (CentOS)?
I upgraded all last week: latest Redmine, Ruby, Passenger, etc...
I tried about all I could find in the Redmine forum and other posts of getting it to speed up faster, that is: when requesting the Redmine website after a few hours being idle, it starts slow, but then it's blazing fast.
I am using Apache web server with Passenger. Below my current apache config, please some advice, as I am out of ideas:
LoadModule passenger_module /usr/local/rvm/gems/ruby-1.9.3-p448/gems/passenger-4.0.19/buildout/apache2/mod_passenger.so
PassengerRoot /usr/local/rvm/gems/ruby-1.9.3-p448/gems/passenger-4.0.19
PassengerDefaultRuby /usr/local/rvm/wrappers/ruby-1.9.3-p448/ruby
# Refs:
# http://stackoverflow.com/questions/8235309/redmine-perfomance-inconsistency
# http://www.redmine.org/boards/2/topics/31783
# This option should be 0, but has an issue: https://code.google.com/p/phusion-passenger/issues/detail?id=904
PassengerPoolIdleTime 999999
PassengerMinInstances 2
PassengerHighPerformance on
PassengerPreStart https://myhost/redmine
PassengerMaxPoolSize 5
PassengerMaxInstancesPerApp 4
PassengerStatThrottleRate 10
RailsAppSpawnerIdleTime 0
PassengerMaxPreloaderIdleTime 0
RailsBaseURI /redmine
RailsEnv production
I solved this by setting up a cron job to request the redmine homepage every 15 minutes:
*/15 * * * * /usr/bin/curl http://redmine_server/ --stderr - > /dev/null
Related
UPDATED: OK, problem solved.
The different way I started the server makes the different result .
# this gives 2800 req/s in a production server, server based on thin
$ bundle exec thin start -R config.ru -e production
# this gives 1600 req/s in the same server, server based on Rack( seems that)
$ bundle exec rackup config.ru -s thin
so the ways of starting sinatra:
wrong: $ ruby main.rb (based on rack?)
wrong: $ rackup config.ru (based on rack)
wrong: $ rackup config.ru -s thin ( event based on rack)
correct: $ thin start -R config.ru -e production
------------------- Original question --------------------
Today I am coding Sinatra for an API application, and found that:
Classic sinatra code can process :
1.1 1800 request/s, with thin as the server.
1.2 2000 request/s, with puma as the server.
Modular sinatra code can only process:
2.1 1100 requests/s, with thin as the server
2.2 800 requests/s , with puma as the server.
How to reproduce this:
classic sinatra
# test_classic_sinatra.rb
require 'sinatra'
get '/' do
'hihihi'
end
run :
siwei $ ruby test.rb
== Sinatra (v2.0.5) has taken the stage on 4567 for development with backup from Thin
Thin web server (v1.7.2 codename Bachmanity)
Maximum connections set to 1024
Listening on localhost:4567, CTRL+C to stop
test:
$ ab -n 1000 -c 100 http://localhost:4567/
and got result:
Concurrency Level: 100
Time taken for tests: 0.530 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 211000 bytes
HTML transferred: 6000 bytes
Requests per second: 1885.43 [#/sec] (mean)
Time per request: 53.038 [ms] (mean)
Time per request: 0.530 [ms] (mean, across all concurrent requests)
Transfer rate: 388.50 [Kbytes/sec] received
Modular sinatra:
# config.ru
require 'sinatra/base'
class App < Sinatra::Application
set :environment, :production
get '/' do
'hihihi'
end
end
run App
with thin as server
run:
$ rackup config.ru -s thin
Thin web server (v1.7.2 codename Bachmanity)
Maximum connections set to 1024
Listening on localhost:9292, CTRL+C to stop
test:
Concurrency Level: 100
Time taken for tests: 0.931 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 211000 bytes
HTML transferred: 6000 bytes
Requests per second: 1073.58 [#/sec] (mean)
Time per request: 93.146 [ms] (mean)
Time per request: 0.931 [ms] (mean, across all concurrent requests)
Transfer rate: 221.22 [Kbytes/sec] received
with puma as server
run:
siwei$ rackup config.ru
Puma starting in single mode...
* Version 3.11.4 (ruby 2.3.8-p459), codename: Love Song
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://localhost:9292
Use Ctrl-C to stop
test:
$ab -n 1000 -c 100 http://localhost:9292/
got result:
Concurrency Level: 100
Time taken for tests: 1.266 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 178000 bytes
HTML transferred: 6000 bytes
Requests per second: 789.62 [#/sec] (mean)
Time per request: 126.643 [ms] (mean)
Time per request: 1.266 [ms] (mean, across all concurrent requests)
Transfer rate: 137.26 [Kbytes/sec] received
Before deciding to use Sinatra, I read many posts about "sinatra, grape and rails api", and I ran test agaist these frameworks, and finally decide to use Sinatra.
But now , I found Modular Sinatra seems not so good as expected. Could someone give me a clue about how to use "Classic Sinatra" or "Modular Sinatra" ?
If I don't use Modular Sinatra, how to write code for big applications?
thanks a lot!
Your benchmark is incorrect.
According to the snippets you posted, in the first case you use Thin as a server and in the second it's Puma.
These servers implement absolutely different concurrency models: as far as I remember it is a single-threaded event loop for the former and multiple threads for the latter. As a result, Thin performs better for light non-blocking tasks while Puma beats it in the scenarios with relatively heavy computations or blocking ones.
Your dummy example just better fits to the Thin's model and this causes the difference... Modular Sinatra should be absolutely fine, just compare apples to apples :)
OK, I found the root cause, and with The Tin Man's suggestion, I post my answer here.
The problem is the "web server" , but not the "framework"
The different way I started the server makes the different result .
# this gives 2800 req/s in a production server, server based on thin
$ bundle exec thin start -R config.ru -e production
# this gives 1600 req/s in the same server, server based on Rack( seems that)
$ bundle exec rackup config.ru -s thin
so the ways of starting sinatra:
wrong: $ ruby main.rb (based on rack?)
wrong: $ rackup config.ru (based on rack)
wrong: $ rackup config.ru -s thin ( event based on rack)
correct: $ thin start -R config.ru -e production
I updated one of my apps to Rails 5 and upgraded the Ruby version to 2.3.1 as well. The app already used Puma prior to the Rails 5 upgrade as well and was deployed on a Digital Ocean droplet.
When I start rails server locally, I get the normal output in my Rails log, which I've copied below.
=> Booting Puma
=> Rails 5.0.0 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
[14669] Puma starting in cluster mode...
[14669] * Version 3.4.0 (ruby 2.3.1-p112), codename: Owl Bowl Brawl
[14669] * Min threads: 5, max threads: 5
[14669] * Environment: development
[14669] * Process workers: 2
[14669] * Preloading application
[14669] * Listening on tcp://localhost:3000
[14669] Use Ctrl-C to stop
[14669] - Worker 1 (pid: 14684) booted, phase: 0
[14669] - Worker 0 (pid: 14683) booted, phase: 0
Everything looks normal to me. When I visit localhost:3000, the browser has a pending request that is pending indefinitely. There is no further activity in the Rails log acknowledging that any request is being received.
Has anyone encountered this type of issue, or know of any potential causes for that?
Resolved this issue, and confirmed by #marvindanig who was experiencing the same issue, that the 'tmp' folder needed to be cleared. There is a rake task in rails to do so...
rake tmp:clear
I'm experiencing an unresponsive socket in with my Puma setup after random time. Up to this point I don't have a clue what's causing the issue. I was hoping somebody over here can help we with some answers or point me in the right direction. I'm having the following setup:
I'm using the official docker ruby-2.2.3-slim image together with the latest puma release 2.15.3, I've also installed Nginx as a reverse proxy. But I'm already sure Nginx isn't the problem over here because and I've tried to verify if the socket was working using this script. And the socket wasn't working, I got a timeout over there as well so I could ignore Nginx.
This is a testing environment so the server isn't experiencing any extreme load, I've also check memory consumption it has still several GB's of free space so that couldn't be the issue either.
What triggered me to look at the puma socket was the error message I got in my Nginx error logging:
upstream timed out (110: Connection timed out) while reading response header from upstream
Also I couldn't find anything in the logs of puma indicating what is going wrong, over here are my puma setup:
threads 0, 16
app_dir = ENV.fetch('APP_HOME')
environment ENV['RAILS_ENV']
daemonize
bind "unix://#{app_dir}/sockets/puma.sock"
stdout_redirect "#{app_dir}/log/puma.stdout.log", "#{app_dir}/log/puma.stderr.log", true
pidfile "#{app_dir}/pids/puma.pid"
state_path "#{app_dir}/pids/puma.state"
activate_control_app
on_worker_boot do
require 'active_record'
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[ENV['RAILS_ENV']])
end
And this it the output in my puma state file:
---
pid: 43
config: !ruby/object:Puma::Configuration
cli_options:
conf:
options:
:min_threads: 0
:max_threads: 16
:quiet: false
:debug: false
:binds:
- unix:///APP/sockets/puma.sock
:workers: 1
:daemon: true
:mode: :http
:before_fork: []
:worker_timeout: 60
:worker_boot_timeout: 60
:worker_shutdown_timeout: 30
:environment: staging
:redirect_stdout: "/APP/log/puma.stdout.log"
:redirect_stderr: "/APP/log/puma.stderr.log"
:redirect_append: true
:pidfile: "/APP/pids/puma.pid"
:state: "/APP/pids/puma.state"
:control_url: unix:///tmp/puma-status-1449260516541-37
:config_file: config/puma.rb
:control_url_temp: "/tmp/puma-status-1449260516541-37"
:control_auth_token: cda8879717be7a645ea323d931b88d4b
:tag: APP
The application itself is a Rails app on the latest version 4.2.5, it's deployed on GCE (Google Container Engine).
If somebody could give me some pointer's on how to debug this any further would be very much appreciated. Because now I don't see any output anywhere which could help me any further.
EDIT
I replaced the unix socket with tcp connection to Puma with the same result, still hangs after x time
I'd start with:
How many requests get processed successfully per instance of puma?
Make sure you log the beginning and end of each request with the thread id of the thread executing it, what do you see?
Not knowing more about your application, I'd say it's likely the threads get stuck doing some long/blocking calls without timeouts or spinning on some computation until the whole thread pool gets depleted.
We'll see.
I finally found out why my application was behaving the way it was.
After trying to use a tcp connection and switching to Unicorn I start looking into other possible sources.
That's when I thought maybe my connection to Google Cloud SQL could be the problem. Once I read the faq of Cloud SQL, they mentioned that you have to tweak you Compute instances to ensure they keep open your DB connection. So I performed the next steps they recommend and that solved the problem for me, I added them just in case:
# Display the current tcp_keepalive_time value.
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
# Set tcp_keepalive_time to 60 seconds and make it permanent across reboots.
$ echo 'net.ipv4.tcp_keepalive_time = 60' | sudo tee -a /etc/sysctl.conf
# Apply the change.
$ sudo /sbin/sysctl --load=/etc/sysctl.conf
# Display the tcp_keepalive_time value to verify the change was applied.
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
I am trying to use the Rufus Scheduler (within Dashing) to schedule a cron job, but also have it run once upon the server spinning up. I am following the readme here where it is saying to do the following:
scheduler.cron '00 14 * * *', :first_in => '3d' do
# ... every day at 14h00, but start after 3 * 24 hours
end
When I try to do this, I get the following error in my job:
`cron': unknown option: :first_in (ArgumentError)
Has anyone come across this?
Dashing is using rufus-scheduler 2.0.24 ( https://github.com/Shopify/dashing/blob/55f90939eae4d6eb64822fd3590f694418396510/dashing.gemspec#L24 ) which doesn't support the first_in feature for cron.
First_in was introduced for cron in rufus-scheduler 3.0.
It seems you're reading the rufus-scheduler 3.x documentation instead of the 2.x one.
The documentation for rufus-scheduler is at https://github.com/jmettraux/rufus-scheduler#rufus-scheduler , on top of it, there is the link to the 2.x documentation ( https://github.com/jmettraux/rufus-scheduler/blob/two/README.rdoc ). You'll have better luck there.
A 2.x alternative would be:
scheduler.in '3d' do
scheduler.cron '00 14 * * *' do
# ... every day at 1400
end
end
I'm working on a very large puppet deployment, but seem to be hitting a brick wall. My ideal setup is to use Nginx + Passenger to serve puppet. The problem I am having is that Puppet throws errors when running through passenger. If I start puppetmasterd, everything works fine, but serving through Passenger gives the following errors:
Jun 22 07:33:04 $master_hostname puppet-master[15710]: Starting Puppet master version 2.6.8
Jun 22 07:33:04 $master_hostname puppet-master[15720]: No support for http method POST
Jun 22 07:33:04 $master_hostname puppet-master[15720]: Denying access: Forbidden request: $client_hostname($client_ip) access to /report/$client_hostname [save] authenticated at line 0
Jun 22 07:33:04 $master_hostname puppet-master[15720]: Forbidden request: $client_hostname($client_ip) access to /report/$client_hostname [save] authenticated at line 0
Everything seems to point to an auth.conf problem, but my auth.conf file is about as generic as it could get, and like I said, everything works when I serve puppet using Rack directly.
Has anybody ever ran into this issue?
Sounds like this:
http://groups.google.com/group/puppet-users/browse_frm/thread/910994e88f21a497/cae809c17a9acd8a?#cae809c17a9acd8a
The concept being that you need to configure NGINX to pass information through to Puppet as it now provides the SSL layers.