How can I pass along attributes to my tasks in capistrano?
My goal is to deploy to multiple servers in a load balancer. I'd like to take each one out, deploy, and add it back in sequentially so no more than one server is down at any time.
I'm thinking it would be something along the lines of... (and the hosts array would be generated dynamically after querying my load balancer)...
role :app,
[["server_one", {:instanceId => "alice"}],
["server_two", {:instanceId => "bob"}],
["server_three", {:instanceId => "charles"}]]
And then for my tasks...
before :deploy, :deregister_instance_from_lb
after :deploy, :register_instance_with_lb
task deregister_instance_from_lb
#TODO - Deregister #{instanceId} from load balancer
end
task register_instance_with_lb
#TODO - Register #{instanceId} with load balancer
end
Any ideas?
I use this to restart my servers in series, instead of in parallel.
task :my_task, :roles => :web do
find_servers_for_task(current_task).each do |server|
run "[task command here]", :hosts => server.host
end
end
Justin, I'm sorry that's not possible, once the stream pool is opened (first run on a server set) there's no way to access server properties. (as the run code isn't run per-server, but against all-matching in the pool). Some people have had some success with doing something like this, but really it's a symptom that your scripts need too much information that you should be able to extract from your production environment.
As in this case it seems you are doing something like using the host's name to pass to a script, use what Unix gives you:
run "./my_script.rb `hostname`"
WIll that work?
References:
• http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html (Section 3.4.5)
• http://unixhelp.ed.ac.uk/CGI/man-cgi?hostname (or $ man (1) hostname)
No one knows? I found something about the sequential block below, but thats as far as I got...
find_servers.each do |server|
#TODO - remove from load balancer
#TODO - deploy
#TODO - add back to load balancer
end
I find it hard to believe that no one has ever needed to do sequential tasks with cap.
Related
We are using Sidekiq to process a number of backend jobs. One in particular is used very heavily. All I can really say about it is that it sends emails. It doesn't do the email creation (that's a separate job), it just sends them. We spin up a new worker for each email that needs to be sent.
We are trying to upgrade to Ruby 3 and having problems, though. Ruby 2.6.8 has no issues; in 3 (as well as 2.7.3 IIRC), if there is a large number of queued workers, it will get through maybe 20K of them, then it will start hemorrhaging FIFO pipes, on the order of 300-1000 ever 5 seconds or so. Eventually it gets to the ulimit on the system (currently set at 64K) and all sockets/connections fail due to insufficient resources.
In trying to debug this issue I did a run with 90% of what the email worker does entirely commented out, so it does basically nothing except make a couple database queries and do some string templating. I thought I was getting somewhere with that approach, as one run (of 50K+ emails) succeeded without the pipe explosion. However, the next run (identical parameters) did wind up with the runaway pipes.
Profiling with rbspy and ruby-prof did not help much, as they primarily focus on the Sidekiq infrastructure, not the workers themselves.
Looking through our code, I did see that nothing we wrote is ever using IO.* (e.g. IO.popen, IO.select, etc), so I don't see what could be causing the FIFO pipes.
I did see https://github.com/mperham/sidekiq/wiki/Batches#huge-batches, which is not necessarily what we're doing. If you look at the code snippet below, we're basically creating one large batch. I'm not sure whether pushing jobs in bulk as per the link will help with the problem we're having, but I'm about to give it a try once I rework things a bit.
No matter what I do I can't seem to figure out the following:
What is making these pipes? Why are they being created?
What is the condition by which the pipes start getting made exponentially? There are two FIFO pipes that open when we start Sidekiq, but until enough work has been done, we don't see more than 2-6 pipes open generally.
Any advice is appreciated, even along the lines of where to look next, as I'm a bit stumped.
Initializer:
require_relative 'logger'
require_relative 'configuration'
require 'sidekiq-pro'
require "sidekiq-ent"
module Proprietary
unless const_defined?(:ENVIRONMENT)
ENVIRONMENT = ENV['RACK_ENV'] || ENV['RAILS_ENV'] || 'development'
end
# Sidekiq.client_middleware.add Sidekiq::Middleware::Client::Batch
REDIS_URL = if ENV["REDIS_URL"].present?
ENV["REDIS_URL"]
else
"redis://#{ENV["REDIS_SERVER"]}:#{ENV["REDIS_PORT"]}"
end
METRICS = Statsd.new "10.0.9.215", 8125
Sidekiq::Enterprise.unique! unless Proprietary::ENVIRONMENT == "test"
Sidekiq.configure_server do |config|
# require 'sidekiq/pro/reliable_fetch'
config.average_scheduled_poll_interval = 2
config.redis = {
namespace: Proprietary.config.SIDEKIQ_NAMESPACE,
url: Proprietary::REDIS_URL
}
config.server_middleware do |chain|
require 'sidekiq/middleware/server/statsd'
chain.add Sidekiq::Middleware::Server::Statsd, :client => METRICS
end
config.error_handlers << Proc.new do |ex,ctx_hash|
Proprietary.report_exception(ex, "Sidekiq", ctx_hash)
end
config.super_fetch!
config.reliable_scheduler!
end
Sidekiq.configure_client do |config|
config.redis = {
namespace: Proprietary.config.SIDEKIQ_NAMESPACE,
url: Proprietary::REDIS_URL,
size: 15,
network_timeout: 5
}
end
end
Code snippet (sanitized)
def add_targets_to_batch
#target_count = targets.count
queue_counter = 0
batch.jobs do
targets.shuffle.each do |target|
send(campaign_target)
queue_counter += 1
end
end
end
def send(campaign_target)
TargetEmailWorker.perform_async(target[:id],
guid,
is_draft ? target[:email_address] : nil)
begin
Target.where(id: target[:id]).update(send_at: Time.now.utc)
rescue Exception => ex
Proprietary.report_exception(ex, self.class.name, { target_id: target[:id], guid: guid })
end
end
end
First I tried auditing our external connections for connection pooling, etc. That did not help the issue. Eventually I got to the point where I disabled all external connections and let the job run doing virtually nothing outside of a database query and some logging. This allowed one run to complete without issue, but on the second one, the FIFO pipes still grew exponentially after a certain (variable) amount of work was done.
I have a bare metal server, and I want to install multiple services on this server.
My inventory looks like that
[Mygroup]
Server port_service=9990 service_name="service1"
Server port_service=9991 service_name="service2"
When I launch my ansible job,only service 2 is installed because I have the same server in each line of my group. There is way to force ansible to take all lines of a group?
I don't want to create a group for each service
Q: "There is a way to force Ansible to take all lines of a group?"
A: No. There is not. In a group, the hosts shall be unique. If there are multiple hosts with the same name the last one will be taken.
Put the variables into one line e.g.
[Mygroup]
Server port_services="[9990, 9991]" service_names="['service1', 'service2']"
(and change the code).
See How to build your inventory. There is a lot of other options e.g.
[Mygroup]
Server
[Mygroup:vars]
port_services="[9990, 9991]"
service_names="['service1', 'service2']"
I hope I got u right but this should do the trick.
https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html
Greets
Harry
Another solution is to use an alias.
This solution works fine for me
[Mygroup]
service_1 ansible_host=Server port_service=9990 service_name="service1"
service_2 ansible_host=Server port_service=9991 service_name="service2"
I am performing this query with my Ruby mongo driver:
begin
User.collection.find({}).count()
rescue => e
Rails.logger.error e.to_s
end
I would like to catch all situations where this operation fails. The main reason it would fail is if the server was unavailable.
For example, one of the errors I see occasionally is:
Mongo::Error::NoServerAvailable (No server is available matching preference: #<Mongo::ServerSelector::Primary:0x70302744731080 tag_sets=[] max_staleness=nil> using server_selection_timeout=30 and local_threshold=0.015)
I want to catch errors after just 6 seconds.
From the docs I see that there are a few different timeout options (connect_timeout, server_selection_timeout, socket_timeout). But I am not sure which to pass, and how to pass them.
You're on the right track. server_selection_timeout is the correct option in this case -- it tells the driver how long to wait to find a suitable server before timing out. You can set that option on a new client in the Mongoid configuration file (config/mongoid.yml).
You want your configuration file to look something like this:
development: # (or production or whatever environment you're using)
clients:
default:
# your default client here
new_client: # the name of your new client with different options
database: mongoid
hosts:
- localhost:27017
options:
server_selection_timeout: 6
...
Read the Mongoid Configuration documentation to learn more about setting up config files.
Then, you want to use the new client you defined to perform your query, and rescue any Mongo::Error::NoServerAvailable errors.
begin
User.with(client: 'new_client').collection.find({}).count()
rescue Mongo::Error::NoServerAvailable => e
# perform error handling here
end
Note that this starts up a new Mongo::Client instance, which is an expensive operation if done repeatedly. I would recommend that you close the extra client once you are done using it, like so:
Mongoid::Clients.with_name('new_client').close
I have a Daemon I am trying to start but I would like to set a few variables in the daemon when starting it. Here is the script I am using to control my daemons locates in RAILSAPP/script/daemon
#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
ENV["APP_ROOT"] ||= File.expand_path("#{File.dirname(__FILE__)}/..")
ENV["RAILS_ENV_PATH"] ||= "#{ENV["APP_ROOT"]}/config/environment.rb"
script = "#{ENV["APP_ROOT"]}/daemons/#{ARGV[1]}"
Daemons.run(script, dir_mode: :normal, dir: "#{ENV["APP_ROOT"]}/tmp/pids")
When I start this daemon I would like to pass a variable to it like a reference to an active record so I can base the daemon's initial run off of it.
If you want to fetch a specific ActiveRecord object, you can either pass just the id, or the classname + id as an additional parameter on the commandline. As you're already using ARGV[1] for the script name, you could pass it as ARGV[2] and something like Product_123 that you then parse via split, and do a Product.find(123) to get the actual record.
Another approach would be to put the object information into a queue like memcached or redis, and have the daemon fetch the information out of the queue. That would keep your daemon startup a bit simpler, and would allow you to queue up multiple records for the daemon to process. (Something just processing a single record would probably be better written as a script anyway.)
The other concern I have about your script is using ENV["APP_ROOT"]. Does that really need to go in the environment? What if you have a second daemon? It seems that it would be better as a local variable, and if you need it in the daemon, you can always get it relative to where the daemon's file is located anyway.
I'm writing an rails 3 application which requires performing small tasks on a custom schedule for each user. The scheduled tasks will be defined dynamically. Right now my plan is to use resque scheduler with redis.
Once I set the schedule for a specify task (for eg. run task A every 48 hours) I would like to run that task indefinitely. So I would like to store those schedules in a db or something so in case an app crashes when it restarts it would load queue those task again.
Is this something Resque supports by default by storing it in redis or do I need to write my own custom thing? I was also looking at ruby-taskr (http://code.google.com/p/ruby-taskr/). I am not sure if taskr supports storing it in a database and registering it on start?
Also it would be helpful if there are applications/demo that I can look at it.
Thanks
I have a similar setup for batch jobs. The user adds them on a web dashboard and they get run however often is specified.
I use active-record to store the scheduling definitions, use resque for execution and a single cron entry for enqueueing using a rake task.
so then in the rake task:
to_run = Report.daily
to_run += Report.weekly if Time.now.monday?
to_run += Report.monthly if Time.now.day == 1
to_run.each{|r| r.enqueue!}
where daily, weekly, monthly are named scopes on the model:
class Report < ActiveRecord::Base
scope :daily, where(:when_to_run => 'daily')
scope :weekly, where(:when_to_run => 'weekly')
scope :monthly, where(:when_to_run => 'monthly')
end
This is a little hacky, but it works well and I stay within the stack nicely. Hope that is useful