I have a set of of ~60 sorted sets each containing a ~200 members which I am trying to process. Previously I built a Redis (Lua) serverside script but the big(O) time values of the request bog down under load.
I am now trying to offload the processing to Ruby/Sinatra and refreshing the results on every request proving to be inefficient. Given the code below, is there an way to cache the "scores" results in Sinatra so I don't have to pull from Redis on every request?
global = redis.smembers("id_list")
i=0
scores = redis.pipelined do
global.each do |key|
redis.zrange("user:#{global[i]}",0,100,:with_scores => true)
i+=1
end
end
Sinatra has a global scope where objects will persist between requests. If you have a scorekeeper class defined which maintains an instance variable for your score, then you can have a lookup method for the score which holds the value. For example:
class Scorekeeper
def initialize
#scores = nil
end
def scores
#scores ||= get_scores
end
def get_scores
global = redis.smembers("id_list")
i=0
scores = redis.pipelined do
global.each do |key|
redis.zrange("user:#{global[i]}",0,100,:with_scores => true)
i+=1
end
end
scores
end
end
Now your Sinatra app just needs to instantiate a scorekeeper outside of any resource declaration:
require 'sinatra'
keeper = Scorekeeper.new
get '/scores' do
keeper.scores
end
This way on the first request, the scores property will get populated, on all further requests, it will use the cached value.
Related
I have recently started using Rubocop to "standardise" my code, and it has helped me optimise a lot of my code, as well as help me learn a lot of Ruby "tricks". I understand that I should use my own judgement and disable Cops where necessary, but I have found myself quite stuck with the below code:
def index
if params[:filters].present?
if params[:filters][:deleted].blank? || params[:filters][:deleted] == "false"
# if owned is true, then we don't need to filter by admin
params[:filters][:admin] = nil if params[:filters][:admin].present? && params[:filters][:owned] == "true"
# if admin is true, then must not filter by owned if false
params[:filters][:owned] = nil if params[:filters][:owned].present? && params[:filters][:admin] == "false"
companies_list =
case params[:filters][:admin]&.to_b
when true
current_user.admin_companies
when false
current_user.non_admin_companies
end
if params[:filters][:owned].present?
companies_list ||= current_user.companies
if params[:filters][:owned].to_b
companies_list = companies_list.where(owner: current_user)
else
companies_list = companies_list.where.not(owner: current_user)
end
end
else
# Filters for deleted companies
companies_list = {}
end
end
companies_list ||= current_user.companies
response = { data: companies_list.alphabetical.as_json(current_user: current_user) }
json_response(response)
end
Among others, the error that I'm getting is the following:
C: Metrics/AbcSize: Assignment Branch Condition size for index is too high. [<13, 57, 16> 60.61/15]
I understand the maths behind it, but I don't know how to simplify this code to achieve the same result.
Could someone please give me some guidance on this?
Thanks in advance.
Well first and foremost, is this code fully tested, including all the myriad conditions? It's so complex that refactoring will surely be disastrous unless the test suite is rigorous. So, write a comprehensive test suite if you don't already have one. If there's already a test suite, make sure it tests all the conditions.
Second, apply the "fat model skinny controller" paradigm. So move all the complexity into a model, let's call it CompanyFilter
def index
companies_list = CompanyFilter.new(current_user, params).list
response = { data: companies_list.alphabetical.as_json(current_user: current_user) }
json_response(response)
end
and move all those if/then/else statements into the CompanyFilter#list method
tests still pass? great, you'll still get the Rubocop warnings, but related to the CompanyFilter class.
Now you need to untangle all the conditions. It's a bit hard for me to understand what's going on, but it looks as if it should be reducible to a single case statement, with 5 possible outcomes. So the CompanyFilter class might look something like this:
class CompanyFilter
attr_accessors :current_user, :params
def initialize(current_user, params)
#current_user = current_user
#params = params
end
def list
case
when no_filter_specified
{}
when user_is_admin
#current_user.admin_companies
when user_is_owned
# etc
when # other condition
# etc
end
end
private
def no_filter_specified
#params[:filter].blank?
end
def user_is_admin
# returns boolean based on params hash
end
def user_is_owned
# returns boolean based on params hash
end
end
tests still passing? perfect! [Edit] Now you can move most of your controller tests into a model test for the CompanyFilter class.
Finally I would define all the different companies_list queries as scopes on the Company model, e.g.
class Company < ApplicationRecord
# some examples, I don't know what's appropriate in this app
scope :for_user, ->(user){ where("...") }
scope :administered_by, ->(user){ where("...") }
end
When composing database scopes ActiveRecord::SpawnMethods#merge is your friend.
Post.where(title: 'How to use .merge')
.merge(Post.where(published: true))
While it doesn't look like much it lets you programatically compose scopes without overelying on mutating assignment and if/else trees. You can for example compose an array of conditions and merge them together into a single ActiveRecord::Relation object with Array#reduce:
[Post.where(title: 'foo'), Post.where(author: 'bar')].reduce(&:merge)
# => SELECT "posts".* FROM "posts" WHERE "posts"."title" = $1 AND "posts"."author" = $2 LIMIT $3
So lets combine that with a skinny controllers approach where you handle filtering in a seperate object:
class ApplicationFilter
include ActiveModel::Attributes
include ActiveModel::AttributeAssignment
attr_accessor :user
def initialize(**attributes)
super()
assign_attributes(attributes)
end
# A convenience method to both instanciate and apply the filters
def self.call(user, params, scope: model_class.all)
return scope unless params[:filters].present?
scope.merge(
new(
permit_params(params).merge(user: user)
).to_scope
)
end
def to_scope
filters.map { |filter| apply_filter(filter) }
.compact
.select {|f| f.respond_to?(:merge) }
.reduce(&:merge)
end
private
# calls a filter_by_foo method if present or
# defaults to where(key => value)
def apply_filter(attribute)
if respond_to? "filter_by_#{attribute}"
send("filter_by_#{attribute}")
else
self.class.model_class.where(
attribute => send(attribute)
)
end
end
# Convention over Configuration is sexy.
def self.model_class
name.chomp("Filter").constantize
end
# filters the incoming params hash based on the attributes of this filter class
def self.permit_params
params.permit(filters).reject{ |k,v| v.blank? }
end
# provided for modularity
def self.filters
attribute_names
end
end
This uses some of the goodness provided by Rails to setup objects with attributes that will dynamically handle filtering attributes. It looks at the list of attributes you have declared and then slices those off the params and applies a method for that filter if present.
We can then write a concrete implementation:
class CompanyFilter < ApplicationFilter
attribute :admin, :boolean, default: false
attribute :owned, :boolean
private
def filter_by_admin
if admin
user.admin_companies
else
user.non_admin_companies
end
end
# this should be refactored to use an assocation on User
def filter_by_owned
case owned
when nil
nil
when true
Company.where(owner: user)
when false
Company.where.not(owner: user)
end
end
end
And you can call it with:
# scope is optional
#companies = CompanyFilter.call(current_user, params), scope: current_user.companies)
We have a Rails 3.2 website which is fairly large with thousands of URLs. We implemented Cache_Digests gem for Russian Doll caching. It is working well. We want to further optimize by warming up the cache overnight so that user gets a better experience during the day. I have seen answer to this question: Rails: Scheduled task to warm up the cache?
Could it be modified for warming up large number of URLs?
To trigger cache hits for many pages with expensive load times, just create a rake task to iteratively send web requests to all record/url combinations within your site. (Here is one implementation)
Iteratively Net::HTTP request all site URL/records:
To only visit every page, you can run a nightly Rake task to make sure that early morning users still have a snappy page with refreshed content.
lib/tasks/visit_every_page.rake:
namespace :visit_every_page do
include Net
include Rails.application.routes.url_helpers
task :specializations => :environment do
puts "Visiting specializations..."
Specialization.all.sort{ |a,b| a.id <=> b.id }.each do |s|
begin
puts "Specialization #{s.id}"
City.all.sort{ |a,b| a.id <=> b.id }.each do |c|
puts "Specialization City #{c.id}"
Net::HTTP.get( URI("http://#{APP_CONFIG[:domain]}/specialties/#{s.id}/#{s.token}/refresh_city_cache/#{c.id}.js") )
end
Division.all.sort{ |a,b| a.id <=> b.id }.each do |d|
puts "Specialization Division #{d.id}"
Net::HTTP.get( URI("http://#{APP_CONFIG[:domain]}/specialties/#{s.id}/#{s.token}/refresh_division_cache/#{d.id}.js") )
end
end
end
end
# The following methods are defined to fake out the ActionController
# requirements of the Rails cache
def cache_store
ActionController::Base.cache_store
end
def self.benchmark( *params )
yield
end
def cache_configured?
true
end
end
(If you want to directly include cache expiration/recaching into this task, check out this implementation.)
via a Custom Controller Action:
If you need to bypass user authentication restrictions to get to your pages, and/or you don't want to screw up (too badly) your website's tracking analytics, you can create a custom controller action for hitting cache digests that use tokens to bypass authentication:
app/controllers/specializations.rb:
class SpecializationsController < ApplicationController
...
before_filter :check_token, :only => [:refresh_cache, :refresh_city_cache, :refresh_division_cache]
skip_authorization_check :only => [:refresh_cache, :refresh_city_cache, :refresh_division_cache]
...
def refresh_cache
#specialization = Specialization.find(params[:id])
#feedback = FeedbackItem.new
render :show, :layout => 'ajax'
end
def refresh_city_cache
#specialization = Specialization.find(params[:id])
#city = City.find(params[:city_id])
render 'refresh_city.js'
end
def refresh_division_cache
#specialization = Specialization.find(params[:id])
#division = Division.find(params[:division_id])
render 'refresh_division.js'
end
end
Our custom controller action renders the views of other expensive to load pages, causing cache hits to those pages. E.g. refresh_cache renders the same view page & data as controller#show, so requests to refresh_cache will warm up the same cache digests as controller#show for those records.
Security Note:
For security reasons, I recommend before providing access to any custom refresh_cache controller request that you pass in a token and check it to make sure that it corresponds with a unique token for that record. Matching URL tokens to database records before providing access (as seen above) is trivial because your Rake task has access to the unique tokens of each record -- just pass the record's token in with each request.
tl;dr:
To trigger thousands of site URL's/cache digests, create a rake task to iteratively request every record/url combination in your site. You can bypass your app's user authentication restrictions for this task by creating a a custom controller action that authenticates access via tokens instead.
I realize this question is about a year old, but I just worked out my own answer, after scouring a bunch of partial & incorrect solutions.
Hopefully this will help the next person...
Per my own utility class, which can be found here:
https://raw.githubusercontent.com/JayTeeSF/cmd_notes/master/automated_action_runner.rb
You can simply run this (per it's .help method) and pre-cache your pages, without tying-up your own web-server, in the process.
class AutomatedActionRunner
class StatusObject
def initialize(is_valid, error_obj)
#is_valid = !! is_valid
#error_obj = error_obj
end
def valid?
#is_valid
end
def error
#error_obj
end
end
def self.help
puts <<-EOH
Instead tying-up the frontend of your production site with:
`curl http://your_production_site.com/some_controller/some_action/1234`
`curl http://your_production_site.com/some_controller/some_action/4567`
Try:
`rails r 'AutomatedActionRunner.run(SomeController, "some_action", [{id: "1234"}, {id: "4567"}])'`
EOH
end
def self.common_env
{"rack.input" => "", "SCRIPT_NAME" => "", "HTTP_HOST" => "localhost:3000" }
end
REQUEST_ENV = common_env.freeze
def self.run(controller, controller_action, params_ary=[], user_obj=nil)
success_objects = []
error_objects = []
autorunner = new(controller, controller_action, user_obj)
Rails.logger.warn %Q|[AutomatedAction Kickoff]: Preheating cache for #{params_ary.size} #{autorunner.controller.name}##{controller_action} pages.|
params_ary.each do |params_hash|
status = autorunner.run(params_hash)
if status.valid?
success_objects << params_hash
else
error_objects << status.error
end
end
return process_results(success_objects, error_objects, user_obj.try(:id), autorunner.controller.name, controller_action)
end
def self.process_results(success_objects=[], error_objects=[], user_id, controller_name, controller_action)
message = %Q|AutomatedAction Summary|
backtrace = (error_objects.first.try(:backtrace)||[]).join("\n\t").inspect
num_errors = error_objects.size
num_successes = success_objects.size
log_message = %Q|[#{message}]: Generated #{num_successes} #{controller_name}##{controller_action}, pages; Failed #{num_errors} times; 1st Fail: #{backtrace}|
Rails.logger.warn log_message
# all the local-variables above, are because I typically call Sentry or something with extra parameters!
end
attr_reader :controller
def initialize(controller, controller_action, user_obj)
#controller = controller
#controller = controller.constantize unless controller.respond_to?(:name)
#controller_instance = #controller.new
#controller_action = controller_action
#env_obj = REQUEST_ENV.dup
#user_obj = user_obj
end
def run(params_hash)
Rails.logger.warn %Q|[AutomatedAction]: #{#controller.name}##{#controller_action}(#{params_hash.inspect})|
extend_with_autorun unless #controller_instance.respond_to?(:autorun)
#controller_instance.autorun(#controller_action, params_hash, #env_obj, #user_obj)
end
private
def extend_with_autorun
def #controller_instance.autorun(action_name, action_params, action_env, current_user_value=nil)
self.params = action_params # suppress strong parameters exception
self.request = ActionDispatch::Request.new(action_env)
self.response = ActionDispatch::Response.new
define_singleton_method(:current_user, -> { current_user_value })
send(action_name) # do it
return StatusObject.new(true, nil)
rescue Exception => e
return StatusObject.new(false, e)
end
end
end
I have spent the last two weeks building a Sinatra app, and I have hit my first major roadblock. I have spent the last 3 days trying to find a solution, but need to ask for help.
I am building a small game using Sinatra, and I am having trouble understanding a concept.
The game tracks rounds and scores.
Here is some of my code:
#Player Class
class Player
attr_accessor :name, :guess, :round_score, :total_score
def initialize
#guess = 0
#round_score = 0
#total_score = 0
end
end
#Tracks Round
class RoundCount
attr_accessor :round
def initialize
#round = 0
end
def count
#round += 1
end
end
helpers do
#New game function. Makes set number of player classes in ##player array.
def new_game(players)
##i = RoundCount.new
##player = []
players.times do |x|
##player[x] = Player.new
##player[x].name = "Player#{x}"
end
end
#Calculates score
def player_score()
params.each do |x, y|
##player.each do |z|
if z.name == x
z.guess = y.to_i
z.round_score = (y.to_i - ##movie[##i.round].ratings['critics_score'].to_i).abs
z.total_score += (y.to_i - ##movie[##i.round].ratings['critics_score'].to_i).abs
end
end
end
end
get '/' do
slim :home
end
get '/players' do
slim :players
end
post '/numplayers' do
new_game(params[:numplayers].to_i)
slim :names
end
post '/names' do
slim :titles
end
Currently the program gives me the error "warning: class variable access from topelevel" and is affected by other users who play at the same time on seperate computers.
I need help understanding how to track players scores and the round of the game in each view, in a way that isn't affected by other players running a different game.
Any suggestions on how to avoid using class variables for round and score, while retaining a way to access them in each route and view?
Use Sessions or Cookies
Generally, keeping server-side state through class variables will make your life harder. There's also usually more than one way to do anything, but for your use case I'd recommend that you store your state in a session variable or a cookie.
You could then store your state in any serializable object, such as a Hash, Array, Struct, or OpenSruct. This will allow for concurrent users, and do away with the need for the class variables you're currently using.
This a simplified version of what I am trying to solve:
In Ruby 2.0.0, Rails 4.0.0, Activerecord
Segment has_many Sales
Sale.find(1).bid = 1
Sale.find(1).ask = 2
Sale.find(2).bid = 10
Sale.find(2).ask = 20
etc
When I write this method:
class Segment
def add_stuff(param)
sales.map{ |s| s.param }.inject(:+)
end
end
Question: How to pass in bid or ask as param and interpolate that param properly within the block? Thanks.
Never use map on active record association, when you can use pluck or other querying method like, for example sum:
def add_staff(param)
sales.sum(param)
end
Just for the completeness, here is how to do it with map - note however that map is much slower and error prone than querying methods:
def add_staff(param)
sales.map {|s| s[param]} # if param is name of the column on sales model.
end
Or more general, where param is name of association or custom (non-column) method
def add_staff(param)
sales.map(¶m) # Or map {|s| s.send(param) }
end
I'm trying out Rubymotion and can't seem to do figure how to accomplish what seems like a simple task.
I've set up a UITableView for a directory of people. I've created a rails back end that returns json.
Person model has a get_people class method defined:
def self.get_people
BubbleWrap::HTTP.get("http://myapp.com/api.json") do |response|
#people = BW::JSON.parse(response.body.to_str)
# p #people prints [{"id"=>10, "name"=>"Sam"}, {etc}] to the console
end
end
In the directory_controller I just want to set an instance variable for #data to the array that my endpoint returns such that I can populate the table view.
I am trying to do #data = Person.get_people in viewDidLoad, but am getting an error message that indicates the BW response object is being passed instead: undefined methodcount' for #BubbleWrap::HTTP::Query:0x8d04650 ...> (NoMethodError)`
So if I hard code my array into the get_people method after the BW response block everything works fine. But I find that I am also unable to persist an instance variable through the close of the BW respond block.
def self.get_people
BubbleWrap::HTTP.get("http://myapp.com/api.json") do |response|
#people = BW::JSON.parse(response.body.to_str)
end
p #people #prints nil to the console
# hard coding [{"id"=>10, "name"=>"Sam"}, {etc}] here puts my data in the table view correctly
end
What am I missing here? How do I get this data out of bubblewrap's response object and in to a usable form to pass to my controllers?
As explained in the BW documentation "BW::HTTP wraps NSURLRequest, NSURLConnection and friends to provide Ruby developers with a more familiar and easier to use API. The API uses async calls and blocks to stay as simple as possible."
Due to async nature of the call, in your 2nd snippet you are printing #people before you actually update it. THe right way is to pass the new data to the UI after parsing ended (say for instance #table.reloadData() if #people array is supposed to be displayed in a UITableView).
Here's an example:
def get_people
BubbleWrap::HTTP.get("http://myapp.com/api.json") do |response|
#people = BW::JSON.parse(response.body.to_str)
update_result()
end
end
def update_result()
p #people
# do stuff with the updated content in #people
end
Find a more complex use case with a more elaborate explanation at RubyMotion async programming with BubbleWrap
Personally, I'd skip BubbleWrap and go for something like this:
def self.get_people
people = []
json_string = self.get_json_from_http
json_data = json_string.dataUsingEncoding(NSUTF8StringEncoding)
e = Pointer.new(:object)
hash = NSJSONSerialization.JSONObjectWithData(json_data, options:0, error: e)
hash["person"].each do |person| # Assuming each of the people is stored in the JSON as "person"
people << person
end
people # #people is an array of hashes parsed from the JSON
end
def self.get_json_from_http
url_string = ("http://myapp.com/api.json").stringByAddingPercentEscapesUsingEncoding(NSUTF8StringEncoding)
url = NSURL.URLWithString(url_string)
request = NSURLRequest.requestWithURL(url)
response = nil
error = nil
data = NSURLConnection.sendSynchronousRequest(request, returningResponse: response, error: error)
raise "BOOM!" unless (data.length > 0 && error.nil?)
json = NSString.alloc.initWithData(data, encoding: NSUTF8StringEncoding)
end