I'm building a simple app on the side using an API I made with Sinatra that returns some JSON. It's quite a bit of JSON, my app's API relies on a few hundred requests to other APIs.
I can probably cache the results for 5 days or so, no problem with the data at all. I'm just not 100% sure how to implement the caching. How would I go about doing that with Sinatra?
Personally, I prefer to use redis for this type of things over memcached. I have an app that I use redis in pretty extensively, using it in a similar way to what you described. If I make a call that is not cached, page load time is upwards of 5 seconds, with redis, the load time drops to around 0.3 seconds. You can set an expires time as well, which can be changed quite easily. I would do something like this to retrieve the data from the cache.
require 'redis'
get '/my_data/:id' do
redis = Redis.new
if redis[params[:id]]
send_file redis[params[:id]], :type => 'application/json'
end
end
Then when you wanted to save the data to the cache, perhaps something like this:
require 'redis'
redis = Redis.new
<make API calls here and build your JSON>
redis[id] = json
redis.expire(id, 3600*24*5)
get '/my_data/:id' do
# security check for file-based caching
raise "invalid id" if params[:id] =~ /[^a-z0-9]/i
cache_file = File.join("cache",params[:id])
if !File.exist?(cache_file) || (File.mtime(cache_file) < (Time.now - 3600*24*5))
data = do_my_few_hundred_internal_requests(params[:id])
File.open(cache_file,"w"){ |f| f << data }
end
send_file cache_file, :type => 'application/json'
end
Don't forget to mkdir cache.
alternatively you could use memcache-client, but it will require you to install memcached system-wide.
Related
I am developing sinatra web app and
I would like to cache in server-side with sinatra-cache gem.
http://www.rubydoc.info/gems/sinatra-cache/0.3.7/frames
I can install it and it worked.
But now it cache all method.
Next what I want to do is limit a specific method to cache.
For example,
get '/cache-me'
will be cached but
get '/nocache'
won't be cached.
How can I control this ?
And also once cached , I want to expire after specified time duration.
How can I do it ?
You have to disable cache by your self.
In your get methods, just add no-cache parameter :cache => false to erb or haml calls. E.g.:
# To turn off caching on certain pages:
get('/nocahce') {
haml(:view_name, :cache => false) # <- here
}
Take a look on documentation for more details. Have a nice day!
I have a working Sinatra app that uses redis-namespace for its Redis connections. It works fine, but on Heroku it keeps running out of its 10 Redis connections, despite having very little traffic - they seem to stay open for ages and the app keeps opening new ones.
So, there might be a better way to structure what I've got, so it doesn't keep opening new connections. Or maybe I can use connection_pool... although I'm not sure how to use that with redis-namespace.
The Sinatra front end (myapp/frontend.rb) is something like:
require 'sinatra/base'
require 'myapp/store'
module MyApp
class Frontend < Sinatra::Base
registration_store = MyApp::Store::Registration.new
subscription_store = MyApp::Store::Subscription.new
get '/' do
...
end
...
end
end
And the Redis-accessing Store classes are in myapp/store.rb:
require 'redis'
require 'redis-namespace'
module MyApp
module Store
class RedisBase
attr_accessor :redis
def initialize
uri = URI.parse(ENV['REDISCLOUD_URL'])
redis = ::Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
#redis = ::Redis::Namespace.new(:myapp, :redis => redis)
end
class Registration < RedisBase
def add(user_id)
redis.sadd(:registrations, user_id)
end
...
end
class Subscription < RedisBase
...
end
end
end
end
The frontend stores data via the Store classes: registration_store.add(37).
Am I doing something wrong that keeps opening new connections unnecessarily? Or, how can I add connection_pool or similar?
I bumped into a similar problem and stumbled upon this question. I think you should add redis.quit somewhere in your code. Doing some manual testing monitoring connections with client list on the redis command line gives that the connection disappears on a quit. The object can still be used later and will open a new connection it the connection is closed. No need for pooling! (At least when the load is low.... I guess you may end up with calls not getting a connection under higher loads.)
I'm not sure how to solve this big performance issue of my application. I'm using open-uri to request the most popular videos from youtube and when I ran perftools https://github.com/tmm1/perftools.rb
It shows that the biggest performance issue is Timeout.timeout. Can anyone suggest me how to solve the problem?
I'm using ruby 1.8.7.
Edit:
This is the output from my profiler
https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0B4bANr--YcONZDRlMmFhZjQtYzIyOS00YjZjLWFlMGUtMTQyNzU5ZmYzZTU4&hl=en_US
Timeout is wrapping the function that is actually doing the work to ensure that if the server fails to respond within a certain time, the code will raise an error and stop execution.
I suspect that what you are seeing is that the server is taking some time to respond. You should look at caching the response in some way.
For instance, using memcached (pseudocode)
require 'dalli'
require 'open-uri'
DALLI = Dalli.client.new
class PopularVideos
def self.get
result = []
unless result = DALLI.get("videos_#{Date.today.to_s}")
doc = open("http://youtube/url")
result = parse_videos(doc) # parse the doc somehow
DALLI.set("videos_#{Date.today.to_s}", result)
end
result
end
end
PopularVideos.get # calls your expensive parsing script once
PopularVideos.get # gets the result from memcached for the rest of the day
I have the following code (just as a test) and I want to create an HTTP proxy using EventMachine. The code below is an example on the es-proxy GitHub page. However, when I run this and open up a website that has a moderate amount of images, the images start loading incorrectly. What I mean by this is that some images are loaded twice or if I request my icon for the navigation bar, I instead get the profile picture. This is especially evident if I refresh the page a few times.
It seems that the responses do not correspond to the matching request; causing everything to be jumbled. However, I'm not sure why this is. The code below seems simple enough for this to not be a problem.
require 'rubygems'
require 'em-proxy'
require 'http/parser' # gem install http_parser.rb
require 'uuid' # gem install uuid
# > ruby em-proxy-http.rb
# > curl --proxy localhost:9889 www.google.com
host = "0.0.0.0"
port = 9889
puts "listening on #{host}:#{port}..."
Proxy.start(:host => host, :port => port) do |conn|
#p = Http::Parser.new
#p.on_headers_complete = proc do |h|
session = UUID.generate
puts "New session: #{session} (#{h.inspect})"
host, port = h['Host'].split(':')
conn.server session, :host => host, :port => (port || 80)
conn.relay_to_servers #buffer
#buffer = ''
end
#buffer = ''
conn.on_connect do |data,b|
puts [:on_connect, data, b].inspect
end
conn.on_data do |data|
#buffer << data
#p << data
data
end
conn.on_response do |backend, resp|
#puts [:on_response, backend, resp].inspect
resp
end
conn.on_finish do |backend, name|
puts [:on_finish, name].inspect
end
end
Update
I believe I have insight as to what is happening but, still no way of solving my problem. I am creating a server for each request and when I relay my requests I have multiple servers. Then in the on response I should only be returning the response if it is from the correct server. However, I don't have a way to correlate this as of yet.
Here a proper response:
Try removing every puts in the example so the main loop can concentrate on doing the actual network I/O, it works for me like that.
I think there may be some kind of timeout playing behind this, maybe the client does not wait long enough for the full answer to come back while the server is stuck outputing text to the console.
That's the downside of using an event reactor, you have to make sure nothing blocks it.
The code doesn't seem to account for persistent http connections. Maybe you could try a HTTP 1.0 browser.
Rails' ActiveRecord has a feature called Query Caching (ActiveRecord::QueryCache) which saves the result of SQL query for the life-span of a request. While I'm not very familiar with the internals of the implementation, I think that it saves the query results somewhere in the Rack env, which is discarded in the end of the request.
The Mongoid, unfortunately, doesn't currently provide such feature, and this is exacerbated by the fact, that some queries occur implicitly (references).
I'm considering to implement this feature, and I'm curious, where and how Mongoid (or, perhaps, mongo driver?) should be hooked in order to implement this.
Mongoid has caching, described under http://mongoid.org/en/mongoid/docs/extras.html
Also MongoDB itself has caching ability: http://www.mongodb.org/display/DOCS/Caching
The mongoid caching extra knows 2 different cases: Caching of all queries of a model or caching of a query.
Mongoid caching seems to work slightly different: it looks like mongoid delegates caching to mongodb. (In the sources of mongoid I only can find option settings for caching but no cache module.)
Finally would say, there is no real difference in the caching in general -- in memory is in fact in memory! No matter if it's in the app or in the database.
I don't prefer to implement an extra caching algorithm, because this seems to be redundant and a RAM killer.
BTW: If your really want to cache results in-app you could try Rails.cache or another cache gem as a workaround.
The other answer is obviously wrong. Not only mongoid or mongo driver doesn't cache the query, even if mongo would - it still might be on other machine across the network.
My solution was to wrap the receive_message in Mongo::Connection.
Pros: one definite place
Cons: deserialization still takes place
require 'mongo'
module Mongo
class Connection
module QueryCache
extend ActiveSupport::Concern
module InstanceMethods
# Enable the selector cache within the block.
def cache
#query_cache ||= {}
old, #query_cache_enabled = #query_cache_enabled, true
yield
ensure
clear_query_cache
#query_cache_enabled = old
end
# Disable the selector cache within the block.
def uncached
old, #query_cache_enabled = #query_cache_enabled, false
yield
ensure
#query_cache_enabled = old
end
def clear_query_cache
#query_cache.clear
end
def cache_receive_message(operation, message)
#query_cache[operation] ||= {}
key = message.to_s.hash
log = "[MONGO] CACHE %s"
if entry = #query_cache[operation][key]
Mongoid.logger.debug log % 'HIT'
entry
else
Mongoid.logger.debug log % 'MISS'
#query_cache[operation][key] = yield
end
end
def receive_message_with_cache(operation, message, log_message=nil, socket=nil, command=false)
if query_cache_enabled
cache_receive_message(operation, message) do
receive_message_without_cache(operation, message, log_message, socket, command)
end
else
receive_message_without_cache(operation, message, log_message, socket, command)
end
end
end # module InstanceMethods
included do
alias_method_chain :receive_message, :cache
attr_reader :query_cache, :query_cache_enabled
end
end # module QueryCache
end # class Connection
end
Mongo::Connection.send(:include, Mongo::Connection::QueryCache)
OK, Mongoid 4 supports QueryCache middleware.
Just add middleware in application.rb
config.middleware.use "Mongoid::QueryCache::Middleware"
And then profit:
MOPED: 127.0.0.1:27017 QUERY database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564dabb6d61631e21d70000')}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil runtime: 0.4397ms
MOPED: 127.0.0.1:27017 QUERY database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564dacf6d61631e21dc0000')}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil runtime: 0.4590ms
QUERY CACHE database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564c9596d61631e21d30000')}, "$orderby"=>{:_id=>1}}
QUERY CACHE database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564dabb6d61631e21d70000')}, "$orderby"=>{:_id=>1}}
Source:
Mongoid changelog
https://github.com/mongoid/mongoid/blob/master/CHANGELOG.md#new-features-2
3410 Mongoid now has a query cache that can be used as a middleware in Rack applications. (Arthur Neves)
For Rails:
config.middleware.use(Mongoid::QueryCache::Middleware)
Mongoid 4.0+ now has a QueryCaching module: http://www.rubydoc.info/github/mongoid/mongoid/Mongoid/QueryCache
You can use it on finds by wrapping your lookups like so:
QueryCache.cache { MyCollection.find("xyz") }