We are using Datamapper in a Sinatra application and would like to use case insensitive like that works on both Sqlite (locally in development) and Postgresql (on Heroku in production).
We have statements like
TreeItem.all(:name.like =>"%#{term}%",:unique => true,:limit => 20)
If termis "BERL" we get the suggestion "BERLIN" from both the Sqlite and Postgresql backends. However if termis "Berl" we only get that result from Sqlite and not Postgresql.
I guess this has to do with the fact that both dm-postgres-adapter and dm-sqlite-adapter outputting a LIKE in the resulting SQL query. Since Postgresql has a case sensitive LIKE we get this (for us unwanted) behavior.
Is there a way to get case insensitive like in Datamapper without resorting to use a raw SQL query to the adapter or patching the adapter to use ILIKEinstead of LIKE?
I could of course use something in between, such as:
TreeItem.all(:conditions => ["name LIKE ?","%#{term}%"],:unique => true,:limit => 20)
but then we would be tied to the use of Postgresql within our own code and not just as a configuration for the adapter.
By writing my own data object adapter that overrides the like_operator method I managed to get Postgres' case insensitive ILIKE.
require 'do_postgres'
require 'dm-do-adapter'
module DataMapper
module Adapters
class PostgresAdapter < DataObjectsAdapter
module SQL #:nodoc:
private
# #api private
def supports_returning?
true
end
def like_operator(operand)
'ILIKE'
end
end
include SQL
end
const_added(:PostgresAdapter)
end
end
Eventually I however decided to port the application in question to use a document database.
For other people who happen to use datamapper wanting support for ilike as well as 'similar to' in PostgreSQL: https://gist.github.com/Speljohan/5124955
Just drop that in your project, and then to use it, see these examples:
Model.all(:column.ilike => '%foo%')
Model.all(:column.similar => '(%foo%)|(%bar%)')
Related
I am working on an app that allows Members to take a survey (Member has a one to many relationship with Response). Response holds the member_id, question_id, and their answer.
The survey is submitted all or nothing, so if there are any records in the Response table for that Member they have completed the survey.
My question is, how do I re-write the query below so that it actually works? In SQL this would be a prime candidate for the EXISTS keyword.
def surveys_completed
members.where(responses: !nil ).count
end
You can use includes and then test if the related response(s) exists like this:
def surveys_completed
members.includes(:responses).where('responses.id IS NOT NULL')
end
Here is an alternative, with joins:
def surveys_completed
members.joins(:responses)
end
The solution using Rails 4:
def surveys_completed
members.includes(:responses).where.not(responses: { id: nil })
end
Alternative solution using activerecord_where_assoc:
This gem does exactly what is asked here: use EXISTS to to do a condition.
It works with Rails 4.1 to the most recent.
members.where_assoc_exists(:responses)
It can also do much more!
Similar questions:
How to query a model based on attribute of another model which belongs to the first model?
association named not found perhaps misspelled issue in rails association
Rails 3, has_one / has_many with lambda condition
Rails 4 scope to find parents with no children
Join multiple tables with active records
You can use SQL EXISTS keyword in elegant Rails-ish manner using Where Exists gem:
members.where_exists(:responses).count
Of course you can use raw SQL as well:
members.where("EXISTS" \
"(SELECT 1 FROM responses WHERE responses.member_id = members.id)").
count
You can also use a subquery:
members.where(id: Response.select(:member_id))
In comparison to something with includes it will not load the associated models (which is a performance benefit if you do not need them).
If you are on Rails 5 and above you should use left_joins. Otherwise a manual "LEFT OUTER JOINS" will also work. This is more performant than using includes mentioned in https://stackoverflow.com/a/18234998/3788753. includes will attempt to load the related objects into memory, whereas left_joins will build a "LEFT OUTER JOINS" query.
def surveys_completed
members.left_joins(:responses).where.not(responses: { id: nil })
end
Even if there are no related records (like the query above where you are finding by nil) includes still uses more memory. In my testing I found includes uses ~33x more memory on Rails 5.2.1. On Rails 4.2.x it was ~44x more memory compared to doing the joins manually.
See this gist for the test:
https://gist.github.com/johnathanludwig/96fc33fc135ee558e0f09fb23a8cf3f1
where.missing (Rails 6.1+)
Rails 6.1 introduces a new way to check for the absence of an association - where.missing.
Please, have a look at the following code snippet:
# Before:
Post.left_joins(:author).where(authors: { id: nil })
# After:
Post.where.missing(:author)
And this is an example of SQL query that is used under the hood:
Post.where.missing(:author)
# SELECT "posts".* FROM "posts"
# LEFT OUTER JOIN "authors" ON "authors"."id" = "posts"."author_id"
# WHERE "authors"."id" IS NULL
As a result, your particular case can be rewritten as follows:
def surveys_completed
members.where.missing(:response).count
end
Thanks.
Sources:
where.missing official docs.
Pull request.
Article from the Saeloun blog.
Notes:
where.associated - a counterpart for checking for the presence of an association is also available starting from Rails 7.
See offical docs and this answer.
I am using DataMapper to interface with MySql. Is there any check I can do with Datamapper to ensure that the database is up?
If you want to test whether you can actually do some work with your database, something like this could be helpful:
begin
DataMapper.repository(:default).adapter.execute('SHOW TABLES;')
rescue
puts "Problem!"
end
This will make sure that the server is up and that the database you chose is valid (that's why something like SELECT 1 wouldn't work).
I've to add several fields in a mongoid model, I know there is not migration with MongoDB but if I go on without dropping the DB, making rails to "regenerate" the DB entirely, it doesn't display or use the new fields at all !
What's the best way to go here ? Is there something softer than drop/reopen mongodb ?
Thanks in advance
luca
In general it should be possible to update old documents with the new fields at runtime. There is no need for migrations in MongoDB.
You maybe want to write rake tasks to update your old documents with the new fields and default values.
You could find out these documents by checking those new fields which have per default a nil value.
Update
Easy style:
If you define a new field with a default value, this value should always be used as long as you set a new one:
app/models/my_model.rb
class MyModel
include Mongoid::Document
field :name, type: String
field :data, type: String
# NEW FIELD
field :note, type: String, default: "no note given so far!"
end
If you query your database you should get your default value for documents which haven't this field before your extension:
(rails console)
MyModel.first
#=> #<MyModel …other fields…, note: "no note given so far!">
I tested this with a fresh rails stack with a current mongoid on Ruby 1.9.2 - should work with other stacks, too.
More complicated/complex style:
If you didn't set a default value, you'll get nil for this new field.
app/models/my_model.rb
class MyModel
include Mongoid::Document
field :name, type: String
field :data, type: String
# NEW FIELD
field :note, type: String
end
(rails console)
MyModel.first
#=> #<MyModel …other fields…, note: nil>
Then you could set up a rake task and migration file like in this example:
lib/tasks/my_model_migration.rake:
namespace :mymodel do
desc "MyModel migration task"
task :migrate => :environment do
require "./db/migrate.rb"
end
end
db/migrate.rb:
olds = MyModel.where(note: nil)
# Enumerator of documents without a valid :note field (= nil)
olds.each do |doc|
doc.note = "(migration) no note given yet"
# or whatever your desired default value should be
doc.save! rescue puts "Could not modify doc #{doc.id}/#{doc.name}"
# the rescue is only a failsafe statement if something goes wrong
end
Run this migration with rake mymodel:migrate.
This is only a starting point and you can extend this to a full mongoid migration engine.
The task :migrate => :environment do … is necessary, otherwise rake won't load models.
It is a little ridiculous to say that you don't need migrations with mongodb or mongoid. Any sophisticated app needs to be refactored from time to time and that can mean pulling fields out of disparate documents into a new one.
Writing one off rake tasks is way less convenient and error prone than having migrations be part of your deploy script so that it always gets run on every environment.
https://github.com/adacosta/mongoid_rails_migrations brings AR style migrations to mongoid.
You might need them less often, but you will certainly need them as an app grows.
Below is a nice code example for data migration script with mongoid and the ruby mongo driver - to be used when your updated model no longer match production data.
http://pivotallabs.com/users/lee/blog/articles/1548-mongoid-migrations-using-the-mongo-driver
I whish we would stop using "no migrations with mongoid" as slogan. It'll turn people to MongoDB for the wrong reasons, and it's only partially true. No schema, true, but data still needs to be maintained, which IMO is harder with MongoDB than RDBMs. There are other, great reasons for choosing MongoDB and it depends on your problem.
Rails' ActiveRecord has a feature called Query Caching (ActiveRecord::QueryCache) which saves the result of SQL query for the life-span of a request. While I'm not very familiar with the internals of the implementation, I think that it saves the query results somewhere in the Rack env, which is discarded in the end of the request.
The Mongoid, unfortunately, doesn't currently provide such feature, and this is exacerbated by the fact, that some queries occur implicitly (references).
I'm considering to implement this feature, and I'm curious, where and how Mongoid (or, perhaps, mongo driver?) should be hooked in order to implement this.
Mongoid has caching, described under http://mongoid.org/en/mongoid/docs/extras.html
Also MongoDB itself has caching ability: http://www.mongodb.org/display/DOCS/Caching
The mongoid caching extra knows 2 different cases: Caching of all queries of a model or caching of a query.
Mongoid caching seems to work slightly different: it looks like mongoid delegates caching to mongodb. (In the sources of mongoid I only can find option settings for caching but no cache module.)
Finally would say, there is no real difference in the caching in general -- in memory is in fact in memory! No matter if it's in the app or in the database.
I don't prefer to implement an extra caching algorithm, because this seems to be redundant and a RAM killer.
BTW: If your really want to cache results in-app you could try Rails.cache or another cache gem as a workaround.
The other answer is obviously wrong. Not only mongoid or mongo driver doesn't cache the query, even if mongo would - it still might be on other machine across the network.
My solution was to wrap the receive_message in Mongo::Connection.
Pros: one definite place
Cons: deserialization still takes place
require 'mongo'
module Mongo
class Connection
module QueryCache
extend ActiveSupport::Concern
module InstanceMethods
# Enable the selector cache within the block.
def cache
#query_cache ||= {}
old, #query_cache_enabled = #query_cache_enabled, true
yield
ensure
clear_query_cache
#query_cache_enabled = old
end
# Disable the selector cache within the block.
def uncached
old, #query_cache_enabled = #query_cache_enabled, false
yield
ensure
#query_cache_enabled = old
end
def clear_query_cache
#query_cache.clear
end
def cache_receive_message(operation, message)
#query_cache[operation] ||= {}
key = message.to_s.hash
log = "[MONGO] CACHE %s"
if entry = #query_cache[operation][key]
Mongoid.logger.debug log % 'HIT'
entry
else
Mongoid.logger.debug log % 'MISS'
#query_cache[operation][key] = yield
end
end
def receive_message_with_cache(operation, message, log_message=nil, socket=nil, command=false)
if query_cache_enabled
cache_receive_message(operation, message) do
receive_message_without_cache(operation, message, log_message, socket, command)
end
else
receive_message_without_cache(operation, message, log_message, socket, command)
end
end
end # module InstanceMethods
included do
alias_method_chain :receive_message, :cache
attr_reader :query_cache, :query_cache_enabled
end
end # module QueryCache
end # class Connection
end
Mongo::Connection.send(:include, Mongo::Connection::QueryCache)
OK, Mongoid 4 supports QueryCache middleware.
Just add middleware in application.rb
config.middleware.use "Mongoid::QueryCache::Middleware"
And then profit:
MOPED: 127.0.0.1:27017 QUERY database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564dabb6d61631e21d70000')}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil runtime: 0.4397ms
MOPED: 127.0.0.1:27017 QUERY database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564dacf6d61631e21dc0000')}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil runtime: 0.4590ms
QUERY CACHE database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564c9596d61631e21d30000')}, "$orderby"=>{:_id=>1}}
QUERY CACHE database=XXX collection=page_variants selector={"$query"=>{"_id"=>BSON::ObjectId('5564dabb6d61631e21d70000')}, "$orderby"=>{:_id=>1}}
Source:
Mongoid changelog
https://github.com/mongoid/mongoid/blob/master/CHANGELOG.md#new-features-2
3410 Mongoid now has a query cache that can be used as a middleware in Rack applications. (Arthur Neves)
For Rails:
config.middleware.use(Mongoid::QueryCache::Middleware)
Mongoid 4.0+ now has a QueryCaching module: http://www.rubydoc.info/github/mongoid/mongoid/Mongoid/QueryCache
You can use it on finds by wrapping your lookups like so:
QueryCache.cache { MyCollection.find("xyz") }
Is there any way that I can fire a raw mongo query directly in Ruby instead of converting them to the native Ruby objects?
I went through Ruby Mongo Tutorial, but I cannot find such a method anywhere.
If it were mysql, I would have fired a query something like this.
ActiveRecord::Base.connection.execute("Select * from foo")
My mongo query is a bit large and it is properly executing in the MongoDB console. What I want is to directly execute the same inside Ruby code.
Here's a (possibly) better mini-tutorial on how to get directly into the guts of your MongoDB. This might not solve your specific problem but it should get you as far as the MongoDB version of SELECT * FROM table.
First of all, you'll want a Mongo::Connection object. If
you're using MongoMapper then you can call the connection
class method on any of your MongoMapper models to get a connection
or ask MongoMapper for it directly:
connection = YourMongoModel.connection
connection = MongoMapper.connection
Otherwise I guess you'd use the from_uri constructor to build
your own connection.
Then you need to get your hands on a database, you can do this
using the array access notation, the db method, or get
the current one straight from MongoMapper:
db = connection['database_name'] # This does not support options.
db = connection.db('database_name') # This does support options.
db = MongoMapper.database # This should be configured like
# the rest of your app.
Now you have a nice shiny Mongo::DB instance in your hands.
But, you probably want a Collection to do anything interesting
and you can get that using either array access notation or the
collection method:
collection = db['collection_name']
collection = db.collection('collection_name')
Now you have something that behaves sort of like an SQL table so
you can count how many things it has or query it using find:
cursor = collection.find(:key => 'value')
cursor = collection.find({:key => 'value'}, :fields => ['just', 'these', 'fields'])
# etc.
And now you have what you're really after: a hot out of the oven Mongo::Cursor
that points at the data you're interested in. Mongo::Cursor is
an Enumerable so you have access to all your usual iterating
friends such as each, first, map, and one of my personal
favorites, each_with_object:
a = cursor.each_with_object([]) { |x, a| a.push(mangle(x)) }
There are also command and eval methods on Mongo::DB that might do what you want.
In case you are using mongoid you will find the answer to your question here.
If you're using Mongoid 3, it provides easy access to its MongoDB driver: Moped. Here's an example of accessing some raw data without using Models to access the data:
db = Mongoid::Sessions.default
# inserting a new document
collection = db[:collection_name]
collection.insert(name: 'my new document')
# finding a document
doc = collection.find(name: 'my new document').first
# "select * from collection"
collection.find.each do |document|
puts document.inspect
end