MongoDB Group using Ruby driver - ruby

I'm trying to bring back a list of year/month combinations with counts for describing blog posts. The idea is that they will be displayed like so:
January 2010 (1 post)
December 2009 (2 posts)
...
I have managed to get this to work using the MongoDB JS shell, and it returns results in a useful format:
db.posts.group({
keyf: function(x){
return {
month: x.datetime.getMonth(),
year:x.datetime.getFullYear()
};
},
reduce: function(x,y){ y.count++ },
initial:{count:0}
})
Results:
[ { "month" : 0, "year" : 2010, "count" : 3 },
{ "month" : 0, "year" : 1970, "count" : 1 } ]
This is great, exactly what I'm after. However, trying to convert this into code appropriate for the ruby driver, I can't get it to work. I have looked at the documentation and from my understanding, the following should yield the same results (I'm using MongoMapper, hence the Post.collection):
#archive = Post.collection.group(
"function(x) { return { month: x.datetime.getMonth(), year:x.datetime.getFullYear() }; }",
nil, { :count => 0 }, 'function(x,y){y.count++}', true)
Instead of giving back the nice array of useful data, I'm getting this mess:
{
"function(x) { return { month: x.datetime.getMonth(), year:x.datetime.getFullYear() }; }" => nil,
"count" => 4.0
}
It seems that either it is completely defying its own documentation (and my understanding of the source code!) or I am missing something fundamental here. I'm almost pulling my hair out, any help gratefully accepted.

That's pretty strange behavior. I just ran your code locally, and everything worked. Can you verify that you're using the driver version 0.18.2? If so, make sure that that's the only version installed (just as a sanity check).
I don't think it should make any difference, but I wasn't running #group from MongoMapper -- I was using the gem alone. You might try that, too. Here's the code I ran:
require 'rubygems'
require 'mongo'
d = Mongo::Connection.new.db('blog')
c = d['post']
p c.group("function(x) { return { month: x.date.getMonth(), year:x.date.getFullYear() }; }",
nil,
{ :count => 0 },
"function(x,y){y.count++}",
true)

Related

How can I sort a request to a view - using nodejs API

I'm querying a cloudant DB from my nodejs App.
I am now trying to sort results from a view query.
My index (keys) are like this:
[ "FR000001", 1577189089166 ]
[ "FR000001", 1577189089165 ]
etc
from the following view:
function(doc) {
emit([doc.siteId, doc.creationDate],{"id" :doc._id, "rev": doc._rev, "siteId": doc.siteId, "creationDate": doc.creationDate, "scores": doc.scores, locationId: doc.locationId});
}
I managed to make that work on a real index using the syntax "sort: "-creationDate" " using syntax found in the bugs sections of cloudant github.
var ddoc = {
q: "site:\"" + id + "\"",
include_docs: false,
sort: "-creationDate",
};
const tmp = await cloudant.use('alarms').search('alarmSearch', 'IndexBySite', ddoc);
I can't make it work on my view with an array of query parameters. I have tried different variation around:
var ddoc_view = {
startkey: ["siteid1",0000000000000],
endkey: ["siteid1",9999999999999],
include_docs: true,
sort: "creationDate"
};
Can anyone help finding the right syntax, or pointing me to where I can find good "cloudant API for nodejs" documentation? for instance there is nothing on how to use sort" on the github... Thanks...
ok after another day of searching:
- best documentation I found is directly the couchdb doc: https://docs.couchdb.org/en/stable/ddocs/views/intro.html
- I ended up modifying the view:
emit([doc.creationDate, doc.siteId], {"id" :doc._id, "rev": doc._rev, "siteId": doc.siteId, "locationTag":doc.locationTag});
and the request:
var ddoc_view = {
endkey: [0000000000000, siteid],
startkey: [9999999999999, siteid],
include_docs: false,
descending: true,
limit: docsReturned,
};
To get a sorted response.

Chai assertion, unable to identify property using should / expect

Hello: Need your help on a chai assertion.
I have a JSON response as shown below. I want to assert that it contains "Lastname is mandatory" only.
I tried using this statement but the error i get is AssertionError: expected [ Array(2) ] to have a deep property '#text'. Please help how to write this correctly.
using expect
chai.expect(data.response.error).to.have.deep.property('#text', 'Lastname is mandatory.');
using should
data.response.error.should.have.deep.property('#text', 'Lastname is mandatory.');
Response JSON
{
response: {
error: [
{
'#id': '1000',
'#text': 'Firstname is mandatory.'
},
{
'#id': '10001',
'#text': 'Lastname is mandatory.'
}
],
result:
{
status: '0'
}
}
}
Prior to Chai version 4
The use of deep with property requires that you pass a complete path to the property you want to test. In other words, deep.property won't do a search through all the properties for you. As the documentation puts it:
If the deep flag is set, you can use dot- and bracket-notation for deep references into objects and arrays.
Something like:
data.response.should.have.deep.property("error[0].#text");
Or you can start the path to the property with an array index if the object on which you use should is an array:
data.response.error.should.have.deep.property("[0].#text");
Here is a complete example derived from the code you show:
const chai = require("chai");
chai.should();
const data = {
response: {
error: [
{
'#id': '1000',
'#text': 'Firstname is mandatory.'
},
{
'#id': '10001',
'#text': 'Lastname is mandatory.'
}
],
result:
{
status: '0'
}
}
};
it("works", () => {
data.response.should.have.deep.property("error[0].#text");
// Or this, which looks weird but is allowed...
data.response.error.should.have.deep.property("[0].#text");
});
Chai version 4 and later
The OP was using a release of Chai earlier than version 4. If you are using Chai version 4 and over, the flag to use to is not .deep anymore but .nested. So in earlier versions where you would use data.response.should.have.deep.property("error[0].#text"); in version 4 or later you'd use data.response.should.have.nested.property("error[0].#text");
Thanks to answer from #shvaikalesh at github. It has the relevant answer to my question which i provide here for future reference + the code extract is also below for quick reference.
chai.expect(data.response.error.some(e => e['#text'] == 'Lastname is mandatory.')).to.be.true

Ruby finding duplicates in MongoDB

I am struggling to get this working efficiently I think map reduce is the answer but can't getting anything working, I know it is probably a simple answer hopefully someone can help
Entry Model looks like this:
field :var_name, type: String
field :var_data, type: String
field :var_date, type: DateTime
field :external_id, type: Integer
If the external data source malfunctions we get duplicate data. One way to stop this was when consuming the results we check if a record with the same external_id already exists, as one we have already consumed. However this is slowing down the process a lot. The plan now is to check for duplicates once a day. So we are looking get a list of Entries with the same external_id. Which we can then sort and delete those no longer needed.
I have tried adapting the snippet from here https://coderwall.com/p/96dp8g/find-duplicate-documents-in-mongoid-with-map-reduce as shown below but get
failed with error 0: "exception: assertion src/mongo/db/commands/mr.cpp:480"
def find_duplicates
map = %Q{
function() {
emit(this.external_id, 1);
}
}
reduce = %Q{
function(key, values) {
return Array.sum(values);
}
}
Entry.all.map_reduce(map, reduce).out(inline: true).each do |entry|
puts entry["_id"] if entry["value"] != 1
end
end
Am I way off? Could anyone suggest a solution? I am using Mongiod, Rails 4.1.6 and Ruby 2.1
I got it working using the suggestion in the comments of the question by Stennie using the Aggregation framework. It looks like this:
results = Entry.collection.aggregate([
{ "$group" => {
_id: { "external_id" => "$external_id"},
recordIds: {"$addToSet" => "$_id" },
count: { "$sum" => 1 }
}},
{ "$match" => {
count: { "$gt" => 1 }
}}
])
I then loop through the results and delete any unnecessary entries.

How to group in mongo from ruby

I'm trying to group by 2 fields in my mongo collection from ruby. I've got it working from within the mongo client however can't get the ruby syntax right.
My command on the mongo client is
db.truckroutes.group({key: { "route" : 0, "assetId" : "" }, reduce: function (curr, result) {}, initial: {} })
Could anyone give me any ideas what the ruby syntax would be like to perform this command?
Thank you in advance!
OK, managed to fix this after spending a while experimenting.
trucks = #mongoCollection.group( ["route", "assetId"], { }, { }, "function() {}")

Upsert Multiple Records with MongoDb

I'm trying to get MongoDB to upsert multiple records with the following query, ultimately using MongoMapper and the Mongo ruby driver.
db.foo.update({event_id: { $in: [1,2]}}, {$inc: {visit:1}}, true, true)
This works fine if all the records exist, but does not create new records for records that do not exist. The following command has the desired effect from the shell, but is probably not ideal from the ruby driver.
[1,2].forEach(function(id) {db.foo.update({event_id: id}, {$inc: {visit:1}}, true, true) });
I could loop through each id I want to insert from within ruby, but that would necessitate a trip to the database for each item. Is there a way to upsert multiple items from the ruby driver with only a single trip to the database? What's the best practice here? Using mongomapper and the ruby driver, is there a way to send multiple updates in a single batch, generating something like the following?
db.foo.update({event_id: 1}, {$inc: {visit:1}}, true); db.foo.update({event_id: 2}, {$inc: {visit:1}}, true);
Sample Data:
Desired data after command if two records exist.
{ "_id" : ObjectId("4d6babbac0d8bb8238d02099"), "event_id" : 1, "visit" : 11 }
{ "_id" : ObjectId("4d6baf56c0d8bb8238d0209a"), "event_id" : 2, "visit" : 2 }
Actual data after command if two records exist.
{ "_id" : ObjectId("4d6babbac0d8bb8238d02099"), "event_id" : 1, "visit" : 11 }
{ "_id" : ObjectId("4d6baf56c0d8bb8238d0209a"), "event_id" : 2, "visit" : 2 }
Desired data after command if only the record with event_id 1 exists.
{ "_id" : ObjectId("4d6babbac0d8bb8238d02099"), "event_id" : 1, "visit" : 2 }
{ "_id" : ObjectId("4d6baf56c0d8bb8238d0209a"), "event_id" : 2, "visit" : 1 }
Actual data after command if only the record with event_id 1 exists.
{ "_id" : ObjectId("4d6babbac0d8bb8238d02099"), "event_id" : 1, "visit" : 2 }
This - correctly - will not insert any records with event_id 1 or 2 if they do not already exist
db.foo.update({event_id: { $in: [1,2]}}, {$inc: {visit:1}}, true, true)
This is because the objNew part of the query (see http://www.mongodb.org/display/DOCS/Updating#Updating-UpsertswithModifiers) does not have a value for field event_id. As a result, you will need at least X+1 trips to the database, where X is the number of event_ids, to ensure that you insert a record if one does not exist for a particular event_id (the +1 comes from the query above, which increases the visits counter for existing records). To say it in a different way, how does MongoDB know you want to use value 2 for the event_id and not 1? And why not 6?
W.r.t. batch insertion with ruby, I think it is possible as the following link suggests - although I've only used the Java driver: Batch insert/update using Mongoid?
What you are after is the Find and Modify command with the upsert option set to true. See the example from the Mongo test suite (same one linked to in the Find and Modify docs) for an example that looks very much like what you describe in your question.
I found a way to do this using the eval operator for server-side code execution. Here is the code snippit:
def batchpush(body, item_opts = {})
#batch << {
:body => body,
:duplicate_key => item_opts[:duplicate_key] || Mongo::Dequeue.generate_duplicate_key(body),
:priority => item_opts[:priority] || #config[:default_priority]
}
end
def batchprocess()
js = %Q|
function(batch) {
var nowutc = new Date();
var ret = [];
for(i in batch){
e = batch[i];
//ret.push(e);
var query = {
'duplicate_key': e.duplicate_key,
'complete': false,
'locked_at': null
};
var object = {
'$set': {
'body': e.body,
'inserted_at': nowutc,
'complete': false,
'locked_till': null,
'completed_at': null,
'priority': e.priority,
'duplicate_key': e.duplicate_key,
'completecount': 0
},
'$inc': {'count': 1}
};
db.#{collection.name}.update(query, object, true);
}
return ret;
}
|
cmd = BSON::OrderedHash.new
cmd['$eval'] = js
cmd['args'] = [#batch]
cmd['nolock'] = true
result = collection.db.command(cmd)
#batch.clear
#pp result
end
Multiple items are added with batchpush(), and then batchprocess() is called. The data is sent as an array, and the commands are all executed. This code is used in the MongoDequeue GEM, in this file.
Only one request is made, and all the upserts happen server-side.

Resources