Upserting a MongoDB collection doc with $max - max

TL;DR Only last $max statement seems to get implemented
Hi there I am trying to add or update data based on if the new incoming value is greater than the stored value using pymongo
{
'site': 'xyz.com',
'site_data':
{'particular_aspect_about_site':{'score_1': 2, 'score_2': 2, 'score_3': 1}},
`enter code here`{'a_different_aspect_about_site': {'score_a': 3, 'score_b': 1, 'score_c': 4}},
}
what I am trying is something like
def upsert_site_data():
site_to_upsert = None
data_to_upsert = None
json_object = request.get_json()
if "site" in json_object:
site_to_upsert = json_object["site"]
data_to_upsert = { "site" : site_to_upsert}
###### check if data was collected
if "site_data" in json_object:
data_to_upsert.update(json_object["site_data"])
collection_name = mongo.db.SiteData # establish mongo db instance to work with
try:
result = collection_name.update_one({"site" : site_to_upsert,
},
{
"$max" : { "particular_aspect_about_site.score_2" : data_to_upsert["particular_aspect_about_site"]["score_2"]} ,
"$max" : { "particular_aspect_about_site.score_3" : data_to_upsert["particular_aspect_about_site"]["score_3"]},
"$max" : { "a_different_aspect_about_site.score_b" : data_to_upsert["a_different_aspect_about_site"]["score_b"]},
}
,
upsert = True)
if result.raw_result["updatedExisting"] != True:
return jsonify({"status": "ok"}) , 200
if result.raw_result["updatedExisting"] == True:
return jsonify({"error": "Site data was not updated, thanks though :D "}), 200
except Exception as e:
return jsonify({"error": e}) , 400
else:
return jsonify({"error": "A site must be refenced"}) , 400
The issue is with the $max statements in the update section of the update function.
It only ever implements the final max.
I have also tried other methods to less success such as
{
"$max" : [
{ "particular_aspect_about_site.score_2" : data_to_upsert["particular_aspect_about_site"]["score_2"]},
{ "particular_aspect_about_site.score_3" : data_to_upsert["particular_aspect_about_site"]["score_3"]},
{ "a_different_aspect_about_site.score_b" : data_to_upsert["a_different_aspect_about_site"]["score_b"]}
]
}

So it was a mistake by me, I did
{ "$max" : { "particular_aspect_about_site.score_2" : data_to_upsert["particular_aspect_about_site"]["score_2"]} ,
"$max" : { "particular_aspect_about_site.score_3" : data_to_upsert["particular_aspect_about_site"]["score_3"]},
"$max" : { "a_different_aspect_about_site.score_b" : data_to_upsert["a_different_aspect_about_site"]["score_b"]},
}
But I should have used this template
{ $max: { field1: value1, field2: value2 ... } }
from [geeks4geeks][1]
now it looks like
{ "$max" : {
"particular_aspect_about_site.score_2" : data_to_upsert["particular_aspect_about_site"]["score_2"] ,
"particular_aspect_about_site.score_3" : data_to_upsert["particular_aspect_about_site"]["score_3"],
"a_different_aspect_about_site.score_b" : data_to_upsert["a_different_aspect_about_site"]["score_b"]},
}
Does what I want now. I'm sure there is a better way to do this and when i find it I will post, does what I need now
[1]: https://www.geeksforgeeks.org/mongodb-maximum-operator-max/

Related

protobuf map value is object or array of object

I have
message {
message leadsEntity{
optional string name = 1;
}
optional string description = 1;
map <string, leadsEntity> entity = 2;
}
Now I can have 2 types of responses-
// 1st case
{
description : "this is test",
entity : {
key1 : { name : "name1" },
key2 : { name : "name2" }
}
}
Or
// 2nd case
{
description : "this is test",
entity : {
key1 : { name : "name1" },
key2 : [ { name : "name2" } ] //the value in array now
}
}
Now the above given proto message works perfectly for 1st case.
But how to handle 2nd case? where the value is an object array?
I assume by the use of optional that you prefer proto2
Your existing proto is invalid.
Your response types are going to make your life difficult because the entity values are a union of types.
Perhaps something like this:
syntax = "proto2";
message X {
message LeadsEntity{
optional string name = 1;
}
message LeadsEntities {
repeated LeadsEntity leads_entity = 1;
}
message Value {
oneof These {
LeadsEntity leads_entity = 1;
LeadsEntities leads_entities = 2;
}
}
optional string description = 1;
map <string, Value> entity = 2;
}
I gave the root message the name X
I added LeadsEntities to hold the repeated LeadsEntity
I added Value to hold the oneof
from google.protobuf.json_format import MessageToJson
import x_pb2
x = x_pb2.X()
x.description = "this is a test"
x.entity["key1"].leads_entity.name = "name1"
le = x.entity["key2"].leads_entities.leads_entity.add()
le.name = "name2"
print(MessageToJson(x))
Yields:
{
"description": "this is a test",
"entity": {
"key1": {
"leadsEntity": {
"name": "name1"
}
},
"key2": {
"leadsEntities": {
"leadsEntity": [
{
"name": "name2"
}
]
}
}
}
}

Aggregating sequence of connected events

Lets say I have events like this in my log
{type:"approval_revokation", approval_id=22}
{type:"approval", request_id=12, approval_id=22}
{type:"control3", request_id=12}
{type:"control2", request_id=12}
{type:"control1", request_id=12}
{type:"request", request_id=12 requesting_user="user1"}
{type:"registration", userid="user1"}
I would like to do a search that aggregates one bucket for each approval_id containing all events connected to it as above. As you see there is not a single id field that can be used throughout the events, but they are all connected in a chain.
The reason I would like this is to feed this into a anomaly detector to verify things like that all controls where executed and validate registration event for a eventual approval.
Can this be done using aggregation or are there any other suggestion?
If there's no single unique "glue" parameter to tie these events together, I'm afraid the only choice is a brute-force map-reduce iterator on all the docs in the index.
After ingesting the above events:
POST _bulk
{"index":{"_index":"events","_type":"_doc"}}
{"type":"approval_revokation","approval_id":22}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"approval","request_id":12,"approval_id":22}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control3","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control2","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control1","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"request","request_id":12,"requesting_user":"user1"}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"registration","userid":"user1"}
we can link them together like so:
POST events/_search
{
"size": 0,
"aggs": {
"log_groups": {
"scripted_metric": {
"init_script": "state.groups = [];",
"map_script": """
int fetchIndex(List groups, def key, def value, def backup_key) {
if (key == null || value == null) {
// nothing to search
return -1
}
return IntStream.range(0, groups.size())
.filter(i -> groups.get(i)['docs']
.stream()
.anyMatch(_doc -> _doc.get(key) == value
|| (backup_key != null
&& _doc.get(backup_key) == value)))
.findFirst()
.orElse(-1);
}
def approval_id = doc['approval_id'].size() != 0
? doc['approval_id'].value
: null;
def request_id = doc['request_id'].size() != 0
? doc['request_id'].value
: null;
def requesting_user = doc['requesting_user.keyword'].size() != 0
? doc['requesting_user.keyword'].value
: null;
def userid = doc['userid.keyword'].size() != 0
? doc['userid.keyword'].value
: null;
HashMap valueMap = ['approval_id':approval_id,
'request_id':request_id,
'requesting_user':requesting_user,
'userid':userid];
def found = false;
for (def entry : valueMap.entrySet()) {
def field = entry.getKey();
def value = entry.getValue();
def backup_key = field == 'userid'
? 'requesting_user'
: field == 'requesting_user'
? 'userid'
: null;
def found_index = fetchIndex(state.groups, field, value, backup_key);
if (found_index != -1) {
state.groups[found_index]['docs'].add(params._source);
if (approval_id != null) {
state.groups[found_index]['approval_id'] = approval_id;
}
found = true;
break;
}
}
if (!found) {
HashMap nextInLine = ['docs': [params._source]];
if (approval_id != null) {
nextInLine['approval_id'] = approval_id;
}
state.groups.add(nextInLine);
}
""",
"combine_script": "return state",
"reduce_script": "return states"
}
}
}
}
returning the grouped events + the inferred approval_id:
"aggregations" : {
"log_groups" : {
"value" : [
{
"groups" : [
{
"docs" : [
{...}, {...}, {...}, {...}, {...}, {...}, {...}
],
"approval_id" : 22
},
{ ... }
]
}
]
}
}
Keep in mind that such scripts are going to be quite slow, esp. when run on large numbers of events.

Why does Spring Data fail on date queries?

I have records in my mongodb which are like this example record.
{
"_id" : ObjectId("5de6e329bf96cb3f8d253163"),
"changedOn" : ISODate("2019-12-03T22:35:21.126Z"),
"bappid" : "BAPP0131337",
}
I have code which is implemented as:
public List<ChangeEvent> fetchChangeList(Application app, Date from, Date to) {
Criteria criteria = null;
criteria = Criteria.where("bappid").is(app.getBappid());
Query query = Query.query(criteria);
if(from != null && to == null) {
criteria = Criteria.where("changedOn").gte(from);
query.addCriteria(criteria);
}
else if(to != null && from == null) {
criteria = Criteria.where("changedOn").lte(to);
query.addCriteria(criteria);
} else if(from != null && to != null) {
criteria = Criteria.where("changedOn").gte(from).lte(to);
query.addCriteria(criteria);
}
logger.info("Find change list query: {}", query.toString());
List<ChangeEvent> result = mongoOps.find(query, ChangeEvent.class);
return result;
This code always comes up empty. The logging statement generates a log entry like:
Find change list query: Query: { "bappid" : "BAPP0131337", "changedOn" : { "$gte" : { "$date" : 1575418473670 } } }, Fields: { }, Sort: { }
Playing around with variants of the query above in a database which has the record above gets the following results we get.
Returns records:
db["change-events"].find({ "bappid" : "BAPP0131337" }).pretty();
Returns empty set:
db["change-events"].find({ "bappid" : "BAPP0131337", "changedOn" : { "$gte" : { "$date" : 1575418473670 } } }).pretty();
Returns empty set:
db["change-events"].find({ "bappid" : "BAPP0131337", "changedOn" : { "$lte" : { "$date" : 1575418473670 } } }).pretty();
The record returned without the date query should be non empty on one of the two above. But it is empty on both.
What is wrong here?
Since the collection name change-events is different then Class name ChangeEvent so you have to pass the collection name in the find query of mongoOps as below:
List<ChangeEvent> result = mongoOps.find(query, ChangeEvent.class, "change-events");
I have tried it replicating and found that your query without dates in where clause also not working i.e:
Criteria criteria = null;
criteria = Criteria.where("bappid").is(bappid);
Query query = Query.query(criteria);
And the find query on mongoOps as below:
List<ChangeEvent> result = mongoTemplate.find(query, ChangeEvent.class);
Will not work, becuase collection name is missing, below query with collection name execute fine:
List<ChangeEvent> result1 = mongoTemplate.find(query, ChangeEvent.class, "changeEvents");
For details explanation of above discussion you can find out at my Github repo: https://github.com/krishnaiitd/learningJava/blob/master/spring-boot-sample-data-mongodb/src/main/java/sample/data/mongo/main/Application.java#L157

Azure CosmoDB - MongoDb - C# - how to increment a value inside an array of Object?

I've this schema for my db:
{
"_id" : "test_schema",
"t" : 5,
"p" : [
{
"id" : "207",
"v" : 4
},
{
"id" : "309",
"v" : 1
}
....
]
}
I'm trying to $inc the v value of p.id equal to "207".
I'm currently able to inc the t value with this code:
var result = collection.UpdateOneAsync(new BsonDocument("_id", "test_schema"}), new BsonDocument("$inc", new BsonDocument("t", 4)), new UpdateOptions() { IsUpsert = true }).Result;
but whe i try to update a value on array nothing happen(even no error!):
var result = collection.UpdateOneAsync(new BsonDocument(new Dictionary<string, object>() { { "_id", "test_schema" }, { "p.id", "207" } }), new BsonDocument("$inc", new BsonDocument("p.v", 4)), new UpdateOptions() { IsUpsert = true }).Result;
Following MongoDB documentation i noticed that "p.v", 4 should be "p.$.v" but in cosmodb raise a not valid $ symbol exception.
Any suggestion?
Cosmos DB doesn't yet support positional operator, it is on the roadmap. The feedback item https://feedback.azure.com/forums/599059-azure-cosmos-db-mongodb-api/suggestions/20091454-positional-array-update-via-query-support is filed to track interest in this feature. Please vote there if you need this support.

How to cast swiftyjson value if sometimes is detected as int and other as string

I am getting the login info using alamofire and swiftyjson
Alamofire.request(.POST, postsEndpoint, parameters: newPost)
.responseSwiftyJSON({ (request, response, json, error) in
in my post response i have the value
json["id_usuario"]
the problem is that, when the value is -1 or 0 (zero) it can be obtained as int
using
let idUser = json["id_usuario"].int
and example of the reponse with the value in -1
{
"id_usuario" : -1
}
and the response when the value is greater than, a success login
{
"estado" : "Jalisco",
"plan_activo" : "0",
"datos_registro_completos" : 1,
"plan" : 0,
"genero" : "H",
"id_usuario_openpay" : "annvwl3didjylvex0wzh",
"fb_id" : "10205386840402780",
"email" : "steel.edward#hotmail.com",
"postal_code" : "44630",
"address" : "Nueva Escocia #1514 Interior 106",
"nombres" : "Steel Edward",
"app_mat" : "George",
"app_pat" : "Vázquez",
"ciudad" : "Guadalajara",
"id_usuario" : "204",
"admin" : "1",
"phone_number" : "3334691505"
}
but if the value is greater than 0 returns a nil and only could be obtained as string
let idUser = json["id_usuario"].string
my final code works and looks like this
if let idUser = json["id_usuario"].int {
if(idUser == -1) {
//specific error from the server
} else if(idUser == 0) {
//another specific error from the server
}
} else if let idUser = json["id_usuario"].string {
if(idUser != "") {
//success
}
}
i would like to store the value always as Int and perform the validation using it, and to have a code like this
if(idUser == -1) {
//specific error from the server
} else if(idUser == 0) {
//another specific error from the server
} else if (idUser > 0) {
//success
}
var id = 0
if let num = json["id_usuario"].int {
id = num
} else if let str = json["id_usuario"].string,
let num = Int(str) {
id = num
}
//
// do something with id, which is an Int value
if you own the server code, you are better to find a bug there ....
This was my first solution not using Swifty-JSON
let id_usuario_object:NSObject = jsonData.valueForKey("id_usuario") as! NSObject
var id_usuario:Int
if ( id_usuario_object.isKindOfClass(NSString) ) {
let id_usuario_string:NSString = jsonData.valueForKey("id_usuario") as! NSString
id_usuario = Int(id_usuario_string.intValue)
} else {
id_usuario = jsonData.valueForKey("id_usuario") as! NSInteger
}
The problem resides in the JSON response from the PHP...
Before
echo json_encode( $results);
After
echo json_encode( $results, JSON_NUMERIC_CHECK );

Resources