Why does Spring Data fail on date queries? - spring

I have records in my mongodb which are like this example record.
{
"_id" : ObjectId("5de6e329bf96cb3f8d253163"),
"changedOn" : ISODate("2019-12-03T22:35:21.126Z"),
"bappid" : "BAPP0131337",
}
I have code which is implemented as:
public List<ChangeEvent> fetchChangeList(Application app, Date from, Date to) {
Criteria criteria = null;
criteria = Criteria.where("bappid").is(app.getBappid());
Query query = Query.query(criteria);
if(from != null && to == null) {
criteria = Criteria.where("changedOn").gte(from);
query.addCriteria(criteria);
}
else if(to != null && from == null) {
criteria = Criteria.where("changedOn").lte(to);
query.addCriteria(criteria);
} else if(from != null && to != null) {
criteria = Criteria.where("changedOn").gte(from).lte(to);
query.addCriteria(criteria);
}
logger.info("Find change list query: {}", query.toString());
List<ChangeEvent> result = mongoOps.find(query, ChangeEvent.class);
return result;
This code always comes up empty. The logging statement generates a log entry like:
Find change list query: Query: { "bappid" : "BAPP0131337", "changedOn" : { "$gte" : { "$date" : 1575418473670 } } }, Fields: { }, Sort: { }
Playing around with variants of the query above in a database which has the record above gets the following results we get.
Returns records:
db["change-events"].find({ "bappid" : "BAPP0131337" }).pretty();
Returns empty set:
db["change-events"].find({ "bappid" : "BAPP0131337", "changedOn" : { "$gte" : { "$date" : 1575418473670 } } }).pretty();
Returns empty set:
db["change-events"].find({ "bappid" : "BAPP0131337", "changedOn" : { "$lte" : { "$date" : 1575418473670 } } }).pretty();
The record returned without the date query should be non empty on one of the two above. But it is empty on both.
What is wrong here?

Since the collection name change-events is different then Class name ChangeEvent so you have to pass the collection name in the find query of mongoOps as below:
List<ChangeEvent> result = mongoOps.find(query, ChangeEvent.class, "change-events");
I have tried it replicating and found that your query without dates in where clause also not working i.e:
Criteria criteria = null;
criteria = Criteria.where("bappid").is(bappid);
Query query = Query.query(criteria);
And the find query on mongoOps as below:
List<ChangeEvent> result = mongoTemplate.find(query, ChangeEvent.class);
Will not work, becuase collection name is missing, below query with collection name execute fine:
List<ChangeEvent> result1 = mongoTemplate.find(query, ChangeEvent.class, "changeEvents");
For details explanation of above discussion you can find out at my Github repo: https://github.com/krishnaiitd/learningJava/blob/master/spring-boot-sample-data-mongodb/src/main/java/sample/data/mongo/main/Application.java#L157

Related

Spring data jpa "IN" clause using specification

i am trying to get the records from the database using spring data jpa Speicification API.
here i need to put a condition for "In" clause for status column, for that i am code like below.
public static Specification<UserEntity> userSpecificationsforOperator(String orgName, String groupID,
List<String> status, Date startDate, Date endDate) {
return (root, query, builder) -> {
List<Predicate> predicates = new ArrayList<>();
if (orgName != null) {
Join<UserEntity, OrganizationEntity> organization = root.join("organization");
predicates.add(builder.equal(organization.get("name"), orgName));
}
/*
* if (groupID != null) {
* predicates.add(builder.equal(root.get("refApprovalGroupId"), groupID)); }
*/
if (status != null && status.size()>0) {
predicates.add(root.get("status").in(status));
}
if (startDate != null) {
predicates.add(builder.greaterThanOrEqualTo(root.get("createdDate"), startDate.toInstant()));
}
if (endDate != null) {
predicates.add(builder.lessThanOrEqualTo(root.get("createdDate"), endDate.toInstant()));
}
Predicate[] p = predicates.toArray(new Predicate[predicates.size()]);
return p.length == 0 ? null : p.length == 1 ? p[0] : builder.and(p);
};
}
the above code generating the query in cosole like below
SELECT userentity0_.id AS id1_68_,
userentity0_.created_by AS created_2_68_,
userentity0_.created_date AS created_3_68_,
userentity0_.last_modified_by AS last_mod4_68_,
userentity0_.last_modified_date AS last_mod5_68_,
userentity0_.group_id AS group_id6_68_,
userentity0_.group_name AS group_na7_68_,
userentity0_.is_enrollment_updated AS is_enrol8_68_,
userentity0_.is_federated AS is_feder9_68_,
userentity0_.name AS name10_68_,
userentity0_.organization_id AS organiz17_68_,
userentity0_.ref_approval_group_id AS ref_app11_68_,
userentity0_.reference_name AS referen12_68_,
userentity0_.status AS status13_68_,
userentity0_.uims_id AS uims_id14_68_,
userentity0_.user_status AS user_st15_68_,
userentity0_.version AS version16_68_
FROM user userentity0_
INNER JOIN organization organizati1_
ON userentity0_.organization_id = organizati1_.id
WHERE organizati1_.name ='utopia'
AND ( userentity0_.status =(?,?)
when i take the query into db tool and passing the values i am getting the data.
but while running from the application i am not getting the data.
here i understood that i am able to generate the query properly but my values are not passing correctly.
so could you please suggest how i can get my code return the data.
Maybe the implementation of it causes the issue...
Here, use this and let me know if it works
if (status != null && status.size()>0) {
predicates.add(builder.in(root.get("status")).value(status));
}

Aggregating sequence of connected events

Lets say I have events like this in my log
{type:"approval_revokation", approval_id=22}
{type:"approval", request_id=12, approval_id=22}
{type:"control3", request_id=12}
{type:"control2", request_id=12}
{type:"control1", request_id=12}
{type:"request", request_id=12 requesting_user="user1"}
{type:"registration", userid="user1"}
I would like to do a search that aggregates one bucket for each approval_id containing all events connected to it as above. As you see there is not a single id field that can be used throughout the events, but they are all connected in a chain.
The reason I would like this is to feed this into a anomaly detector to verify things like that all controls where executed and validate registration event for a eventual approval.
Can this be done using aggregation or are there any other suggestion?
If there's no single unique "glue" parameter to tie these events together, I'm afraid the only choice is a brute-force map-reduce iterator on all the docs in the index.
After ingesting the above events:
POST _bulk
{"index":{"_index":"events","_type":"_doc"}}
{"type":"approval_revokation","approval_id":22}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"approval","request_id":12,"approval_id":22}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control3","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control2","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control1","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"request","request_id":12,"requesting_user":"user1"}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"registration","userid":"user1"}
we can link them together like so:
POST events/_search
{
"size": 0,
"aggs": {
"log_groups": {
"scripted_metric": {
"init_script": "state.groups = [];",
"map_script": """
int fetchIndex(List groups, def key, def value, def backup_key) {
if (key == null || value == null) {
// nothing to search
return -1
}
return IntStream.range(0, groups.size())
.filter(i -> groups.get(i)['docs']
.stream()
.anyMatch(_doc -> _doc.get(key) == value
|| (backup_key != null
&& _doc.get(backup_key) == value)))
.findFirst()
.orElse(-1);
}
def approval_id = doc['approval_id'].size() != 0
? doc['approval_id'].value
: null;
def request_id = doc['request_id'].size() != 0
? doc['request_id'].value
: null;
def requesting_user = doc['requesting_user.keyword'].size() != 0
? doc['requesting_user.keyword'].value
: null;
def userid = doc['userid.keyword'].size() != 0
? doc['userid.keyword'].value
: null;
HashMap valueMap = ['approval_id':approval_id,
'request_id':request_id,
'requesting_user':requesting_user,
'userid':userid];
def found = false;
for (def entry : valueMap.entrySet()) {
def field = entry.getKey();
def value = entry.getValue();
def backup_key = field == 'userid'
? 'requesting_user'
: field == 'requesting_user'
? 'userid'
: null;
def found_index = fetchIndex(state.groups, field, value, backup_key);
if (found_index != -1) {
state.groups[found_index]['docs'].add(params._source);
if (approval_id != null) {
state.groups[found_index]['approval_id'] = approval_id;
}
found = true;
break;
}
}
if (!found) {
HashMap nextInLine = ['docs': [params._source]];
if (approval_id != null) {
nextInLine['approval_id'] = approval_id;
}
state.groups.add(nextInLine);
}
""",
"combine_script": "return state",
"reduce_script": "return states"
}
}
}
}
returning the grouped events + the inferred approval_id:
"aggregations" : {
"log_groups" : {
"value" : [
{
"groups" : [
{
"docs" : [
{...}, {...}, {...}, {...}, {...}, {...}, {...}
],
"approval_id" : 22
},
{ ... }
]
}
]
}
}
Keep in mind that such scripts are going to be quite slow, esp. when run on large numbers of events.

Spring boot mongodb: Query date as ISODate instead of $date long

I'm building my query like this:
Date date = new Date();
Criteria criteria = Criteria
.where("metadata.value.digitalitzacio.dataDigitalitzacio")
.is(new Date(2018,10,10));
this.mongoTemplate.find(Query.query(criteria));
It builds this query:
Query: { "metadata.value.digitalitzacio.dataDigitalitzacio" : { "$date" : 61499948400000 } }
So, it fails.
It sends query as an $date long, instead of an ISODate.
I mean, metadata.value.digitalitzacio.dataDigitalitzacio is stored as a ISODate into collection:
{
"_id" : "cpd4-175ec7f0-d70f-4b63-a709-69918d98c4f2",
"metadata" : [
{
"user" : "RDOCFO",
"value" : {
"digitalitzacio" : {
"csvDigitalitzacio" : "eeeeeeeeee",
"dataDigitalitzacio" : ISODate("2018-10-10T00:00:00Z"),
"empleatDigitalitzacio" : "empleat-digitalitzacio"
}
}
}
]
}
But it's queried as a $date long. How vould I solve that?
From https://stackoverflow.com/a/30294522/9731186, the following code should work, I didn't tested it though. java.util.Date(int, int, int) is now deprecated.
String string_date = "10-10-2018";
SimpleDateFormat f = new SimpleDateFormat("dd-MM-yyyy");
Date d = new Date();
try {
d = f.parse(string_date);
long milliseconds = d.getTime();
} catch (ParseException ex) {
ex.printStackTrace();
}
Criteria criteria = Criteria
.where("metadata.value.digitalitzacio.dataDigitalitzacio")
.is(d);
}
this.mongoTemplate.find(Query.query(criteria));
}

Why can't I compare two fields in a search predicate in Sitecore 7.5?

I am trying to build a search predicate in code that compares two fields in Sitecore and I am getting a strange error message. Basically I have two date fields on each content item - FirstPublishDate (the date that the content item was first published) and LastPublishDate (the last date that the content item was published). I would like to find all content items where the LastPublishDate falls within a certain date range AND where the LastPublishDate does not equal the FirstPublishDate. Using Linq here is my method for generating the predicate...
protected Expression<Func<T, Boolean>> getDateFacetPredicate<T>() where T : MySearchResultItem
{
var predicate = PredicateBuilder.True<T>();
foreach (var facet in myFacetCategories)
{
var dateTo = System.DateTime.Now;
var dateFrom = dateTo.AddDays(facet.Value*-1);
predicate = predicate.And(i => i.LastPublishDate.Between(dateFrom, dateTo, Inclusion.Both)).And(j => j.LastPublishDate != j.FirstPublishDate);
}
return predicate;
}
Then I use this predicate in my general site search code to perform the search as follows: the above predicate gets passed in to this method as the "additionalWhere" parameter.
public static SearchResults<T> GeneralSearch<T>(string searchText, ISearchIndex index, int currentPage = 0, int pageSize = 20, string language = "", IEnumerable<string> additionalFields = null,
Expression<Func<T, Boolean>> additionalWhere = null, Expression<Func<T, Boolean>> additionalFilter = null, IEnumerable<string> facets = null,
Expression<Func<T, Boolean>> facetFilter = null, string sortField = null, SortDirection sortDirection = SortDirection.Ascending) where T : SearchResultItem {
using (var context = index.CreateSearchContext()) {
var query = context.GetQueryable<T>();
if (!string.IsNullOrWhiteSpace(searchText)) {
var keywordPred = PredicateBuilder.True<T>();
// take into account escaping of special characters and working around Sitecore limitation with Contains and Equals methods
var isSpecialMatch = Regex.IsMatch(searchText, "[" + specialSOLRChars + "]");
if (isSpecialMatch) {
var wildcardText = string.Format("\"*{0}*\"", Regex.Replace(searchText, "([" + specialSOLRChars + "])", #"\$1"));
wildcardText = wildcardText.Replace(" ", "*");
keywordPred = keywordPred.Or(i => i.Content.MatchWildcard(wildcardText)).Or(i => i.Name.MatchWildcard(wildcardText));
}
else {
keywordPred = keywordPred.Or(i => i.Content.Contains(searchText)).Or(i => i.Name.Contains(searchText));
}
if (additionalFields != null && additionalFields.Any()) {
keywordPred = additionalFields.Aggregate(keywordPred, (current, field) => current.Or(i => i[field].Equals(searchText)));
}
//query = query.Where(i => (i.Content.Contains(searchText) || i.Name.Contains(searchText))); // more explicit call to check the content or item name for our term
query = query.Where(keywordPred);
}
if (language == string.Empty) {
language = Sitecore.Context.Language.ToString();
}
if (language != null) {
query = query.Filter(i => i.Language.Equals(language));
}
query = query.Page(currentPage, pageSize);
if (additionalWhere != null) {
query = query.Where(additionalWhere);
}
if (additionalFilter != null) {
query = query.Filter(additionalFilter);
}
query = query.ApplySecurityFilter();
FacetResults resultFacets = null;
if (facets != null && facets.Any()) {
resultFacets = facets.Aggregate(query, (current, fname) => current.FacetOn(i => i[fname])).GetFacets();
}
// calling this before applying facetFilter should allow us to get a total facet set
// instead of just those related to the current result set
// var resultFacets = query.GetFacets();
// apply after getting facets for more complete facet list
if (facetFilter != null) {
query = query.Where(facetFilter);
}
if (sortField != null)
{
if (sortDirection == SortDirection.Ascending)
{
query = query.OrderBy(x => x[sortField]);
}
else
{
query = query.OrderByDescending(x => x[sortField]);
}
}
var results = query.GetResults(); // this enumerates the actual results
return new SearchResults<T>(results.Hits, results.TotalSearchResults, resultFacets);
}
}
When I try this I get the following error message:
Server Error in '/' Application.
No constant node in query node of type: 'Sitecore.ContentSearch.Linq.Nodes.EqualNode'. Left: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'. Right: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.NotSupportedException: No constant node in query node of type: 'Sitecore.ContentSearch.Linq.Nodes.EqualNode'. Left: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'. Right: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'.
Source Error:
Line 548: FacetResults resultFacets = null;
Line 549: if (facets != null && facets.Any()) {
Line 550: resultFacets = facets.Aggregate(query, (current, fname) => current.FacetOn(i => i[fname])).GetFacets();
Line 551: }
Line 552: // calling this before applying facetFilter should allow us to get a total facet set
From what I can understand about the error message it seems to not like that I am trying to compare two different fields to each other instead of comparing a field to a constant. The other odd thing is that the error seems to be pointing to a line of code that has to do with aggregating facets. I did a Google search and came up with absolutely nothing relating to this error. Any ideas?
Thanks,
Corey
I think what you are trying is not possible, and if you look at this that might indeed be the case. A solution that is given there is to put your logic in the index: create a ComputedField that checks your dates and puts a value in the index that you can search on (can be a simple boolean).
You will need to split your logic though - the query on the date range can still be done in the predicate (as it is relative to the current date) but the comparison of first and last should be done on index time instead of on query time.

foreach or switch messing up linq query?

Trying to build up a query to filter on data in the following manner works OK, returning users filtered by whatever filters are in the FilterNamesAndValues parameter.
GetAllUsersFiltered(..., Dictionary<string,string> FilterNamesAndValues)
{
....
List<DataContracts.IUser> lstUsers = new List<DataContracts.IUser>();
....
var query = from u in lstUsers select u;
string firstName = string.Empty;
FilterNamesAndValues.TryGetValue("FirstName", out firstName);
query = query.Where(u => u.FirstName == firstName);
string company = string.Empty;
FilterNamesAndValues.TryGetValue("Company", out company);
query = query.Where(u => u.CompanyName == company);
....
return query.ToList();
}
The example below however doesn't work and I can't see why:
GetAllUsersFiltered(..., Dictionary<string,string> FilterNamesAndValues)
{
....
List<DataContracts.IUser> lstUsers = new List<DataContracts.IUser>();
....
var query = from u in lstUsers select u;
foreach (KeyValuePair<string, string> kv in FilterNamesAndValues)
{
if (kv.Value != null)
{
switch (kv.Key)
{
case "FirstName":
query = query.Where(u => u.FirstName == kv.Value);
break;
case "Company":
query = query.Where(u => u.CompanyName == kv.Value);
break;
}
}
}
return query.ToList();
}
After the application has hit the first switch case, I can do a query.ToList() and see a row in there. But by the time the execution has gone around the loop to hit the second filter, query.ToList() returns nothing. The query is not filtered successively the way it was in the first example and worse than that, the filter conditions have effectively been lost. There's probably an obvious explanation for this, but right now I can't see it.
The problem is that you're closing over kv in the foreach, but the query is executed using deferred execution. This causes it to close over the wrong value. For details on what's happening, I'd recommend Eric Lippert's post titled "Closing over the loop variable considered harmful".
You can solve this via a temporary:
foreach (KeyValuePair<string, string> kvOriginal in FilterNamesAndValues)
{
// Make a temporary in the correct scope!
KeyValuePair<string, string> kv = kvOriginal;
if (kv.Value != null)
{
switch (kv.Key)
{
case "FirstName":
query = query.Where(u => u.FirstName == kv.Value);
break;
case "Company":
query = query.Where(u => u.CompanyName == kv.Value);
break;
}
}
}

Resources