having troubles getting !any() query to work with raven db - linq

I am new to RavenDB and it's been frustrating try to get the Any() LINQ query to work correctly. Here is what my document looks like:
{
"Key": "BKey",
"Text": "B Key",
"IsLocal": false,
"Category": null,
"_destroy": false,
"Translations": [
{
"CultureCode": "es-ES",
"Text": null
},
{
"CultureCode": "ja-JP",
"Text": "Hello"
}
]
}
I would expect the following query to give me all entries that don't have translations for "es-ES":
var nonTranslatedEntries =
_docs.Query<ResourceEntry>().Where( e => e.Translations == null || e.Translations.Count == 0 || !e.Translations.Any(t => t.CultureCode == "es-ES" && t.Text != null))
However, this isn't working. It's bringing back the entries even when a translation for the specified CultureCode exists. It works if I have only have one item inside the translations array. But as soon as I have more than one item inside the translations array, then the query stops working.
As an alternative solution, I tried to do the following:
var translatedEntries = from re in _docs.Query<ResourceEntry>()
where re.Translations.Any(t => t.CultureCode == cultureCode && t.Text != null)
select new {Id = re.Id};
var translatedIds = translatedEntries.ToList().Select(e => e.Id).ToList();
var nonTranslatedEntries =
_docs.Query<ResourceEntry>().Where(e => !e.Id.In(translatedEntryIds));
But that just brings back an empty list.
Any help would be very much appreciated.
Thanks,
Nizar

A static index that will get the job done:
public class Resources_ByTranslation :
AbstractIndexCreationTask<ResourceEntry, Resources_ByTranslation.IndexEntry>
{
public class IndexEntry
{
public string Key { get; set; }
public string CultureCodes { get; set; }
}
public Resources_ByTranslation()
{
Map = entries => from entry in entries
select new {
entry.Key,
CultureCodes = entry.Translations
.Where(x => x.Text != null)
.Select(x => x.CultureCode)
};
}
}
Then query with:
var nonTranslatedEntries =
session.Query<Resources_ByTranslation.IndexEntry, Resources_ByTranslation>()
.Where(x => x.CultureCodes != "es-ES")
.As<ResourceEntry>();
Note that the CultureCodes list is being treated as single string. This is due to how then index matching works internally. It's slightly strange, but it does work.

I ended up fixing my issue as follows:
Upon creation of a resource entry, I go ahead and create all the possible translations and set the translation text to null:
{
"Key": "BKey",
"Text": "B Key",
"IsLocal": false,
"Category": null,
"_destroy": false,
"Translations": [
{
"CultureCode": "es-ES",
"Text": null
},
{
"CultureCode": "ja-JP",
"Text": null
}
]
}
I then created the following index:
public class ResourceEntries_ByCultureCodes : AbstractIndexCreationTask<ResourceEntry>
{
public ResourceEntries_ByCultureCodes()
{
Map = entries => from e in entries
select new
{
e.Id,
e.Key,
e.Text,
e.IsLocal,
e.Category,
_ = e.Translations.Select(t => CreateField(t.CultureCode, t.Text != null, false, true))
};
}
}
That gives me a flat record of translations with the value set to 'true' if a translation exists for a that specific culture. For example, I get back something like this:
{
Id,
Key,
Text,
IsLocal,
Category,
es-ES: <true> or <false> depending upon whether Translation.Text != null
ja-JP:<true> or <false> depending upon whether Translation.Text != null
}
I can then do LuceneQuery as follows to get all entries for "es-ES" that have NOT been translated:
_docs.LuceneQuery<ResourceEntry, ResourceEntries_ByCultureCodes>().WhereEquals("es-ES", false);
I wish there were an easier solution, though.

Related

Aggregating sequence of connected events

Lets say I have events like this in my log
{type:"approval_revokation", approval_id=22}
{type:"approval", request_id=12, approval_id=22}
{type:"control3", request_id=12}
{type:"control2", request_id=12}
{type:"control1", request_id=12}
{type:"request", request_id=12 requesting_user="user1"}
{type:"registration", userid="user1"}
I would like to do a search that aggregates one bucket for each approval_id containing all events connected to it as above. As you see there is not a single id field that can be used throughout the events, but they are all connected in a chain.
The reason I would like this is to feed this into a anomaly detector to verify things like that all controls where executed and validate registration event for a eventual approval.
Can this be done using aggregation or are there any other suggestion?
If there's no single unique "glue" parameter to tie these events together, I'm afraid the only choice is a brute-force map-reduce iterator on all the docs in the index.
After ingesting the above events:
POST _bulk
{"index":{"_index":"events","_type":"_doc"}}
{"type":"approval_revokation","approval_id":22}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"approval","request_id":12,"approval_id":22}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control3","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control2","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"control1","request_id":12}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"request","request_id":12,"requesting_user":"user1"}
{"index":{"_index":"events","_type":"_doc"}}
{"type":"registration","userid":"user1"}
we can link them together like so:
POST events/_search
{
"size": 0,
"aggs": {
"log_groups": {
"scripted_metric": {
"init_script": "state.groups = [];",
"map_script": """
int fetchIndex(List groups, def key, def value, def backup_key) {
if (key == null || value == null) {
// nothing to search
return -1
}
return IntStream.range(0, groups.size())
.filter(i -> groups.get(i)['docs']
.stream()
.anyMatch(_doc -> _doc.get(key) == value
|| (backup_key != null
&& _doc.get(backup_key) == value)))
.findFirst()
.orElse(-1);
}
def approval_id = doc['approval_id'].size() != 0
? doc['approval_id'].value
: null;
def request_id = doc['request_id'].size() != 0
? doc['request_id'].value
: null;
def requesting_user = doc['requesting_user.keyword'].size() != 0
? doc['requesting_user.keyword'].value
: null;
def userid = doc['userid.keyword'].size() != 0
? doc['userid.keyword'].value
: null;
HashMap valueMap = ['approval_id':approval_id,
'request_id':request_id,
'requesting_user':requesting_user,
'userid':userid];
def found = false;
for (def entry : valueMap.entrySet()) {
def field = entry.getKey();
def value = entry.getValue();
def backup_key = field == 'userid'
? 'requesting_user'
: field == 'requesting_user'
? 'userid'
: null;
def found_index = fetchIndex(state.groups, field, value, backup_key);
if (found_index != -1) {
state.groups[found_index]['docs'].add(params._source);
if (approval_id != null) {
state.groups[found_index]['approval_id'] = approval_id;
}
found = true;
break;
}
}
if (!found) {
HashMap nextInLine = ['docs': [params._source]];
if (approval_id != null) {
nextInLine['approval_id'] = approval_id;
}
state.groups.add(nextInLine);
}
""",
"combine_script": "return state",
"reduce_script": "return states"
}
}
}
}
returning the grouped events + the inferred approval_id:
"aggregations" : {
"log_groups" : {
"value" : [
{
"groups" : [
{
"docs" : [
{...}, {...}, {...}, {...}, {...}, {...}, {...}
],
"approval_id" : 22
},
{ ... }
]
}
]
}
}
Keep in mind that such scripts are going to be quite slow, esp. when run on large numbers of events.

linq - condtionally hide a property in a .select() statement

[frangment of a linq query]
query.Select(c => new OdsSummaryResponse
{
Id = c.RISODS_Id,
Driver = c.CDGOREC.Where(a => a.CDGOREC_Conducente).FirstOrDefault().PERSONALE.PERSONE.PERSONE_CognomeNome,
Operators = c.CDGOREC.Where(a => myexternavar == true && !a.CDGOREC_Conducente && a.CDGOREC_PersonaleId != null).Select( d => new Operatori() {
Id = (int)d.CDGOREC_PersonaleId,
Nominativo = d.PERSONALE.PERSONE.PERSONE_CognomeNome,
Codice = d.PERSONALE.PERSONALE_Codice
}).ToList()
}).OrderByDescending(c => Id);
in the subquery Operators = c.CDGOREC.Where when an external variable myexternavar=false i'd like to completely hide the property "Operators" from the output.
But this code shows me an empty array,
{
"Id": 194214,
"Driver": "XXXXXXXX",
"OperatorsNo": 1,
"Operators": [] //i'd like to hide this property if myexternavar=false
}
is it possible to exclude the property at all???
You can use Conditional Property Serialization to instruct the Serializer when to include that property.
For example, the following will not emit the Operators property when it's either null or empty:
public class OdsSummaryResponse
{
// ...
public IList<Operatori> Operators { get; set;}
public bool ShouldSerializeOperators()
{
return this.Operators?.Count > 0;
}
}

How to test null value on nullable enum

I have documents with property _report_type, it supposed to contain a string "FTE", or "CAB".
In legacy document, it can be null or not exist at all (undefined), they are supposed to be "ContractorWorkReport".
My mapping:
public class Order : Document, IBaseEntity<string>
{
[JsonProperty(Order = 70, PropertyName = "orderData")]
public OrderData Data { get; set; }
}
public class OrderData
{
[JsonProperty(Order = 60, PropertyName = "_order_type")]
public OrderType? OrderType { get; set; }
}
[JsonConverter(typeof(StringEnumConverter))]
public enum OrderType
{
[EnumMember(Value = "TFE")]
ContractorWorkReport,
[EnumMember(Value = "CAB")]
BudgetElectricMeter
}
My query based on that :
var query = _client.CreateDocumentQuery<T>($"/dbs/{_databaseId}/colls/{CollectionId}");
if (filter.OrderType.Value == OrderType.ContractorWorkReport)
query = query.Where(o => o.Data.OrderType == null || !o.Data.OrderType.HasValue || o.Data.OrderType == filter.OrderType);
else
query = query.Where(o => o.Data.OrderType == filter.OrderType);
This query crashes because of the o.Data.OrderType == null, error message is :
Unable to cast object of type 'Microsoft.Azure.Documents.Sql.SqlNullLiteral' to type 'Microsoft.Azure.Documents.Sql.SqlNumberLiteral'.'
How can fix this? Right now, i do this, but it's dirty...
if (filter.OrderType.Value == OrderType.ContractorWorkReport)
query = query.Where(o => o.Data.OrderType != OrderType.BudgetElectricMeter);
else
query = query.Where(o => o.Data.OrderType == filter.OrderType);
Thanks!
First, I would recommend you use this for the first parameter in CreateDocumentQuery : UriFactory.CreateDocumentCollectionUri(databaseId, collectionId)
To answer your question, you are comparing an int to a null. o.Data.OrderType, being an enum, resolves to an int. You may want to compare that to the default enum value 0 or query over documents where o.Data.OrderType is not a key at all

Why can't I compare two fields in a search predicate in Sitecore 7.5?

I am trying to build a search predicate in code that compares two fields in Sitecore and I am getting a strange error message. Basically I have two date fields on each content item - FirstPublishDate (the date that the content item was first published) and LastPublishDate (the last date that the content item was published). I would like to find all content items where the LastPublishDate falls within a certain date range AND where the LastPublishDate does not equal the FirstPublishDate. Using Linq here is my method for generating the predicate...
protected Expression<Func<T, Boolean>> getDateFacetPredicate<T>() where T : MySearchResultItem
{
var predicate = PredicateBuilder.True<T>();
foreach (var facet in myFacetCategories)
{
var dateTo = System.DateTime.Now;
var dateFrom = dateTo.AddDays(facet.Value*-1);
predicate = predicate.And(i => i.LastPublishDate.Between(dateFrom, dateTo, Inclusion.Both)).And(j => j.LastPublishDate != j.FirstPublishDate);
}
return predicate;
}
Then I use this predicate in my general site search code to perform the search as follows: the above predicate gets passed in to this method as the "additionalWhere" parameter.
public static SearchResults<T> GeneralSearch<T>(string searchText, ISearchIndex index, int currentPage = 0, int pageSize = 20, string language = "", IEnumerable<string> additionalFields = null,
Expression<Func<T, Boolean>> additionalWhere = null, Expression<Func<T, Boolean>> additionalFilter = null, IEnumerable<string> facets = null,
Expression<Func<T, Boolean>> facetFilter = null, string sortField = null, SortDirection sortDirection = SortDirection.Ascending) where T : SearchResultItem {
using (var context = index.CreateSearchContext()) {
var query = context.GetQueryable<T>();
if (!string.IsNullOrWhiteSpace(searchText)) {
var keywordPred = PredicateBuilder.True<T>();
// take into account escaping of special characters and working around Sitecore limitation with Contains and Equals methods
var isSpecialMatch = Regex.IsMatch(searchText, "[" + specialSOLRChars + "]");
if (isSpecialMatch) {
var wildcardText = string.Format("\"*{0}*\"", Regex.Replace(searchText, "([" + specialSOLRChars + "])", #"\$1"));
wildcardText = wildcardText.Replace(" ", "*");
keywordPred = keywordPred.Or(i => i.Content.MatchWildcard(wildcardText)).Or(i => i.Name.MatchWildcard(wildcardText));
}
else {
keywordPred = keywordPred.Or(i => i.Content.Contains(searchText)).Or(i => i.Name.Contains(searchText));
}
if (additionalFields != null && additionalFields.Any()) {
keywordPred = additionalFields.Aggregate(keywordPred, (current, field) => current.Or(i => i[field].Equals(searchText)));
}
//query = query.Where(i => (i.Content.Contains(searchText) || i.Name.Contains(searchText))); // more explicit call to check the content or item name for our term
query = query.Where(keywordPred);
}
if (language == string.Empty) {
language = Sitecore.Context.Language.ToString();
}
if (language != null) {
query = query.Filter(i => i.Language.Equals(language));
}
query = query.Page(currentPage, pageSize);
if (additionalWhere != null) {
query = query.Where(additionalWhere);
}
if (additionalFilter != null) {
query = query.Filter(additionalFilter);
}
query = query.ApplySecurityFilter();
FacetResults resultFacets = null;
if (facets != null && facets.Any()) {
resultFacets = facets.Aggregate(query, (current, fname) => current.FacetOn(i => i[fname])).GetFacets();
}
// calling this before applying facetFilter should allow us to get a total facet set
// instead of just those related to the current result set
// var resultFacets = query.GetFacets();
// apply after getting facets for more complete facet list
if (facetFilter != null) {
query = query.Where(facetFilter);
}
if (sortField != null)
{
if (sortDirection == SortDirection.Ascending)
{
query = query.OrderBy(x => x[sortField]);
}
else
{
query = query.OrderByDescending(x => x[sortField]);
}
}
var results = query.GetResults(); // this enumerates the actual results
return new SearchResults<T>(results.Hits, results.TotalSearchResults, resultFacets);
}
}
When I try this I get the following error message:
Server Error in '/' Application.
No constant node in query node of type: 'Sitecore.ContentSearch.Linq.Nodes.EqualNode'. Left: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'. Right: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.NotSupportedException: No constant node in query node of type: 'Sitecore.ContentSearch.Linq.Nodes.EqualNode'. Left: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'. Right: 'Sitecore.ContentSearch.Linq.Nodes.FieldNode'.
Source Error:
Line 548: FacetResults resultFacets = null;
Line 549: if (facets != null && facets.Any()) {
Line 550: resultFacets = facets.Aggregate(query, (current, fname) => current.FacetOn(i => i[fname])).GetFacets();
Line 551: }
Line 552: // calling this before applying facetFilter should allow us to get a total facet set
From what I can understand about the error message it seems to not like that I am trying to compare two different fields to each other instead of comparing a field to a constant. The other odd thing is that the error seems to be pointing to a line of code that has to do with aggregating facets. I did a Google search and came up with absolutely nothing relating to this error. Any ideas?
Thanks,
Corey
I think what you are trying is not possible, and if you look at this that might indeed be the case. A solution that is given there is to put your logic in the index: create a ComputedField that checks your dates and puts a value in the index that you can search on (can be a simple boolean).
You will need to split your logic though - the query on the date range can still be done in the predicate (as it is relative to the current date) but the comparison of first and last should be done on index time instead of on query time.

How to Search through all fields in a LINQ table?

in LINQ how do i search all fields in a table, what do i put for ANYFIELD in the below?
Thanks
var tblequipments = from d in db.tblEquipments.Include(t => t.User).Include(t => t.ChangeLog).Include(t => t.AssetType)
where d."ANYFIELD" == "VALUE" select d;
You can't. You must compare each field individually. It doesn't make sense to compare all fields, given a field may not even be of the same type as the object you're comparing to.
You can, using reflection. Try this:
static bool CheckAllFields<TInput, TValue>(TInput input, TValue value, bool alsoCheckProperties)
{
Type t = typeof(TInput);
foreach (FieldInfo info in t.GetFields().Where(x => x.FieldType == typeof(TValue)))
{
if (!info.GetValue(input).Equals(value))
{
return false;
}
}
if (alsoCheckProperties)
{
foreach (PropertyInfo info in t.GetProperties().Where(x => x.PropertyType == typeof(TValue)))
{
if (!info.GetValue(input, null).Equals(value))
{
return false;
}
}
}
return true;
}
And your LINQ query:
var tblequipments = from d in db.tblEquipments.Include(t => t.User).Include(t => t.ChangeLog).Include(t => t.AssetType)
where CheckAllFields(d, "VALUE", true) select d;
The third parameter should be true if you want to check all fields and all properties, and false if you want to check only all fields.
EDIT: Someone already built this...see here.
Not a full answer, but I don't agree with assertion that you simply can't...
You could come up with an extension method that dynamically filtered the IQueryable/IEnumerable (I'm guessing IQueryable by the db variable) based on properties of a similar type for you. Here's something whipped up in Linqpad. It references PredicateBuilder and is by no means complete/fully accurate, but I tested it out in Linq-to-SQL on some of my tables and it worked as described.
void Main()
{
YourDbSet.WhereAllPropertiesOfSimilarTypeAreEqual("A String")
.Count()
.Dump();
}
public static class EntityHelperMethods
{
public static IQueryable<TEntity> WhereAllPropertiesOfSimilarTypeAreEqual<TEntity, TProperty>(this IQueryable<TEntity> query, TProperty value)
{
var param = Expression.Parameter(typeof(TEntity));
var predicate = PredicateBuilder.True<TEntity>();
foreach (var fieldName in GetEntityFieldsToCompareTo<TEntity, TProperty>())
{
var predicateToAdd = Expression.Lambda<Func<TEntity, bool>>(
Expression.Equal(
Expression.PropertyOrField(param, fieldName),
Expression.Constant(value)), param);
predicate = predicate.And(predicateToAdd);
}
return query.Where(predicate);
}
// TODO: You'll need to find out what fields are actually ones you would want to compare on.
// This might involve stripping out properties marked with [NotMapped] attributes, for
// for example.
private static IEnumerable<string> GetEntityFieldsToCompareTo<TEntity, TProperty>()
{
Type entityType = typeof(TEntity);
Type propertyType = typeof(TProperty);
var fields = entityType.GetFields()
.Where (f => f.FieldType == propertyType)
.Select (f => f.Name);
var properties = entityType.GetProperties()
.Where (p => p.PropertyType == propertyType)
.Select (p => p.Name);
return fields.Concat(properties);
}
}
Useful resources for the unresolved part:
Finding the relevant properties
if this help some one.
first find all properties within Customer class with same type as query:
var stringProperties = typeof(Customer).GetProperties().Where(prop =>
prop.PropertyType == query.GetType());
then find all customers from context that has at least one property with value equal to query:
context.Customer.Where(customer =>
stringProperties.Any(prop =>
prop.GetValue(customer, null) == query));

Resources