I am using ElasticSearch and the NEST .Net library for implementing the Search functionality needed in our app. In my model, I have a type that contains Nested objects as per below.
[ElasticType(Name = "x")]
public class X
{
[ElasticProperty(IncludeInAll = false, Index = FieldIndexOption.NotAnalyzed)]
public string Id { get; set; }
[ElasticProperty(Type = FieldType.Nested)]
public List<Y> Ys { get; set; }
}
Any queries executed against X are actually executed against the List of Ys. I would like to highlight the hits in the nested objects and based on https://github.com/elasticsearch/elasticsearch/issues/5245 .
However, in order to use the proposed workaround, the include_in_parent option should be true for the nested object.
How can this option be enabled using the NEST library? Is there any ElasticProperty property (I haven’t found any obvious one) or some other way to do so?
Thank you
Apparently this can be done only by using fluent syntax. For the above case the code would be:
.AddMapping<X>(m => m
.Properties(p => p
.NestedObject<Y>(n => n
.Name("ys")
.IncludeInParent())
Related
I've been trying to update to ES6 and NEST 6 and running into issues with NEST serializing of search requests - specifically serializing Terms queries where the underlying C# type is an enum.
I've got a Status enum mapped in my index as a Keyword, and correctly being stored in its string representation by using NEST.JsonNetSerializer and setting the contract json converter as per Elasticsearch / NEST 6 - storing enums as string
The issue comes when trying to search based on this Status enum. When I try to use a Terms query to specify multiple values, these values are being serialized as integers in the request and causing the search to find no results due to the type mismatch.
Interestingly the enum is serialized correctly as a string in a Term query, so I'm theorizing that the StringEnumConverter is being ignored in a scenario where it's having to serialize a collection of enums rather than a single enum.
Lets show it a little more clearly in code. Here's the enum and the (simplified) model used to define the index:
public enum CampaignStatus
{
Active = 0,
Sold = 1,
Withdrawn = 2
}
public class SalesCampaignSearchModel
{
[Keyword]
public Guid Id { get; set; }
[Keyword(DocValues = true)]
public CampaignStatus CampaignStatus { get; set; }
}
Here's a snippet of constructing the settings for the ElasticClient:
var pool = new SingleNodeConnectionPool(new Uri(nodeUri));
var connectionSettings = new ConnectionSettings(pool, (builtin, serializerSettings) =>
new JsonNetSerializer(builtin,
serializerSettings,
contractJsonConverters: new JsonConverter[]{new StringEnumConverter()}
)
)
.EnableHttpCompression();
Here's the Term query that correctly returns results:
var singleTermFilterQuery = new SearchDescriptor<SalesCampaignSearchModel>()
.Query(x => x.Term(y => y.Field(z => z.CampaignStatus).Value(CampaignStatus.Active)));
Generating the request:
{
"query": {
"term": {
"campaignStatus": {
"value": "Active"
}
}
}
}
Here's the Terms query that does not return results:
var termsFilterQuery = new SearchDescriptor<SalesCampaignSearchModel>()
.Query(x => x.Terms(y => y.Field(z => z.CampaignStatus).Terms(CampaignStatus.Active, CampaignStatus.Sold)));
Generating the request:
{
"query": {
"terms": {
"campaignStatus": [
0,
1
]
}
}
}
So far I've had a pretty good poke around at the options being presented by the JsonNetSerializer, tried a bunch of the available attributes (NEST.StringEnumAttribute, [JsonConverter(typeof(StringEnumConverter))] rather than using the global one on the client, having an explicit filter object with ItemConverterType set on the collection of CampaignStatuses, etc.) and the only thing that has had any success was a very brute-force .ToString() every time I need to query on an enum.
These are toy examples from a reasonably extensive codebase that I'm trying to migrate across to NEST 6, so what I'm wanting is to be able to specify global configuration somewhere rather than multiple developer teams needing to be mindful of this kind of eccentricity.
So yeah... I've been looking at this for a couple of days now. Good chances there's something silly I've missed. Otherwise I'm wondering if I need to be providing some JsonConverter with a contract that would match to an arbitrary collection of enums, and whether NEST and their tweaked Json.NET serializer should just be doing that kind of recursive resolution out of the box already.
Any help would be greatly appreciated, as I'm going a bit crazy with this one.
I have two objects.
[DataContract]
public class Record
{
[DataMember]
public string Id { get; set; }
}
And this class:
public class BatteryStatus : Record
{
[DataMember]
public DateTime RetrieveTime { get; set; }
}
I'm using Lite DB as a local NoSQL option to query and save the data. I'm needing to find and delete the values based after some time. Here's my code doing so:
var col = db.GetCollection<BatteryStatus>(CollectionName);
var test = col.FindAll()
.Where(x => x.Id == status.Id).ToList();
var result = col.Find(Query.EQ("Id", status.Id.ToString())).ToList();
Test returns with the with the object, but the result value doesn't. Lite DB only uses the Query or the BSONId as a way to delete an object. I don't have a BSON id attached to it (it's a referenced definition so I can't change it).
How can I use the "Query" function in order to get a nested value so I can delete it?
Classes has properties, BSON documents has fields. By default, LiteDB convert all property names to same name in BSON document except _id field which is document identifier.
If you want query using Linq, you will use properties expressions. If you are using Query object class, you must use field name.
var result = col.FindById(123);
// or
var result = col.FindOne(x => x.Id == 123);
// or
var result = col.FindOne(Query.EQ("_id", 123));
Find using _id always returns 1 (or zero) document.
I figured out the problem with LiteDB, since I was using the property name of "Id", the BSON interpreted that as the "_id" of the JSON object, and merging their two values. I solve the issue by renaming the "Id" property to something else.
I have this query that hasn't changed since I first got it working:
ISearchResponse<Series> response = await IndexManager.GetClient()
.SearchAsync<Series>(r => r
.Filter(f => f.Term<Role>(t => t.ReleasableTo.First(), Role.Visitor))
.SortDescending(ser => ser.EndDate)
.Size(1));
My IndexManager.GetClient() is simply responsible for setting up my connection to ElasticSearch, and ensuring that the indexes are built properly. The rest of the code gets the most recent article series that is releasable to the general public.
Inside the IndexManager I set up explicit index mapping, and when I did that I got results from my query every time. The code looked like this:
client.Map<Series>(m => m.Dynamic(DynamicMappingOption.Allow)
.DynamicTemplates(t => t
.Add(a => a.Name("releasableTo").Match("*releasableTo").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
.Add(a => a.Name("id").Match("*id").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
.Add(a => a.Name("services").Match("*amPm").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed)))
.Match("*dayOfWeek").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
.Add(a => a.Name("urls").Match("*Url").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
));
While all well and good, doing this for every type we stored wasn't really going to scale well. So I made a conscious decision to use the attributes and map it that way:
// In IndexManager
client.Map<T>(m => m.MapFromAttributes());
// In the type definition
class Series
{
// ....
[DataMember]
[ElasticProperty(Index = FieldIndexOption.NotAnalyzed, Store = true)]
public HashSet<Role> ReleasableTo { get; set; }
// ....
}
As soon as I do this, I no longer get results. When I look at my indexes in Kibana, I see my 'releasableTo' field is not analyzed and it is indexed. However the query I wrote no longer works. If I remove the filter clause I get results, but I really need that to work.
What am I missing? How do I get my query to work again?
It appears that the ElasticSearch attributes to provide indexing hints don't know what to do with enums.
The problem turned out to be the fact that the Role type was an enumeration. The client.Map<Series>(m => m.MapFromAttributes()) call skipped that property. At run time, it dynamically maps the property to a string.
// In the type definition
class Series
{
// ....
[DataMember]
[ElasticProperty(Index = FieldIndexOption.NotAnalyzed, Store = true)]
public HashSet<Role> ReleasableTo { get; set; }
// ....
}
To get the field properly indexed I had to explicitly set it's type in the ElasticProperty attribute. Changing the code to this:
// In the type definition
class Series
{
// ....
[DataMember]
[ElasticProperty(Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String, Store = true)]
public HashSet<Role> ReleasableTo { get; set; }
// ....
}
made my query work again. The moral of the story is that unless it's a primitive type, be explicit when setting the field type.
I know there is a way to use Expressions and Lambdas to accomplish this but I having a hard time piecing it all together. All I need is a method that will dynamically query an Entity Framework DBSet object to find the row where the propery with the given name matches the value.
My context:
public class MyContext : DbContext
{
public IDbSet<Account> Accoounts{ get { return Set<Account>(); } }
}
The method that I'm looking to write:
public T Get<T>(string property, object value) : where T is Account
{...}
I would rather not have to use Dynamic SQL to accomplish this so no need to suggest it because I already know it's possible. What I'm really looking for is some help to accomplish this using Expressions and Lambdas
Thanks in advance, I know it's brief but it should be pretty self-explanatory. Comment if more info is needed
I'm trying to avoid dynamic linq as much as possible because the main point of linq is strongly typed access. Using dynamic linq is a solution but it is exactly the oppose of the linq purpose and it is quite close to using ESQL and building the query from sting concatenation. Anyway dynamic linq is sometimes real time saver (especially when it comes to complex dynamic ordering) and I successfully use it in a large project with Linq-to-Sql.
What I usually do is defining some SearchCriteria class like:
public class SearchCriteria
{
public string Property1 { get; set; }
public int? Property2 { get; set; }
}
And helper query extension method like:
public static IQueryable<SomeClass> Filter(this IQueryable<SomeClass> query, SearchCriteria filter)
{
if (filter.Property1 != null) query = query.Where(s => s.Property1 == filter.Property1);
if (filter.Property2 != null) query = query.Where(s => s.Property2 == filter.Property2);
return query;
}
It is not generic solution. Again generic solution is for some strongly typed processing of classes sharing some behavior.
The more complex solution would be using predicate builder and build expression tree yourselves but again building expression tree is only more complex way to build ESQL query by concatenating strings.
Here's my implementation:
public T Get<T>(string property, object value) : where T is Account
{
//p
var p = Expression.Parameter(typeof(T));
//p.Property
var propertyExpression = Expression.Property(p, property);
//p.Property == value
var equalsExpression = Expression.Equal(propertyExpression, Expression.Constant(value));
//p => p.Property == value
var lambda = Expression.Lambda<Func<T,bool>>(equalsExpression, p);
return context.Set<T>().SingleOrDefault(lambda);
}
It uses EF 5's Set<T>() method. If you are using a lower version, you'll need to implement a way of getting the DbSet based on the <T> type.
Hope it helps.
Dynamic Linq may be an option. Specify your criteria as a string and it will get built as an expression and ran against your data;
An example from something I have done;
var context = new DataContext(ConfigurationManager.ConnectionStrings["c"].ConnectionString);
var statusConditions = "Status = 1";
var results = (IQueryable)context.Contacts.Where(statusConditions);
http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx
This may be a really elementry question but whats a nice way to include multiple children entities when writing a query that spans THREE levels (or more)?
i.e. I have 4 tables: Company, Employee, Employee_Car and Employee_Country
Company has a 1:m relationship with Employee.
Employee has a 1:m relationship with both Employee_Car and Employee_Country.
If i want to write a query that returns the data from all 4 the tables, I am currently writing:
Company company = context.Companies
.Include("Employee.Employee_Car")
.Include("Employee.Employee_Country")
.FirstOrDefault(c => c.Id == companyID);
There has to be a more elegant way! This is long winded and generates horrendous SQL
I am using EF4 with VS 2010
Use extension methods.
Replace NameOfContext with the name of your object context.
public static class Extensions{
public static IQueryable<Company> CompleteCompanies(this NameOfContext context){
return context.Companies
.Include("Employee.Employee_Car")
.Include("Employee.Employee_Country") ;
}
public static Company CompanyById(this NameOfContext context, int companyID){
return context.Companies
.Include("Employee.Employee_Car")
.Include("Employee.Employee_Country")
.FirstOrDefault(c => c.Id == companyID) ;
}
}
Then your code becomes
Company company =
context.CompleteCompanies().FirstOrDefault(c => c.Id == companyID);
//or if you want even more
Company company =
context.CompanyById(companyID);
EF Core
For eager loading relationships more than one navigation away (e.g. grand child or grand parent relations), where the intermediate relation is a collection (i.e. 1 to many with the original 'subject'), EF Core has a new extension method, .ThenInclude(), and the syntax is slightly different to the older EF 4-6 syntax:
using Microsoft.EntityFrameworkCore;
...
var company = context.Companies
.Include(co => co.Employees)
.ThenInclude(emp => emp.Employee_Car)
.Include(co => co.Employees)
.ThenInclude(emp => emp.Employee_Country)
With some notes
As per above (Employees.Employee_Car and Employees.Employee_Country), if you need to include 2 or more child properties of an intermediate child collection, you'll need to repeat the .Include navigation for the collection for each child of the collection.
Personally, I would keep the extra 'indent' in the .ThenInclude to preserve your sanity.
For serialization of intermediaries which are 1:1 (or N:1) with the original subject, the dot syntax is also supported, e.g.
var company = context.Companies
.Include(co => co.City.Country);
This is functionally equivalent to:
var company = context.Companies
.Include(co => co.City)
.ThenInclude(ci => ci.Country);
However, in EFCore, the old EF4 / 6 syntax of using 'Select' to chain through an intermediary which is 1:N with the subject is not supported, i.e.
var company = context.Companies
.Include(co => co.Employee.Select(emp => emp.Address));
Will typically result in obscure errors like
Serialization and deserialization of 'System.IntPtr' instances are not supported
EF 4.1 to EF 6
There is a strongly typed .Include which allows the required depth of eager loading to be specified by providing Select expressions to the appropriate depth:
using System.Data.Entity; // NB!
var company = context.Companies
.Include(co => co.Employees.Select(emp => emp.Employee_Car))
.Include(co => co.Employees.Select(emp => emp.Employee_Country))
.FirstOrDefault(co => co.companyID == companyID);
The Sql generated is by no means intuitive, but seems performant enough. I've put a small example on GitHub here
You might find this article of interest which is available at codeplex.com.
Improving Entity Framework Query Performance Using Graph-Based Querying.
The article presents a new way of expressing queries that span multiple tables in the form of declarative graph shapes.
Moreover, the article contains a thorough performance comparison of this new approach with EF queries. This analysis shows that GBQ quickly outperforms EF queries.
How do you construct a LINQ to Entities query to load child objects directly, instead of calling a Reference property or Load()
There is no other way - except implementing lazy loading.
Or manual loading....
myobj = context.MyObjects.First();
myobj.ChildA.Load();
myobj.ChildB.Load();
...
Might be it will help someone, 4 level and 2 child's on each level
Library.Include(a => a.Library.Select(b => b.Library.Select(c => c.Library)))
.Include(d=>d.Book.)
.Include(g => g.Library.Select(h=>g.Book))
.Include(j => j.Library.Select(k => k.Library.Select(l=>l.Book)))
To doing this:
namespace Application.Test
{
using Utils.Extensions;
public class Test
{
public DbSet<User> Users { get; set; }
public DbSet<Room> Rooms { get; set; }
public DbSet<Post> Posts { get; set; }
public DbSet<Comment> Comments { get; set; }
public void Foo()
{
DB.Users.Include(x => x.Posts, x => x.Rooms, x => x.Members);
//OR
DB.Users.Include(x => x.Posts, x => x.Rooms, x => x.Members)
.ThenInclude(x => x.Posts, y => y.Owner, y => y.Comments);
}
}
}
this extension might be helpful:
namespace Utils.Extensions
{
using Microsoft.EntityFrameworkCore;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
public static partial class LinqExtension
{
public static IQueryable<TEntity> Include<TEntity>(
this IQueryable<TEntity> sources,
params Expression<Func<TEntity, object>>[] properties)
where TEntity : class
{
System.Text.RegularExpressions.Regex regex = new(#"^\w+[.]");
IQueryable<TEntity> _sources = sources;
foreach (var property in properties)
_sources = _sources.Include($"{regex.Replace(property.Body.ToString(), "")}");
return _sources;
}
public static IQueryable<TEntity> ThenInclude<TEntity, TProperty>(
this IQueryable<TEntity> sources,
Expression<Func<TEntity, IEnumerable<TProperty>>> predicate,
params Expression<Func<TProperty, object>>[] properties)
where TEntity : class
{
System.Text.RegularExpressions.Regex regex = new(#"^\w+[.]");
IQueryable<TEntity> _sources = sources;
foreach (var property in properties)
_sources = _sources.Include($"{regex.Replace(predicate.Body.ToString(), "")}.{regex.Replace(property.Body.ToString(), "")}");
return _sources;
}
}
}