Let's assume folowing class which I want to index:
private class User
{
public User()
{
Id = Guid.NewGuid();
Added = DateTime.Now;
}
public Guid Id { get; protected set; }
public string LastName { get; set; }
public DateTime Added { get; protected set; } // Unimportant for search
}
The point is that I need just the Id and LastName property in the index. Using the fluent api everything works fine (just the specified properties will be mapped):
_client.Map<User>(m => m.
Index("nest_test").
Properties(p => p.String(s => s.Name(u => u.LastName))));
Now, when I index an object the mapping will be extended by the remaining properties. How can I avoid these behavior. (Also odd for me: MapFromAttributes() maps all of the properties, while not even one property is decorated?!).
This is a very little example but some of my domain objects stand in many-many-relations. I didn't try it but I don't think it's possible to map these objects when everything will be committed.
Copied from the answer in https://github.com/elastic/elasticsearch-net/issues/1278:
This is due to the dynamic mapping behavior of ES when it detects new fields. You can turn this behavior off by setting dynamic: false or ignore in your mapping:
client.Map<Foo>(m => m
.Dynamic(DynamicMappingOption.Ignore)
...
);
client.Map<Foo>(m => m
.Dynamic(false)
...
);
Keep in mind though, the property will still be present in _source.
Alternatively, you can use the fluent ignore property API mentioned above, which will exclude the property entirely from the mapping and _source since doing so will cause it to not undergo serialization:
var settings = new ConnectionSettings()
.MapPropertiesFor<Foo>(m => m
.Ignore(v => v.Bar)
);
var client = new ElasticClient(settings);
or less ideal, just stick Json.NET's [JsonIgnore] attribute on the properties you want to exclude.
Related
I'm upgrading my Elasticsearch project from v6.6 to v7.6.2. I use Elasticsearch.NET and NEST in order to create my index including mappings, settings, and to ingest my data into Elasticsearch from my SQL db.
Everything works well in v6.6, but when I upgraded to v7.6.2, it no longer accepts my custom mappings and settings. I'm referring to things like my nested objects, my custom analyzers, etc. The data does get ingested, but it defaults to the, well, default mapping (where mostly everything is a keyword or simple data type).
This type of behavior normally happens when there's something syntactically wrong in your mappings or in your POCO. This isn't my case, I don't think.
Is there some breaking change in v7.x that I may have missed? I've been through the documentation fairly extensively.
To give an example of what my mapping should look like, here's an excerpt (from my v6.6 cluster).
...notice things like the 'products' object is of type: nested...
[{"searchdata":{"_all":{"enabled":false},"properties":{"groupid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"products":{"type":"nested","properties":{"adddate":{"type":"date"},"additionaltitles":{"type":"text"},"additionaltitleslist":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"adult":{"type":"integer"},"artists":{"properties":{"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","norms":false,"fields":{"ci":{"type":"text","norms":false,"analyzer":"caseInsensitive"},"nc":{"type":"text","norms":false,"analyzer":"titleNoCharAnalyzer"},"raw":
and here's what it shows as the default version in v7.6.2...
...notice things like the 'products' object not nested, and lots of 'keyword' types, no custom analyzers, etc. ...
[{"_doc":{"properties":{"groupid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"products":{"properties":{"adult":{"type":"long"},"artists":{"properties":{"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"nameidsplit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"nameremarticle":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"catalognum":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"costprice":{"type":"float"},"cover":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"credits":{"properties":{"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"nameidsplit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"nameremarticle":{"type":"text","fields":
As you may be able to notice, the v6.6 version is much more detailed and contains my nested objects, etc. But my v7.6.2 version contains the basic, default structure.
Something must be going wrong here but I don't know what.
Is there some debug logging level that I can enable that will tell me why it's "failing"?
UPDATE
So I'm putting a slightly abbreviated version of how I'm mapping my objects and settings here. Hopefully this helps to understand this issue.
var indexCreate = client.Indices.Create(indexName, nc => nc
.Settings(st => st
.RefreshInterval(60) //slowing the refresh interval so we can get a running count
.NumberOfReplicas(0) //must be set to refresh after created
.NumberOfShards(numOfShards)
.Analysis(a => a
.Analyzers(an => an
.UserDefined("fullTermCaseInsensitive", FullTermCaseInsensitive)
.UserDefined("fullTerm", FullTerm)
.UserDefined("caseInsensitive", CaseInsensitive)
)
.TokenFilters(tf => tf
.UserDefined("syn", Syn)
.UserDefined("myStopFilter", MyStopFilter)
.UserDefined("wordDelimiter", WordDelimiter)
)
)
)
.Map<Store24>(m => m
.Dynamic(false)
.AutoMap()
.Properties(props =>
{
SetPutMappingStore24(props); //see function below
return props;
})
));
private static void SetPutMappingStore24(PropertiesDescriptor<Store24> pm)
{
pm.Nested<Product>(x => x
.AutoMap()
.Name(nm => nm.Products)
.Properties(pr => pr
.Nested<StorePriceSplit>(sp => sp
.Name("storeprice").AutoMap()
)
.Properties(props =>
{
SetPutMappingDescriptorTiWo(props, "tracks");
SetPutMappingDescriptorTiWo(props, "title");
}
)
);
//...... more fluent mappings here
}
[ElasticsearchType(IdProperty = "Groupid")]
public class Store24
{
/// <summary>
/// Identifiers
/// </summary>
[Key]
[Keyword]
public string Groupid { get; set; }
[JsonIgnore]
public string Upc { get; set; }
[JsonIgnore]
public string Titleremarticle { get; set; }
[Nested]
[PropertyName("products")]
public IEnumerable<Product> Products { get; set; }
//... more properties here
}
[ElasticsearchType(RelationName = "product")]
public class Product
{
[Key]
[Text(Analyzer = "fullTermCaseInsensitive")]
public string Upc { get; set; }
[Text(Analyzer = "fullTermCaseInsensitive", Fielddata = true)]
public string Titleremarticle { get; set; }
//...more properties here
}
I use two objects, one called Store24 which maps from the data returned from SQL and one called Product which creates the Elasticsearch mapping.
I figured out my own issue. It seems the creating of the index was throwing an error but I wasn't seeing it. It was being 'hidden' by the response to the Indices.Create() returned object. (the var indexCreate object in my sample above). Once I poked at it, the following error was being thrown: "Token filter [wordDelimiter] cannot be used to parse synonyms".
This was an issue in my settings - I guess a breaking change for 7.x that I didn't see. A rather complex issue actually.
I asked a question a few days ago regarding a net core game I'm making that is using Entity Framework.
I had one issue where a controller was returning duplicate JSON data.
Based on one of the answers, I changed that controller to this:
[HttpGet("GetDungeonAndRoomData/{dungeonId}")]
public async Task<ActionResult<GameDungeon>> GetDungeonAndRoomData(Guid dungeonID)
{
{
var dungeon = await _context.DungeonList
.Select(c => new GameDungeon
{
DungeonId = c.DungeonId,
DungeonName = c.DungeonName,
StartRoom = c.StartRoom,
Rooms = c.Rooms.Select(n => new GameDungeonRoom
{
RoomId = n.RoomId,
RoomText = n.RoomText,
TreasureId = n.TreasureId
})
}).SingleOrDefaultAsync(c => c.DungeonId == dungeonID);
Since I changed the controller, I had to modify this model class, so I added a new property called Rooms.
public partial class GameDungeon
{
[Key]
public string DungeonId { get; set; }
public string DungeonName { get; set; }
public string StartRoom { get; set; }
public IEnumerable<GameDungeonRoom> Rooms { get; set; }
}
Since I added that new "Rooms" property, I had to create a new model called GameDungeonRoom:
public partial class GameDungeonRoom
{
public Guid DungeonId { get; set; }
[Key]
public string RoomId { get; set; }
public string RoomText { get; set; }
public string TreasureId { get; set; }
}
Building and running the game, I now get one set of dungeon data, but it is not returning any rooms.
At first, and based off this Stack Overflow question, .net core : incomplete JSON response,I thought it was because I needed to add this to my Startup.cs file:
services.AddMvc()
.AddJsonOptions(
options => options.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore
);
But that was not the case.
So I then spent the evening trying a bunch of different ways of writing the controller, but they either produced the same results or just threw an error.
Reviewing the code this morning, I realized something. In my controller, the first select statement that creates the new "GameDungeon" is getting data from _context.DungeonList.
DungeonList is a model generated by Entity Framework from a real table in my database.
But GameDungeonRoom is just a new class model I created. It's based off a table in my database called RoomList, but there is nothing in _context that specifically handles GameDungeonRoom.
So I am wondering, should I introduce another controller that kind of looks like this?
var rooms = await _context.RoomList
.Select(c => new GameDungeonRoom ...
And then somehow append that to the GameDungeon object.
I sort of tried that after lunch but ended up just creating a mess of code that created an even bigger mess of errors so I just deleted it all.
Anyway, after all that, this is where my JSON currently stands:
{
"dungeonId" : "293hf938",
"dungeonName" : "Dungeon of Dread",
"startRoom" : "bjgh39811ffr",
"roomId" : "fgf4h635j",
"roomText" : "A big empty room",
"treasureId" : "12a",
"rooms": [
You'll notice that "rooms" is empty. I'm not quite sure why it is, or what's going on.
One idea I had, is maybe I should just create an API controller that get's the dungeon data for a particular dungeon. Then create another API controller that gets the Room data for a particular dungeon.
Then let the client call both controllers(using the same dungeonId) and combine the data on the client side.
So I was wondering if anyone could think of an idea as to why the "rooms" object is empty.
Thanks!
Just guessing you might have hit a cyclic reference in your result set due to Data Context being cached. Hence Json serializer cannot serialize it properly and give incomplete json content. So can you try following to pin point that.
var dungeon = await _context.DungeonList
.Select(c => new GameDungeon
{
DungeonId = c.DungeonId,
DungeonName = c.DungeonName,
StartRoom = c.StartRoom,
Rooms = c.Rooms.Select(n => new GameDungeonRoom
{
RoomId = n.RoomId,
RoomText = n.RoomText,
TreasureId = n.TreasureId
})
})
.AsNoTracking() //This ignore the cached data
.SingleOrDefaultAsync(c => c.DungeonId == dungeonID);
We have a lot of Dto classes in our project and on various occasions SELECT them using Expressions from the entity framework context. This has the benefit, that EF can parse our request, and build a nice SQL statement out of it.
Unfortunatly, this has led to very big Expressions, because we have no way of combining them.
So if you have a class DtoA with 3 properties, and one of them is of class DtoB with 5 properties, and again one of those is of class DtoC with 10 properties, you would have to write one big selector.
public static Expression<Func<ClassA, DtoA>> ToDto =
from => new DtoA
{
Id = from.Id,
Name = from.Name,
Size = from.Size,
MyB = new DtoB
{
Id = from.MyB.Id,
...
MyCList = from.MyCList.Select(myC => new DtoC
{
Id = myC.Id,
...
}
}
};
Also, they cannot be reused. When you have DtoD, which also has a propertiy of class DtoB, you would have to paste in the desired code of DtoB and DtoC again.
public static Expression<Func<ClassD, DtoD>> ToDto =
from => new DtoD
{
Id = from.Id,
Length = from.Length,
MyB = new DtoB
{
Id = from.MyB.Id,
...
MyCList = from.MyCList.Select(myC => new DtoC
{
Id = myC.Id,
...
}
}
};
So this will escalate pretty fast. Please note that the mentioned code is just an example, but you get the idea.
I would like to define an expression for each class and then combine them as required, as well as EF still be able to parse it and generate the SQL statement so to not lose the performance improvement.
How can i achieve this?
Have you thought about using Automapper ? You can define your Dtos and create a mapping between the original entity and the Dto and/or vice versa, and using the projection, you don't need any select statements as Automapper will do it for you automatically and it will project only the dto's properties into SQL query.
for example, if you have a Person table with the following structure:
public class Person
{
public int Id { get; set; }
public string Title { get; set; }
public string FamilyName { get; set; }
public string GivenName { get; set; }
public string Initial { get; set; }
public string PreferredName { get; set; }
public string FormerTitle { get; set; }
public string FormerFamilyName { get; set; }
public string FormerGivenName { get; set; }
}
and your dto was like this :
public class PersonDto
{
public int Id { get; set; }
public string Title { get; set; }
public string FamilyName { get; set; }
public string GivenName { get; set; }
}
You can create a mapping between Person and PersonDto like this
Mapper.CreateMap<Person, PersonDto>()
and when you query the database using Entity Framework (for example), you can use something like this to get PersonDto columns only:
ctx.People.Where(p=> p.FamilyName.Contains("John"))
.Project()
.To<PersonDto>()
.ToList();
which will return a list of PersonDtos that has a family name contains "John", and if you run a sql profiler for example you will see that only the PersonDto columns were selected.
Automapper also supports hierachy, if your Person for example has an Address linked to it that you want to return AddressDto for it.
I think it worth to have a look and check it, it cleans a lot of the mess that manual mapping requires.
I thought about it a little, and I didn't come up with any "awesome" solution.
Essentially you have two general choices here,
Use placeholder and rewrite expression tree entirely.
Something like this,
public static Expression<Func<ClassA, DtoA>> DtoExpression{
get{
Expression<Func<ClassA, DtoA>> dtoExpression = classA => new DtoA(){
BDto = Magic.Swap(ClassB.DtoExpression),
};
// todo; here you have access to dtoExpression,
// you need to use expression transformers
// in order to find & replace the Magic.Swap(..) call with the
// actual Expression code(NewExpression),
// Rewriting the expression tree is no easy task,
// but EF will be able to understand it this way.
// the code will be quite tricky, but can be solved
// within ~50-100 lines of code, I expect.
// For that, see ExpressionVisitor.
// As ExpressionVisitor detects the usage of Magic.Swap,
// it has to check the actual expression(ClassB.DtoExpression),
// and rebuild it as MemberInitExpression & NewExpression,
// and the bindings have to be mapped to correct places.
return Magic.Rebuild(dtoExpression);
}
The other way is to start using only Expression class(ditching the LINQ). This way you can write the queries from zero, and reusability will be nice, however, things get harder & you lose type safety. Microsoft has nice reference about dynamic expressions. If you structure everything that way, you can reuse a lot of the functionality. Eg, you define NewExpression and then you can later reuse it, if needed.
The third way is to basically use lambda syntax: .Where, .Select etc.. This gives you definitely better "reusability" rate. It doesn't solve your problem 100%, but it can help you to compose queries a bit better. For example: from.MyCList.Select(dtoCSelector)
I have a web api 2 application, and in my controller , I have this code :
[Queryable]
public IQueryable<Title> GetTitles()
{
return db.Titles;
}
and here is the Title entity :
public partial class Title
{
public Title()
{
this.People = new HashSet<Person>();
}
public short Id { get; set; }
public string NameT { get; set; }
public virtual ICollection<Person> People { get; set; }
}
When people query the Titles, they must get only "NameT" property. but now they get all of the properties. and yes, I know about $select, but I want another way. means even they use $select, they should not able to get "Id" property for example. if I have to bring more information, please tell me. thanks.
There are two ways to solve your problem when you use ODataController. However, they won't affect ApiController non-query part.
In that condition, you can try what Zoe suggested.
1.Ignore those properties while building your model with model builder.
builder.EntityType<Title>().Ignore(title => title.Id);
2.Add ignore member attributes on those properties.
[IgnoreDataMember]
public short Id { get; set; }
More than these, we provide support for limiting the set of allowed queries in Web API 2.2 for OData v4.0.
http://blogs.msdn.com/b/webdev/archive/2014/03/13/getting-started-with-asp-net-web-api-2-2-for-odata-v4-0.aspx
We can use attributes like Unsortable, NonFilterable, NotExpandable or NotNavigable on the properties of the types in our model, or we can configure this explicitly in the model builder.
Maybe you can have filter in your action GetTitles(), like:
[Queryable]
public IQueryable<Title> GetTitles()
{
return db.Titles.Select(t=>t.Name);
}
Use the ODataModelBuilder class as opposed to the ODataConventionModelBuilder class.
var modelBuilder = new ODataModelBuilder();
var titles = modelBuilder.EntitySet<Title>("titles");
var title = titles.EntityType;
title.HasKey(x => x.Id);
title.Ignore(x => x.Id);
title.Property(x => x.TName);
titles.HasIdLink(x => { return x.GenerateSelfLink(true); }, true);
config.Routes.MapODataRoute("odata", "odata", modelBuilder.GetEdmModel());
I'm messing around with LINQ for the first time, and I'm using EF 4.1 code first.
I have entities containing nested Lists of other entities, for example:
class Release
{
int ReleaseID { get; set; }
string Title { get; set; }
ICollection<OriginalTrack> OriginalTracks { get; set; }
}
class OriginalTrack
{
int OriginalTrackID { get; set; }
string Title { get; set; }
ICollection<Release> Releases { get; set; }
ICollection<OriginalArtist> OriginalArtists { get; set; }
}
class OriginalArtist
{
int OriginalArtistID { get; set; }
string Name { get; set; }
ICollection<OriginalTrack> OriginalTracks { get; set; }
}
I'm wondering what is the quickest way, in one LINQ query, to obtain all the information for where ReleaseID == some value.
I've done my homework, but have found solutions that require implicit rebuilding of an object (usually anonymous) with the required data. I want the data out of the database in the exact format that it is held within the database, i.e. pulling a Release object with relevant ReleaseID pulls and populates all the OriginalTrack and OriginalArtist data in the Lists.
I know about Include(), but am not sure how to apply it for multiple entities.
All help greatly appreciated.
Use Include. This is the purpose of Include, and there's no reason to write a bunch of nested select statements.
context.Releases.Include("OriginalTracks.OriginalArtist")
.Where(release => release.ReleaseID == id);
This is simpler to write, simpler to read, and preserves your existing data structure.
To use Include you need to specify the name of the property you want to return - this means the name as it exists in your code, not in the database. For example:
.Include("OriginalTracks") will include the OriginalTracks property on each Release
.Include("OriginalTracks.OriginalArtist") will include OriginalTracks property on each Release, and the OriginalArtist on each Track (note that it's not possible - syntactically or logically - to include an OriginalArtist within including the OriginalTrack)
.Include("OriginalTracks").Include("OtherProperty") will include the OriginalTracks and OtherProperty objects on each Release.
You can chain as many of these as you like, for example:
.Include("Tracks.Artist").Include("AnotherProperty")
.Include("ThirdProperty.SomeItems").Where(r => r.something);
is perfectly valid. The only requirement is that you put the Include on the EntitySet, not on a query - you can't .Where().Include().
Don't worry about using include here
just do something like the following
var query =
from release in ctx.Releases
select new {
release,
originalTracks = from track in release.OriginalTracks
select new {
track,
releases = track.Releases,
orignialArtist = from artist in track.OriginalArtists
select new {
artist,
artist.OriginalTracks
}
}
}
var Releases = query.Select(x => x.Release);
Should load all of your data
I worked with information from this post here.
http://blogs.msdn.com/b/alexj/archive/2009/10/13/tip-37-how-to-do-a-conditional-include.aspx
To include the nested entities without using string literals, use Select, like this:
context.Releases.Include(r => r.OriginalTracks.Select(t => t.OriginalArtist))
.Where(release => release.ReleaseID == id);