Automatic id generation and mapping _id NEST - elasticsearch

I want the id to be automatically generated when I index a document into elastic search. This works fine when I don't specify an Id property in my poco.
What I would like to do is map the underlying _id field onto my poco class when getting and use auto generated id when indexing. It looks like I can either specify the id or not at all. Are their any nest api options that I am missing?
EDIT
Example gist
https://gist.github.com/antonydenyer/9074159

As #Duc.Duong said in comments you can access Id using DocumentWithMeta.
In current version of NEST DocumentsWithMetaData is replaced with Hits and also you should cast IHit<DataForGet>.Id from string to int.
this is my code:
public class DataForIndex
{
public string Name { get; set; }
// some other fields...
}
public class DataForGet : DataForIndex
{
public int Id { get; set; }
}
var result = client.Search<DataForGet>(x => x.Index("index").MatchAll());
var list = results.Hits.Select(h =>
{
h.Source.Id = Convert.ToInt32(h.Id);
return h.Source;
}).ToList();

Automatic ID Generation
The index operation can be executed without specifying the id. In such a case, an id will be generated automatically. In addition, the op_type will automatically be set to create. Here is an example (note the POST used instead of PUT):
$ curl -XPOST 'http://localhost:9200/twitter/tweet/' -d '{
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}'
Result:
{
"_index" : "twitter",
"_type" : "tweet",
"_id" : "6a8ca01c-7896-48e9-81cc-9f70661fcb32",
"_version" : 1,
"created" : true
}

Seem that NEST auto detect "Id" field in class and map it to "_id" in ES. Your requirement look strange but why not create 2 classes for it ? One for indexing (without Id field), one for getting (inherited from indexing class and declare new field "Id") ?
e.g.:
public class DataForIndex
{
public string Name { get; set; }
// some other fields...
}
public class DataForGet : DataForIndex
{
public int Id { get; set; }
}

Related

Spring Data MongoRepository Saving Objects with Differing Numbers of Fields

I am storing game states in a MongoDB database and am using Spring Data to manage my database interactions. I am new to Spring Data and am unsure how to deal with the following scenario.
I have a document of type "Game" with a bunch of properties such as id, timestamp, etc... One of these properties is a list of actions taken by users. These actions are of the form:
{ type: 2 }, {type: 3, value: 4}, {type: 5, id: 1234}, {type 6}, {type: 5, value: 6, id: 56}
In other words, an action can have three properties: type, value, and id. However, not every action requires all three values to be stored. I want to avoid having a bunch of null values in my database, and would like my database to just not include and id or a value if they are not specified.
Using Spring Data's MongoRepository model, I am not sure how to achieve this. I can create a CRUD Game class and have one of its properties be a list of Action (where Action itself is a CRUD class with properties type, value, and id), but won't that end up storing null values in the database if I do not specify the value or id?
In short, how can I use Spring Data's MongoRepository but still maintain the flexibility of being able to store lists of objects with varying parameters or object types in general.
I will explain how varying fields are handled with an example. The following Game.java POJO class represents the object mapping to the game collection document.
public class Game {
String name;
List<Actions> actions;
public Game(String name, List<Actions> actions) {
this.name = name;
this.actions = actions;
}
public String getName() {
return name;
}
public List<Actions> getActions() {
return actions;
}
// other get/set methods, override, etc..
public static class Actions {
Integer id;
String type;
public Actions() {
}
public Actions(Integer id) {
this.id = id;
}
public Actions(Integer id, String type) {
this.id = id;
this.type = type;
}
public Integer getId() {
return id;
}
public String getType() {
return type;
}
// other methods
}
}
For the Actions class you need to provide constructors with the possible combinations. Use the appropriate constructor with id, type, etc. For example, create a Game object and save to the database:
Game.Actions actions= new Game.Actions(new Integer(1000));
Game g1 = new Game("G-1", Arrays.asList(actions));
repo.save(g1);
This is stored in the database collection game as follows (queried from mongo shell):
{
"_id" : ObjectId("5eeafe2043f875621d1e447b"),
"name" : "G-1",
"actions" : [
{
"_id" : 1000
}
],
"_class" : "com.example.demo.Game"
}
Note the actions array. As we had stored only the id field in the Game.Actions object, only that field is stored. Even though you specify all the fields in the class, only those provided with values are persisted.
These are two more documents with Game.Actions created with type only and id + type using the appropriate constructors:
{
"_id" : ObjectId("5eeb02fe5b86147de7dd7484"),
"name" : "G-9",
"actions" : [
{
"type" : "type-x"
}
],
"_class" : "com.example.demo.Game"
}
{
"_id" : ObjectId("5eeb034d70a4b6360d5398cc"),
"name" : "G-11",
"actions" : [
{
"_id" : 2,
"type" : "type-y"
}
],
"_class" : "com.example.demo.Game"
}

web api : checking odata query

I have a web api exposing a list of sessions. This is my code :
[RoutePrefix("api/data")]
public class SessionController : ApiController
{
[HttpGet]
[Route("sessions")]
[Queryable]
public IQueryable<Session> Get()
{
List<Session> list = new List<Session>();
list.Add(new Session { Id = 1, Name = "name 1", Place = "place 1", SessionOn = Convert.ToDateTime("1/1/2014") });
list.Add(new Session { Id = 2, Name = "name 2", Place = "place 2", SessionOn = Convert.ToDateTime("2/1/2014") });
list.Add(new Session { Id = 3, Name = "name 3", Place = "place 3", SessionOn = Convert.ToDateTime("3/1/2014") });
return list.AsQueryable();
}
}
public class Session
{
public int Id { get; set; }
public string Name { get; set; }
public string Place { get; set; }
public DateTime SessionOn { get; set; }
}
An user can request this api to see all sessions like this :
mydomain/api/data/sessions
I have added the oData to allow user querying and filtering those data like this :
mydomain/api/data/sessions?$filter=Name eq 'name1'
mydomain/api/data/sessions?$filter=Place eq 'place 1'
Everything is working well, the only problem remaining is that I would like to check the query given by the user to tell him that a value is not valid for example :
In my list of sessions, the possible values for the field 'Place' are :
place 1
place 2
place 3
place 4
If the user do the following request :
mydomain/api/data/sessions?$filter=Place eq 'placezzzzz 1'
He will just get an empty sets of Session. What I would like to do, is check in my backend code api the value given (that is to say 'placezzzzz 1') and returns a response to the user telling that this value is invalid.
You could change your method signature to include a ODataQueryOptions<T> parameter:
[EnableQuery]
public IQueryable<Session> Get(ODataQueryOptions<Session> options) {
//Do something fun with the filter:
var filter = options.Filter;
}
The options argument can be added to your Get method without changing the routing. It contains all kinds of info about the request, including the filter that was used. You can still return your IQueryable, just like you are used to.
I use the [System.Web.OData.EnableQuery] attribute instead of [Queryable] because QueryableAttribute is obsolete. (source msdn) I'm not sure this solution will work with the older version.

How to query and return all documents in documentdb using linq

let problemDocument = documentClient.CreateDocumentQuery<ProblemDatabaseModel>("")
problemDocument
doesn't seem to work
(problemDocument.Select(fun problem -> problem))
doesn't seem to work
(problemDocument.Where(fun problem -> problem.id = problem.id))
doesn't seem to work either. Any ideas?
If you want to query all document in document db, please try to below code:
documentClient.CreateDocumentQuery<ProblemDatabaseModel>("").ToList();
Please note that, we can store different json entity in documentDB, if document property does not in your data model, it will give a default value. I have a simple test for this:
Data model:
public class Cred
{
[JsonProperty(PropertyName = "id")]
public string ID { get; set; }
[JsonProperty(PropertyName = "title")]
public string Title { get; set; }
[JsonProperty(PropertyName = "credits")]
public int Credits { get; set; }
[JsonProperty(PropertyName = "authordetails")]
public AuthDetail AuthInfo { get; set; }
}
If json data in documentDB is:
{
"id": "CDC103",
"title": "Fundamentals of database design",
"authordetails": {
"Name": "dave",
"Age": 33
},
"custom":"test"
}
client.CreateDocumentQuery<Cred>(UriFactory.CreateDocumentCollectionUri("jambordb", "jamborcols")).ToList();
Here is the result:
From the screenshot we know that, the property "custom" will not included in our data model. and the credits will give a default value 0.

Dynamic mapping in Nest 1.4

I am working on Nest version 1.4 . Currently I am handling csv files with column names varying in every new insertion. In order to do that I have to chamge my class name every now and then, eg suppose
Suppose there are only three fields in csv file like severity,entity_type and operation then I define my class like this
[ElasticProperty(Name = "severity", Store = true, Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string severity { get; set; }
[ElasticProperty(Name = "entity_type", Store = true, Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string entity_type { get; set; }
[ElasticProperty(Name = "operation", Store = true, Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string operation { get; set; }
But in another csv file i have a new field called "number", then again i need to add this attribute to this class which becomes tedious. Is there something called dynamic mapping in nest version 1.4 that can automatically map the column names from the csv files? Also I tried with MULTI-FIELD MAPPING but it doesnt seem to work. Please suggest.
Thanks
Nilanjana

How can I use nested types with NEST client for Elastic Search

I ran in to some issues whilst trying to use statistical facets on my documents in Elastic Search. This resulted in the following posts on the Elastic Search google group - see https://groups.google.com/forum/#!topic/elasticsearch/wNjrnAC_KOY. I tried to apply the recommendation in the answer about using Nested types with in the document to provide distinct sums on the collections property(see https://groups.google.com/forum/#!topic/elasticsearch/wNjrnAC_KOY)
That is I would have many instances of MyType with a collection of MyItem. Some collections of MyItem will have instances with matching amounts i.e. the first document could have two instances of myitem, both with an amount of 100. Without nested types I don't believe statistical facets will aggregate each amount as they're not unique.
So I've created a document structure (similar to below) and populated my index. Before populating my index I've used the following code in an effort to created a nested document.
client.MapFromAttributes<Page>();
[ElasticType(Name="page", DateDetection = true, NumericDetection = true, SearchAnalyzer = "standard",IndexAnalyzer = "standard")]
public class MyType
{
public int TypeId { get; set; }
public string Name { get; set; }
public ANotherType AnotherProperty { get; set; }
public DateTime Created { get; set; }
[ElasticProperty(Type = FieldType.nested, Name="mycollection")]
public List<MyItem> MyItems { get;
}
public class MyItem
{
public decimal Amount {get;set;}
}
However, when I run the following query via the nest api I don't get any results.
query.Index("pages")
.Type("page")
.From(0)
.Size(100)
.FacetStatistical("TotalAmount", x => x.Nested("donations")
.OnField("amount")));
More over I've also tried the following via the Chrome plugin PostMan :
{
"facets": {
"test": {
"statistical": {
"field": "amount"
},
"nested": "mycollection"
}
},
"size":0
}'
and get a response that notes :
"..facet nested path [mycollection] is not nested.."
Any thoughts on this would be great.
Tim
Try to map you object as followed:
client.MapFluent<MyType>(m=>m
.MapFromAttributes()
.NestedObject<MyItem>(no=>no
.Name(p=>p.MyItems.First())
.Dynamic()
.Enabled()
.IncludeInAll()
.IncludeInParent()
.IncludeInRoot()
.MapFromAttributes()
.Path("full")
.Properties(pprops => pprops
.String(ps => ps
.Name(p => p.FirstName)
.Index(FieldIndexOption.not_analyzed)
)
//etcetera
)
)
);
The client.MapFromAttributes() is very limited and will probably be removed in the 1.0 release. Its great to annotate property names but quickly becomes limitted in what it can express. The MapFromAttributes() in the mapfluent call is still a great way to type int's as int, float's as floats, DateTime's as dates etcetera.

Resources