Is there a way to deserialize properties which have special charaters in names with ElasticLowLevelClient.SearchAsync? - elasticsearch-net

I have logstash pushing logs into elasticsearch which automatically inserts the #timestamp field into each doc. I used Elasticsearch.Net instead of NEST in this project.
Each doc looks something like:
{
"Origin": 0,
"VisitorId": "49f843eb-be38-4912-a3b8-277841cd709d",
"CompanyId": 1,
"ActivityType": 21,
"Referer": "http://cn.bing.com/search?q=sogou&src=IE-SearchBox&FORM=IE10SR",
"ProductId": 1,
"version": "1.0",
"Url": "http://localhost:8001/Product/Detail/1",
"DeviceType": 1,
"#timestamp": "2017-03-21T02:04:13.136Z",
"UserId": 4,
"host": "ZBL-PC",
"#version": "1",
"IpAddress": "14.197.145.136",
"timestamp": "2017-03-21T02:04:13.136Z"
}
Currently, my search query looks like:
var res = client.SearchAsync<SearchResult<ProductVisit>>(index: _indexName,
type: "viewsingleproduct",
body: new PostData<object>(new
{
size = 10
}));
public class ProductVisit
{
public int? UserId { get; set; }
public Guid? VisitorId { get; set; }
public int DeviceType { get; set; }
public string timestamp { get; set; }
public int CompanyId { get; set; }
public long ProductId { get; set; }
}
public class Hit<TDocument>
{
public TDocument _source { get; set; }
}
public class Hits<TDocument>
{
public int total { get; set; }
public List<Hit<TDocument>> hits { get; set; }
}
public class SearchResult<TDocument>
{
public Hits<TDocument> hits { get; set; }
}
The Elasticsearch.Net way of retrieving docs is to declare a DTO which can be automatically deserialized into from the json in the response. However, in this case, I cannot declare a property that is a valid C# property name such that #timestamp values can be filled into.
In Nest which uses Newtonsoft.json, I can specify the actual field name in elasticsearch by attribute annotations (I remember JsonProperty will do the trick and also NEST has its own attribute that lets you specify the property name). However, Elasticsearch.Net doesn't use Newtonsoft.json internally. It uses SimpleJson which could've supported similar tricks via DataMember attribute but it requires SIMPLE_JSON_DATACONTRACT preprocessor switch to be enabled and Elasticsearch.Net disabled it by default.
Currently my solution is to copy #timestamp field into timestamp in logstash config, as shown in the json doc above. But it is a hack.
I originally had problems in the request side where official examples mostly construct queries using anonymous types like this:
new
{
term = new
{
_TopicPostId = new {
value = 4
}
}
},
But occasionally I need to construct queries with field names containing special characters. I finally figured out I could use Dictionary instead of anonymous types and it worked. For example:
new {
range = new Dictionary<string, object>{{
"#timestamp",
new
{
gte = "now-1d/d",
lte = "now/d"
}
}}
},

Related

How can I fix null object being sent from PUT json in Postman?

I am PUTing json in the body from postman in hopes of updating an entity called MusicListListItem.
When I debug, I am finding that the MusicListListItem item is null.
Here is my method on on my MusicListsController:
[HttpPut("{id}")]
[Route("API/[Controller]/UpdateMusicListListItem")]
public async Task<IActionResult> PutMusicListListItem(int id, [FromBody]MusicListListItem item)
{
MusicListListItem myMLLI = item;
if (id != item.MusicListListItemId)
{
return BadRequest();
}
_context.Entry(item).State = EntityState.Modified;
await _context.SaveChangesAsync();
return NoContent();
}
and here is the json i put into Postman's Body section with JSON(application/json) selected:
{
"musicListListItemId": 4,
"musicListId": 1,
"listItemId": 1019,
"listItem": {
"artist": null,
"listItemId": 1019,
"title": "test33_tryAddRelation",
"dateCreated": "0001-01-01T00:00:00",
"musicListListItem": []
},
"rank": 2,
"comments": null,
"imageUrl": null,
"dateModified": "0001-01-01T00:00:00",
"deleted": false
}
I expect the API Controller's PutMusicListListItem method to receive the MusicListListItem object from the body of Postman's PUT request, but it comes back as null.
I eventually get a
NullReferenceException: Object reference not set to an instance of an object.
error.
Here is the additional information that was requested:
The musicListListItem object definition:
using MusicListsApp.Domain;
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
using System.Text;
namespace Domain
{
public class MusicListListItem
{
public int MusicListListItemId { get; set; }
public int MusicListId { get; set; }
public int ListItemId { get; set; }
public virtual MusicList MusicList { get; set; }
public virtual ListItem ListItem { get; set; }
public int Rank { get; set; }
public string Comments { get; set; }
public string ImageUrl { get; set; }
public DateTime DateModified { get; set; }
public bool Deleted { get; set; }
}
}
A screenshot of Postman:
I was able to successfully PUT to the API by using a simpler MusicListListItem json object that I obtained by creating a GETMusicListListItemById method:
{
"musicListListItemId": 4,
"musicListId": 1,
"listItemId": 1019,
"musicList": null,
"listItem": null,
"rank": 2,
"comments": null,
"imageUrl": null,
"dateModified": "0001-01-01T00:00:00",
"deleted": false
}

file.filename is returned null in NEST elastic search query

I want to search on content field and return content and file name. The query below is taken from NEST github page
Connection string:
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
var client = new ElasticClient(settings);
My class:
The class of search type is below (I feel problem might be here):
public class myclass
{
public string Content { get; set; }
public string filename { get; set; }
}
So I need only content and filename which is in file.filename, but in my search it return null for file.filename but content do return in same query.
NEST API CALL:
var request = new SearchRequest
{
From = 0,
Size = 10,
Query = new TermQuery { Name="Web", Field = "content", Value = "findthis" }
};
var response = client.Search<myclass>(request);
var twet = response.Documents.Select(t=>t.Content).ToList();
As I am new to elastic search so can't understand it. I even don't know why I am using term query to search a document while in kibana I user different queries and quite understandable match and match_phrase queries. So please help me get file.filename.
Edit: I have tried to include this too (later removed):
Source = new SourceFilter { Includes = ("file.filename") }
KIBANA Call:
This is the call from kibana console:
GET /extract/_search
{
"from" : 0, "size" : 1
, "query": {
"match": {
"content": "findthis"
}
}
}
The call returns following result I have used 1 result to show here:
Document in Elastic Search Index:
{
"took": 322,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 3330,
"max_score": 4.693223,
"hits": [
{
"_index": "extract",
"_type": "doc",
"_id": "29ebfd23bd7b276d7f3afc2bfad146d",
"_score": 4.693223,
"_source": {
"content": """
my title
A looooong document text to findthis.
""",
"meta": {
"title": "my title",
"raw": {
"X-Parsed-By": "org.apache.tika.parser.DefaultParser",
"Originator": "Microsoft Word 11",
"dc:title": "my title",
"Content-Encoding": "windows-1252",
"Content-Type-Hint": "text/html; charset=windows-1252",
"resourceName": "filename.htm",
"ProgId": "Word.Document",
"title": "my title",
"Content-Type": "text/html; charset=windows-1252",
"Generator": "Microsoft Word 11"
}
},
"file": {
"extension": "htm",
"content_type": "text/html; charset=windows-1252",
"last_modified": "2015-10-27T15:44:07.093+0000",
"indexing_date": "2018-02-10T08:16:23.329+0000",
"filesize": 32048,
"filename": "filename.htm",
"url": """file://D:\tmp\path\to\filename.htm"""
},
"path": {
"root": "e1a38f7da342f641e3eefad1ed1ca0f2",
"virtual": "/path/to/filename.htm",
"real": """D:\tmp\path\to\filename.htm"""
}
}
}
]
}
}
I am using NEST Api to get document file.filename from elastic search 6 on same server.
ISSUE:
Even though I have mentioned above too. Problem is filename is returned null in NEST API while content does return.
SOLUTION 1:
Using settings.DisableDirectStreaming(); I retrieved JSON result and created Following Class:
New Class:
public class Rootobject
{
public int took { get; set; }
public bool timed_out { get; set; }
public _Shards _shards { get; set; }
public Hits hits { get; set; }
}
public class _Shards
{
public int total { get; set; }
public int successful { get; set; }
public int skipped { get; set; }
public int failed { get; set; }
}
public class Hits
{
public int total { get; set; }
public float max_score { get; set; }
public Hit[] hits { get; set; }
}
public class Hit
{
public string _index { get; set; }
public string _type { get; set; }
public string _id { get; set; }
public float _score { get; set; }
public _Source _source { get; set; }
}
public class _Source
{
public string content { get; set; }
public Meta meta { get; set; }
public File file { get; set; }
public Path path { get; set; }
}
public class Meta
{
public string title { get; set; }
public Raw raw { get; set; }
}
public class Raw
{
public string XParsedBy { get; set; }
public string Originator { get; set; }
public string dctitle { get; set; }
public string ContentEncoding { get; set; }
public string ContentTypeHint { get; set; }
public string resourceName { get; set; }
public string ProgId { get; set; }
public string title { get; set; }
public string ContentType { get; set; }
public string Generator { get; set; }
}
public class File
{
public string extension { get; set; }
public string content_type { get; set; }
public DateTime last_modified { get; set; }
public DateTime indexing_date { get; set; }
public int filesize { get; set; }
public string filename { get; set; }
public string url { get; set; }
}
public class Path
{
public string root { get; set; }
public string _virtual { get; set; }
public string real { get; set; }
}
Query:
Instead of TermQuery I used MatchQuery here is my query, connection string is as above:
var request = new SearchRequest
{
From = 0,
Size = 1,
Query = new MatchQuery { Field = "content", Query = txtsearch.Text }
};
New Problem:
I tried much, though response does contain whole JSON result, but it is not being mapped properly.
I tried using Rootobject, Hits and Hit class but results only returned for _source as:
var response = client.Search<_Source>(request);
var twet = response.Documents.Select(t => t.file.filename).ToList();
Now I can retrieve content and file name but if I try using previous classes. The Hits and hits.total are returned as null.
I tried following queries:
var twet = response.Documents.SelectMany(t => t.hits.hits.Select(k => k._source.content)).ToList();
and
var twet1 = response.Hits.SelectMany(t => t.Source.hits.hits.Select(k => k._source.content)).ToList();
and
var twet1 = response.Documents.Select(t => t.Filename.fileName).ToList();
and
var twet = response.HitsMetadata.Hits.Select(t => t.Source.filename).ToList();
using Rootobject , Hits, Hit classes. While response does contain it.
So how can I use Rootobject class instead so that I can get whatever I want.
The elastic search server returns response as a JSON string and then Nest deserializes it into your required class.
In your case filename is a nested property inside the file property. To deserialize nested JSON properties check this link How to access nested object from JSON with Json.NET in C#
public class MyClass
{
public string Content { get; set; }
public FileClass File { get; set; }
}
public class Fileclass
{
public string Filename { get; set; }
}
And then you can read filename like response.Documents.Select(t=>t.File.Filename).ToList();

Using Delta<T> with complex objects

Can Delta<T> be used with complex object graphs, rather than single objects? I have tried without success to use it and am wondering if I'm missing something or if the functionality simply is not supported.
For example, here's a model:
public class Person {
public int Id { get;set; }
public string Name { get;s set; }
public Address Address { get; set; }
}
public class Address {
public string Street { get; set; }
public string City { get; set; }
public string PostCode { get; set; }
}
And here's the OData model building:
var builder = new ODataConventionModelBuilder();
builder.ComplexType<Address>();
builder.EntitySet<Person>("Person");
For the POST method of my controller, I can use this JSON and it will be deserialised fine:
{
"Name": "Mr Smith",
"Address": {
"Street": "Some Street",
"City": "Some City",
"PostCode": "Some PostCode"
}
}
However, problems arise when I use PATCH. If I send this:
{
"Name": "Mr Doe",
"Address": {
"Street": "Another Street"
}
}
And my controller method signature looks like this:
public IHttpActionResult Patch([FromODataUri] key, Delta<Person> delta) { ... }
Then the delta will contain an Address property, with null for City and PostCode and "Another Street" for Street. This seems to be correct to me.
When I then use delta.Patch(person) to apply the changes to the person object it copies this address wholesale over to the person, rather than just updating the Street property of the address.
Am I missing something, or is patching of complex type properties not supported by Delta<T>?
That's the current behavior. Complex type property, one of the structural property in OData, is treated as an unit when updating an entity.
However, Web API has the issue (Support PATCH to a complex type) to track this problem.

Nest elasticsearch-nested properties have incorrect mappings

I have a type that I inserted into my elastic search db with
public class Capture
{
public Guid Id { get; set; }
...
[ElasticProperty(Type = FieldType.Nested)]
public IList<CustomerInformation> CustomerInformations { get; set; }
}
However when I do a
var mapping=client.GetMapping<Capture>(o => o.Index("captures").Type("capture"));
the mapping of the customerinformations property is still object and not nested. Why is this? I have even deleted the index and recreated it and that does not solve anything.

Strange behavior of MongoDB LINQ provider for fields called "id"

Here's a JSON document where Mongo LINQ provider fails:
{"results":
{"text":"#twitterapi http://tinyurl.com/ctrefg",
"to_user_id":396524,
"to_user":"TwitterAPI",
"from_user":"jkoum",
"metadata":
{
"result_type":"popular",
"recent_retweets": 109
},
"id":1478555574,
"from_user_id":1833773,
"iso_language_code":"nl",
"source":"<a href=\"http://twitter.com/\">twitter<\/a>",
"profile_image_url":"http://s3.amazonaws.com/twitter_production/profile_images/118412707/2522215727_a5f07da155_b_normal.jpg",
"created_at":"Wed, 08 Apr 2009 19:22:10 +0000",
"since_id":0,
"max_id":1480307926,
"refresh_url":"?since_id=1480307926&q=%40twitterapi",
"results_per_page":15,
"next_page":"?page=2&max_id=1480307926&q=%40twitterapi",
"completed_in":0.031704,
"page":1,
"query":"%40twitterapi"}
}
Note an "id" field. Here are related C# entity definitions:
class Twitter
{
[BsonId]
public ObjectId Id { get; set; }
public Result results { get; set; }
}
private class Result
{
public string text { get; set; }
public int to_user_id { get; set; }
public string to_user { get; set; }
public string from_user { get; set; }
public Metadata metadata { get; set; }
public int id { get; set; }
public int from_user_id { get; set; }
public string iso_language_code { get; set; }
public string source { get; set; }
public string profile_image_url { get; set; }
public string created_at { get; set; }
public int since_id { get; set; }
public int max_id { get; set; }
public string refresh_url { get; set; }
public int results_per_page { get; set; }
public string next_page { get; set; }
public double completed_in { get; set; }
public int page { get; set; }
public string query { get; set; }
}
class Metadata
{
public string result_type { get; set; }
public int recent_retweets { get; set; }
}
If I create a "Twitter" collection and save the document above, then when I query it using Mongo LINQ provider, it throws a FileFormatException exception:
"Element 'id' does not match any field or property of class Mongo.Context.Tests.NativeTests+Result"
However there are two alternative workarounds to fix this problem:
Rename Result "id" field, e.g. to "idd" both in JSON doc and the Result class. Then LINQ query works.
Keep "id" field but in addition add a field "Id" to the Result class and mark it with attribute [BsonId]. Now the Result class contains both "Id" and "id" fields, but the query works!
I use Mongo API to query the collection, everything works fine, so I guess this must be a bug in a MongoDB LINQ provider. "id" in a nested JSON element should not be a reserved work, should it?
UPDATE: Here's the result of a native API query execution:
> db.Twitter.find().limit(1);
{ "_id" : ObjectId("50c9d3a4870f4f17e049332b"),
"results" : {
"text" : "#twitterapi http://tinyurl.com/ctrefg",
"to_user_id" : 396524,
"to_user" : "TwitterAPI",
"from_user" : "jkoum",
"metadata" : { "result_type" : "popular", "recent_retweets" : 109 },
"id" : 1478555574,
"from_user_id" : 1833773, "
iso_language_code" : "nl",
"source" : "twitter",
"profile_image_url" : "http://s3.amazonaws.com/twitter_production/profile_images/118412707/2522215727_a5f07da155_b_normal.jpg",
"created_at" : "Wed, 08 Apr 2009 19:22:10 +0000",
"since_id" : 0,
"max_id" : 1480307926,
"refresh_url" : "?since_id=1480307926&q=%40twitterapi", "results_per_page" : 15, "next_page" : "?page=2&max_id=1480307926&q=%40twitterapi",
"completed_in" : 0.031704,
"page" : 1,
"query" : "%40twitterapi"
}
}
MongoDB requires that every document stored in the database have a field (at the root level) called "_id".
The C# driver assumes that any field in your class called "Id", "id" or "_id" is meant to be mapped to the special "_id" field. This is a convention, one that can be overridden. The C# driver doesn't know that your Result class isn't meant to be used as the root document of a collection, so it finds your "id" field and maps it to "_id" in the database.
One way you can override this is to change the name of the field in your class (as you discovered). What you can then also do is use the [BsonElement] attribute to map your C# field name (e.g. "idd") to whatever name is actually being used in the database (e.g. "id"). For example:
public class Result
{
[BsonElement("id")]
public int idd; // matches "id" in the database
// other fields
}
Another alternative is to override the convention that finds the "Id" member of a class to suppress the default behavior of the C# driver for your Result class. You can do this by registering a new ConventionProfile for your Result class. For example:
var noIdConventions= new ConventionProfile();
noIdConventions.SetIdMemberConvention(new NamedIdMemberConvention()); // no names
BsonClassMap.RegisterConventions(noIdConventions, t => t == typeof(Result));
You must be sure to do this very early in your program, before your Result class gets mapped.
The problem is your POCO doesn't match your JSON
Your Twitter class has an ID and a class called results
Yet, your JSON just has results. That's where the problem lies.
So in reality, you're bypassing the Twitter class, and just creating an instance of Result
Your JSON should look something like:
{
_id: ObjectId("50c9c8f3e0ae76405f7d2b5e"),
"results": {
"text":"#twitterapi http://tinyurl.com/ctrefg",
"to_user_id":396524,
"to_user":"TwitterAPI",
"from_user":"jkoum",
"metadata":
{
"result_type":"popular",
"recent_retweets": 109
},
"id":1478555574,
"from_user_id":1833773,
"iso_language_code":"nl",
"source":"<a href=\"http://twitter.com/\">twitter<\/a>",
"profile_image_url":"http://s3.amazonaws.com/twitter_production/profile_images/118412707/2522215727_a5f07da155_b_normal.jpg",
"created_at":"Wed, 08 Apr 2009 19:22:10 +0000",
"since_id":0,
"max_id":1480307926,
"refresh_url":"?since_id=1480307926&q=%40twitterapi",
"results_per_page":15,
"next_page":"?page=2&max_id=1480307926&q=%40twitterapi",
"completed_in":0.031704,
"page":1,
"query":"%40twitterapi"}
}
}
Edit
Your results is actually an embedded document (in the case of your POCO model currently) so you should also mark Result.ID with [BsonId]

Resources