NEST elasticsearch DateRange query - elasticsearch

I am trying to get results from elasticsearch on a predefined range of dates using NEST api but i dont manage. This example code i am trying to retrieve a week of data but it is ignoring my date Filter and returning older data than my initial date.
Thanks in advance for the help!
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
var client = new ElasticClient(settings);
settings.DefaultIndex("wholelog-2017.04*");
client.CreateIndex("wholelog-2017.04*",
create => create.Mappings(
mappings => mappings.Map<LogLine>(type =>
type.AutoMap()
)
));
DateTime InitDate = DateTime.Now.AddDays(-7);
var filterClauses = new List<QueryContainer>();
filterClauses.Add(new DateRangeQuery
{
Field = new Field("logTime"),
LessThanOrEqualTo = DateTime.Now,
GreaterThanOrEqualTo = InitDate
});
var searchRequest = new SearchRequest<LogLine>()
{
Size = 10000,
From = 0,
Scroll = "1m",
Query = new BoolQuery
{
Filter = filterClauses
}
};
var searchResult = client.Search<LogLine>(searchRequest);
[ElasticsearchType(Name="logLine")]
public class LogLine
{
public string Message {get;set;}
public DateTime logTime {get;set;}
public string type {get;set;}
public string logServer {get;set;}
}
logstash config file
output {
elasticsearch{
hosts => "localhost"
#user => ####
#password => ####
index => "wholelog-%{+YYYY.MM.dd}"
}
}

I assume that you didn't explicitly put your mapping and just started adding documents to your index. Try to create a new index like this
client.CreateIndex(indexName,
create => create.Mappings(
mappings => mappings.Map<LogLine>(
type => type.AutoMap()
)
)
);
And use DateTime for your field. DateMath is intended for building search queries
[ElasticsearchType(Name = "logLine")]
public class LogLine
{
public string Message { get; set; }
public DateTime logTime { get; set; }
public string type { get; set; }
public string logServer { get; set; }
}
After explicitly putting your mapping try to add docs and search against new index.

Related

How can I skip indexing a specific property, but still retrieve its contents when querying in Elasticsearch and NEST?

Given this index-model class:
public class ProductIndexModel
{
public Guid Id { get; set; }
public DateTime Created { get; set; }
public string Name { get; set; }
public JObject DynamicContent { get; set; }
}
I am struggling to do the following:
store the entire object in Elasticsearch
index all properties, except DynamicContent property
upon querying, retrieve the entire object (including DynamicContent prop)
My reason for not indexing DynamicContent property is, that it's a json-blob, where occasionally there will be clashes between property-paths, that are of different types (e.g. object vs string, int vs string, and so on). For example, trying to index path /dynamiccontent.id with 2 objects, where value is respectively of type int and string might give me:
error: Type: mapper_parsing_exception Reason: "failed to parse
[dynamiccontent.id]" CausedBy: Type: json_parse_exception Reason:
"Current token (START_OBJECT) not numeric, can not use numeric value
accessors
I create the index in this way:
var createIndexResponse = await _elasticClient.CreateIndexAsync(indexName, c => c
.InitializeUsing(settingsState)
.Mappings(ms => ms
.Map<ProductIndexModel>(m => m
.AutoMap()
)
)
);
Where settingsState is of type Nest.IndexState with some tokenizers and more, that is irrelevant to the question.
ISearchRequest SearchRequest(SearchDescriptor<ProductIndexModel> x) => x
.Index(indexName)
.Query(q => q.Bool(bq => bq.Filter(filters)))
.From(filter.Offset)
.Size(filter.PageSize)
;
var searchResponse = await _elasticClient.SearchAsync<ProductIndexModel>(SearchRequest);
Where filters is a dynamically constructed generic list of filters to reduce results by.
So I want to keep DynamicContent un-indexed, but still be able to get its (raw) contents when querying.
I have tried to annotate DynamicContent with the Nest.IgnoreAttribute which leaves it out entirely, thus resulting in a null value when retrieving. Any suggestions as to how to just "store" the value, but not index it, using NEST?
Since DynamicContent is a Json.NET JObject type, if you're using NEST 6.x, you will need to hook up the JsonNetSerializer to be able to correctly index an instance of JObject.
Once this serializer is hooked up, you can attribute the model with [Object(Enabled = false)], which sets enabled=false for the field, meaning the property is persisted in _source but not parsed or indexed.
With JObject in particular, NEST's automapping (which is needed to take the attribute into account when mapping) will generate a large "properties" object for JObject which is wholly unneccessary, since the field will not be parsed or indexed. In this particular case, fluent mapping would be a better choice than attribute mapping. Here's an example:
private static void Main()
{
var defaultIndex = "default_index";
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool, JsonNetSerializer.Default)
.DefaultIndex(defaultIndex)
.DefaultTypeName("_doc");
var client = new ElasticClient(settings);
if (client.IndexExists(defaultIndex).Exists)
client.DeleteIndex(defaultIndex);
var createIndexResponse = client.CreateIndex(defaultIndex, c => c
.Mappings(m => m
.Map<ProductIndexModel>(mm => mm
.AutoMap() // <-- automap
.Properties(p => p
.Object<JObject>(o => o
.Name(n => n.DynamicContent) <-- override the automap inferred mapping for DynamicContent
.Enabled(false)
)
)
)
)
);
var indexResponse = client.Index(new ProductIndexModel
{
Id = Guid.NewGuid(),
Created = DateTime.UtcNow,
Name = "foo",
DynamicContent = new JObject
{
{ "prop1", "value1" },
{ "prop2", new JArray(1, 2, 3, 4) }
}
}, i => i.Refresh(Refresh.WaitFor));
var searchResponse = client.Search<ProductIndexModel>(s => s
.MatchAll()
);
}
public class ProductIndexModel
{
public Guid Id { get; set; }
public DateTime Created { get; set; }
public string Name { get; set; }
[Object(Enabled = false)]
public JObject DynamicContent { get; set; }
}

C# Mongo db driver updating an existing document

I have an existing document in mongo db with certain properties. When i am trying to update this document with a document with the same id but having more properties it is not updating it. I am using mongo db driver version 2.4.4
Following is the existing document in my db
{
"_id" : "1234",
"AccountNumber" : "3453535345354",
"Name" : "new1",
"PhoneNumber" : "34534535353534543",
"ETag" : "6ba32e6e-3808-41f5-9b28-0ea882d9c629",
"Id" : "1234"
}
Now when i am trying to update/ replace this document with the following document it does not work:
{
"AccountNumber": "3453535345354",
"Name":"new1",
"PhoneNumber":"34534535353534543",
"TempProperty":"something new",
"ETag" : "6ba32e6e-3808-41f5-9b28-0ea882d9c629",
"Id" : "1234",
"_id" : "1234"
}
Following is the C# code for my upsert method:
public async Task UpsertDocument(string collectionId, string documentId, BsonDocument item)
{
var collection = database.GetCollection<BsonDocument>(collectionId);
var initialCorrelationId = item["ETag"].AsString;
item["ETag"] = Guid.NewGuid().ToString();
var builders = Builders<BsonDocument>.Filter;
var filter = builders.Eq(x => x["Id"], documentId) & builders.Eq(x => x["ETag"], initialCorrelationId);
var options = new FindOneAndReplaceOptions<BsonDocument, BsonDocument>
{
ReturnDocument = ReturnDocument.Before,
IsUpsert = true
};
try
{
item["Id"] = documentId;
item["_id"] = documentId;
await collection.FindOneAndReplaceAsync(filter, item, options);
}
catch (MongoCommandException ex) when (ex.Code == 11000)
{
throw new ConcurrencyException(
$"Error upserting item with id {documentId} and etag {item["Etag"]}{Environment.NewLine}{item.ToJson()}");
}
}
Any help would be much appreciated.
Assuming you have a class as follows:
public class Thing
{
public ObjectId Id { get; set; }
public string AccountNumber { get; set; }
public string Name { get; set; }
public string PhoneNumber { get; set; }
public string ETag { get; set; }
public string SecondId { get; set; }
}
The following code does what you want:
var context = new Context();
var tempProperty = "newValue";
var id = "1234"; // This should be unique
var builder = Builders<Thing>.Filter;
var filter = builder.Eq(x => x.SecondId, id);
var update = Builders<Thing>.Update
.Set("TempProperty", tempProperty);
context.ThingCollection.UpdateOne(filter, update, new UpdateOptions() {IsUpsert = true});
I just renamed Id to SecondId.

Elasticsearch - Trying to index MS Word attachment & making a full text search within

As the title is already indicating, I am trying to index MS Word documents and making a full text search within.
I have seen several examples, but I am not able to figure out what I am doing incorrectly.
Relevant Code:
[ElasticsearchType(Name = "AttachmentDocuments")]
public class Attachment
{
[String(Name = "_content")]
public string Content { get; set; }
[String(Name = "_content_type")]
public string ContentType { get; set; }
[String(Name = "_name")]
public string Name { get; set; }
public Attachment(Task<File> file)
{
Content = file.Result.FileContent;
ContentType = file.Result.FileType;
Name = file.Result.FileName;
}
}
The "Content" property above is set to "file.Result.FileContent" in the constructor. The "Content" property is a base64 string.
public class Document
{
[Number(Name = "Id")]
public int Id { get; set; }
[Attachment]
public Attachment File { get; set; }
public String Title { get; set; }
}
Below is the method for indexing documents to elasticsearch database.
public void IndexDocument(Attachment attachmentDocument)
{
// Create the index if it does not already exist
var indexExists = _client.IndexExists(new IndexExistsRequest(ElasticsearchIndexName));
if (!indexExists.Exists)
{
var indexDescriptor =
new CreateIndexDescriptor(new IndexName {Name = ElasticsearchIndexName}).Mappings(
ms => ms.Map<Document>(m => m.AutoMap()));
_client.CreateIndex(indexDescriptor);
}
var doc = new Document()
{
Id = 1,
Title = "Test",
File = attachmentDocument
};
_client.Index(doc);
}
Based on the code above, the document get indexed into the correct index(Screenshot from Elasticsearch host - Searchly):
Searchly Screenshot
The content in the file is : "VCXCVXCVXCVXCVXVXCVXCV" and with the following query I get zero hits in return:
QueryContainer queryContainer = null;
queryContainer |= new MatchQuery()
{
Field = "file",
Query = "VCXCVXCVXCVXCVXVXCVXCV"
};
var searchResult =
await _client.LowLevel.SearchAsync<string>(ApplicationsIndexName, "document", new SearchRequest()
{
From = 0,
Size = 10,
Query = queryContainer,
Aggregations = GetAggregations()
});
I would appericiate if someone could hint me what I am doing incorrectly or should look into?
Providing screenshot of mapping in my Elasticsearch database:
Elasticsearch - Mapping
Because you refer to wrong field. Field should be file.content
queryContainer |= new MatchQuery()
{
Field = "file.content",
Query = "VCXCVXCVXCVXCVXVXCVXCV"
};

How to disable source field and use store with Nest?

I want to disable source as it is done here initially. I am not sure how to achieve similar like in the sample
PUT tweets
{
"mappings": {
"tweet": {
"_source": {
"enabled": false
}
}
}
}
I tried below but when I enter http://localhost:9200/_plugin/head/; I can see that all the properties are stored. I expect to store and index only id and name properties.
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node, defaultIndex: "mydatabase");
var client = new ElasticClient(settings);
var createIndexResult = client.CreateIndex("mydatabase");
var mapResult = client.Map<Product>(c => c.MapFromAttributes().SourceField(s=>s.Enabled(false)).IgnoreConflicts().Type("product").Indices("mydatabase"));
client.Index(sampleproduct);
[ElasticType(Name ="product", IdProperty = "ProductId" )]
[Table("Product")]
public partial class Product
{
[ElasticProperty(Name = "id",Index = FieldIndexOption.NotAnalyzed, Store = true)]
public int ProductId { get; set; }
[ElasticProperty(Index = FieldIndexOption.Analyzed, Store = true)]
public string Name { get; set; }
[ElasticProperty(Index = FieldIndexOption.No, Store = false)]
public int? ProductTypeId { get; set; }
[ElasticProperty(Index = FieldIndexOption.No, Store = false)]
public int? ManufacturerId { get; set; }
}
EDIT: without adding any documents after creating index, index metadata looks like in the image. i dont see any source enabled false.
EDIT2: after changing creating index first then mapping. Name field is displaying store=true as in the image but there is no value stored. I debugged and i am surely passing value where i index the sampleproduct
You need to add the mapping first and then index the document. Only then will you see the get the expected mapping. By indexing first without the mapping, a mapping gets dynamically created with default options (source being enabled).

Elastic search Nest index query always returning false

var local = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(local, null);
var elastic = new ElasticClient(settings);
var res = elastic.CreateIndex(ci => ci
.Index("my_first_index_final2")
.AddMapping<BlogPost>(m => m.MapFromAttributes()));
Console.WriteLine(res.RequestInformation.Success);
var blogPost = new BlogPost
{
Id = Guid.NewGuid(),
Title = "First blog post",
Body = "This is very long blog post!"
};
var firstId = blogPost.Id;
var result = elastic.Index(blogPost, p => p
.Index("my_first_index_final2")
.Id(blogPost.Id.ToString())
.Refresh());
Console.WriteLine(result.RequestInformation.Success);
Blogpost class:
[ElasticType(IdProperty = "Id", Name = "blog_post")]
public class BlogPost
{
[ElasticProperty(Name = "_id", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public Guid? Id { get; set; }
[ElasticProperty(Name = "title", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Title { get; set; }
[ElasticProperty(Name = "body", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Body { get; set; }
public override string ToString()
{
return string.Format("Id: '{0}', Title: '{1}', Body: '{2}'", Id, Title, Body);
}
}
This is my code. Everytime it returns:
true
false
Meaning, it creates index but unable to insert document into the index. I don't understand the reason.
Also, I have to rename my index name everytime i run this demo console application as i think we cannot insert index with same name. how can i avoid doing this?
I am following this tutorial:
https://www.devbridge.com/articles/getting-started-with-elastic-using-net-nest-library-part-two/
Any other resource for learning nest and elastic search, please feel free to suggest.
My guess is you are using Elasticsearch 2.x. Your code won't break in Elasticsearch 1.x. The problem is, you are trying to add a field _id inside a document. It being one of the metadata fields, Elasticsearch 2.x prohibits you from indexing it inside the document. To make your code work, simply change the name of the Id field from _id to something different, say, id.

Resources