I'm trying to use NEST from MVC app, however I'm getting the request serialized incorrectly:
iisexpress.exe Error: 0 : NEST POST http://localhost:9200/_search (00:00:00.8188240):
StatusCode: 400,
Method: POST,
Url: http://localhost:9200/_search,
Request: {
"$type": "Nest.SearchDescriptor`1[[System.Object, mscorlib]], Nest",
"aggs": {
"Period": {
"$type": "Nest.AggregationDescriptor`1[[System.Object, mscorlib]], Nest",
"date_histogram": {
"$type": "Nest.DateHistogramAggregationDescriptor`1[[System.Object, mscorlib]], Nest",
"field": "Timestamp",
"interval": "day",
"format": "yyyy-MM-dd"
}
}
}
}
The query is very simple:
var cs2 = new ConnectionSettings(new Uri("http://localhost:9200")).EnableTrace();
var client = new ElasticClient(cs2);
var res3 = client.Search<object>(q =>q.Aggregations(agg =>
agg.DateHistogram("DayAgg", t => t.Field("Timestamp").Interval("day"))));
The exact same code works fine in console application, so I'm thinking that could be related to the serialization, since in the bad case the "$type" property is added.
Found related issue: Serialization error with Elasticsearch NEST/C#
The rootcause is following settings:
config.Formatters.JsonFormatter.SerializerSettings.TypeNameHandling = TypeNameHandling.Objects;
And now there is a new api supported for settings: SetJsonSerializerSettingsModifier
var cs2 = new ConnectionSettings(new Uri("http://localhost:9200"))
.SetJsonSerializerSettingsModifier(settings => settings.TypeNameHandling = TypeNameHandling.None)
.EnableTrace();
Related
I have a json and I have to remove whole filter block from that ,How can i remove that?I tried below in JSR223PostProcessor but it is not working
var response = prev.getResponseDataAsString();
var responseWithoutIds = response.replaceAll("\"filter\"[ ]*:[^,}\\]]*[,]?", "");
{
"templateId": "1e5eef7d-9581-40a1-98ed-774dcf68ce11",
"ownerId": "241e992f-a1c7-430a-be9f-347337643697",
"orgId": "4c41a6d7-dfbf-485b-84b6-3a3991546d9c",
"createdDate": "2018-12-24T10:51:57.336Z",
"shareWithOrg": false,
"templateName": "custom template",
"templateDesc": null,
"system": false,
"optionIds": [
"9c9b93c9-4ada-4489-9129-604cf1fa35be",
"65285021-9c2c-4787-b810-390c50f48f9f",
"222c832d-8db6-4ca1-88ef-150b6a388d87",
"be22ba29-144b-4971-99de-0d4258220558",
"b7ab883b-fe51-4e02-bdb1-0d7321d30ac3",
"510660fa-d255-45e2-b6b8-c1c09c9009d6",
"51f78f08-b655-4a82-b0c5-61365881fb0a",
"ac1d4b39-f41c-4b69-bdc4-b76fd1d2b9af",
"2de3591e-c9eb-48bd-a9bf-acfcee695ac8",
"c9e2a1f1-f786-4e9a-9d11-525a5aa01345",
"441a0627-9983-4edc-95ce-a5b8e79f7f1d",
"087a1ac8-e7c1-4cee-ae31-4f0aaccba8c9",
"dba760fb-9e97-4dcb-b1ab-4a2795e59ee6",
"53bc5244-b33e-4732-8bf0-cd051e017089",
"2bf04285-fe19-4534-8467-44c08661da60",
"020eb946-6ba0-4519-b6f9-23ffafd949d0",
"ced89c82-5ba1-4fe6-bc83-5bddff820c85",
"18ea7006-2e29-4a3f-a28f-59a1323a4bd0",
"c7efa0d1-094c-4626-90aa-80e1343bbdef",
"6504605f-3a41-4406-9f0e-150365c8ee35",
"3c28c622-83b1-4b46-ac16-3c5cbff80999",
"6a227b18-519f-49c4-8c59-7fe0eaf73fea",
"f9127d4e-c27e-48fa-8286-42a045d8fc40",
"bb695a06-50fe-4419-924f-7fee96530b63",
"731393bc-92e0-4ddb-8113-a7b3942527c5",
"a355ee2d-c654-4406-9027-8e801cb1a4a9"
],
"optionGroupIds": [
"09f1302b-6ad0-486b-8f8c-e65ae14f5831",
"2041206d-cc87-4173-912c-fee52cefcf2b",
"6fc154fb-8313-488c-a2cb-e1a5b88b3028",
"cc0767f2-3e7f-42f1-b9c7-9f76991a6fcf",
"4cd75ade-db20-43fe-8b15-7effa961b4b5",
"6af8ddba-8a7c-4ff7-8043-6358b6e6a31b"
],
"filter": [
{
"optionId": "222c832d-8db6-4ca1-88ef-150b6a388d87",
"selectedValues": [
"all"
],
"selectedComplexValues": null,
"contextualFilterId": null,
"operator": "pastDays",
"filterType": "date"
}
],
"sortDirection": "desc",
"sortedOptionId": "222c832d-8db6-4ca1-88ef-150b6a388d87",
"templateCategory": "patients"
}
Using regular expressions is not the best way to proceed. Also consider switching to Groovy language, it provides JsonSlurper and JsonBuilder classes which way more reliable.
Example code:
def response = new groovy.json.JsonSlurper().parse(prev.getResponseData())
response.remove('filter')
def responseWithoutIds = new groovy.json.JsonBuilder(response).toPrettyString()
log.info(responseWithoutIds)
References:
Groovy: Parsing and producing JSON
Apache Groovy - Why and How You Should Use It
Use below code in JSR223PostProcessor :
var body = prev.getResponseDataAsString();
var parsed = JSON.parse(body);
delete parsed.filter[0];
delete parsed.filter;
vars.put('value', JSON.stringify(parsed));
Below is a portion of my code, that I've taken from here. I had to make few changes to make work with the new v6.X of Elasticsearch.
It runs without any errors and creates new documents BUT with empty field values. If I take the same JSON payload and PUT in Elasticsearch using Postman, the document gets indexed fine, with all fields populated. Please let me know if I am using the right Elasticsearch API methods, and using them right.
string strJsonMessage = #"
{
message : {
content: 'Test Message Content'
}
}";
ConnectionSettings connectionSettings = new ConnectionSettings(new Uri("xxx")).BasicAuthentication("xx", "xx");
ElasticClient client = new ElasticClient(connectionSettings);
JObject msg = JObject.Parse(strJsonMessage);
var result = await client.LowLevel.IndexAsync<BytesResponse>("events-2018.03.27", "event", PostData.Serializable(msg));
if (result.Success)
{
log.Info("Data successfully sent.");
log.Verbose(result.Body);
}
else
{
log.Error("Failed to send data.");
}
****OUTPUT TRACE:****
2018-03-27T18:37:18.961 [Info] Data successfully sent.
2018-03-27T18:37:18.961 [Verbose] {"_index":"events-2018.03.27","_type":"event","_id":"u9HPaGIBBm3ZG7GB5jM_","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":29,"_primary_term":1}
Querying elasticsearch for this document by its ID gives me this:
{
"_index": "events-2018.03.27",
"_type": "event",
"_id": "u9HPaGIBBm3ZG7GB5jM_",
"_version": 1,
"found": true,
"_source": {
"message": {
"content": []
}
}
}
In case someone comes across the same issue, I got the solution here.
According to Retrieving a document documentation
GET /website/blog/123/_source
would directly return the document stored inside the _source field.
I'm currently using Node JS's express framework. How should I implement this in my code?
esClient.search({
index: "myIndex",
type: "myType",
body: {
"query": {
"match_all": {}
},
"size": 3,
"from": 1
}
}).then(function (resp) {
var result = resp.hits.hits;
res.status(200).send({data: {recommendations: result, showItemFrom: showItemFrom}})
}, function (err) {
console.trace(err.message)
res.status(500).send({data: err.message})
})
I'm getting the response this way...
[
"_source":{
{
"id": 1,
"title": "Test"
}
}
]
However, I want it this way...
[
{
id:1,
title:"Test"
}
]
I don't think the Elasticsearch API has a method to do that for searches, the one that Val mentioned works, but it is only usable to GET documents directly through its id.
But you can map the result using the Javascript Array#map() method:
var result = resp.hits.hits.map(hit => hit._source);
After
index:"myIndex"
Add:
source:true
You need to call the getSource() function, like this:
esClient.getSource({
index: "website",
type: "blog",
id: "123"
}).then(function (source) {
// do something with source
}, function (err) {
// error happened
})
I'm using the object initializer syntax with NEST to form a search query. When I include the second pdfQuery with the logical OR operator, I get no results. If I exclude it, I get results.
QueryContainer titleQuery = new MatchQuery
{
Field = Property.Path<ElasticBook>(p => p.Title),
Query = query,
Boost = 50,
Slop = 2,
MinimumShouldMatch = "55%"
};
QueryContainer pdfQuery = new MatchQuery
{
Field = Property.Path<ElasticBook>(p => p.Pdf),
Query = query,
CutoffFrequency = 0.001
};
var result = _client.Search<ElasticBook>(new SearchRequest("bookswithstop", "en")
{
From = 0,
Size = 10,
Query = titleQuery || pdfQuery,
Timeout = "20000",
Fields = new []
{
Property.Path<ElasticBook>(p => p.Title)
}
});
If I debug and inspect the result var, I copy-value one of request properties to get:
{
"timeout": "20000",
"from": 0,
"size": 10,
"fields": [
"title"
],
"query": {
"bool": {
"should": [
{
"match": {
"title": {
"query": "Proper Guide To Excel 2010",
"slop": 2,
"boost": 50.0,
"minimum_should_match": "55%"
}
}
},
{
"match": {
"pdf": {
"query": "Proper Guide To Excel 2010",
"cutoff_frequency": 0.001
}
}
}
]
}
}
}
The problem is that if I copy that query into sense - it returns about 100 results (albeit slowly). I've checked the header info and that seems to be correct from NEST as well:
ConnectionStatus = {StatusCode: 200,
Method: POST,
Url: http://elasticsearch-blablablamrfreeman/bookswithstop/en/_search,
Request: {
"timeout": "20000",
"from": 0,
"size": 10,
"fields": [
"title"
],
"query": {
"bool": {
"shoul...
The pdf field uses the elastic search attachment plugin (located # https://github.com/elastic/elasticsearch-mapper-attachments) and I was getting Newtonsoft.JSON system.outofmemoryexceptions being thrown before (but not now for some reason).
My only suggestion therefore is that perhaps there's some serialization issue via my query and NEST? If that were the case I'm not sure why it would just execute successfully with a 200 code and give 0 documents in the Documents property
Could anyone please explain to me how I would go about troubleshooting this please? It clearly doesn't like my second search query (pdfQuery) but I'm not sure why - and the resultant JSON request syntax seems to be correct as well!
I think this part is causing problems
Fields = new []
{
Property.Path<ElasticBook>(p => p.Title)
}
When do you use Fields option, elasticsearch is not returning _source field, so you can't access results through result.Documents. Instead, you have to use result.FieldSelections, which is quite unpleasant.
If you want to return only specific fields from elasticsearch and still be able to use result.Documents you can take advantage of source includes / excludes. With NEST you can do this as follows:
var searchResponse = client.Search<Document>(s => s
.Source(source => source.Include(f => f.Number))
.Query(q => q.MatchAll()));
Hope this helps you.
I'm trying to mimic a query that I wrote in Sense (chrome plugin) using NEST in C#. I can't figure out what the difference between the two queries is. The Sense query returns records while the nest query does not. The queries are as follows:
var searchResults = client.Search<File>(s => s.Query(q => q.Term(p => p.fileContents, "int")));
and
{
"query": {
"term": {
"fileContents": {
"value": "int"
}
}
}
What is the difference between these two queries? Why would one return records and the other not?
You can find out what query NEST uses with the following code:
var json = System.Text.Encoding.UTF8.GetString(searchResults.RequestInformation.Request);
Then you can compare the output.
I prefer this slightly simpler version, which I usually just type in .NET Immediate window:
searchResults.ConnectionStatus;
Besides being shorter, it also gives the url, which can be quite helpful.
? searchResults.ConnectionStatus;
{StatusCode: 200,
Method: POST,
Url: http://localhost:9200/_all/filecontent/_search,
Request: {
"query": {
"term": {
"fileContents": {
"value": "int"
}
}
}
}
Try this:
var searchResults2 = client.Search<File>(s => s
.Query(q => q
.Term(p => p.Field(r => r.fileContents).Value("int")
)
));
Followup:
RequestInformation is not available in newer versions of NEST.
I'd suggest breaking down your code in steps (Don't directly build queries in client.Search() method.
client.Search() takes Func<SearchDescriptor<T>, ISearchRequest> as input (parameter).
My answer from a similar post:
SearchDescriptor<T> sd = new SearchDescriptor<T>()
.From(0).Size(100)
.Query(q => q
.Bool(t => t
.Must(u => u
.Bool(v => v
.Should(
...
)
)
)
)
);
And got the deserialized JSON like this:
{
"from": 0,
"size": 100,
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
...
]
}
}
]
}
}
}
It was annoying, NEST library should have something that spits out the JSON from request. However this worked for me:
using (MemoryStream mStream = new MemoryStream()) {
client.Serializer.Serialize(sd, mStream);
Console.WriteLine(Encoding.ASCII.GetString(mStream.ToArray()));
}
NEST library version: 2.0.0.0.
Newer version may have an easier method to get this (Hopefully).