I need to make some calculation with each doc's some fields and then return it as response's part
I've written this
TermsAggregationBuilder subAggregation = AggregationBuilders
.terms("price")
.script(new Script(ScriptType.INLINE, "painless",
" int total = 0;\n" +
" for (int i = 0; i < 5; ++i) {\n" +
" total += doc['age'].value;\n" +
" }\n" +
" return total ;\n"
, Collections.emptyMap()));
NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.addAggregation(subAggregation)
.build();
SearchHits<Stories>
searchHits = elasticsearchOperations.search(nativeSearchQuery, Stories.class
, IndexCoordinates.of("long_stories"));
and this returns this one
{
"totalHits": 1,
"totalHitsRelation": "EQUAL_TO",
"maxScore": 1,
"scrollId": null,
"searchHits": [
{
"id": "44",
"score": 1,
"sortValues": [],
"content": {
"id": "44",
"innerId": 5,
"age": 44,
"salary": 433,
"mark": 10,
"state": "state",
"stores": [
{
"storiesId": "44",
"innerId": 1
},
{
"storiesId": "44b",
"innerId": 2
},
{
"storiesId": "44c",
"innerId": 3
}
],
"esh": null
},
"highlightFields": {}
}
],
"aggregations": {
"asMap": {
"price": {
"name": "price",
"metadata": null,
"buckets": [
{
"aggregations": {
"asMap": {},
"fragment": true
},
"keyAsString": "220",
"docCount": 1,
"docCountError": 0,
"key": "220",
"keyAsNumber": 220,
"fragment": true
}
],
"type": "sterms",
"docCountError": 0,
"sumOfOtherDocCounts": 0,
"fragment": true
}
},
"fragment": true
},
"empty": false
}
this response gives separate calculation I need it to be with above part ,and also if key which is 220 now if I had some another doc which would give the same answer then response would print 220 and doc count 2 I need it to be separately and in above response
like doc1 220 doc2 220 and in above response where actual data holds
Terms aggregation works as "group by".
So it will have only unique keys and documents clubbed under those keys. If you need it on per document basis use scripted field in query instead of aggregation.
With scripted field you will need to specify "source" to fetch rest of the document.
You can also explore runtime field in place of scripted field. It offer some benefits over scripted field
You can use script fields to access values in _source and return
calculated values based on a script valuation. Runtime fields have
these same capabilities, but provide greater flexibility because you
can query and aggregate on runtime fields in a search request. Script
fields can only fetch values.
Related
Having array as:
[France, Switzerland, Croatia, Spain, Brondby, Sonderjyske, CFR Cluj, Ujpest FC, Young Boys , Xamax, Bedford Town, Biggleswade Town]
Having JSON fetched from API call as (NOT full JSON provided)
{
"payload": [
{
"id": 104,
"externalId": "e5b93978-0960-4c01-a201-b15c9e6afb49",
"shortName": "FRA",
"name": "France",
"countryCode": "en",
"leagueId": 1,
"logoUrl": "http://www.example.com/logo.png"
},
{
"id": 110,
"externalId": "7f1f76c1-483c-4c14-b5f9-61e58a5d9328",
"shortName": "Swi",
"name": "Switzerland",
"countryCode": "en",
"leagueId": 1,
"logoUrl": "http://www.example.com/logo.png"
},
{
"id": 2189,
"externalId": "8d23300c-85f4-41b4-b0f1-65e5948f2c80",
"shortName": "Cro",
"name": "Croatia",
"countryCode": "en",
"leagueId": 1,
"logoUrl": "http://www.sdfgsdf.com/image.png"
}
],
"errorCode": 0,
"errorTimeStamp": null,
"errorMessage": null,
"hasError": false
}
For every single member of the array, I need to get its ID's.
I know how match for signle member using JSON path
But, how to fetch all the ID's based on the particullar array?
Given that you have this "array" in a JMeter Variable called array you can get all the IDs by adding a JSR223 PostProcessor and using the following Groovy code in the "Script" area:
vars.get('array').replace('[', '').replace(']', '').tokenize(',').each { country ->
def result = new groovy.json.JsonSlurper().parse(prev.getResponseData()).payload.find { entry -> entry.name == country.trim() }
if (result != null) {
log.info('Country: ' + country.trim() + ', id: ' + result.id)
}
}
Demo:
More information:
JsonSlurper
Apache Groovy - Parsing and producing JSON
I have documents in the following format:
{
"id": number
"chefId: number
"name": String,
"ingredients": List<String>,
"isSpecial": boolean
}
Here is a list of 5 documents:
{
"id": 1,
"chefId": 1,
"name": "Roasted Potatoes",
"ingredients": ["Potato", "Onion", "Oil", "Salt"],
"isSpecial": false
},
{
"id": 2,
"chefId": 1,
"name": "Dauphinoise potatoes",
"ingredients": ["Potato", "Garlic", "Cream", "Salt"],
"isSpecial": true
},
{
"id": 3,
"chefId": 2,
"name": "Boiled Potatoes",
"ingredients": ["Potato", "Salt"],
"isSpecial": true
},
{
"id": 4,
"chefId": 3
"name": "Mashed Potatoes",
"ingredients": ["Potato", "Butter", "Milk"],
"isSpecial": false
},
{
"id": 5,
"chefId": 4
"name": "Hash Browns",
"ingredients": ["Potato", "Onion", "Egg"],
"isSpecial": false
}
I will be doing a search where "Potatoes" is contained in the name field. Like this:
{
"query": {
"wildcard": {
"status": {
"value": "*Potatoes*"
}
}
}
}
But I also want to add some extra criteria when returning documents:
If the ingredients contain onion or milk, then return the documents. So documents with the id 1 and 4 will be returned. Note that this means that we have documents returned where chef ids are 1 and 3.
Then, for the documents where we haven't already got another document with the same chef id, return where the isSpecial flag is set to true. So only document 3 will be returned. 2 wouldn't be returned as we already have a document where the chef id is equal to one.
Is it possible to do this kind of chaining in Elasticsearch? I would like to be able to do this in a single query so that I can avoid adding logic to my (Java) code.
You can't have that sort of logic in one elasticsearch query. You could have a tricky query with aggregations / post_filter and so to have all the data you need in one query and then transform it in your Java application.
But the best approach (and the more maintainable) is to have two queries.
I've started the process of learning ElasticSearch and I was wondering if somebody could help me shortcut the process by providing some examples of how I would a build couple of queries.
Here's my example schema...
PUT /sales/_mapping
{
"sale": {
"properties": {
"productCode: {"type":"string"},
"productTitle": {"type": "string"},
"quantity" : {"type": "integer"},
"unitPrice" : {"type": double}
}
}
}
POST /sales/1
{"productCode": "A", "productTitle": "Widget", "quantity" : 5, "unitPrice":
5.50}
POST /sales/2
{"productCode": "B", "productTitle": "Gizmo", "quantity" : 10, "unitPrice": 1.10}
POST /sales/3
{"productCode": "C", "productTitle": "Spanner", "quantity" : 5, "unitPrice":
9.00}
POST /sales/4
{"productCode": "A", "productTitle": "Widget", "quantity" : 15, "unitPrice":
5.40}
POST /sales/5
{"productCode": "B", "productTitle": "Gizmo", "quantity" : 20, "unitPrice":
1.00}
POST /sales/6
{"productCode": "B", "productTitle": "Gizmo", "quantity" : 30, "unitPrice":
0.90}
POST /sales/7
{"productCode": "B", "productTitle": "Gizmo", "quantity" : 40, "unitPrice":
0.80}
POST /sales/8
{"productCode": "C", "productTitle": "Spanner", "quantity" : 100,
"unitPrice": 7.50}
POST /sales/9
{"productCode": "C", "productTitle": "Spanner", "quantity" : 200,
"unitPrice": 5.50}
What query would I need to generate the following results?
a). Show the show the number of documents grouped by product code
Product code Title Count
A Widget 2
B Gizmo 4
C Spanner 3
b). Show the total units sold by product code, i.e.
Product code Title Total units sold
A Widget 20
B Gizmo 100
C Spanner 305
TIA
You can accomplish that using aggregations, in particular Terms Aggregations. And it can be done in just one run, by including them within your query structure; in order to instruct ES to generate analytic data based in aggregations, you need to include the aggregations object (or aggs), and specify within it the type of aggregations you would like ES to run upon your data.
{
"query": {
"match_all": {}
},
"aggs": {
"group_by_product": {
"terms": {
"field": "productCode"
},
"aggs": {
"units_sold": {
"sum": {
"field": "quantity"
}
}
}
}
}
}
By running that query, besides the resulting hits from your search (in this case we are doing a match all), and additional object will be included, within the response object, holding the corresponding resulting aggregations. For example
{
...
"hits": {
"total": 6,
"max_score": 1,
"hits": [ ... ]
},
"aggregations": {
"group_by_product": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "b",
"doc_count": 3,
"units_sold": {
"value": 60
}
},
{
"key": "a",
"doc_count": 2,
"units_sold": {
"value": 20
}
},
{
"key": "c",
"doc_count": 1,
"units_sold": {
"value": 5
}
}
]
}
}
}
I omitted some details from the response object for brevity, and to highlight the important part, which is within the aggregations object. You can see how the aggregated data consists of different buckets, each representing the distinct product types (identified by the key key) that were found within your documents, doc_count has the number of occurrences per product type, and the unit_sold object, holds the total sum of units sold per each of the product types.
One important thing to keep into consideration is that in order to perform aggregations on string or text fields, you need to enable the fielddata setting within your field mapping, as that setting is disabled by default on all text based fields. In order to update the mapping, for ex. of the product code field, you just need to to a PUT request to the corresponding mapping type within the index, for example
PUT http://localhost:9200/sales/sale/_mapping
{
"properties": {
"productCode": {
"type": "string",
"fielddata": true
}
}
}
(more info about the fielddata setting)
My REST contorller:
#GetMapping("/test")
public Page<MyObject> pathParamTest(Pageable pageable) {
return myService.getPage(pageable);
}
I send a request like following:
localhost:8091/endpoint/test?page=0&size=3&sort=id&direction=DESC
It's my response from server:
{
"content": [
{
"id": 1
},
{
"id": 2
},
{
"id": 3
}
],
"last": true,
"totalPages": 1,
"totalElements": 3,
"first": true,
"sort": [
{
"direction": "ASC",
"property": "id",
"ignoreCase": false,
"nullHandling": "NATIVE",
"descending": false,
"ascending": true
}
],
"numberOfElements": 3,
"size": 3,
"number": 0
}
but the request has still direction = ASC.
How can I send to server direction = DESC?
And why response has a field "last" = true, because next page has one element more?
try
localhost:8091/endpoint/test?page=0&size=3&sort=id,DESC
from spring data rest 6.2. Sorting
curl -v
"http://localhost:8080/people/search/nameStartsWith?name=K&sort=name,desc"
sort Properties that should be sorted by in the format
property,property(,ASC|DESC). Default sort direction is ascending. Use
multiple sort parameters if you want to switch directions, e.g.
?sort=firstname&sort=lastname,asc.
I am currently using the Java API for Elasticsearch.
A suggestion query, returns multiple results 'Suggestion', which I want to be able to iterate over and access the variables. This is done by:
val builder = es.prepareSuggest("companies").addSuggestion(new CompletionSuggestionBuilder("companies").field("name_suggest").text(text).size(count()) )
val suggestResponse = builder.execute().actionGet()
val it = suggestResponse.getSuggest.iterator()
Now
while (it.hasNext()){
val info = it.next()
info.....
}
I need to be able to access info from the payloads from 'info'. An example of the return looks like:
{
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"companies": [
{
"text": "wells",
"offset": 0,
"length": 16,
"options": [
{
"text": "Wells Fargo",
"score": 123.0,
"payload": {
"industry_id": 130,
"id": 776,
"popularity": 123
}
},
{
"text": "Wells Real Estate Funds",
"score": 140.0,
"payload": {
"industry_id": 100,
"id": 778,
"popularity": 123
}
},
{
"text": "Wellstar Health System",
"score": 126.0,
"payload": {
"industry_id": 19,
"id": 1964,
"popularity": 123
}
}
]
}
]
}
When iterating through each suggestion, I seem unable to get the payload. Any ideas as to how I can do this?
To access the payloads, you have to iterate over the Option-Objects. (Right now, you are iterating over the suggestions)
import org.elasticsearch.search.suggest.Suggest.Suggestion
import org.elasticsearch.search.suggest.completion.CompletionSuggestion.Entry
...
// Get your "companies" suggestion
val suggestions: Suggestion[Entry] = suggestResponse.getSuggest.getSuggestion("companies")
val it = suggestions.getEntries.iterator()
// Iterate over all entries in the "company" suggestion
while(it.hasNext) {
val entry = it.next()
val optionsIt = entry.getOptions.iterator()
// Iterate over the options of the entry
while (optionsIt.hasNext) {
val option = optionsIt.next()
// As soon as you have the option-object, you can access the payload with various methods
val payloadAsMap = option.getPayloadAsMap
}
}
I found the information in the tests of ElasticSearch on Github