In my Spring-Data-Elasticsearch application, I am trying to use SearchQuery to search through Elasticsearch, according to some given QueryBuilder and FilterBuilder.
However, Elasticsearch docs talk about SearchResponse, which to me, seems to do the same work as SearchQuery.
I don't understand the difference between SearchQuery and SearchResponse.
Can someone please point out the difference?
If you pass the query object to an elasticsearch client and execute the query you get a response back.
The response type is dependent on the query type.
executed SearchQuery object -> SearchResponse object
executed IndexQuery object -> IndexResponse object
and so on...
In the code snippet of your link the SearchQuery object is build with the prepareSearch method. Afterwards it gets executed by the client.
SearchResponse response =
// Query creation part
client.prepareSearch("index1", "index2")
.setTypes("type1", "type2")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(QueryBuilders.termQuery("multi", "test"))
.setPostFilter(FilterBuilders.rangeFilter("age").from(12).to(18))
.setFrom(0).setSize(60).setExplain(true)
//query execution part
.execute()
.actionGet();
The search query is the query you send to Elastic, the search response is Elasticsearch's response to that query.
For example, this could be your query:
POST /your_index/_search
{
"query": {
"term": {
"available": {
"value": true
}
}
}
And a possible query response from ES:
{
"took": 99,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 58188,
"max_score": 0.99998283,
"hits": [
...
]
}
}
Related
My requirement is to create a spring boot application controller/service that calls elastic search reactive apis with out using traditional model and repositories. i need to pass dynamic json object to create documents in elastic search using spring controller and return dynamic json as a result for queries.
Could you please help me with below questions
Does elastic search has reactive document apis for CRUDQ operations.if exists can we tie those to springboot application for CRUDQ operations without defining a model
I did check few blogs where we can use ReactiveElasticsearchClient (Elasticsearch APIs centric) but takes a model as input to the api. can we create an api without providing model and pass dynamic json for create and return dynamic object as query results.
Thank you in advance!!!
With the repository approach it may not be possible without the models. Having said that , the other way is to use the HTTP API using webflux to approach this problem.
As an example , I have set up a sample index with some data including an employee that matches this name
curl -XPOST "http://localhost:9200/_search" -H 'Content-Type: application/json' -d'{"query": {"match": {"name":"JohnSmith77067"}}}' |jq
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 3.205453,
"hits": [
{
"_index": "sample",
"_type": "employee",
"_id": "KbxD6IEBCsN8USu280mF",
"_score": 3.205453,
"_source": {
"id": null,
"organization": {
"id": 31,
"name": "TestO31",
"address": "Test Street No. 31"
},
"department": {
"id": 3631,
"name": "TestD3631"
},
"name": "JohnSmith77067",
"age": 66,
"position": "Developer"
}
}
]
}
}
will try and emulate this through a free query run through reactive web client
#RestController
#RequestMapping("/employees")
public class EmployeeController {
private static final Logger LOGGER = LoggerFactory.getLogger(EmployeeController.class);
private final WebClient client = WebClient.create();
#PostMapping("/customselect")
public Mono<String> generateMulti() throws URISyntaxException {
return client.post().uri(new URI("http://localhost:9200/_search")).
accept(MediaType.APPLICATION_JSON).
header("Content-Type", "application/json").
body(Mono.just("{\"query\": {\"match\": {\"name\":\"JohnSmith77067\"}}}"), String.class)
.retrieve()
.bodyToMono(String.class);
}
}
starting the app up and then hitting the endpoint will return similar results You could pass a custom query in the body , but you get the idea of this.
[![Postman results][1]][1]
Having a repository based approach takes away all the boilerplate and provides a lot of goodies & error handling through config driven approaches. Having said that if that does not suit , for ES the way would be to use the raw HTTP API (there are many things you could attach to the web clients such as Auth , timeout etc that i have done) and then take it from there. Hope this helps a bit
[1]: https://i.stack.imgur.com/FIAXR.png
I'm trying to run a pretty simple EQL query using the Elasticsearch Java API.
I'm running this using scala, and my query is:
val response = connection.client.eql().search(
new EqlSearchRequest.Builder()
.index("interactions")
.timestampField("start_time")
.eventCategoryField("info.node_type")
.query("sequence by sequence_id [any where info.node_type == \"DType\"] [any where info.node_type == \"RType\"] ")
.build(),
classOf[Interaction]
)
Putting in a breakpoint, and looking at the request, the request looks like a POST to http://localhost:9200/interactions/_eql/search of:
{
"event_category_field":"info.node_type",
"query":"sequence by id [any where info.node_type == \"DType\"] [any where info.node_type == \"RType\"] ",
"timestamp_field":"start_time"
}
If I submit this query manually using Postman, I get back a perfectly good response:
{
"is_partial": false,
"is_running": false,
"took": 9,
"timed_out": false,
"hits": {
"total": {
"value": 10,
"relation": "eq"
},
"sequences": [
{
"join_keys": [
0
],
"events": [
...
But when submitted through the Java API, I get an error:
Exception in thread "main" co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch.eql.HitsSequence: java.lang.UnsupportedOperationException (JSON path: hits.sequences[0].join_keys[0]) (line no=1, column no=140, offset=-1)
at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:125)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:320)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:280)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:85)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:280)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56)
at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:328)
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:294)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.eql.ElasticsearchEqlClient.search(ElasticsearchEqlClient.java:184)
at <my code>
Caused by: java.lang.UnsupportedOperationException
at co.elastic.clients.json.JsonpDeserializer$2.deserialize(JsonpDeserializer.java:117)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
... 29 more
The problem seems to be in the join_keys key - A hit sequence doesn't recognize that as a possible property.
Does anyone know what I'm doing wrong?
Thanks,
-Nathan
I'm trying to return multiple "buckets" of results from Elasticsearch in one HTTP request.
I'm using the _msearch API.
I'm using the following query:
POST /_msearch
{"index" : "[INDEXNAME]", "type":"post"}
{"query" : {"match" : {"post_type":"team-member"}}, "from" : 0, "size" : 10}
{"index" : "[INDEXNAME]", "type": "post"}
{"query" : {"match" : {"post_type": "article"}}, "from" : 0, "size" : 10}
The query executes without error, but the results only return one object, where it seems it shoul be two (one for the 10 team-members, and one for the 10 articles):
{
"responses": [
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"failed": 0
},
"hits": {
"total": 191,
"max_score": 3.825032,
"hits": [
{...}
]
}
}, // second query should be here, no?
]
}
Is my query construction wrong, or am I misunderstanding how this should work?
The format of a _msearch request must follow the bulk API format. It must look something like this:
header\n
body\n
header\n
body\n
The header part includes which index / indices to search on, optional (mapping) types to search on, the search_type, preference, and routing. The body includes the typical search body request (including the query, aggregations, from, size, and so on).
NOTE: the final line of data must end with a newline character \n.
Make sure your query follows this format (from your code example, depending on the environment, as you've added two new lines after POST /_msearch, your query may or may not work; you should only add one new line) . If the responses array only has one result, then, in your case, the last query is somehow discarded - again, check its format.
I don't see any problem actually, but you should check "Bulk API", it's similar.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html
I want to know the time taken by count query in elastic search, just like the search query query which contain took - time taken.
My Query looks like -
curl -XGET "http://localhost:9200/index1/type1/_count"
And result for that query -
{
"count": 136,
"_shards": {
"total": 15,
"successful": 15,
"failed": 0
}
}
Is there is any way so that I can get the time taken for count query just like search api?
Document for count API - Count API
At the time of writing this answer still its not supported by Elastic, raised a feature request and mostly I will work on to add a support of it.
A trick that can help with that is to use _search
with size zero (so no restult will be returned);
track_total_hits set to true (so it will count all hits, not only the ones in the result window); and
filter_path equal to took,htis.total.value.
For example, I executed the query above in a cluster of mine...
GET viagens-*/_search?filter_path=took,hits.total.value
{
"size": 0,
"track_total_hits": true,
"query": {
"match_all": {}
}
}
...and got this result:
{
"took": 2,
"hits": {
"total": {
"value": 2589552
}
}
}
It does not profile the Count API itself, unfortunately, but has a similar result. Can be very useful as an alternative in some situations!
I have following elastic search query
SearchResponse response = DbContext.INSTANCE.client()
.prepareSearch(indexList)
.setTypes("docs")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(queryBuilder)
.setFilter(FilterBuilders.andFilter(f1, f2))
.addFacet(FacetBuilders.termsFacet("f").field("frm").size(10))
.setFrom(pageNumber).setSize(size).execute().actionGet();
Facet count return by this query is global, how to get the facet for the search response?
I found the answer: What I done is add filter to the main query
QueryBuilder queryBuilder = QueryBuilders.boolQuery()
.must(QueryBuilders.queryString(criterion.getFullText()).field("text"))
.must(QueryBuilders.termsQuery("from", criterion.getFrom().toArray((new String[0]))))
.must(QueryBuilders.termsQuery("to", criterion.getTo().toArray((new String[0]))))
.must(QueryBuilders.rangeQuery("date").from(dtf.print(criterion.getFromDate())).to(dtf.print(criterion.getToDate())));
SearchResponse response =
DbContext.INSTANCE.client()
.prepareSearch(indexList)
.setTypes("docs")
.setQuery(queryBuilder)
.addFacet(fb)
.addFacet(tf)
.addSort("date", SortOrder.DESC)
.setFrom(pageNumber).setSize(size).execute().actionGet();
This is what i did unless you want "main" search and facet has different search requirement.
global=false should do the trick for you
"facets": {
"volume_by_day": {
"date_histogram": {
"field": "crdate",
"interval": "day"
},
"global": "false"
}
}