My requirement is to create a spring boot application controller/service that calls elastic search reactive apis with out using traditional model and repositories. i need to pass dynamic json object to create documents in elastic search using spring controller and return dynamic json as a result for queries.
Could you please help me with below questions
Does elastic search has reactive document apis for CRUDQ operations.if exists can we tie those to springboot application for CRUDQ operations without defining a model
I did check few blogs where we can use ReactiveElasticsearchClient (Elasticsearch APIs centric) but takes a model as input to the api. can we create an api without providing model and pass dynamic json for create and return dynamic object as query results.
Thank you in advance!!!
With the repository approach it may not be possible without the models. Having said that , the other way is to use the HTTP API using webflux to approach this problem.
As an example , I have set up a sample index with some data including an employee that matches this name
curl -XPOST "http://localhost:9200/_search" -H 'Content-Type: application/json' -d'{"query": {"match": {"name":"JohnSmith77067"}}}' |jq
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 3.205453,
"hits": [
{
"_index": "sample",
"_type": "employee",
"_id": "KbxD6IEBCsN8USu280mF",
"_score": 3.205453,
"_source": {
"id": null,
"organization": {
"id": 31,
"name": "TestO31",
"address": "Test Street No. 31"
},
"department": {
"id": 3631,
"name": "TestD3631"
},
"name": "JohnSmith77067",
"age": 66,
"position": "Developer"
}
}
]
}
}
will try and emulate this through a free query run through reactive web client
#RestController
#RequestMapping("/employees")
public class EmployeeController {
private static final Logger LOGGER = LoggerFactory.getLogger(EmployeeController.class);
private final WebClient client = WebClient.create();
#PostMapping("/customselect")
public Mono<String> generateMulti() throws URISyntaxException {
return client.post().uri(new URI("http://localhost:9200/_search")).
accept(MediaType.APPLICATION_JSON).
header("Content-Type", "application/json").
body(Mono.just("{\"query\": {\"match\": {\"name\":\"JohnSmith77067\"}}}"), String.class)
.retrieve()
.bodyToMono(String.class);
}
}
starting the app up and then hitting the endpoint will return similar results You could pass a custom query in the body , but you get the idea of this.
[![Postman results][1]][1]
Having a repository based approach takes away all the boilerplate and provides a lot of goodies & error handling through config driven approaches. Having said that if that does not suit , for ES the way would be to use the raw HTTP API (there are many things you could attach to the web clients such as Auth , timeout etc that i have done) and then take it from there. Hope this helps a bit
[1]: https://i.stack.imgur.com/FIAXR.png
Related
We're importing logs that contain the the full request/response for a given endpoint. Using the escLayout c# library. The import is working fine, however the 'structured' part of the log is stored under metadata, like so:
"metadata": {
"event": {
"controller": "main",
"method": "GetData",
"request": {
"userId": 1,
"clientTypeId": 2
},
"response": {
"marketOpen": true,
"price": 18.56
}
}
}
How do I go about querying this metadata as the fields do not appear in the 'Lens' page.
Is it a case of creating an index of some description? There are a lot of different (and occasionally large) data sets so this seems really impactable.
Is querying 'ad-hoc' data like this not a good use of Kibana? Should I look elsewhere, say Grafana before I go too far down the Elastic road?
Note: We're on Elastic 8.2.0
I have Kafka set-up running with the Elasticsearch connector and I am successfully indexing new documents into an ES index based on the incoming messages on a particular topic.
However, based on incoming messages on another topic, I need to append data to a field on a specific document in the same index.
Psuedo-schema below:
{
"_id": "6993e0a6-271b-45ef-8cf5-1c0d0f683acc",
"uuid": "6993e0a6-271b-45ef-8cf5-1c0d0f683acc",
"title": "A title",
"body": "A body",
"created_at": 164584548,
"views": []
}
^ This document is being created fine in ES based on the data in the topic mentioned above.
However, how do I then add items to the views field using messages from another topic. Like so:
article-view topic schema:
{
"article_id": "6993e0a6-271b-45ef-8cf5-1c0d0f683acc",
"user_id": 123456,
"timestamp: 136389734
}
and instead of simply creating a new document on the article-view index (which I dont' want to even have). It appends this to the views field on the article document with corresponding _id equal to article_id from the message.
so the end result after one message would be:
{
"_id": "6993e0a6-271b-45ef-8cf5-1c0d0f683acc",
"uuid": "6993e0a6-271b-45ef-8cf5-1c0d0f683acc",
"title": "A title",
"body": "A body",
"created_at": 164584548,
"views": [
{
"user_id": 123456,
"timestamp: 136389734
}
]
}
Using the ES API it is possible using a script. Like so:
{
"script": {
"lang": "painless",
"params": {
"newItems": [{
"timestamp": 136389734,
"user_id": 123456
}]
},
"source": "ctx._source.views.addAll(params.newItems)"
}
}
I can generate scripts like above dynamically in bulk, and then use the helpers.bulk function in the ES Python library to bulk update documents this way.
Is this possible with Kafka Connect / Elasticsearch? I haven't found any documentation on Confluent's website to explain how to do this.
It seems like a fairly standard requirement and an obvious thing people would need to do with Kafka / A sink connector like ES.
Thanks!
Edit: Partial updates are possible with write.method=upsert (src)
The Elasticsearch connector doesn't support this. You can update documents in-place but need to send the full document, not a delta for appending which I think it what you're after.
How to loop the paged Rest service with the Java DSL Http.outboundGatewaymethod?
The rest URL is for example
http://localhost:8080/people?page=3
and it returns for example
"content": [
{"name": "Mike",
"city": "MyCity"
},
{"name": "Peter",
"city": "MyCity"
},
...
]
"pageable": {
"sort": {
"sorted": false,
"unsorted": true
},
"pageSize": 20,
"pageNumber": 3,
"offset": 60,
"paged": true,
"unpaged": false
},
"last": false,
"totalElements": 250,
"totalPages": 13,
"first": false,
"sort": {
"sorted": false,
"unsorted": true
},
"number": 3,
"numberOfElements": 20,
"size": 20
}
where the variable totalPages tells the total pages amount.
So if the implementation
integrationFlowBuilder
.handle(Http
.outboundGateway("http://localhost:8080/people?page=3")
.httpMethod(HttpMethod.GET)
.expectedResponseType(String.class))
access one page, how to loop all the pages?
The easiest way to do this is like wrapping the call to this Http.outboundGateway() with the #MessagingGateway and provide a page number as an argument:
#MessagingGateway
public interface HttpPagingGateway {
#Gateway(requestChannel = "httpPagingGatewayChannel")
String getPage(int page);
}
Then you get a JSON as a result, where you can convert it into some domain model or just perform a JsonPathUtils.evaluate() (based on json-path) to get the value of the last attribute to be sure that you need to call that getPage() for the page++ or not.
The page argument is going to be a payload of the message to send and that can be used as an uriVariable:
.handle(Http
.outboundGateway("http://localhost:8080/people?page={page}")
.httpMethod(HttpMethod.GET)
.uriVariable("page", Message::getPayload)
.expectedResponseType(String.class))
Of course, we can do something similar with Spring Integration, but there are going to be involved filter, router and some other components.
UPDATE
First of all I would suggest you to create a domain model (some Java Bean), let's say PersonPageResult, to represent that JSON response and this type to the expectedResponseType(PersonPageResult.class) property of the Http.outboundGateway(). The RestTemplate together with the MappingJackson2HttpMessageConverter out-of-the-box will do the trick for you to return such an object as a reply for the downstream processing.
Then, as I said before, looping would be better done from some Java code, which you could wrap to the service activator call. For this purpose you should daclare a gateway like this:
public interface HttpPagingGateway {
PersonPageResult getPage(int page);
}
Pay attention: no annotations at all. The trick is done via IntegrationFlow:
#Bean
public IntegrationFlow httpGatewayFlow() {
return IntegrationFlows.from(HttpPagingGateway.class)
.handle(Http
.outboundGateway("http://localhost:8080/people?page={page}")
.httpMethod(HttpMethod.GET)
.uriVariable("page", Message::getPayload)
.expectedResponseType(PersonPageResult.class))
}
See IntegrationFlows.from(Class<?> aClass) JavaDocs.
Such a HttpPagingGateway can be injected into some service with hard looping logic:
int page = 1;
boolean last = false;
while(!last) {
PersonPageResult result = this.httpPagingGateway.getPage(page++);
last = result.getLast();
List<Person> persons = result.getPersons();
// Process persons
}
For processing those persons I would suggest to have separate IntegrationFlow, which may start from the gateway as well or you can just send a Message<List<Person>> to its input channel.
This way you will separate concerns about paging and processing and will have a simple loop logic in some POJO method.
I'm trying to return multiple "buckets" of results from Elasticsearch in one HTTP request.
I'm using the _msearch API.
I'm using the following query:
POST /_msearch
{"index" : "[INDEXNAME]", "type":"post"}
{"query" : {"match" : {"post_type":"team-member"}}, "from" : 0, "size" : 10}
{"index" : "[INDEXNAME]", "type": "post"}
{"query" : {"match" : {"post_type": "article"}}, "from" : 0, "size" : 10}
The query executes without error, but the results only return one object, where it seems it shoul be two (one for the 10 team-members, and one for the 10 articles):
{
"responses": [
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"failed": 0
},
"hits": {
"total": 191,
"max_score": 3.825032,
"hits": [
{...}
]
}
}, // second query should be here, no?
]
}
Is my query construction wrong, or am I misunderstanding how this should work?
The format of a _msearch request must follow the bulk API format. It must look something like this:
header\n
body\n
header\n
body\n
The header part includes which index / indices to search on, optional (mapping) types to search on, the search_type, preference, and routing. The body includes the typical search body request (including the query, aggregations, from, size, and so on).
NOTE: the final line of data must end with a newline character \n.
Make sure your query follows this format (from your code example, depending on the environment, as you've added two new lines after POST /_msearch, your query may or may not work; you should only add one new line) . If the responses array only has one result, then, in your case, the last query is somehow discarded - again, check its format.
I don't see any problem actually, but you should check "Bulk API", it's similar.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html
In my Spring-Data-Elasticsearch application, I am trying to use SearchQuery to search through Elasticsearch, according to some given QueryBuilder and FilterBuilder.
However, Elasticsearch docs talk about SearchResponse, which to me, seems to do the same work as SearchQuery.
I don't understand the difference between SearchQuery and SearchResponse.
Can someone please point out the difference?
If you pass the query object to an elasticsearch client and execute the query you get a response back.
The response type is dependent on the query type.
executed SearchQuery object -> SearchResponse object
executed IndexQuery object -> IndexResponse object
and so on...
In the code snippet of your link the SearchQuery object is build with the prepareSearch method. Afterwards it gets executed by the client.
SearchResponse response =
// Query creation part
client.prepareSearch("index1", "index2")
.setTypes("type1", "type2")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(QueryBuilders.termQuery("multi", "test"))
.setPostFilter(FilterBuilders.rangeFilter("age").from(12).to(18))
.setFrom(0).setSize(60).setExplain(true)
//query execution part
.execute()
.actionGet();
The search query is the query you send to Elastic, the search response is Elasticsearch's response to that query.
For example, this could be your query:
POST /your_index/_search
{
"query": {
"term": {
"available": {
"value": true
}
}
}
And a possible query response from ES:
{
"took": 99,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 58188,
"max_score": 0.99998283,
"hits": [
...
]
}
}