Using Mustache API to parse Elasticsearch JSON Template requests - elasticsearch

I have been using the SearchTemplateRequest class to execute my requests which uses Mustache templating to parse my template string with the passed parameters.
Elasticsearch Template - Converting Parameters to JSON
However, I have to change my implementation where I will be switching to the Java Low-Level Client. I want to use the Mustache implementation that SearchTemplateRequest uses internally to parse the template.
I'm okay to use the Mustache dependency or use the Elasticsearch implementation of it. Could someone help me out here?
My Template String:
{
"query": {
"bool": {
"filter": "{{#toJson}}clauses{{/toJson}}"
}
}
}
My Params Object:
{
"clauses": [
{
"term": {
"field1": "field1Value"
}
}
]
}
My test code:
StringWriter writer = new StringWriter();
MustacheFactory mustacheFactory = new DefaultMustacheFactory();
mustacheFactory.compile(new StringReader(requestTemplate), "templateName").execute(writer, params);
writer.flush();
The above code returns me the request template string with empty strings replacing the template.
Returned Response:
{
"query": {
"bool": {
"filter": ""
}
}
}
Expected Response:
{
"query": {
"bool": {
"filter": [
{
"term": {
"field1": "field1Value"
}
}
]
}
}
}

I finally figured out the solution.
import org.elasticsearch.script.Script;
import org.elasticsearch.script.ScriptEngine;
import org.elasticsearch.script.TemplateScript;
import org.elasticsearch.script.mustache.MustacheScriptEngine;
import java.util.Map;
import static java.util.Collections.singletonMap;
public class CustomMustacheScriptEngine {
private final String JSON_MIME_TYPE_WITH_CHARSET = "application/json; charset=UTF-8";
private final String JSON_MIME_TYPE = "application/json";
private final String PLAIN_TEXT_MIME_TYPE = "text/plain";
private final String X_WWW_FORM_URLENCODED_MIME_TYPE = "application/x-www-form-urlencoded";
private final String DEFAULT_MIME_TYPE = JSON_MIME_TYPE;
private final Map<String, String> params = singletonMap(Script.CONTENT_TYPE_OPTION, JSON_MIME_TYPE_WITH_CHARSET);
public String compile(String jsonScript, final Map<String, Object> scriptParams) {
jsonScript = jsonScript.replaceAll("\"\\{\\{#toJson}}", "{{#toJson}}").replaceAll("\\{\\{/toJson}}\"", "{{/toJson}}");
final ScriptEngine engine = new MustacheScriptEngine();
TemplateScript.Factory compiled = engine.compile("ScriptTemplate", jsonScript, TemplateScript.CONTEXT, params);
TemplateScript executable = compiled.newInstance(scriptParams);
String renderedJsonScript = executable.execute();
return renderedJsonScript;
}
}

Related

Using #Query to access a data object?

I have a Spring Elastic Repository function that has way too many parameters (11 parameters).
I want to be able to access members of class objects to reduce the parameter count. Is there a way to access class members via #Query? For example, I want to access members of the following class:
public class Header {
int version;
string timestamp;
}
and want to do a query as follows:
#Query("""
{
"bool":{
"must":[
{ "match": { "header_version": {"query": "<header version substitution>"}}},
{ "match": { "header_timestamp": {"query": "<header timestamp substitution>"}}},
{ "match": { "body": {"query": "<body value>"}}}
]
}
}
""")
List<Result> findEntry(Header header, String body);
Where:
public class Result {
int version;
String timestamp;
String body;
}

restful webservice get list of objects

My rest webservice returns the following output:
{
"result": {
"TICKET1": {
"number": "TICKET1",
"description": "aa"
},
"TICKET2": {
"number": "TICKET2",
"description": "dd"
}
}
}
To convert this into a list of Tickets I tried as below.
class TicketResponse {
private List<Ticket> result;
// Get Set
}
class Ticket {
private String number;
private String description;
// Get Set
}
TicketResponse response = restTemplate.getForObject(WEB_SERVICE_URL, TicketResponse.class);
But I get response as null. How to do this.
I'll provide two ways to do with the JSON structure you have.
Option 1:
Modify your TicketResponse class like below:
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.stream.Collectors;
import com.fasterxml.jackson.annotation.JsonAnySetter;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
public class TicketResponse {
#JsonProperty("result")
private Map<String, Ticket> ticketsMap = new HashMap<>();
#JsonAnySetter
public void setUnknownField(String name, Ticket value) {
ticketsMap.put(name, value);
}
#JsonIgnore private List<Ticket> ticketsList;
public List<Ticket> getTicketsList() {
return ticketsMap.entrySet().stream().map(Entry::getValue).collect(Collectors.toList());
}
}
then you can get your list of tickets from:
response.getTicketsList();
Option 2:
Read your response in to a String
String response = restTemplate.getForObject(WEB_SERVICE_URL, String.class);
and use below code to convert it to a List<Ticket>
ObjectMapper mapper = new ObjectMapper();
JsonNode jsonNode = mapper.readTree(response);
JsonNode wantedJsonNode = jsonNode.get("result");
Map<String, Ticket> map =
mapper.convertValue(wantedJsonNode, new TypeReference<Map<String, Ticket>>() {});
List<Ticket> tickets =
map.entrySet().stream().map(Entry::getValue).collect(Collectors.toList());
The object you provided doesn't contain a list/array, which would be inside square brackets, like this:
{
"result": {
"tickets": [
{
"number": "TICKET1",
"description": "aa"
},
{
"number": "TICKET2",
"description": "dd"
}
]
}
}
Change your service if possible to return a list/array. Otherwise what you have is an object with individual fields named TICKET1 and TICKET2, so you'll need a field for each.
TicketResponse must have a structure that corresponds to response of the service.
You can change your TicketResponse class and add getTicketArray method:
public class TicketResponse {
private Map<String,Ticket> result;
// getter setter
public List<Ticket> getTicketsAsArray(){
return new ArrayList<Ticket>(result.values());
}
}

Get data as a JSON format in spring boot

I want to build a request endepoints using spring boot: I have to consume restful api and convert that into another rest endpoint.
I have a json Response on www.exampleapiurl.com/details
[{
"name": "age",
"value": "Child"
},
{
"name": "recommendable",
"value": true
},
{
"name": "supported",
"value": yes
},
]
[{
"name": "age",
"value": "Adult"
},
{
"name": "recommendable",
"value": true
},
{
"name": "supported",
"value": no
},
]
I want the response to be:
[{
"age": "Child"
},
{
"recommendable": true
},
{
"supported": "yes"
},
]
[{
"age": "Adult"
},
{
"recommendable": true
},
{
"supported": "no"
},
]
For this I have a attribute class with getter and setter:
Attributes.class
#JsonIgnoreProperties(ignoreUnknown = true)
public class Attributes {
private String age;
private boolean recommendable;
private String supported;
getter and setter for these:
}
This is my service.java class
#Service
public class CService {
private static RestTemplate restTemplate;
public String url;
#Autowired
public CService(String url) {
this.url = url;
}
public Attributes getAttributes() {
HttpHeaders headers= new HttpHeaders();
headers.add("Authorization", "some value");
HttpEntity<String> request = new HttpEntity<String>(headers);
ResponseEntity<String> response = restTemplate.exchange(url, HttpMethod.GET, request, Attributes.class);
return response.getBody();
}
}
And this is my controller.class
#Controller
public class CController {
private CService cService;
#Autowired
public CController(CService cService) {
this.cService = cService;
}
#RequestMapping(value="/example")
#ResponseBody
public Attributes getCAttributes() {
return cService.getAttributes(); }
}
The Authorization is successful but,
I am not getting any response for now
What you can do is create a model class to recive the response from example API
as follows.
#JsonIgnoreProperties(ignoreUnknown = true)
public class Details{
private String name;
private String value;
#JsonProperty("value")
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
#JsonProperty("name")
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
Then change your invoking RestTemplate code as follows
Details[] details = null;
//Keeps the example API's data in array.
ResponseEntity<Details[]> response = restTemplate.exchange(url, HttpMethod.GET, request, Details[].class);
details = response.getBody();
//Next step is to process this array and send the response back to your client
List<Attributes> attributes = new ArrayList<Attributes>();
Attributes attr = null;
for(Details detail : details) {
attr = new Attributes();
//set the values here
}
//returns the attributes here
attributes.toArray(new Attributes[attributes.size()]);

Custom sorting using script in Elasticsearch

I want to make use of the scripting to sort the results in the elasticsearch with custom logic. I have read the docs from the elasticsearch and could not make it up. After seeing some links on the internet, I tried a bit and below is the source code for the same. I am using native (Java) for it. I am not sure whether this is the correct approach.
import org.elasticsearch.script.ExecutableScript;
import org.elasticsearch.script.NativeScriptFactory;
import customSortProject.Sorted;
import java.util.Map;
public class CustomScriptFactory implements NativeScriptFactory {
public ExecutableScript newScript(Map<String, Object> params) {
return new Sorted(params);
}
}
And the class where i am implementing the logic for the sore. Currently I am just getting the future dates.
import java.util.ArrayList;
import java.util.Collections;
import java.util.Date;
import java.util.List;
import java.util.Map;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.script.AbstractSearchScript;
public class Sorted extends AbstractSearchScript {
String fieldParam;
int lengthParam;
public Sorted(#Nullable Map<String,Object> params){
fieldParam = (String)params.get("field");
lengthParam = new Integer(params.get("length").toString()).intValue();
}
public Object run() {
if(source().containsKey(fieldParam) && source().get(fieldParam)!= null && source().get(fieldParam).toString() != null) {
String field = doc().get(fieldParam).toString();
field = field.replaceAll("\\[", "").replaceAll("\\]","");
long fieldLong = 0;
Date today = new Date();
fieldLong = Long.parseLong(field);
Date date = new Date(fieldLong);
List<Date> futureList = new ArrayList<Date>();
if (date.after(today))
futureList.add(date);
Collections.sort(futureList);
return futureList;
}
else {
return "";
}
}
}
With this logic and using the query_dsl where I am trying to call this script which is register in .yml file.
Query :
{
"query": {
"match": {
"title": "cancer"
}
},
"sort": {
"_script": {
"script": "sorted",
"lang": "native",
"type": "string",
"ignore_unmapped": true,
"params": {
"field": "startdate",
"length": 6
}
}
}
}
Please let me know is this correct approach to custom sorting. I want to call it from the query dsl, as our application is in PHP and we are using PHP's es-client to search.

How to use Elasticsearch plugin-defined filter

I have created a plugin for Elasticsearch and have installed it successfully (http://localhost:9200/_nodes/plugins/ shows it installed.) But I can't seem to use it in my queries - I only get errors. "ScriptException[dynamic scripting for [groovy] disabled]". It seems like I need a different lang setting. But I've tried 'lang': 'java'. No joy. I've tried lang: expression. Then I get "ExpressionScriptCompilationException[Unknown variable [maxmind] in expression". How do I access the plugin I've created? Or do I need to do something more to register it?
I've been following this excellent guide:
https://github.com/imotov/elasticsearch-native-script-example
But it says nothing about how queries should be written.
My AbstractPlugin:
package org.elasticsearch.plugin.maxmind;
import java.util.Collection;
import org.elasticsearch.common.collect.Lists;
import org.elasticsearch.common.inject.Module;
import org.elasticsearch.plugins.AbstractPlugin;
import org.elasticsearch.script.ScriptModule;
import org.elasticsearch.plugin.maxmind.GeoLoc;
public class MaxMind extends AbstractPlugin {
#Override public String name() {
return "maxmind";
}
#Override public String description() {
return "Plugin to annotate ip addresses with maxmind geo data";
}
// Thanks https://github.com/imotov/elasticsearch-native-script-example
public void onModule(ScriptModule module) {
module.registerScript("geoloc", GeoLoc.Factory.class);
}
}
Note the name "geoloc". Is that the name I use in my query?
My GeoLoc module:
package org.elasticsearch.plugin.maxmind;
import java.util.HashMap;
import java.util.Map;
import org.elasticsearch.script.ScriptException;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.xcontent.support.XContentMapValues;
import org.elasticsearch.index.fielddata.ScriptDocValues;
import org.elasticsearch.script.AbstractSearchScript;
import org.elasticsearch.script.ExecutableScript;
import org.elasticsearch.script.NativeScriptFactory;
public class GeoLoc extends AbstractSearchScript {
public static class Factory implements NativeScriptFactory {
// called on every search on every shard
#Override
public ExecutableScript newScript
(#Nullable Map<String, Object> params)
{
String fieldName = params == null ? null:
XContentMapValues.nodeStringValue(params.get("field"), null);
if (fieldName == null) {
throw new ScriptException("Missing field parameter");
}
return new GeoLoc(fieldName);
}
}
private final String fieldName;
private GeoLoc(String fieldName) {
this.fieldName = fieldName;
}
#Override
public Object run() {
ScriptDocValues docValue = (ScriptDocValues) doc().get(fieldName);
if (docValue != null && !docValue.isEmpty()) {
// TODO: real geolocation here
HashMap fakeloc = new HashMap<String, String>();
fakeloc.put("lat", "1.123");
fakeloc.put("lon", "44.001");
fakeloc.put("basedon", docValue);
return fakeloc;
}
return false;
}
}
My query:
{
"_source": [
"uri",
"user_agent",
"server_ip",
"server_port",
"client_ip",
"client_port"
],
"query": {
"filtered": {
"filter": {}
}
},
"script_fields": {
"test1": {
"params": {
"field": "client_ip"
},
"script": "geoloc" // is this right?
}
},
"size": 1
}
You should be able to specify lang: "native" with your script, any script written in Java and registered with registerScript is the "native" type.

Resources