I have implemented Elastic Search - Java API to search data based on the distance.
QueryBuilder geoQuery = QueryBuilders.geoDistanceQuery("article_location.location").point(lat , lon).distance(5 , DistanceUnit.KILOMETERS);
QueryBuilders.boolQuery().filter(geoQuery);
However, I would like to get distance in each result from input lat/lon. Is there any way to specify that?
I see in elastic DSL we can do it via script_fields but do not know how to get it via Java API.
"script_fields": {
"distance": {
"script": "doc['latlong'].distanceInKm(lat, lon)"
}
Im using
elasticsearch-rest-high-level-client 7.9.2 version.
The Java client also allows you to add script fields to your query, simply like this:
sourceBuilder.query(...);
// add lat/lon as parameters
Map<String, Object> params = new HashMap<String, Object>();
params.put("lat", lat);
params.put("lon", lon);
// create the script
Script script = new Script(ScriptType.INLINE, "painless", "doc['latlong'].distanceInKm(params.lat, params.lon)", params);
// add the script field to the source query builder
sourceBuilder.scriptField("distance", script);
Related
I'm trying to learn Painless so that I could use it while trying to enrich and manipulate incoming documents. However, every way I've seen for accessing the document just results in errors.
Having input this in the Painless Lab in Kibana, these are the errors I'm getting:
def paths = new String[3];
paths[0]= '.com';
paths[1] = 'bar.com';
paths[2] = 'foo.bar.com';
doc['my_field'] = paths; // does not work: '[Ljava.lang.String; cannot be cast to org.elasticsearch.index.fielddata.ScriptDocValues'
ctx.my_field = paths; // does not compile: 'cannot resolve symbol [ctx.my_field]'
return doc['my_field'] == 'field_value'; // does not work: 'No field found for [my_field] in mapping'
doc['my_field'] == 'field_value' complains despite the field being present in the test document, though doc.containsKey('my_field') does return false.
How should I actually be accessing and manipulating the incoming document? I'm using ElasticSearch 7.12.
You can create ingest pipeline with set processor for adding hardcode value to incoming document.
{
"description" : "sets the value of count to 1",
"set": {
"field": "count",
"value": 1
}
}
There are very specific context available for painless API. you are using String[] which may be causing issue so you need to use either Arrays or ArraysList. you can check example of painless lab here.
Below is script i have tried in painless lab and it is working as expcted:
def ctx = params.ctx;
ArrayList paths = new ArrayList();
paths.add('.com');
paths.add('bar.com');
paths.add('foo.bar.com');
ctx['my_field'] = paths;
return ctx
Add below in parameters tab, i missed to add this in answer. this required because in actual implmentation you will get value from context and update context.
{
"ctx":{
"my_field":["test"]
}
}
I'm using spring data elastic search, Now my document do not have any static fields, and it is accumulated data per qtr, I will be getting ~6GB/qtr (we call them as versions). Lets say we get 5GB of data in Jan 2021 with 140 columns, in the next version I may get 130 / 120 columns, which we do not know, The end user requirement is to get the information from the database and show it in a tabular format, and he can filter the data. In MongoDB we have BasicDBObject, do we have anything in springboot elasticsearch
I can provide, let say 4-5 columns which are common in every version record and apart from that, I need to retrieve the data without mentioning the column names in the pojo, and I need to use filters on them just like I can do in MongoDB
List<BaseClass> getMultiSearch(#RequestBody Map<String, Object>[] attributes) {
Query orQuery = new Query();
Criteria orCriteria = new Criteria();
List<Criteria> orExpression = new ArrayList<>();
for (Map<String, Object> accounts : attributes) {
Criteria expression = new Criteria();
accounts.forEach((key, value) -> expression.and(key).is(value));
orExpression.add(expression);
}
orQuery.addCriteria(orCriteria.orOperator(orExpression.toArray(new Criteria[orExpression.size()])));
return mongoOperations.find(orQuery, BaseClass.class);
}
You can define an entity class for example like this:
public class GenericEntity extends LinkedHashMap<String, Object> {
}
To have that returned in your calling site:
public SearchHits<GenericEntity> allGeneric() {
var criteria = Criteria.where("fieldname").is("value");
Query query = new CriteriaQuery(criteria);
return operations.search(query, GenericEntity.class, IndexCoordinates.of("indexname"));
}
But notice: when writing data into Elasticsearch, the mapping for new fields/properties in that index will be dynamically updated. And there is a limit as to how man entries a mapping can have (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html). So take care not to run into that limit.
I am trying to build a recommendation system by integrating Elasticsearch with Apache Spark. I am using Java. I am using movilens dataset as example data. I have indexed the data to Elasticsearch as well. So far, I have been able to read the input from Elasticsearch index as follows:
SparkConf conf = new SparkConf().setAppName("Example App").setMaster("local");
conf.set("spark.serializer", org.apache.spark.serializer.KryoSerializer.class.getName());
conf.set("es.nodes", "localhost");
conf.set("es.port", "9200");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaPairRDD<String, Map<String, Object>> esRDD = JavaEsSpark.esRDD(sc, "movielens/recommendation");
Using esRDD.collect() function, I can see that I am retrieving the data from elastic search correctly. Now I need to feed the user id, item id and preference from the Elasticsearch result to Spark's recommendation. If I am using a csv file, I would be able to do it as follows:
String path = "resources/user_data.data";
JavaRDD<String> data = sc.textFile(path);
JavaRDD<Rating> ratings = data.map(
new Function<String, Rating>() {
public Rating call(String s) {
String[] sarray = s.split(" ");
return new Rating(Integer.parseInt(sarray[0]), Integer.parseInt(sarray[1]),
Double.parseDouble(sarray[2]));
}
}
);
What could be an equivalent mapping if I need to iterate through the elastic search output stored in esRDD and create a similar map as above? If there is any example code that I could refer to, that would be of great help.
Apologies for not answering the Spark question directly, but in case you missed it, there is a description of doing recommendations on MovieLens data using elasticsearch here: http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_significant_terms_demo.html
You have not specified the format of the data in ElasticSearch. But let's assume it has fields userId, movieId and rating so an example document looks something like {"userId":1,"movieId":1,"rating":4}.
Then you should be able to do (ignoring null checks etc):
JavaRDD<Rating> ratings = esRDD.map(
new Function<Map<String, Object>, Rating>() {
public Rating call(Map<String, Object> m) {
Int userId = Integer.parseInt(m.get("userId"));
Int movieId = Integer.parseInt(m.get("movieId"));
Double rating = Double.parseDouble(m.get("rating"));
return new Rating(userId, movieId, rating);
}
}
);
I am trying to update multiple value in index using Java Api through Elastic Search Script. But not able to update fields.
Sample code :-
1:
UpdateResponse response = request.setScript("ctx._source").setScriptParams(scriptParams).execute().actionGet();
2:
UpdateResponse response = request.setScript("ctx._source.").setScriptParams(scriptParams).execute().actionGet();
if I mentioned .(dot) in ("ctx._source.") getting illegalArgument Exception and if i do not use dot, not getting any exception but values not getting updated in Index.
Can any one tell me the solutions to resolve this.
First of all, your script (ctx._source) doesn't do anything, as one of the commenters already pointed out. If you want to update, say, field "a", then you would need a script like:
ctx._source.a = "foobar"
This would assign the string "foobar" to field "a". You can do more than simple assignment, though. Check out the docs for more details and examples:
http://www.elasticsearch.org/guide/reference/api/update/
Updating multiple fields with one script is also possible. You can use semicolons to separate different MVEL instructions. E.g.:
ctx._source.a = "foo"; ctx._source.b = "bar"
In Elastic search have an Update Java API. Look at the following code
client.prepareUpdate("index","typw","1153")
.addScriptParam("assignee", assign)
.addScriptParam("newobject", responsearray)
.setScript("ctx._source.assignee=assignee;ctx._source.responsearray=newobject ").execute().actionGet();
Here, assign variable contains object value and response array variable contains list of data.
You can do the same using spring java client using the following code. I am also listing the dependencies used in the code.
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.index.query.QueryBuilder;
import org.springframework.data.elasticsearch.core.query.UpdateQuery;
import org.springframework.data.elasticsearch.core.query.UpdateQueryBuilder;
private UpdateQuery updateExistingDocument(String Id) {
// Add updatedDateTime, CreatedDateTime, CreateBy, UpdatedBy field in existing documents in Elastic Search Engine
UpdateRequest updateRequest = new UpdateRequest().doc("UpdatedDateTime", new Date(), "CreatedDateTime", new Date(), "CreatedBy", "admin", "UpdatedBy", "admin");
// Create updateQuery
UpdateQuery updateQuery = new UpdateQueryBuilder().withId(Id).withClass(ElasticSearchDocument.class).build();
updateQuery.setUpdateRequest(updateRequest);
// Execute update
elasticsearchTemplate.update(updateQuery);
}
XContentType contentType =
org.elasticsearch.client.Requests.INDEX_CONTENT_TYPE;
public XContentBuilder getBuilder(User assign){
try {
XContentBuilder builder = XContentFactory.contentBuilder(contentType);
builder.startObject();
Map<String,?> assignMap=objectMap.convertValue(assign, Map.class);
builder.field("assignee",assignMap);
return builder;
} catch (IOException e) {
log.error("custom field index",e);
}
IndexRequest indexRequest = new IndexRequest();
indexRequest.source(getBuilder(assign));
UpdateQuery updateQuery = new UpdateQueryBuilder()
.withType(<IndexType>)
.withIndexName(<IndexName>)
.withId(String.valueOf(id))
.withClass(<IndexClass>)
.withIndexRequest(indexRequest)
.build();
I want to try new solr collapsing/grouping included in solr 3.3, i have tried queries on solr Admin page and that works absolutely right but when I try to query in my c# code using solr net that does not seem to work as expected. Here is how I am setting the param values
options.ExtraParams = new List<KeyValuePair<string, string>>
{
new KeyValuePair<string,string>("group","true"),
new KeyValuePair<string,string>("group.field","AuthorID"),
};
Yes, you can use Grouping (formerly known as Field Collapsing) with SolrNet, it was introduced in the SolrNet 0.4.0 alpha1 release. Here are the release notes on the author's blog about this support being added in. So you will need to grab that version (or later) from Google Code(binaries) or GitHub(source). Also here is an example of using grouping from the unit tests in the source - Grouping Tests
public void FieldGrouping()
{
var solr = ServiceLocator.Current.GetInstance<ISolrBasicOperations<Product>>();
var results = solr.Query(SolrQuery.All, new QueryOptions
{
Grouping = new GroupingParameters()
{
Fields = new [] { "manu_exact" },
Format = GroupingFormat.Grouped,
Limit = 1,
}
});
Console.WriteLine("Group.Count {0}", results.Grouping.Count);
Assert.AreEqual(1, results.Grouping.Count);
Assert.AreEqual(true, results.Grouping.ContainsKey("manu_exact"));
Assert.GreaterThanOrEqualTo(results.Grouping["manu_exact"].Groups.Count,1);
}