My resolver template
{
"version" : "2017-02-28",
"operation" : "Query",
"query" : {
"expression" : "receiverusername = :receiverusername and createdat > :createdat",
"expressionValues" : {
":receiverusername" : $util.dynamodb.toDynamoDBJson($context.identity.username),
":createdat" : $util.dynamodb.toDynamoDBJson($ctx.args.input.lastDate),
}
},
"index" : "receiverusername-createdat-index",
}
I am trying to query the date and username but I keep on getting this error
{
"data": {
"listMyMessages": null
},
"errors": [
{
"path": [
"listMyMessages"
],
"data": null,
"errorType": "DynamoDB:DynamoDbException",
"errorInfo": null,
"locations": [
{
"line": 2,
"column": 3,
"sourceName": null
}
],
"message": "One or more parameter values were invalid: Condition parameter type does not match schema type (Service: DynamoDb, Status Code: 400, Request ID: 7I18ODEK46H52NMBSF99OTNSQ7VV4KQNSO5AEMVJF66Q9ASUAAJG)"
}
]
How do I query this ?
I tried to learn from documentations but nothing gives me any clue!
Querying 2 values at the same time is not possible for DynamoDB, since every GSI creates a duplicate table with different index. A good solution I can think of is querying indexes in parallel and returning their intersaction.
I just did this and it worked.
And yes I am not touching or investigating why.
{
"version" : "2017-02-28",
"operation" : "Query",
"index" : "receiverusername-createdat-index",
"query" : {
## Provide a query expression. **
"expression": "receiverusername = :receiverusername and createdat > :createdat",
"expressionValues" : {
":receiverusername" : {
"S" : "${context.identity.username}"
},
":createdat": {
"S": "${ctx.args.lastDate}"
}
}
}
}
I did not use sort key but just index keys
Related
Opensearch ingests documents similar to this example (its just a minimal example):
PUT nested_test/_doc/4
{
"log": "This is a fourth log message",
"function": "4 test function",
"related_objects": [
{ "type": "user", "id": "10" },
{ "type": "offer", "id": "120" }
]
}
PUT nested_test/_doc/5
{
"log": "This is a fifth log message",
"function": "5 test function",
"related_objects": [
{ "type": "user", "id": "120" },
{ "type": "offer", "id": "90" }
]
}
With many of these documents, I'd like to filter those which have a specific related object (e.g. type=user and id=120). With the example data above, this should only return the document with id 5. Using simple filters (DQL syntax) as follows does not work:
related_objects.type:user and related_objects.id:120
As this would also match a document 5, as there is a related_object with type user and a related object with id 120, although its not the related user object with id 120, its the related offer.
If Array[object] is used, the field type is nested, The document reference
Elasticsearch query example:
{
"query" : {
"nested" : {
"path" : "related_objects",
"query" : {
"bool" : {
"must" : [
{
"term" : {"related_objects.type" : "MYTYPE"}
},
{
"term" : {"related_objects.id" : "MYID"}
}
]
}
}
}
}
}
Basically just go into a nested query and specify all your AND conditions as MUST clauses inside a bool query.
As soon as the field is declared as nested field, it is possible to run a simple DQL query to get the desired information:
related_objects:{type:"user" and id:120}
This requires that the field has been defined as nested before:
PUT my-index-000001
{
"mappings": {
"properties": {
"related_objects": {
"type": "nested"
}
}
}
}
this is part of my elasticsearch query:
{
"match": {
"value": {
"query": "1",
}
}
}
value is a boolean field in my index, does elasticsearch accept 1 as true and 0 as false in search on the boolean field?
as specified in the doc ES accept true/"true"/false/"false" as boolean values.
Other values like 0/1 will throw error in recent versions
error: failed to create query: Can't parse boolean value 1, expected
[true] or [false]
NB: you should use a term query to filter in boolean field
This is easy to test...
First create the index:
PUT test
{
"mappings": {
"properties": {
"bool_field": {
"type": "boolean"
}
}
}
}
Then index a document:
PUT test/_doc/1
{
"bool_field": true
}
Try to query using 0/1 instead of a boolean
POST test/_search
{
"query": {
"term": {
"bool_field": "1"
}
}
}
Response: Can't parse boolean value [1], expected [true] or [false]
{
"error" : {
"root_cause" : [
{
"type" : "query_shard_exception",
"reason" : "failed to create query: Can't parse boolean value [1], expected [true] or [false]",
"index_uuid" : "bZpN3j1kT9KtMBnGkpOmKQ",
"index" : "test"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "test",
"node" : "CyVrqrOtR0CP3RfZtdBTag",
"reason" : {
"type" : "query_shard_exception",
"reason" : "failed to create query: Can't parse boolean value [1], expected [true] or [false]",
"index_uuid" : "bZpN3j1kT9KtMBnGkpOmKQ",
"index" : "test",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Can't parse boolean value [1], expected [true] or [false]"
}
}
}
]
},
"status" : 400
}
PS: It used to be possible up to ES 5. As of ES 6, only true/false are accepted values for boolean fields.
Gist: Trying to write a custom filter on nested documents using painless. Want to write error checks when there are no nested documents to surpass null_pointer_exception
I have a mapping as such (simplified and obfuscated)
{
"video_entry" : {
"aliases" : { },
"mappings" : {
"properties" : {
"captions_added" : {
"type" : "boolean"
},
"category" : {
"type" : "keyword"
},
"is_votable" : {
"type" : "boolean"
},
"members" : {
"type" : "nested",
"properties" : {
"country" : {
"type" : "keyword",
},
"date_of_birth" : {
"type" : "date",
}
}
}
}
Each video_entry document can have 0 or more members nested documents.
Sample Document
{
"captions_added": true,
"category" : "Mental Health",
"is_votable: : true,
"members": [
{"country": "Denmark", "date_of_birth": "1998-04-04T00:00:00"},
{"country": "Denmark", "date_of_birth": "1999-05-05T00:00:00"}
]
}
If one or more nested document exist, we want to write some painless scripts that'd check certain fields across all the nested documents. My script works on mappings with a few documents but when I try it on larger set of documents I get null pointer exceptions despite having every null check possible. I've tried various access patterns, error checking mechanisms but I get exceptions.
POST /video_entry/_search
{
"query": {
"script": {
"script": {
"source": """
// various NULL checks that I already tried
// also tried short circuiting on finding null values
if (!params['_source'].empty && params['_source'].containsKey('members')) {
def total = 0;
for (item in params._source.members) {
// custom logic here
// if above logic holds true
// total += 1;
}
return total > 3;
}
return true;
""",
"lang": "painless"
}
}
}
}
Other Statements That I've Tried
if (params._source == null) {
return true;
}
if (params._source.members == null) {
return true;
}
if (!ctx._source.contains('members')) {
return true;
}
if (!params['_source'].empty && params['_source'].containsKey('members') &&
params['_source'].members.value != null) {
// logic here
}
if (doc.containsKey('members')) {
for (mem in params._source.members) {
}
}
Error Message
&& params._source.members",
^---- HERE"
"caused_by" : {
"type" : "null_pointer_exception",
"reason" : null
}
I've looked into changing the structure (flattening the document) and the usage of must_not as indicated in this answer. They don't suit our use case as we need to incorporate some more custom logic.
Different tutorials use ctx, doc and some use params. To add to the confusion Debug.explain(doc.members), Debug.explain(params._source.members) return empty responses and I'm having a hard time figuring out the types.
Gist: Trying to write a custom filter on nested documents using painless. Want to write error checks when there are no nested documents to surpass null_pointer_exception
Any help is appreciated.
TLDr;
Elastic flatten objects. Such that
{
"group" : "fans",
"user" : [
{
"first" : "John",
"last" : "Smith"
},
{
"first" : "Alice",
"last" : "White"
}
]
}
Turn into:
{
"group" : "fans",
"user.first" : [ "alice", "john" ],
"user.last" : [ "smith", "white" ]
}
To access members inner value you need to reference it using doc['members.<field>'] as members will not exist on its own.
Details
As you may know, Elastic handles inner documents in its own way. [doc]
So you will need to reference them accordingly.
Here is what I did to make it work.
Btw, I have been using the Dev tools of kibana
PUT /so_test/
PUT /so_test/_mapping
{
"properties" : {
"captions_added" : {
"type" : "boolean"
},
"category" : {
"type" : "keyword"
},
"is_votable" : {
"type" : "boolean"
},
"members" : {
"properties" : {
"country" : {
"type" : "keyword"
},
"date_of_birth" : {
"type" : "date"
}
}
}
}
}
POST /so_test/_doc/
{
"captions_added": true,
"category" : "Mental Health",
"is_votable" : true,
"members": [
{"country": "Denmark", "date_of_birth": "1998-04-04T00:00:00"},
{"country": "Denmark", "date_of_birth": "1999-05-05T00:00:00"}
]
}
PUT /so_test/_doc/
{
"captions_added": true,
"category" : "Mental breakdown",
"is_votable" : true,
"members": []
}
POST /so_test/_doc/
{
"captions_added": true,
"category" : "Mental success",
"is_votable" : true,
"members": [
{"country": "France", "date_of_birth": "1998-04-04T00:00:00"},
{"country": "Japan", "date_of_birth": "1999-05-05T00:00:00"}
]
}
And then I did this query (it is only a bool filter, but I guess making it work for your own use case should not prove too difficult)
GET /so_test/_search
{
"query":{
"bool": {
"filter": {
"script": {
"script": {
"lang": "painless",
"source": """
def flag = false;
// /!\ notice how the field is referenced /!\
if(doc['members.country'].size() != 0)
{
for (item in doc['members.country']) {
if (item == params.country){
flag = true
}
}
}
return flag;
""",
"params": {
"country": "Japan"
}
}
}
}
}
}
}
BTW you were saying you were a bit confused about the context for painless. you can find in the documentation so details about it.
[doc]
In this case the filter context is the one we want to look at.
I need to implement a solution that aims to filter one of my search query using location. You will find here my entity and how I used #Spatial annotation :
#Entity
#Indexed
#Spatial(spatialMode = SpatialMode.RANGE)
#Table(name = "ORGANIZATION", uniqueConstraints = { #UniqueConstraint(columnNames = { "CODE" }) })
public class Organization implements Serializable, FileEntity {
...
#Latitude
#Column(name = "LATITUDE")
private Double latitude;
#Longitude
#Column(name = "LONGITUDE")
private Double longitude;
...
}
Indexing does not figure any errors, here's the result I found using elasticsearch querying :
GET http://localhost:9201/com.supralog.lexis.model.organization.organization
{
"com.supralog.lexis.model.organization.organization" : {
"aliases" : { },
"mappings" : {
"com.supralog.lexis.model.organization.Organization" : {
"properties" : {
"_hibernate_default_coordinates" : {
"properties" : {
"lat" : {
"type" : "float"
},
"lon" : {
"type" : "float"
}
}
},
...
}
}
}
}
GET http://localhost:9201/com.supralog.lexis.model.organization.organization/_search?from=0&size=1
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 15628,
"max_score" : 1.0,
"hits" : [
{
"_index" : "com.supralog.lexis.model.organization.organization",
"_type" : "com.supralog.lexis.model.organization.Organization",
"_id" : "...",
"_score" : 1.0,
"_source" : {
...
"_hibernate_default_coordinates" : {
"lat" : 49.1886203,
"lon" : -0.38740259999997306
},
...
}
}
]
}
}
After checking indexation looks OK, I tried to query all Organization objects within a given radius of 100km :
final Coordinates coordinates = Point.fromDegrees(form.getLatitude(), form.getLongitude());
final String search = StringUtils.join(terms, " ");
final FullTextSession fullTextSession = Search.getFullTextSession(sessionFactory.getCurrentSession());
final QueryBuilder queryBuilder = fullTextSession.getSearchFactory().buildQueryBuilder()
.forEntity(Organization.class).get();
final org.apache.lucene.search.Query elasticQuery = queryBuilder.spatial().within(100,Unit.KM).ofCoordinates(coordinates).createQuery();
final FullTextQuery fullTextQuery = fullTextSession.createFullTextQuery(elasticQuery, Organization.class);
fullTextQuery.setMaxResults(form.getMaximumNumberOfResult());
fullTextQuery.setProjection(FullTextQuery.THIS, FullTextQuery.SCORE);
And my problem is here, when I try to execute this query, I'm having the following return statement :
Request: POST /com.supralog.lexis.model.organization.organization/_search with parameters {from=0, size=50}
Response: 400 'Bad Request' with body
{
"error": {
"root_cause": [
{
"type": "query_shard_exception",
"reason": "failed to find geo_point field [_hibernate_default_coordinates]",
"index_uuid": "phOfJTOyRvetHyZrfeUmrA",
"index": "com.supralog.lexis.model.organization.organization"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "com.supralog.lexis.model.organization.organization",
"node": "9DCzSp6kS5KGtMq6tzywzg",
"reason": {
"type": "query_shard_exception",
"reason": "failed to find geo_point field [_hibernate_default_coordinates]",
"index_uuid": "phOfJTOyRvetHyZrfeUmrA",
"index": "com.supralog.lexis.model.organization.organization"
}
}
]
},
"status": 400
}
To fix it, I tried to set a name to #Spatial record, I tried to make my entity implements Coordinates, etc. However I'm always having the same result. It looks like hibernate-search is not indexing my location as a geo_point, reason why it's failing on querying...
Do you have any idea on what I missed in documentation ?
Versions used :: hibernate : 5.3 ; hibernate-search : 5.10 ; elasticsearch : 5.6
The Elasticsearch mapping is wrong:
"com.supralog.lexis.model.organization.Organization" : {
"properties" : {
"_hibernate_default_coordinates" : {
"properties" : {
"lat" : {
"type" : "float"
},
"lon" : {
"type" : "float"
}
}
},
...
}
}
The fact that there's a "properties" attribute under "_hibernate_default_coordinates" means that "_hibernate_default_coordinates" is of type "object", whereas it should be of type "geo_point".
The most likely explanation is that you didn't generate the schema before indexing, and Elasticsearch tried to automatically generate it on the fly based on the documents it received. As you can see it's a very bad idea, since the risk of Elasticsearch guessing the schema wrong is quite high.
You should have a look at the documentation about configuration. In particular, you should pick a suitable index schema management strategy.
In short, put the following into hibernate.properties
In development environment: Hibernate Search will try its best, but may fail, and data won't be reindexed magically, you'll have to do it yourself
hibernate.search.default.elasticsearch.index_schema_management_strategy update
In production environment: you'll have to update the schema carefully yourself, and plan a mass reindexing when updating your application.
hibernate.search.default.elasticsearch.index_schema_management_strategy create
We have a field "name" in the index. We recently added a new field "alias".
I want to copy name field value to the new field alias for all documents.
Is there any Update query that will do this?
If that is not possible , Help me to achieve this.
Thanks in advance
I am trying this query
http://URL/index/profile/_update_by_query
{
"query": {
"constant_score" : {
"filter" : {
"exists" : { "field" : "name" }
}
}
},
"script" : "ctx._source.alias = name;"
}
In the script , I am not sure how to give name field.
I getting error
{
"error": {
"root_cause": [
{
"type": "class_cast_exception",
"reason": "java.lang.String cannot be cast to java.util.Map"
}
],
"type": "class_cast_exception",
"reason": "java.lang.String cannot be cast to java.util.Map"
},
"status": 500
}
Indeed, the syntax has changed a tiny little bit since. You need to modify your query to this:
POST index/_update_by_query
{
"query": {
"constant_score" : {
"filter" : {
"exists" : { "field" : "name" }
}
}
},
"script" : {
"inline": "ctx._source.alias = ctx._source.name;"
}
}
UPDATE for ES 6
Use source instead of inline