How to write query for following condition in elasticsearch
Select * from table1 where (cnd1 or cond2) and (cnd3)
My cond2 value is from nested object . My json object is below
details={ "name"="name1",
"address":"{
"city":"city1"
}"
}
I need to take city from above object
details.address.city
Is above syntax is right , if not how to get value of second object city.
{
"bool" : {
"must" : cond3,
"should" : [
cond1,
cond2
],
"minimum_should_match" : 1
}
}
go through this link for more info https://www.elastic.co/guide/en/elasticsearch/reference/2.3/query-dsl-bool-query.html
You can easily create a conditional queries with Elasticsearch. But there is some weird situation of your data section.
details={ "name"="name1",
"address":"{
"city":"city1"
}"
}
Elasticsearh save your data as a json object, but you should give your data as a json. In this section, there is an object, you try to sent. Let us examine:
There is a name attribute of detail object, it is a string. And also there is a address attribute, and it is a string too. It should be an object which has to include a city attribute if you want to reach this object via details.address.city. Now we try to fix:
{
"id":...,
...
"details": {
"name": "name1",
"address": {
"city": "city1"
}
}
}
In this case, I remove double quotation marks of details object. Now, you can reach city attribute of json as a json object. Now, we create a query to reach cities:
{
"query": {
"bool": {
"must": {
"term": {
"your-json-attribute": "???"
}
},
"should": [
{
"term": {
"your-json-attribute": "???"
}
},
{
"term": {
"your-json-attribute": "???"
}
}
]
}
}
}
I use term query but there is lots of another query types. You can check them on documentation. But for And and Or, you can use bool query. Check https://www.elastic.co/guide/en/elasticsearch/reference/2.0/query-dsl-bool-query.html
Related
I'm using Elasticsearch DSL, I'm trying to use a query result as a parameter for another query like below:
{
"query": {
"bool": {
"must_not": {
"terms": {
"request_id": {
"query": {
"match": {
"processing.message": "OUT Followup Synthesis"
}
},
"fields": [
"request_id"
],
"_source": false
}
}
}
}
}
}
As you can see above I'm trying to search for sources that their request_id is not one of the request_idswith processing.message equals to OUT Followup Synthesis.
I'm getting an error with this query:
Error loading data [x_content_parse_exception] [1:1660] [terms_lookup] unknown field [query]
How can I achieve my goal using Elasticsearch DSL?
Original question extracted from the comments
I'm trying to fetch data with processing.message equals to 'IN Followup Sythesis' with their request_id doesn't appear in data with processing.message equals to 'OUT Followup Sythesis'. In SQL language:
SELECT d FROM data d
WHERE d.processing.message = 'IN Followup Sythesis'
AND d.request_id NOT IN (SELECT request_id FROM data WHERE processing.message = 'OUT Followup Sythesis');
Answer: generally speaking, neither application-side joins nor subqueries are supported in Elasticsearch.
So you'll have to run your first query, take the retrieved IDs and put them into a second query — ideally a terms query.
Of course, this limitation can be overcome by "hijacking" a scripted metric aggregation.
Taking these 3 documents as examples:
POST reqs/_doc
{"request_id":"abc","processing":{"message":"OUT Followup Synthesis"}}
POST reqs/_doc
{"request_id":"abc","processing":{"message":"IN Followup Sythesis"}}
POST reqs/_doc
{"request_id":"xyz","processing":{"message":"IN Followup Sythesis"}}
you could run
POST reqs/_search
{
"size": 0,
"query": {
"match": {
"processing.message": "IN Followup Sythesis"
}
},
"aggs": {
"subquery_mock": {
"scripted_metric": {
"params": {
"disallowed_msg": "OUT Followup Synthesis"
},
"init_script": "state.by_request_ids = [:]; state.disallowed_request_ids = [];",
"map_script": """
def req_id = params._source.request_id;
def msg = params._source.processing.message;
if (msg.contains(params.disallowed_msg)) {
state.disallowed_request_ids.add(req_id);
// won't need this particular doc so continue looping
return;
}
if (state.by_request_ids.containsKey(req_id)) {
// there may be multiple docs under the same ID
// so concatenate them
state.by_request_ids[req_id].add(params._source);
} else {
// initialize an appendable arraylist
state.by_request_ids[req_id] = [params._source];
}
""",
"combine_script": """
state.by_request_ids.entrySet()
.removeIf(entry -> state.disallowed_request_ids.contains(entry.getKey()));
return state.by_request_ids
""",
"reduce_script": "return states"
}
}
}
}
which'd return only the correct request:
"aggregations" : {
"subquery_mock" : {
"value" : [
{
"xyz" : [
{
"processing" : { "message" : "IN Followup Sythesis" },
"request_id" : "xyz"
}
]
}
]
}
}
⚠️ This is almost guaranteed to be slow and goes against the suggested guidance of not accessing the _source field. But it also goes to show that subqueries can be "emulated".
💡 I'd recommend to test this script on a smaller set of documents before letting it target your whole index — maybe restrict it through a date range query or similar.
FYI Elasticsearch exposes an SQL API, though it's only offered through X-Pack, a paid offering.
Using ElasicSearch's JSON Query DSL through Kibana, how do I retrieve all documents which have:
messageTemplate equals My message
or
level equals Error
You have to use a Bool query for that :
... If the bool query is a filter context or has neither must or filter then at least one of the should queries must match a document for it to match the bool query
POST <your_index>/_search
{
"query": {
"bool": {
"should": [
{ "match_phrase" : { "messageTemplate" : "My message" } },
{ "term" : { "level" : "Error" } }
]
}
}
}
Alternatively, you could type in the Kibana search bar:
messageTemplate:"My message" || level:"Error"
or
messageTemplate:"My message" OR level:"Error"
This may be a repeted question but I'm not findin' a good solution.
I'm trying to search elasticsearch in order to get documents that contains:
- "event":"myevent1"
- "event":"myevent2"
- "event":"myevent3"
the documents must not contain all of them in the same document but the result should contain only documents that are only with those types of events.
And this is simple because elasticsearch helps me with the clause should
which returns exactly what i want.
But then, I want that all the documents must contain another condition that is I want the field result.example.example = 200 and this must be in every single document PLUS the document should be 1 of the previously described "event".
So, for example, a document has "event":"myevent1" and result.example.example = 200 another one has "event":"myevent2" and result.example.example = 200 etc etc.
I've tried this configuration:
{
"query": {
"bool": {
"must":{"match":{"operation.result.http_status":200}},
"should": [
{
"match": {
"event": "bank.account.patch"
}
},
{
"match": {
"event": "bank.account.add"
}
},
{
"match": {
"event": "bank.user.patch"
}
}
]
}
}
}
but is not working 'cause I also get documents that not contain 1 of the should field.
Hope I explained well,
Thanks in advance!
As is, your query tells ES to look for documents that must have "operation.result.http_status":200 and to boost those that have a matching event type.
You're looking to combine two must queries
one that matches one of your event types,
one for your other condition
The event clause accepts multiple values and those values are exact matches : you're looking for a terms query.
Try
{
"query": {
"bool": {
"must": [
{"match":{"operation.result.http_status":200}},
{
"terms" : {
"event" : [
"bank.account.patch",
"bank.account.add",
"bank.user.patch"
]
}
}
]
}
}
}
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html says that it is possible to query for documents that have at least one non-null value in the original field.
If the value of the original fields is an object, is it possible to query for the existence of a key in the object?
Example: a document is
{
"user": {
"name": "XY",
"passport_id": 1234
}
}
Can one make an exists query for user.name? I tried
{
"query": {
"exists" : { "field" : "user.name" }
}
}
but it does not give any results.
I have a document which looks like this:
{
"foo": {
"orgnr": "1"
},
"bar": {
"orgnr" : "2"
},
"created": "2015-02-12",
...
}
I have an API where a user can query for:
orgnr (required)
role (optional) - ANY by default ANY means must match at least one of bar.orgnr or foo.orgnr, but could also be role:BAR, and then it must match bar.orgnr:
created (optional)
query (optional)
Orgnr must match foo.orgnr OR bar.orgnr, and can then have a lot of other field and text queries. I match this doing a query string query. So for a request where orgnr is 1, the following query string would be generated:
(foo.orgnr:1 OR bar.orgnr:1) AND (rest of query)
Where rest of query can be for example
created:[2015-01-01 TO *]
created:[2015-01-01 TO *] AND *query*
But i'm not sure this is actually the correct way or doing this. Reading https://www.elastic.co/guide/en/elasticsearch/guide/current/_queries_and_filters.html makes me insecure.
I could also use a boolean match, with must for the orgnr.
With role:BAR the following query would be generated:
(bar.orgnr:1) AND (rest of query)
The most important here is that orgnr is actually matched towards foo OR bar orgnr.
Or should I use a filter for this instead?
A bool query in filter context is a fine way of doing it. The Should requires at least one clause to match.
POST _search
{
"query": {
"constant_score": {
"filter": {
"bool": {
"should": [
{
"term": {
"foo.orgnr": "1"
}
},
{
"term": {
"bar.orgnr": "1"
}
}
]
}
}
}
}
}