Elasticsearch: Exists Query on Binary Field - elasticsearch

I am looking for a way to look up the number of documents that has a certain binary field "not set" in a mapped index. However, standard "Exists" query does not seem to work. Example:
{
"some-index": {
"mappings": {
"some-type": {
"properties": {
"data": {
"type": "binary"
}
}
}
}
}
}
Query:
POST http://.../some-index/some-type/_search?size=1
{
"query":{
"exists":{
"field":"data"
}
}
}
The query above would return 0 result no matter what. My guess is this is because Elasticsearch does not store binary fields in source by default, and "Exists" query only looks up the source.
Is there an alternative to using Exists query, ideally without using extra boolean field in mapping?

Does the following do what you want... I'm creating a template with field1 set as binary type, then indexed a document with just field2 (which I didn't bother defining) then I'm searching for docs without field1. You can run these in the Dev Console in Kibana
PUT _template\binary
{
"template": "binary",
"mappings": {
"binary": {
"properties": {
"field1": {
"type": "binary"
}
}
}
}}
PUT /binary/type/1
{
"field2":"abc"
}
GET binary/_search
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "field1"
}
}
}
}
}
That should return the doc you just indexed... if you change it to the following, it shouldn't return anything because field2 is present!
GET binary/_search
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "field2"
}
}
}
}
}

Related

Doing aggregation on object in Elasticsearch

I would like to do an aggregation on one of the object type but I couldn't make it work. I created a mapping from dynamic_templates because my object is dictionary and the key is list of constants. Here is my object, mapping and aggregate query.. even I can't access indexed field by must->exists query.
Document
{
"property":{
"innerProperty":{
"constantKey":{
"someArrays":[
{
"id":"12345"
}
]
}
}
}
}
Mapping
{
"dynamic_templates": [
{
"property_map": {
"path_match": "property.innerProperty.*",
"mapping": {
"type": "object",
"dynamic": false
}
}
}
]
}
Mapping result after adding a document
"property": {
"properties": {
"innerProperty": {
"properties": {
"constantKey": {
"type": "object",
"dynamic": "false"
}
}
}
}
}
Aggregation
GET /index/_search
{
"query": {
"bool": {
"must": {
"exists": {
"field": "property.innerProperty.constantKey"
}
}
}
}
}
Query
GET /index/_search
{
"aggs": {
"property-agg": {
"terms": {
"field": "property.innerProperty.constantKey"
}
}
},
"size": 0
}
Both of aren't working. I would like to do an aggregation by constantKey so that I would get the correct document count to facets make it work.

Could query contain multi sub query type in es

I have following simple code snippet. I want to query the documents that contains value field and also the name is hello,
PUT /lib59/_doc/1
{
"name": "hello",
"value":1
}
GET /lib59/_search
{
"query":{
"term": {
"name":"hello"
},
"exists": {
"field": "value"
}
}
}
But when I run it, an error occurs, the error message is:
I would ask whether ES supports the query as I wrote above, if not, how could I express my query? Thanks.
You need to combine your query using bool/must clause
{
"query": {
"bool": {
"must": [
{
"term": {
"name": "hello"
}
},
{
"exists": {
"field": "value"
}
}
]
}
}
}

terms query not working in Elastic search with value having space in it

We need to get the data based on multiple values.
So I am trying to use terms query in elastic search for modelNumber field.
But it is not working as expected.can anyone let me know what is wrong with the query.
POST index_name/_Search
{
"query": {
"bool": {
"must": [
{
"terms": {
"modelNumber": [
"test 1234rthg-1234-1234512-2345",
"testMode11l-123-rtyu-xyz11"
]
}
},
{
"terms": {
"userId": [
"123",
"VALUE2"
]
}
}
]
}
}
}
Terms query returns documents that contain one or more exact terms in
a provided field.
If you have not explicitly defined any index mapping, then you need to add .keyword to the modelNumber field. This uses the keyword analyzer instead of the standard analyzer (notice the ".keyword" after modelNumber field).
{
"query": {
"bool": {
"must": [
{
"terms": {
"modelNumber.keyword": [ // note this
"test 1234rthg-1234-1234512-2345",
"testMode11l-123-rtyu-xyz11"
]
}
},
{
"terms": {
"userId": [
"123",
"VALUE2"
]
}
}
]
}
}
}
OR you need to modify the mapping of modelNUmber field as -
{
"mappings": {
"properties": {
"modelNumber": {
"type": "keyword"
}
}
}
}

Can anyone help me to query from elastic search

I want to access the flights array as in the image and get records 10 by 10. I tried different ways by mapping the flights array.
Here is what i tried in postman
PUT http://12.234.17.134:9200/index-flights
{
"mappings": {
"properties": {
"flights": {
"type": "nested"
}
}
}
}
GET http://12.234.17.134:9200/index-flights/_search
{
"query": {
"match": {
"result.id": "2erfc096-3db0-4817-88fc-69db286e95b8"
},
"query": {
"nested": {
"path": "flights"
}
}
}
}
Image of the structure of my data
https://i.stack.imgur.com/K378p.png
Go with an _id query -- no need for the nested one (which if malformed anyways):
{
"query": {
"terms": {
"_id": [
"2erfc096-3db0-4817-88fc-69db286e95b8"
]
}
}
}
or
{
"query": {
"ids": {
"values": [
"2erfc096-3db0-4817-88fc-69db286e95b8"
]
}
}
}
Tip: never share public cluster IPs in online forums.

Find documents with empty string value on elasticsearch

I've been trying to filter with elasticsearch only those documents that contains an empty string in its body. So far I'm having no luck.
Before I go on, I should mention that I've already tried the many "solutions" spread around the Interwebz and StackOverflow.
So, below is the query that I'm trying to run, followed by its counterparts:
{
"query": {
"filtered":{
"filter": {
"bool": {
"must_not": [
{
"missing":{
"field":"_textContent"
}
}
]
}
}
}
}
}
I've also tried the following:
{
"query": {
"filtered":{
"filter": {
"bool": {
"must_not": [
{
"missing":{
"field":"_textContent",
"existence":true,
"null_value":true
}
}
]
}
}
}
}
}
And the following:
{
"query": {
"filtered":{
"filter": {
"missing": {"field": "_textContent"}
}
}
}
}
None of the above worked. I get an empty result set when I know for sure that there are records that contains an empty string field.
If anyone can provide me with any help at all, I'll be very grateful.
Thanks!
If you are using the default analyzer (standard) there is nothing for it to analyze if it is an empty string. So you need to index the field verbatim (not analyzed). Here is an example:
Add a mapping that will index the field untokenized, if you need a tokenized copy of the field indexed as well you can use a Multi Field type.
PUT http://localhost:9200/test/_mapping/demo
{
"demo": {
"properties": {
"_content": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
Next, index a couple of documents.
/POST http://localhost:9200/test/demo/1/
{
"_content": ""
}
/POST http://localhost:9200/test/demo/2
{
"_content": "some content"
}
Execute a search:
POST http://localhost:9200/test/demo/_search
{
"query": {
"filtered": {
"filter": {
"term": {
"_content": ""
}
}
}
}
}
Returns the document with the empty string.
{
took: 2,
timed_out: false,
_shards: {
total: 5,
successful: 5,
failed: 0
},
hits: {
total: 1,
max_score: 0.30685282,
hits: [
{
_index: test,
_type: demo,
_id: 1,
_score: 0.30685282,
_source: {
_content: ""
}
}
]
}
}
Found solution here https://github.com/elastic/elasticsearch/issues/7515
It works without reindex.
PUT t/t/1
{
"textContent": ""
}
PUT t/t/2
{
"textContent": "foo"
}
GET t/t/_search
{
"query": {
"bool": {
"must": [
{
"exists": {
"field": "textContent"
}
}
],
"must_not": [
{
"wildcard": {
"textContent": "*"
}
}
]
}
}
}
Even with the default analyzer you can do this kind of search: use a script filter, which is slower but can handle the empty string:
curl -XPOST 'http://localhost:9200/test/demo/_search' -d '
{
"query": {
"filtered": {
"filter": {
"script": {
"script": "_source._content.length() == 0"
}
}
}
}
}'
It will return the document with empty string as _content without a special mapping
As pointed by #js_gandalf, this is deprecated for ES>5.0. Instead you should use: query->bool->filter->script as in https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html
For those of you using elastic search 5.2 or above, and still stuck. Easiest way is to reindex your data correctly with the keyword type. Then all the searches for empty values worked. Like this:
"query": {
"term": {"MY_FIELD_TO_SEARCH": ""}
}
Actually, when I reindex my database and rerun the query. It worked =)
The problem was that my field was type: text and NOT a keyword. Changed the index to keyword and reindexed:
curl -X PUT https://username:password#host.io:9200/mycoolindex
curl -X PUT https://user:pass#host.io:9200/mycoolindex/_mapping/mycooltype -d '{
"properties": {
"MY_FIELD_TO_SEARCH": {
"type": "keyword"
},
}'
curl -X PUT https://username:password#host.io:9200/_reindex -d '{
"source": {
"index": "oldindex"
},
"dest": {
"index": "mycoolindex"
}
}'
I hope this helps someone who was as stuck as I was finding those empty values.
OR using lucene query string syntax
q=yourfield.keyword:""
See Elastic Search Reference https://www.elastic.co/guide/en/elasticsearch/reference/6.5/query-dsl-query-string-query.html#query-string-syntax
in order to find the empty string of one field in your document, it's highly relevant to the field's mapping, in other word, its index/analyzer setting .
If its index is not_analyzed, which means the token is just the empty string, you can just use term query to find it, as follows:
{"from": 0, "size": 100, "query":{"term": {"name":""}}}
Otherwise, if the index setting is analyzed and I believe most analyzer will treat empty string as null value So
you can use the filter to find the empty string.
{"filter": {"missing": {"existence": true, "field": "name", "null_value": true}}, "query": {"match_all": {}}}
here is the gist script you can reference: https://gist.github.com/hxuanji/35b982b86b3601cb5571
BTW, I check the commands you provided, it seems you DON'T want the empty string document.
And all my above command are just to find these, so just put it into must_not part of bool query would be fine.
My ES is 1.0.1.
For ES 1.3.0, currently the gist I provided cannot find the empty string. It seems it has been reported: https://github.com/elasticsearch/elasticsearch/issues/7348 . Let's wait and see how it go.
Anyway, it also provides another command to find
{ "query": {
"filtered": {
"filter": {
"not": {
"filter": {
"range": {
"name": {
}
}
}
}
}
} } }
name is the field name to find the empty-string. I've tested it on ES 1.3.2.
I'm using Elasticsearch 5.3 and was having trouble with some of the above answers.
The following body worked for me.
{
"query": {
"bool" : {
"must" : {
"script" : {
"script" : {
"inline": "doc['city'].empty",
"lang": "painless"
}
}
}
}
}
}
Note: you might need to enable the fielddata for text fields, it is disabled by default. Although I would read this: https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html before doing so.
To enable the fielddata for a field e.g. 'city' on index 'business' with type name 'record' you need:
PUT business/_mapping/record
{
"properties": {
"city": {
"type": "text",
"fielddata": true
}
}
}
If you don't want to or can't re-index there is another way. :-)
You can use the negation operator and a wildcard to match any non-blank string *
GET /my_index/_search?q=!(fieldToLookFor:*)
For nested fields use:
curl -XGET "http://localhost:9200/city/_search?pretty=true" -d '{
"query" : {
"nested" : {
"path" : "country",
"score_mode" : "avg",
"query" : {
"bool": {
"must_not": {
"exists": {
"field": "country.name"
}
}
}
}
}
}
}'
NOTE: path and field together constitute for search. Change as required for you to work.
For regular fields:
curl -XGET 'http://localhost:9200/city/_search?pretty=true' -d'{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "name"
}
}
}
}
}'
I didn't manage to search for empty strings in a text field. However it seems to work with a field of type keyword. So I suggest the following:
delete /test_idx
put test_idx
{
"mappings" : {
"testMapping": {
"properties" : {
"tag" : {"type":"text"},
"content" : {"type":"text",
"fields" : {
"x" : {"type" : "keyword"}
}
}
}
}
}
}
put /test_idx/testMapping/1
{
"tag": "null"
}
put /test_idx/testMapping/2
{
"tag": "empty",
"content": ""
}
GET /test_idx/testMapping/_search
{
"query" : {
"match" : {"content.x" : ""}}}
}
}
You need to trigger the keyword indexer by adding .content to your field name. Depending on how the original index was set up, the following "just works" for me using AWS ElasticSearch v6.x.
GET /my_idx/_search?q=my_field.content:""
I am trying to find the empty fields (in indexes with dynamic mapping) and set them to a default value and the below worked for me
Note this is in elastic 7.x
POST <index_name|pattern>/_update_by_query
{
"script": {
"lang": "painless",
"source": """
if (ctx._source.<field name>== "") {
ctx._source.<field_name>= "0";
} else {
ctx.op = "noop";
}
"""
}
}
I followed one of the responses from the thread and came up with below it will do the same
GET index_pattern*/_update_by_query
{
"script": {
"source": "ctx._source.field_name='0'",
"lang": "painless"
},
"query": {
"bool": {
"must": [
{
"exists": {
"field": "field_name"
}
}
],
"must_not": [
{
"wildcard": {
"field_name": "*"
}
}
]
}
}
}
I am also trying to find the documents in the index that dont have the field and add them with a value
one of the responses from this thread helped me to come up with below
GET index_pattern*/_update_by_query
{
"script": {
"source": "ctx._source.field_name='0'",
"lang": "painless"
},
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "field_name"
}
}
]
}
}
}
Thanks to every one who contributed to this thread I am able to solve my problem

Resources