Elasticsearch _validate API fails queries that _search API allows - elasticsearch

I’m trying to use the _validate API and it rejects as invalid queries where the exact body submitted works when sent to the _search API. Does _validate request body need to be different in some way?
For the “explain” result I get things like “no query registered for [fields]". Removing return fields list, it complains about filter.
Here is an example:
curl -s -XGET 'http://localhost:9200/4af9aae4-7ec1-458d-8c50-692ddb0f2c6d/msg,file,file-info/_validate/query?explain=true' -d '{"fields":["id"],"filter":{"not":{"and":[{"numeric_range":{"msg-size":{"gte":1000}}},{"query":{"prefix":{"content-type.verbatim":"application/"}}}]}}}' | python -mjson.tool
{
"_shards": {
"failed": 0,
"successful": 1,
"total": 1
},
"explanations": [
{
"error": "org.elasticsearch.index.query.QueryParsingException: [4af9aae4-7ec1-458d-8c50-692ddb0f2c6d-0] request does not support [fields]",
"index": "4af9aae4-7ec1-458d-8c50-692ddb0f2c6d-0",
"valid": false
}
],
"valid": false
}
Removing fields, it reports that query does not support filter
curl -s -XGET 'http://localhost:9200/4af9aae4-7ec1-458d-8c50-692ddb0f2c6d/msg,file,file-info/_validate/query?explain=true' -d '{"filter":{"not":{"and":[{"numeric_range":{"msg-size":{"gte":1000}}},{"query":{"prefix":{"content-type.verbatim":"application/"}}}]}}}' | python -mjson.tool
{
"_shards": {
"failed": 0,
"successful": 1,
"total": 1
},
"explanations": [
{
"error": "org.elasticsearch.index.query.QueryParsingException: [4af9aae4-7ec1-458d-8c50-692ddb0f2c6d-0] request does not support [filter]",
"index": "4af9aae4-7ec1-458d-8c50-692ddb0f2c6d-0",
"valid": false
}
],
"valid": false
}
Some queries do work with validate API, so it's not an across-the-board failure.
curl -s -XGET 'http://localhost:9200/4af9aae4-7ec1-458d-8c50-692ddb0f2c6d/msg,file,file-info/_validate/query?explain=true' -d '{"query": { "match": { "file-name": "PLEASE READ: something not important" }}}' | python -mjson.tool
{
"_shards": {
"failed": 0,
"successful": 1,
"total": 1
},
"explanations": [
{
"explanation": "filtered(file-name:PLEASE READ: something not important)->cache(_type:file _type:file-info _type:msg)",
"index": "4af9aae4-7ec1-458d-8c50-692ddb0f2c6d-0",
"valid": true
}
],
"valid": true
}
My understanding is _validate runs the same syntax checks etc as when you actually actually execute a query, so I am not sure what is going on.
Other details:
Elasticsearch v 1.2.1
Ubuntu Linux Precise 64

You need to wrap your query in a query key to use the _validate endpoint - that's why your last example is working but your first two do not.
Note
The query being sent in the body must be nested in a query key, same
as the search api works [1.0.0.RC1] Added in 1.0.0.RC1. The query was
previously the top-level object..
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-validate.html

Related

no shard available action exception on kibana discover

when i wanted to view the logs on kibana, i recieved this error :
1 of 37 shards failed The data you are seeing might be incomplete or wrong.
this is Response:
{
"took": 10,
"timed_out": false,
"_shards": {
"total": 21,
"successful": 20,
"skipped": 20,
"failed": 1,
"failures": [
{
"shard": 0,
"index": "tourism-2022.12.11",
"node": null,
"reason": {
"type": "no_shard_available_action_exception",
"reason": null,
"index_uuid": "j2J6dUvTQ_q7qeyyU56bag",
"shard": "0",
"index": "tourism-2022.12.11"
}
}
]
},
"hits": {
"total": 0,
"max_score": 0,
"hits": []
}
}
i delete some indexes
expand pvc
but doesnt work anything
Via Kibana console check the cluster status, if you haven't got Kibana available convert the command to curl request.
Check the Elasticsearch cluster status:
GET /_cluster/health?wait_for_status=yellow&timeout=50s
Check index status:
GET /_cat/indices/tourism-2022.12.11?v=true&s=index
All shards are green, have you got documents available in your index ?

Error while remote indexing with elasticsearch

I'am trying to move from an ES cluster to another, in order to plan an update.
The both are same version (6.4). To achieve this, i'am using this command :
curl -XPOST -H "Content-Type: application/json" http://new_cluster/_reindex -d#reindex.json
And the reindex.json, is looking like this :
{
"source": {
"remote": {
"host": "http://old_cluster:9199"
},
"index": "megabase.33.2",
"query": {
"match_all": {}
}
},
"dest": {
"index": "megabase.33.2"
}
}
I whitelisted one the new cluster the old cluster, and its works but i can't go to the end of the migration of data, because i have this error, and i don't understand what it means here :
{
"took":1762,
"timed_out":false,
"total":8263428,
"updated":5998,
"created":5001,
"deleted":0,
"batches":11,
"version_conflicts":0,
"noops":0,
"retries":{
"bulk":0,
"search":0
},
"throttled_millis":0,
"requests_per_second":-1.0,
"throttled_until_millis":0,
"failures":[
{
"index":"megabase.33.2",
"type":"persona",
"id":"noYOA3IBTWbNbLJUqk6T",
"cause":{
"type":"mapper_parsing_exception",
"reason":"failed to parse [adr_inse]",
"caused_by":{
"type":"illegal_argument_exception",
"reason":"For input string: \"2A004\""
}
},
"status":400
}
]
}
The record in the original cluster look like this :
{
"took": 993,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0,
"hits": [
{
"_index": "megabase.33.2",
"_type": "persona",
"_id": "noYOA3IBTWbNbLJUqk6T",
"_score": 0,
"_source": {
"address": "Obfucated",
"adr_inse": "2A004",
"age": 10,
"base": "Obfucated",
"city": "Obfucated",
"cp": 20167,
"email_md5": "Obfucated",
"fraicheur": "2020-01-12T19:39:04+01:00",
"group": 1,
"latlon": "Obfucated",
"partner": "Obfucated",
"partnerbase": 2,
"sex": 2,
"sms_md5": "Obfucated"
}
}
]
}
}
Any clue on what i'am doing wrong ?
Thanks a lot
Found out, the mapping is not well created when using only the reindex method.
So i drop the new indice, recreate mapping using elasticdump :
elasticdump --input=http://oldcluster/megabase.33.2 --output=http://newcluster/megabase.33.2 --type=mapping
Then run the previous script, everything works flawless (and was rather quick)

No results from search when passing more than one parameter in user metadata

I want to apply document level security in elastic, but once I provide more than one value in user metadata I get no matches.
I am creating a role and a user in elastic and passing values inside user metadata to the role on whose basis the search should happen. It works fine if I give one value.
For creating role:
PUT _xpack/security/role/my_policy
{
"indices": [{
"names": ["my_index"],
"privileges": ["read"],
"query": {
"template": {
"source": "{\"bool\": {\"filter\": [{\"terms_set\": {\"country_name\": {\"terms\": {{#toJson}}_user.metadata.country_name{{/toJson}},\"minimum_should_match_script\":{\"source\":\"params.num_terms\"}}}}]}}"
}
}
}]
}
And for user:
PUT _xpack/security/user/jack_black
{
"username": "jack_black",
"password":"testtest",
"roles": ["my_policy"],
"full_name": "Jack Black"
"email": "jb#tenaciousd.com",
"metadata": {
"country_name": ["india" , "japan"]
}
}
I expect the output to be results for india and japan only. If the user searches for anything else they should get no results.
However, I do not see any results at all:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}

Convert any Elasticsearch response to simple field value format

On elastic search, when doing a simple query like:
GET miindex-*/mytype/_search
{
"query": {
"query_string": {
"analyze_wildcard": true,
"query": "*"
}
}
}
It returns a format like:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 28,
"max_score": 1,
"hits": [
...
So I parse like response.hits.hits to get the actual records.
However if you are doing another type of query e.g. aggregation, the response is totally different like:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 28,
"max_score": 0,
"hits": []
},
"aggregations": {
"myfield": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
...
and I need to look actually in another json property: response.aggregations.myfield.buckets which gets even more complicated if you have more than one aggregation.
So, my question is very simple, isn't there a way that I can get Elasticsearch to response always with just the fields I want just like in SQL format:
E.g.
SELECT author, bookid FROM books
Would return:
{"author":"rogers", "bookid":099991}
{"author":"peter", "bookid":099992}
SELECT COUNT(author) As count_author, author, count(bookid) As count_bookid, bookid FROM books GROUP BY author, bookid
Would return:
{"count_author":4, "author":"rogers", "count_bookid":9, "bookid":099991}
{"count_author":8, "author":"peter", "count_bookid":9, "bookid":099992}
Is there a way to show only the fields I want and nothing else?(not having to look within nested json objects and all that stuff). (I want this because I'm doing many reports and I want to have a simple function that parses each response easily in a uniform way.)

Timeout in Elastic search query

I have following elastic search query, I want to apply timeout. So I used
"timeout" param.
GET testdata-2016.04.14/_search
{
"size": 10000,
"timeout": "1ms"
}
I have set timeout to be 1ms, but I observed that query is taking time about more than 5000ms. I have tried the query as below also:
GET testdata-2016.04.14/_search?timeout=1ms
{
"size": 10000
}
IN both cases, I am getting below response after approx. 5000ms.
{
"took": 126,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 26536,
"max_score": 1,
"hits": [
{
...................
...................
}
}
}
I am not sure what is happening here. Is anything missing in above queries ?
Please help.
I have tried to find out solution on google but didn't find any working solution.

Resources