I have a situation where I want to filter the results not when performing a search but rather a GET using elasticsearch. Basically I have a Document that has a status field indicating that the entity has a state of discarded. When performing the GET I need to check the value of this field thereby excluding it if the status is indeed one of "discarded".
I know i can do this using a search with a term query, but what about when using a GET against the index based on Document ID?
Update: Upon further investigation, it seems the only way to do this is to use percolation or a search. I hope I am wrong if anyone has any suggestions I am all ears.
Just to clarify I am using the Java API.
thanks
Try something like this:
curl http://domain/my_index/_search -d '{
"filter": {
"and": [
{
"ids" : {
"type" : "my_type",
"values" : ["123"]
}
},
{
"term" : {
"discarded" : "false"
}
}
]
}
}
NOTE: you can also use a missing filter if the discarded field does not exist on some docs.
NOTE 2: I don't think this will be markedly slower than a normal get request either...
Related
I have field in index with mapping as :
"sequence_number" : {
"type" : "long",
"copy_to" : [
"_custom_all"
]
}
and using search query as
POST /my_index/_search
{
"query": {
"term": {
"sequence_number": {
"value": "we"
}
}
}
}
I am getting error message :
,"index_uuid":"FTAW8qoYTPeTj-cbC5iTRw","index":"my_index","caused_by":{"type":"number_format_exception","reason":"For input string: \"we\""}}}]},"status":400}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260) ~[elasticsearch-rest-client-7.1.1.jar:7.1.1]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238) ~[elasticsearch-rest-client-7.1.1.jar:7.1.1]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[elasticsearch-rest-client-7.1.1.jar:7.1.1]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433) ~[elasticsearch-rest-high-level-client-7.1.1.jar:7.1.1]
at
How can i ignore number_format_exception errors, so the query just doesn't return anything or ignores this filter in particular - either is acceptable.
Thanks in advance.
What you are looking for is not possible, ideally, you should have coherce enabled on your numeric fields so that your index doesn't contain dirty data.
The best solution is that in your application which generated the Elasticsearch query(you should have a check for NumberFormatExcepton if you are searching for numeric fields as your index doesn't contain the dirty data in the first place and reject the query if you get an exception in your application).
Edit: Another interesting approach is to validate the data before inserting into ES, using the Validate API as suggested by #prakash, only thing is that it would add another network call but if your application is not latency-sensitive, it can be used as a workaround.
I am trying to perform the simplest filter for a specific property value, as a JSON input, in a Kibana visualization, thoroughly without success.
I can't, to my surprise, find concrete examples in doing that (have been searching for a couple of minutes now).
Say we have a document with the following structure:
{
a: true,
b: 10
}
How can I add a Filter aggregation for all documents with a = true ?
I tried using "script", "query", "filters" api, but all give me parse errors. My filter jsons are all valid, my problem is with the exact syntax elastic is expecting, but all examples I found out there and tried - give me parsing errors (after making the amendments to my index structure).
Kibana's version: 6.4.3
How is this accomplished ?
An example:
POST /sales/_search?size=0
{
"aggs" : {
"docs" : {
"filter" : { "term": { "a": "true" } },
}
}
}
Here is the link to the official documentation with example.
I have 100GB ES index now. Right now I need to change one field to multi-fields, such as: username to username.username and username.raw (not_analyzed). I know it will apply to the incoming data. But how can I make this change affect on the old data? Should I using index scroll to copy the whole index to a new one, Or there is a better solution to just copy one filed please.
There's a way to achieve this without reindexing all your data by using the update by query plugin.
Basically, after installing the plugin, you can run the following query and all your documents will get the multi-field re-populated.
curl -XPOST 'localhost:9200/your_index/_update_by_query' -d '{
"query" : {
"match_all" : {}
},
"script" : "ctx._source.username = ctx._source.username;"
}'
It might take a while to run on 100GB docs, but after this runs, the username.raw field will be populated.
Note: for this plugin to work, one needs to have scripting enabled.
POST index/type/_update_by_query
{
"query" : {
"match_all" : {}
},
"script" :{
"inline" : "ctx._source.username = ctx._source.username;",
"lang" : "painless"
}
}
This worked for me on es 5.6, above one did not!
I have been given a problem where I need to perform a search based on different fields.For example,On UI the user is giving several search option like company name,department,state/province,title country and region.
The user selects few of these options like company name,department,state.I need to perform the search on these fields and return the results.
Can I do this with the help of aggregation in elastic search?Can anyone give me detailed example on how this can be done.
I did a few example like performing aggregation on gender.the query is as follows:-
"aggs" :{"group_by_gender" :{"terms" :{"field" : "gender"}}
When I ran this type of query all the sources(from documents) were returned.So,I was kind of confused whether aggregation is actually performed.
Thanks in Advance
Aggregations are meant to make statistics over the values of fields. If you need to search documents depending on fields, you need to make (boolean) queries.
Example:
POST myIndex/_search
{
"bool" : {
"must" : [
{"term" : { "name" : "kimchy" }},
{"term" : { "state" : "unicorn planet" }}
]
}
}
Elastic search boolquery
boolean query has different parameters like must , should , match ,match all , filter.
hope this will help.
I prepare query object and do search in elasticsearch.
For making query object, I give key and their value.
Problem is, when key and value is like "brand":"Men's Wear" then In this case elasticsearch is unable to give me related docs. I think problem is with comma or may be space. everything is fine if I use other json property for key and value (having no space and comma like "priority":"high")
Any help please!
Update:
no match query still not working! one more problem i found in creating search query. query i am using is:
var qryObj1 = {
"query" : {
"text" : {"name":"Tom"}
}
};
This will return all docs having name Tom. Now I want to get all docs having name Tom and profession is developer. So, here modified one:
qryObj1 = {
"query" : {
"text" : {"name":"Tom","profession":"developer"}
},"operator" : "and"
};
but search result is old one. any help!
Sounds like you are using TermQuery, aren't you?
TermQuery are not analyzed so they don't match with your analyzed content.
Try with a MatchQuery. It should work.
You need to use boolean query
http://www.elasticsearch.org/guide/reference/query-dsl/bool-query.html
Here you can ask ES to take AND or OR of various queries
"bool" : {
"must" : [
"text" : {"name":"Tom"},
"text" : {"profession":"developer"}
]
}