With executing searches, I know that if I want to query a pamareter lower than a certain value i have to execute the following script:
{"query": {
"bool": {
"must": [
{
"range": {
"length": {
"lte": "22"
}
}
}
]
}
}
}
However, i want to do the same thing through the elastic plugin in logstash.
elasticsearch{
query =>= "...."
}
But I didn't find how to do that. (and the website doesn't give any help https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html)
Thank you for your attention and your help.
Joe
Using the query string query syntax, you can do it like this
elasticsearch{
query =>= "length:{* to 22]"
}
Also note that at some point, we might be able to use the query DSL if this issue gets some traction.
Related
I would like to do the Elasticsearch equivalent of the following SQL statement:
SELECT * FROM Users WHERE UserName NOT LIKE '%something%'
I don't care about efficiency or scoring... This only gets executed on occasion. I am using request body syntax.
Use a normal wildcard query, and negate it using a bool must_not query.
GET Users/_search
{
"query": {
"bool": {
"must_not": [{
"wildcard": {
"UserName": {
"value": "*something*"
}
}
}]
}
}
}
I'm not sure if your Users are your index in you Elastic - but this is the main idea anyway:
You could go with the regexp query by using Complement - "~" for negation:
GET Users/_search
{
"query": {
"regexp":{
"UserName": {
"value": ".*~(something).*"
}
}
}
}
For more useful reference, you can check here
P.S:
You will not get the best performance but it will do the job
I run the query below on a large elastic search cluster. The cluster bcomes unresponsive
{
"size": 10000,
"query": {
"bool": {
"must": [
{
"regexp": {
"message": {
"value": ".*exception.*"
}
}
},
{
"bool": {
"should": [
{
"term": {
"beat.hostname": "ip-xxx-xx-xx-xx"
}
}
]
}
},
{
"range": {
"#timestamp": {
"lt": 1518459660000,
"format": "epoch_millis",
"gte": 1518459600000
}
}
}
]
}
}
}
When I remove the wildcarded .*exception.* and replace it with any non wildcarded string like xyz it returns fast. Though the query uses a wildcarded expression, it also looks for a small time range and a specific host. I would think this is a very simple query. Any reason why elasticsearch server can't handle this query? The cluster has 10 nodes and 20 TB of data.
See the documentation for Regexp Query. It clearly states the following:
Note: The performance of a regexp query heavily depends on the regular
expression chosen. Matching everything like .* is very slow
What would be ideal is to change the text analysis on the message field with a WordDelimiterTokenFilter and set split_on_case_change to true. Then something like NullPointerException will get indexed as three separate tokens [Null, Pointer, Exception]. This can help you search on exception without using a regex. Caveat is you need to reindex all your documents.
Another quick thing to try might be to keep your filter conditions on the hostname and timestamp in a filter context, which will prefilter documents before running your regexp query. This may be a short-term solution for you until you fix the text analysis.
I have a couple of items in my ES database with fields containing 2020-02-26T05:24:55.757Z for example. Is it possible to (with the URI Search, _search?q=...) search for exact dates? For example, in this case, I would like to find items from 2020-02-26. Is that possible?
Yes, It is possible. You could refer to query string documentation for more info.
curl localhost:9200/your_index_name/_search?q=your_date_field:%7B2020-02-26%20TO%20*%7D
You would need to encode the url. query part looks like q=your_date_field:{2020-02-26 TO *}
Above query in REST api would look like
{
"query": {
"range": {
"your_date_field": {
"gte": "2020-02-26"
}
}
}
}
For exact dates following would work
curl localhost:9200/your_index_name/_search?q=your_date_field:2020-02-26
Although this question is old, I came across it, so maybe others will do so too.
If you want to only work in UTC, you can use a match query, like:
{
"query": {
"match": {
"your_date_field": {
"query": "2020-02-26"
}
}
}
}
If you need to consider things matching on a particular date in a different timezone, you have to use a range query, like:
{
"query": {
"range": {
"your_date_field": {
"gte": "2020-02-26",
"lte": "2020-02-26",
"time_zone": "-08:00"
}
}
}
}
Let's say I have a structure like:
{"account_number":171,"balance":7091,
"firstname":"Nelda","lastname":"Hopper",
"age":39,"gender":"M",
"address":"742 Prospect Place","employer":"Equicom",
"email":"neldahopper#equicom.com",
"city":"Finderne","state":"SC"}
(the data comes from here).
If I write the following query in ElasticSearch:
POST /bank/_search?pretty
{
"query":
{ "bool":
{ "must":
[ { "range":
{ "balance": { "gte": 30000 } } } ] }
},
"fields":["gender", "balance", "age"],
"aggs":{
"age_filter":{
"filter":{
"match":{
"age":"30"
}
},
"aggs":{
"gender_stats":{
"terms":{"field":"gender"}
}
}
}
}
}
I'll get (1) 402 query results for the main query and (2) aggregation on the 18 results that passed the filter "age:30".
I've tried to do the similar trick in Solr 5.1, but the closes I could get was this:
q=balance:[30000%20TO%20*]&facet=true&facet.field=gender&fq=age:30
with the big difference that the filter is now applied to the main query results, so I get only 18 results at all, and then apply a corresponding faceting.
Is there a way to write a Solr query that is entirely equivalent to the ElasticSearch one? I.e. getting full results and then applying filtering only to the aggregation/faceting?
NB: I've tried exclusion by tag:
q={!ex=tagForAge}balance:[30000%20TO%20*]&facet=true&facet.field=gender&fq={!tag="tagForAge"}age:30
but it does not seem to apply to the main query.
Try appending &facet.query=age:30 to your query.
This will basically generate your facets from a particular search query which in your case is age:30.
For more information check here.
This query is returning fine only one item "steve_jobs".
{
"query": {
"constant_score": {
"filter": {
"term": {
"name":"steve_jobs"
}
}
}
}
}
So, now I want to get all people with name prefix steve_. So I try this:
{
"query": {
"constant_score": {
"filter": {
"term": {
"name": "steve_"
}
}
}
}
}
This is returning nothing. Why?
I'm confused about when to use term query / term filter / terms filter / querystring query.
What you need is Prefix Query.
If you are indexing your document like so:
POST /testing_nested_query/class/
{
"name": "my name is steve_jobs"
}
And you are using the default analyzer, then the problem is that the term steve_jobs will be indexed as one term. So your Term Query will never be able to find any docs matching the term steve as there is no term like in the index. Prefix Query helps you solve your problem by searching for a prefix in all the indexed terms.
You can solve the same problem by making your custom analyzers (read this and this) so that steve_jobs is stored as steve and jobs.