I am using Kibana 7.1.
I have successfully created Vega line plot. I can allow it to show month of data however I want user to play around with time filter in dashboard and allow vega visualization to change with it.
From https://www.elastic.co/blog/getting-started-with-vega-visualizations-in-kibana and in Vega documentation I've read that inserting
"%context%": true,
"%timefield%": "#timestamp"
inside url will resolve this issue however when I do this it gives me
url.%context% and url.%timefield% must not be used when url.body.query is set
my full elasticsearch code look like this:
"data": {
"url": {
"%context%":"true",
"index": "access_log",
"body": {
"query": {
"bool": {
"must": [
{"term": {"request_1": "rent"}},
{"term": {"status": 200}}
]
}
},
"aggs": {
"histo": {
"date_histogram": {
"field": "date",
"interval": "day"
},
"aggs": {
"start_agg": {
"filter": {
"term": {"request_2": "start"}
}
},
"check_agg": {
"filter": {
"term": {"request_2": "check"}
}
},
"start_check": {
"bucket_script": {
"buckets_path": {
"start_count": "start_agg._count",
"check_count": "check_agg._count"
},
"script": "params.start_count / params.check_count"
}
}
}
}
}
}
},
"format": {
"property": "aggregations.histo.buckets"
}
},
"mark": {
"type":"line"
},
"encoding": {
"x": {
"field": "key",
"type": "temporal",
"axis": {"title": false}
},
"y": {
"field": "start_check.value",
"type": "quantitative",
"axis": {"title": "Document count"}
},
"tooltip":[
{"field":"start_check.value",
"type" : "quantitative"},
{"field":"key",
"type" :"temporal"}
]
}
}
Quoting Elastic's Vega reference for Kibana:
When using "%context%": true or defining a value for "%timefield%" the body cannot contain a query. To customize the query within the VEGA specification (e.g. add an additional filter, or shift the timefilter), define your query and use the placeholders as in the example above. The placeholders will be replaced by the actual context of the dashboard or visualization once parsed.
The "example above" they are talking about is the following:
{
body: {
query: {
bool: {
must: [
// This string will be replaced
// with the auto-generated "MUST" clause
"%dashboard_context-must_clause%"
{
range: {
// apply timefilter (upper right corner)
// to the #timestamp variable
#timestamp: {
// "%timefilter%" will be replaced with
// the current values of the time filter
// (from the upper right corner)
"%timefilter%": true
// Only work with %timefilter%
// Shift current timefilter by 10 units back
shift: 10
// week, day (default), hour, minute, second
unit: minute
}
}
}
]
must_not: [
// This string will be replaced with
// the auto-generated "MUST-NOT" clause
"%dashboard_context-must_not_clause%"
]
filter: [
// This string will be replaced
// with the auto-generated "FILTER" clause
"%dashboard_context-filter_clause%"
]
}
}
}
}
And, just as already defined in the docs:
"%dashboard_context-must_clause%": String replaced by object containing filters
"%dashboard_context-filter_clause%": String replaced by an object containing filters
"%dashboard_context-must_not_clause%": String replaced by an object containing filters
So, in case you want to use user-defined filters or the time filter having a customized query at the same time, you must use these three strings instead of "%context%": true. They will be parsed by Kibana and substituted by Elasticsearch query objects: one for "MUST", one for "FILTER" and one for "MUST_NOT", respectively.
A simple schema like this one might be useful:
{
body: {
query: {
bool: {
must: [
// {
// A "MUST" clause of yours
// },
"%dashboard_context-must_clause%"
]
must_not: [
// {
// A "MUST_NOT" clause of yours
// },
"%dashboard_context-must_not_clause%"
]
filter: [
// {
// A "FILTER" clause of yours
// },
"%dashboard_context-filter_clause%"
]
}
}
}
}
In case you don't have any clause in some of the categories, just leave the corresponding "%dashboard_context-XXXXX_clause%" string without further objects - just like in the first example for "FILTER" or "MUST_NOT".
inserting "%timefield%":"date" where date is one of my field worked.
Related
I have a situation where I need to do elastic search based on multi-field. For Example: I have multiple fields in my postindex and I want to apply condition on four these fields (i.e. userid, channelid, createat, teamid) to meet my search requirement. When value of all these fields matched then search query displays results and if one of these is not match with values in postindex then it display no result.
I am trying to make a multifield search query for go-elasticsearch to search data from my post index. For the searcquery result four field must match otherwise it display 0 hit/no-result.
So, I think you need to write a following query :
GET postindex/_search
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"bool": {
"must": [
{
"term": {
"userid": {
"value": "mcqmycxpyjrddkie9mr13txaqe"
}
}
},
{
"term": {
"channelid": {
"value": "dnoihmrinins3qrm6bb9175ume"
}
}
},
{
"range": {
"createat": {
"gt": 1672909114890
}
}
}
]
}
},
{
"term": {
"teamid": {
"value": "qomrg11o8b8ijxoy8hrcnweoay"
}
}
}
]
}
}
}
In here, there is a bool query with should in parent scope, which is like OR. And inside the should there is another bool query with must which is like AND. We can also write the query shorter, but this will be better for you to understand.
I need to add a parameter to my search that filters results containing a specific word in a value. The query is searching for user history records and contains a url key. I need to filter out /history and any other url containing that string.
Here's my current query:
GET /user_log/_search
{
"size" : 50,
"query": {
"match": {
"user_id": 56678
}
}
}
Here's an example of a record, boiled down to just the value we're looking at:
"_source": {
"url": "/history?page=2&direction=desc",
},
How can the parameters of the search be changed to filter out this result.
You can use the filter param of boolean query in Elasticsearch.
if your url field is of type keyword, you can use the below query
{
"query": {
"bool": {
"must": {
"match": {
"user_id": 56678
}
},
"filter": { --> note filter
"term": {
"url": "/history"
}
}
}
}
}
I found a way to solve my specific issue. Instead of filtering on the url I'm filtering on a different value. Here's what I'm using now:
{
"size" : 50,
"query": {
"bool" : {
"must" : {
"match" : { "user_id" : 56678 }
},
"must_not": {
"match" : { "controller": "History" }
}
}
}
}
I'm still going to leave this question open for a while to see if anyone has other ways of solving the original problem.
I have a document like this:
{
"listings": {
"mappings": {
"listing": {
"properties": {
"auctionOn": {
"type": "date"
},
"inspections": {
"type": "nested",
"properties": {
"endsOn": {
"type": "date"
},
"startsOn": {
"type": "date"
}
}
},
// more fields.. snipped for brevity
}
}
}
}
}
and i would like to perform the following search: (needs to be a bool filter.. no scoring req'd)
return documents any of the inspections.startsOn matches any of the dates provided (if they are provided)
OR
return documents where auctionOn matches the date provided (if it's provided)
they can also specify to search for a) inspections only, b) auctions only. if not provided, either of the dates need to match.
So in other words, possible searches:
Search where there are any inspections/auctions
Search where there are any inspections
Search where there are any auctions
Search where there are any inspections/auctions on the dates provided
Search where there are any inspections on the dates provided
Search where there are any auctions on the dates provided
Now, i'm already in a bool query filter:
{
"query":
{
"bool":
{
"filter":[{"terms":{"location.suburb":["Camden"]}}
}
}
}
and i need this new filter to be seperate. so.. this is like a nested or filter, within a main bool filter?
So if provided "Suburb = Camden, Dates = ['2018-11-01','2018-11-02']'
then it should return documents where the suburb = Camden and either the inspections or auction date includes one of the dates provided.
I'm kinda stumped on how to do it, so any help would be much appreciated!
There will lot of bool query combinations for the cases you mentioned in the question. Taking the example you mention i.e.
So if provided "Suburb = Camden, Dates = ['2018-11-01','2018-11-02']'
then it should return documents where the suburb = Camden and either
the inspections or auction date includes one of the dates provided.
Assuming your location filter is working as expected, for dates part in the above e.g. additions to the query will be:
{
"query": {
"bool": {
"filter": [
{
"terms": {
"location.suburb": [
"Camden"
]
}
},
{
"bool": {
"should": [
{
"terms": {
"auctionOn": [
"2018-11-01",
"2018-11-02"
]
}
},
{
"nested": {
"path": "inspections",
"query": {
"bool": {
"should": [
{
"terms": {
"inspections.startsOn": [
"2018-11-01",
"2018-11-02"
]
}
},
{
"terms": {
"inspections.endsOn": [
"2018-11-01",
"2018-11-02"
]
}
}
]
}
}
}
}
]
}
}
]
}
}
}
I'm using Elasticsearch with the python library and I have a problem using the search query when the object become a little bit complex. I have objects build like that in my index:
{
"id" : 120,
"name": bob,
"shared_status": {
"post_id": 123456789,
"text": "This is a sample",
"urls" : [
{
"url": "http://test.1.com",
"displayed_url": "test.1.com"
},
{
"url": "http://blabla.com",
"displayed_url": "blabla.com"
}
]
}
}
Now I want to do a query that will return me this document only if in one of the displayed URL's a substring "test" and there is a field "text" in the main document. So I did this query:
{
"query": {
"bool": {
"must": [
{"exists": {"field": "text"}}
]
}
}
}
}
But I don't know what query to add for the part: one of the displayed URL's a substring "test"
Is that posssible? How does the iteration on the list works?
If you didn't define an explicit mapping for your schema, elasticsearch creates a default mapping based on the data input.
urls will be of type object
displayed_url will be of type string and using standard analyzer
As you don't need any association between url and displayed_url, the current schema will work fine.
You can use a match query for full text match
GET _search
{
"query": {
"bool": {
"must": [
{
"exists": {
"field": "text"
}
},
{
"match": {
"urls.displayed_url": "test"
}
}
]
}
}
}
Per our requirement we need to find the max ID of the document before adding new document. Problem here is doc may contain string data also So had to use inline script on the elastic query to find out max id only for the document which has integer data otherwise returning 0. am using following inline script query to find max-key but not working. can you help me onthis ?.
{
"size":0,
"query":
{"bool":
{"filter":[
{"term":
{"Name":
{
"value":"Test2"
}
}}
]
}},
"aggs":{
"MaxId":{
"max":{
"field":"Key","script":{
"inline":"((doc['Key'].value).isNumber()) ? Integer.parseInt(doc['Key'].value) : 0"}}
}
}
}
The error is because the max aggregation only supports numeric fields, i.e. you cannot specify a string field (i.e. Key) in a max aggregation.
Simply remove the "field":"Key" part and only keep the script part
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"Name": "Test2"
}
}
]
}
},
"aggs": {
"MaxId": {
"max": {
"script": {
"source": "((doc['Key'].value).isNumber()) ? Integer.parseInt(doc['Key'].value) : 0"
}
}
}
}
}