I have connected Kibana to my ES instance.
cat/indices returns:
yellow open .kibana 1 1 1 0 3.1kb 3.1kb
yellow open tests 5 1 413042 0 3.4gb 3.4gb
However I get the following on the kibana configuration screen. What am I missing?
Update:
My sample document looks like this
"_index": "tests",
"_type": "test7",
"_id": "AVGlIKIM1CQ8BZRgLZVg",
"_score": 1.7840601,
"_source": {
"severity": "ERROR",
"code": "CODE,
"message": "MESSAGE",
"environment": "TEST",
"error_uuid": "cbe99080-0bf3-495c-a417-77384ba0fd39",
"correlation_id": "cf5a1fd5-4fd2-40bb-9cdf-405b91dcbd6f",
"timestamp": "2015-11-20 15:24:39.831"
Disable the option Use event times to create index names and put the index name instead of the pattern (tests).
The option you are trying to use is used when you have index names based on timestamp (imagine you create a new index per day with tests-2015.12.01, tests-2015.12.02...). It's quite clear if you read the message when you enable that option:
Patterns allow you to define dynamic index names. Static text in an index name is denoted using brackets. Example: [logstash-]YYYY.MM.DD. Please note that weeks are setup to use ISO weeks which start on Monday
EDIT: The problem with an empty dropdown in the time-field name is because you don't have any field with date type in the mapping of your index. You can actually check if you do GET /<index-name>/_mapping?pretty, that the timestamp is a "string" type and not "date". This happens because the format didn't match the regex for the date detection (yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z). To solve this:
You can change the format of the timestamp you are inserting to match the default regex.
You can modify the dynamic_date_format property and put a regex that matches the current format of your timestamp.
You can set an index template and set the type "date" for the "timestamp" field.
In any of the cases, you would need to delete the index and create a new one or reindex the data.
Related
I have an index which contains information about some objects. I want to display some of the information on my Kibana's dasboard. Lets assume an object looks as follows:
{
"_index": "obj",
"_type": "_doc",
"_id": "KwDPAHABfo5V345r4IYV",
"_version": 1,
"_score": 0,
"_source": {
"value_1": "some value",
"value_2": "some_other value",
"owner": "jason",
"modified_date": "2020-02-01T12:53:08.210317+00:00",
"created_date": "2020-02-01T12:53:08.243980+00:00"
}
}
I need to show (live) number of objects that has owner: 'UNKNOWN'. Thing is, that this value changes in time. Each change is a new document - they are not being updated. I need to track how many UNKNOWN owners currently I see. Updates (new documents) are being sent to elk in fixed intervals.
When I try to set up a metric, it sometimes shows 0, during the window between one update and another - when there is no documents flowing into elk. How can I make Kibana display only last documents with owner: 'UNKNOWN'?
How can I make Kibana display only last documents with owner: 'UNKNOWN'?
You could set up a data table visualization for that as an alternative to the one-dimensional metric visualization.
This is how I personally would configure the data table:
Set a filter with 'owner(.keyword) is UNKNOWN'.
Use the metric 'Top Hit' on the field created_date (or #timestamp, thats up to you) instead of the count metric.
Set the order to descending based on the timestamp field.
Split the rows (Term Aggregations) for every field you want to display in the rows. This will create 'columns' in your table.
Go to the options tab and enable count on the sum of all rows.
Set an appropriate time interval, e.g. last 1 hour.
This will display all the relevant data of your documents that have the field owner equal to UNKNOWN. Also, you see the ingestion/creation date timestamp of these documents in a descending order. Furthermore, you see the number of documents that match (configured via the options tab as described above).
I hope I could help you.
so I'm trying to get good search results out of an Elasticsearch installation.
But I run into problems when I'm trying to make a fuzzy search on some very simple data.
Somehow multiple (some of them partial) words are scored too low and only get scored higher, when more letters of the word are present in the search query.
Let me explain:
I have a simple index built with two simple documents.
{
"name": "Product with good qualities and awesome sound system"
},
{
"name": "Another Product that has better acustics than the other one"
}
Now I query the index with this parameters:
{
"query": {
"multi_match": {
"fields": ["name"],
"query": "product acust",
"fuzziness": "auto"
}
}
}
And the results look like this:
"hits": [
{
"_index": "test_products",
"_type": "_doc",
"_id": "1",
"_score": 0.19100355,
"_source": {
"name": "Product with good qualities and awesome sound system"
}
},
{
"_index": "test_products",
"_type": "_doc",
"_id": "2",
"_score": 0.17439455,
"_source": {
"name": "Another Product that has better acustics than the other one"
}
}
]
As you can see the product with the ID 2 is scored less than the other product even though it has possibly more similarity with the given query string than the other product because it has 1 full word match and 1 partial word match.
When the query would looke like "product acusti" the results would start to behave correctly.
I've already fiddled around with bool search but the results are identical.
Any ideas how I can get the wanted results back faster than having to have almost the whole second word typed in?
As far as I know, Elasticsearch does not do partial word matching by default, so the term acust is not matched in neither of your documents.
The reason you are getting a higher score in the first document is that your matched term, product, appears in a shorter sentence:
Product with good qualities and awesome sound system
But as for the second document, product appears in a longer sentence:
Another Product that has better acoustics than the other one
So your second document is getting a lower score because the ratio of your match term (product) to the number of terms in the sentence is lower.
In other words in has lower Field length normalization:
norm = 1/sqrt(numFieldTerms)
Now if you you want to be able to do partial prefix matching, you need to tokenize your term into ngrams, for example you can create the following ngrams for the term "acoustics":
"ac", "aco", "acou", "acous", "acoust", "acousti", "acoustic", "acoustics"
You have 2 options to achieve this, see the answer by Russ Cam on this question
use Analyze API
with an analyzer that will tokenize the field into tokens/terms from
which you would want to partial prefix match, and index this
collection as the input to the completion field. The Standard analyzer
may be a good one to start with...
Don't use the Completion Suggester here and instead set up your field (name) as a text datatype with
multi-fields
that include the different ways that name should be analyzed (or not
analyzed, with a keyword sub field for example). Spend some time with the Analyze API to build an analyzer that will
allow for partial prefix of terms anywhere in the name. As a start,
something like the Standard tokenizer, Lowercase token filter,
Edgengram token filter and possibly Stop token filter would get you
running...
You may also find this guide helpful.
I have an Elasticsearch index which uses the #timestamp field to store the date in a date field.
There are many records which are missing the #timestamp field, but have a timestamp field containing a unix timestamp. (Generated from PHP, so seconds, not milliseconds)
Note, the timestamp field is of date type, but numeric data seems to be stored there.
How can I use Painless script in a reindex and set #timestamp where it is missing, IF there is a numeric timestamp field with a unix timestamp?
Here's an example record that I would want to transform.
{
"_index": "my_log",
"_type": "doc",
"_id": "AWjEkbynNsX24NVXXmna",
"_score": 1,
"_source": {
"name": null,
"pid": "148651",
"timestamp": 1549486104
}
},
Did you have a look at the ingest module of Elasticsearch??
https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html
Parses dates from fields, and then uses the date or timestamp as the
timestamp for the document. By default, the date processor adds the
parsed date as a new field called #timestamp. You can specify a
different field by setting the target_field configuration parameter.
Multiple date formats are supported as part of the same date processor
definition. They will be used sequentially to attempt parsing the date
field, in the same order they were defined as part of the processor
definition.
It does exactly what you want :) In your reindex statement you can direct documents through this ingest processor.
If you need more help let me know, then I can jump behind a computer and help out :D
I need a field, here score, to be a number, and other times a string. Like:
{
"name": "Joe"
"score": 32.5
}
{
"name": "Sue"
"score": "NOT_AVAILABLE"
}
How can I express this in this in the index settings in Elastic Search?
I basically want "dynamic typing" on the field. The code may not make sense to you (like: why not split it into 2 different fields), but it's necessary to be this way on my end (for consistency reasons).
I don't need/want the property to be indexed/"searchable" btw. I just need the data to be in the json response. I need something like "any object will fit here".
Finally figured it out. All I had to do was to set enabled to false, and elastic search will not attempt to do anything with the data - but it's still present in the json response.
Like so:
"score": {
"enabled": false
}
Just define "score" field to be of type "string" in your mapping. That's it. Make sure you do define the mapping before indexing any document though. Otherwise if you let the mapping be created dynamically and the type of value of "score" field is anything but string in the first document you index, you won't be able to index any document next in which "score" holds a string.
I'm pretty new to Kibana and just set up an instance to look at some ElasticSearch data.
I have one index in Elastic Search, which has a few fields including _timestamp. When I go to the 'Discover' tab and look at my documents, each have the _timestamp field but with a yellow warning next to the field saying "No cached mapping for this field". As a result, I can't seem to sort/filter by time.
When I try and create a new index pattern and click on "Index contains time-based events", the 'Time-field name' dropdown doesn't contain anything.
Is there something else I need to do to get Kibana to recognise the _timestamp field?
I'm using Kibana 4.0.
You'll need to take these quick steps first :
Go to Settings → Advanced.
Edit the metaFields and add "_timestamp". Hit save.
Now go back to Settings → Indices and _timestamp will be available in the drop-down list for "Time-field name".
In newer versions you are required to specify the date field before you send your data.
Your date field must be in a standard format such as miliseconds after Epoch (long number) or - just as suggested by MrE - in ISO8601.
See more info here: https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html
Again, before you send your data to the index, you must specify the mapping for this field. In python:
import requests
mapping = '{"mappings": {"your_index": {"properties": {"your_timestamp_field": { "type": "date" }}}}}'
requests.put('http://yourserver/your_index', data=mapping)
...
send_data()
My es version is 2.2.0
You have to the right schema.
I follow the guide
Eg:
{
"memory": INT,
"geo.coordinates": "geo_point"
"#timestamp": "date"
}
If you have the #timestamp, you will see the
ps: if your schema doesn't have "date" field, do not check "Index
contains time-based events
The accepted answer is obsolete as of Kibana 2.0
you should use a simple date field in your data and set it explicitly using either a timestamp, or a date string in ISO 8601 format.
https://en.wikipedia.org/wiki/ISO_8601
you also need to set a mapping to date BEFORE you start sending data apparently.
curl -XPUT 'http://localhost:9200/myindex' -d '{
"mappings": {
"my_type": {
"properties": {
"date": {
"type": "date"
}
}
}
}
}'
Go to Settings->Indices, select your index, and click the yellow "refresh" icon. That will get rid of the warning, and perhaps make the field available in your visualization.