The fields are not displaying in visualize section neither under Terms nor Significant Terms but are available in discover page. What should i change to make fields available in Visualize?
I have tried with keywords and text as type.
Mapping:
PUT /vcmts-new
{
"mappings": {
"build": {
"properties": {
"rel_ver": {
"type": "text"
},
"build_time": {
"type": "date"
},
"rel_status": {
"type": "text"
},
"rel_tags": {
"type": "text"
},
"job_num": {
"type": "integer"
}
}
}
}
}
Document Example:
POST /vcmts-new/build
{
"rel_ver": "2.6.7.1-1-auto14",
"build_time": "2018-05-21",
"rel_status": "fail",
"rel_tags": "PI09",
"job_num": 22
}
Discover page
Missing fields in Visualize page
PS: The data is added manually, there is no logstash or other forwarders configured
Using keyword instead of text as the type helped make those fields as aggregatable. One also needs to either refresh the Index Patterns under Management or restart Kibana to reflect the changes. Simply refreshing the visualize tab wont help.
Related
I put an object with some field and i wanna figure out how to mapping the index to handle and show the values like elasticsearch. I dunno why opensearch separate to individual fields the values. Both app has the same index mappings but the display is different for something.
I tried to map the object type set to nested but nothing changes
PUT test
{
"mappings": {
"properties": {
"szemelyek": {
"type": "nested",
"properties": {
"szam": {
"type": "integer"
},
"nev": {
"type": "text"
}
}
}
}
}
}
When I create an index with mapping like this one, what does it mean the _template/ word? what does the _ mean? I ask your help to understand more about creating an index, are they stored in a kind of folder, like template/packets folder?
PUT _template/packets
{
"template": "packets-*",
"mappings": {
"pcap_file": {
"dynamic": "false",
"properties": {
"timestamp": {
"type": "date"
},
"layers": {
"properties": {
"frame": {
"properties": {
"frame_frame_len": {
"type": "long"
},
"frame_frame_protocols": {
"type": "keyword"
}
}
},
"ip": {
"properties": {
"ip_ip_src": {
"type": "ip"
},
"ip_ip_dst": {
"type": "ip"
}
}
},
"udp": {
"properties": {
"udp_udp_srcport": {
"type": "integer"
},
"udp_udp_dstport": {
"type": "integer"
}
}
}
}
}
}
}
}
}
I ask this because after typing this, I recieve he following error
! Deprecation: Deprecated field [template] used, replaced by [index_patterns]
{
"acknowledged": true
}
I copied the pattern from this link:
https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana
And I'm trying to do exactly what is taught in the link, and I already can capture files with tshark and parse copy them into a packets.json file, and I will use filebeat to transfer the data to Elasticsearch, I already uploaded some data to Elasticsearch, but it wasn't indexed correctly, I just saw a lot of information with a lot of data.
My aim is to inderstand exactly how to create a new index pattern, and also how to relate what I upload to that index.
Thank you very much.
Just replace word template with index_patterns:
PUT _template/packets
{
"index_patterns": ["packets-*"],
"mappings": {
...
Index templates allow you to define templates that will automatically be applied when new indices are created.
After version 5.6 the format of Elasticsearch index templates has changed; the template field, which was used to specify one or more patterns for matching index names that would use the template at create time, was deprecated and superseded by the more appropriately named field index_patterns which works exactly the same way.
To solve the issue and get rid of the deprecation warnings you will have to update all your pre-6.0 index templates, changing the template to index_patterns.
You can list all your index templates by running this command:
curl -XGET 'http://localhost:9200/_template/*?pretty'
Or replace the asterisk with the name of one specific index template.
More about ES templates is here.
One of the fields called "resources" has the following 2 inner documents.
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::sms_vild/servers_backup/db_1246/db/reports_201706.schema"
},
{
"accountId": "934331768510612",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::sms_vild"
}
I need to split the ARN field and get the last part of it. i.e. "reports_201706.schema" preferably using scripted field.
What I have tried:
1) I checked the fileds list and found only 2 entries resources.accountId and resources.type
2) I tried with date-time field and it worked correctly in the scripted filed option (expression).
doc['eventTime'].value
3) But the same does not work with other text fields for e.g.
doc['eventType'].value
Getting this error:
"caused_by":{"type":"script_exception","reason":"link error","script_stack":["doc['eventType'].value","^---- HERE"],"script":"doc['eventType'].value","lang":"expression","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [eventType] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."}}},"status":500}
It means I need to change the mapping. Is there any other way to extract text from nested arrays in an object?
Update:
Please visit sample kibana here...
https://search-accountact-phhofxr23bjev4uscghwda4y7m.us-east-1.es.amazonaws.com/_plugin/kibana/
search for "ebs_attach.png" and then check resources field. You will see 2 nested arrays like this...
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::datameetgeo/ebs_attach.png"
},
{
"accountId": "513469704633",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::datameetgeo"
}
I need to split ARN field and extract the last part that is again "ebs_attach.png"
If I can some-how display it as scripted field, then I can see the bucket name and the file name side-by-side on discovery tab.
Update 2
In other words, I am trying to extract the text shown in this image as a new field on discovery tab.
While you can use scripting for this, I highly encourage you to extract those kind of information at index time. I have provided two examples here, which are far from failsafe (you need to test with different path or with this field missing at all), but it should provide a base to start with
PUT foo/bar/1
{
"resources": [
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::sms_vild/servers_backup/db_1246/db/reports_201706.schema"
},
{
"accountId": "934331768510612",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::sms_vild"
}
]
}
# this is slow!!!
GET foo/_search
{
"script_fields": {
"document": {
"script": {
"inline": "return params._source.resources.stream().filter(r -> 'AWS::S3::Object'.equals(r.type)).map(r -> r.ARN.substring(r.ARN.lastIndexOf('/') + 1)).findFirst().orElse('NONE')"
}
}
}
}
# Do this on index time, by adding a pipeline
PUT _ingest/pipeline/my-pipeline-id
{
"description" : "describe pipeline",
"processors" : [
{
"script" : {
"inline": "ctx.filename = ctx.resources.stream().filter(r -> 'AWS::S3::Object'.equals(r.type)).map(r -> r.ARN.substring(r.ARN.lastIndexOf('/') + 1)).findFirst().orElse('NONE')"
}
}
]
}
# Store the document, specify the pipeline
PUT foo/bar/1?pipeline=my-pipeline-id
{
"resources": [
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::sms_vild/servers_backup/db_1246/db/reports_201706.schema"
},
{
"accountId": "934331768510612",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::sms_vild"
}
]
}
# lets check the filename field of the indexed document by getting it
GET foo/bar/1
# We can even search for this file now
GET foo/_search
{
"query": {
"match": {
"filename": "reports_201706.schema"
}
}
}
Note: Considered "resources" is kind of array
NSArray *array_ARN_Values = [resources valueForKey:#"ARN"];
Hope it will work for you!!!
I have an index with a text field.
"state": {
"type": "text"
}
Now suppose there are two data.
"state": "vail"
and
"state": "eagle vail"
For one of my requirements,
- I need to do a term level query, such that if I type "vail", the search results should only return states with "vail" and not "eagle vail".
But another requirement for different search on the same index,
- I need to do a match query for full text search, such that if I type "vail", "eagle vail" should display as well.
So my question is, how do I do both term level and full text search in this field, as for doing a term level query, I would have to set it as "keyword" type such that it wont be analyzed.
You can use "multi-field" feature to achieve this. Here is a mapping:
{
"mappings": {
"my_type": {
"properties": {
"state": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}
}
In this case state will act as text field (tokenized) whereas state.raw will be keyword (single-token). When indexing a document you should only set state. state.raw will be created automatically.
I am working on a based facebook comments dashboard from facebook graph api using elasticsearch5 & kibana5. I add some analyzed fields and they are appearing in the discover part on Kibana but when going to the visualization i don't find those fields.
My facebook comments index :
PUT fb_comments
{
"settings": {
"analysis": {},
"mapping.ignore_malformed": true
},
"mappings": {
"fb_comment": {
"dynamic_templates": [
{
"created_time": {
"match": "created_time",
"mapping": {
"type": "date",
"format": "epoch_second"
}
}
},
{
"message": {
"match": "message",
"mapping": {
"type": "string",
"analyzer": "simple"
}
}
},
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
The field message the analyzed one is appearing in discover
The field message the analyzed one is not appearing in visualization part
I think it might be related to a memory limitation. As per Kibana 5 help, analyzed fields might required more memory.
I checked my memory and it is indeed used at its max capacity.
I finally found the solution.
So in elasticsearch 4.X we had string type and then you specified the type of analyzer if you wish to be analyzed. In elasticsearch 5.X we have two types keyword which is automatically aggregated and not analyzed, and the 2nd is text which is autmatically analyzed and not aggregated. The solution is if you want an analyzed field and aggregated at the same time you should add a property "fielddata":true and it will be analyzed and aggregated.