I have been trying to update my mapping but not able to do that. Majorly this question is related to updating the nested part. Suppose there is a field "Anand" which contains a field "hello"
{
"properties": {
"anand": {
"hello": {
"type": "short"
}
}
}
}
But I am getting the error
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "No type specified for field [anand]"
}
],
"type" : "mapper_parsing_exception",
"reason" : "No type specified for field [anand]"
},
"status" : 400
}
Current Mapping is
{
"anandschool" : {
"mappings" : {
"properties" : {
"anand" : {
"type" : "nested"
},
"doc" : {
"properties" : {
"properties" : {
"properties" : {
"shop_tier" : {
"type" : "long"
}
}
}
}
},
"message" : {
"type" : "byte"
},
"properties" : {
"properties" : {
"shop_tier" : {
"type" : "long"
},
"shop_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"shop" : {
"type" : "long"
}
}
}
}
}
I even created a nested type anand so that it can work
{
"properties": {
"anand": {
"type": "nested"
}
}
}
Self Answer
When updating mapping for nested things need to update properties of the nested field.
For above example update by
"properties": {
"anand": {
"properties":{
"hello": {
"type": "short"
}
}
}
}
}
THough This will not work inside a field which is nested.Ex if anand type was "nested", it would not work. If anyone knows solution for that let me know.
Related
I would like to count and sort the number of occurred message on a field of type match_only_text. Using a DSL query the output needed to have to look like this:
{" Text message 1":615
" Text message 2":568
....}
So i tried this on kibana:
GET my_index_name/_search?size=0
{
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message"
}
}
}
}
However i get this error:
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "match_only_text fields do not support sorting and aggregations"
}
I am interested in the field "message" this is its mapping:
"message" : {
"type" : "match_only_text"
}
This is a part of the index mapping:
"mappings" : {
"_meta" : {
"package" : {
"name" : "system"
},
"managed_by" : "ingest-manager",
"managed" : true
},
"_data_stream_timestamp" : {
"enabled" : true
},
"dynamic_templates" : [
{
"strings_as_keyword" : {
"match_mapping_type" : "string",
"mapping" : {
"ignore_above" : 1024,
"type" : "keyword"
}
}
}
],
"date_detection" : false,
"properties" : {
"#timestamp" : {
"type" : "date"
}
.
.
.
"message" : {
"type" : "match_only_text"
},
"process" : {
"properties" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
},
"pid" : {
"type" : "long"
}
}
},
"system" : {
"properties" : {
"syslog" : {
"type" : "object"
}
}
}
}
}
}
}
Please Help
Yes, by design, match_only_text is of the text field type family, hence you cannot aggregate on it.
You need to:
A. create a message.keyword sub-field in your mapping of type keyword:
PUT my_index_name/_mapping
{
"properties": {
"message" : {
"type" : "match_only_text",
"fields": {
"keyword": {
"type" : "keyword"
}
}
}
}
}
B. update the whole index (using _update_by_query) so the sub-field gets populated and
POST my_index_name/_update_by_query?wait_for_completion=false
Then, depending on the size of your index, call GET _tasks?actions=*byquery&detailed regularly to check the progress of the task.
C. run the aggregation on that sub-field.
POST my_index_name/_search
{
"size": 0,
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message.keyword"
}
}
}
}
I have parent-child relation for Customer(Parent) and PromotionCustomer(Child).
Customer data:
{
"id": "b7818d4d-566e-24f5-89d2-3995bb97fd5e",
**"externalId": "9200191",**
"name": "LOBLAW NFLD",
"fullName": "LOBLAW NFLD",
"businessSystem": "EBJ/001"
}
PromotionCustomer data:
{
"id" : "31f2e065-a046-9c3a-808b-83545ddb07d1",
"externalId" : "T-000195542",
"businessSystem" : "EBJ/001",
"promotionDescription" : "PM-RT-LOBLAW NFLD-BB",
"promotionType" : "Bill Back",
"promotionStatus" : "Approved",
**"promotionCustomer" : "9200191",**
"validFrom" : "02/28/2019",
"validTo" : "03/20/2019",
"promotionDateTypeCode" : "1"
}
This the mapping details(schema)
{
"promotionsearch" : {
"mappings" : {
"properties" : {
"_class" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"customer" : {
"type" : "nested",
"include_in_parent" : true,
"properties" : {
"businessSystem" : {
"type" : "keyword"
},
"externalId" : {
"type" : "keyword"
},
"fullName" : {
"type" : "text"
},
"id" : {
"type" : "text"
},
"name" : {
"type" : "text"
}
}
},
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"promotioncustomer" : {
"type" : "nested",
"include_in_parent" : true,
"properties" : {
"businessSystem" : {
"type" : "keyword"
},
"externalId" : {
"type" : "keyword"
},
"id" : {
"type" : "text"
},
"promotionCustomer" : {
"type" : "text"
},
"promotionDateTypeCode" : {
"type" : "keyword"
},
"promotionDescription" : {
"type" : "text"
},
"promotionProducts" : {
"type" : "text"
},
"promotionStatus" : {
"type" : "keyword"
},
"promotionType" : {
"type" : "keyword"
},
"validFrom" : {
"type" : "date",
"format" : "MM/dd/yyyy"
},
"validTo" : {
"type" : "date",
"format" : "MM/dd/yyyy"
}
}
},
"promotionjoin" : {
"type" : "join",
"eager_global_ordinals" : true,
"relations" : {
"customer" : "promotioncustomer"
}
},
"routing" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
I need to get the get the promotionCustomer data based on the externalId which is the customerId which maps to promotionCustomer along with the date range.
I've written a query as below.
GET /promotionsearch/_search
{
"query": {
"has_parent": {
"parent_type": "customer",
"inner_hits": {},
"query": {
"has_child": {
"type": "promotioncustomer",
"query": {
"bool": {
"must": [
{
"match": {
"customer.externalId": "9200191"
}
},
{
"range": {
"promotioncustomer.validFrom": {
"gte": "02/28/2019"
}
}
},
{
"range": {
"promotioncustomer.validTo": {
"lte": "03/20/2019"
}
}
}
]
}
}
}
}
}
}
}
But it's not yielding the result. I know the reason as well. In the has_child clause i'm making use of parent field i.e "customer.externalId". Is there a way to include/add this condition in the parent_type query and then based on the result apply range condition inside has_child
I was able to get the solution. Hope it helps others as well. No need make multiple query call to ES inorder get the desired result.
GET /promotionsearch/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"promotioncustomer.promotionDescription": "PM-RT"
}
},
{
"range": {
"promotioncustomer.validFrom": {
"gte": "02/28/2019"
}
}
},
{
"range": {
"promotioncustomer.validTo": {
"lte": "03/28/2019"
}
}
},
{
"has_parent": {
"parent_type": "customer",
"query": {
"match": {
"customer.fullName": "sdjhfb"
}
}
}
}
]
}
}
}
I have a simple doc structure as follows.
{
"did" : "1",
"uid" : "user1",
"mid" : "pc-linux1",
"path" : "/tmp/path1"
}
I need to query elastic ,that matches all fields exactly
GET index2/_search
{
"query": {
"bool":{
"must": [
{
"term" : { "uid" : "user1"}
},
{
"term" : { "mid" : "pc-linux1"}
},
{
"term" : { "did" : "1"}
},
{
"term" : { "path" : "/tmp/path1"}
}
]
}
}
}
The matching should happen without any kind of elastic 'analysis' on keywords, so that "/tmp/path1" is matched as a full term.
I tried to use a custom mapping: with
"index" : false
which does not work.
PUT /index2?include_type_name=true
{
"mappings" : {
"_doc": {
"properties" : {
"did" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"mid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
}
}
}
}
}
I am using elastic7.0 and few posts suggesting a custom mapping with
"index" : "not_analysed"
does not get accepted as a valid mapping in elastic 7.0
Any suggestions?
If you want to match exact terms, try this query:
GET index2/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"uid": "user1"
}
},
{
"match": {
"mid": "pc-linux1"
}
},
{
"match": {
"did": "1"
}
},
{
"match": {
"path": "/tmp/path1"
}
}
]
}
}
}
I am trying to do geomap of a value in Elasticsearch but the value type of the client_location is set as a string and I would like to change it to geo_point. When I run the following I am getting:
#curl -XGET "http://core.z0z0.tk:9200/_all/_mappings/http?pretty"
{
"packetbeat-2015.12.04" : {
"mappings" : {
"http" : {
"properties" : {
"#timestamp" : {
"type" : "date",
"format" : "strict_date_optional_time||epoch_millis"
},
"beat" : {
"properties" : {
"hostname" : {
"type" : "string"
},
"name" : {
"type" : "string"
}
}
},
"bytes_in" : {
"type" : "long"
},
"bytes_out" : {
"type" : "long"
},
"client_ip" : {
"type" : "string"
},
"client_location" : {
"type" : "string"
},
"client_port" : {
"type" : "long"
},
"client_proc" : {
"type" : "string"
},
"client_server" : {
"type" : "string"
},
"count" : {
"type" : "long"
},
"direction" : {
"type" : "string"
},
"http" : {
"properties" : {
"code" : {
"type" : "long"
},
"content_length" : {
"type" : "long"
},
"phrase" : {
"type" : "string"
}
}
},
"ip" : {
"type" : "string"
},
"method" : {
"type" : "string"
},
"notes" : {
"type" : "string"
},
"params" : {
"type" : "string"
},
"path" : {
"type" : "string"
},
"port" : {
"type" : "long"
},
"proc" : {
"type" : "string"
},
"query" : {
"type" : "string"
},
"responsetime" : {
"type" : "long"
},
"server" : {
"type" : "string"
},
"status" : {
"type" : "string"
},
"type" : {
"type" : "string"
}
}
}
}
}
}
When I run the following command to change the type of the value from string to geo_point I am getting the following error:
# curl -XPUT "http://localhost:9200/_all/_mappings/http" -d '
> {
> "http" : {
> "properties" : {
> "client_location" : {
> "type" : "geo_point"
> }
> }
> }
> }
> '
{"error":{"root_cause":[{"type":"merge_mapping_exception","reason":"Merge failed with failures {[mapper [client_location] of different type, current_type [string], merged_type[geo_point]]}"}],"type":"merge_mapping_exception","reason":"Merge failed with failures {[mapper [client_location] of different type, current_type [string], merged_type [geo_point]]}"},"status":400}
Any suggestion how should I correctly change the type?
Thanks in advance.
Unfortunately, once you've created a field you cannot change its type anymore. The best thing to do is to delete the index and recreate it properly with the adequate mapping.
Another temporary solution if you don't want to delete your index immediately, is to create a sub-field of your existing field:
# curl -XPUT "http://localhost:9200/_all/_mappings/http" -d '{
"http": {
"properties": {
"client_location": {
"type": "string",
"fields": {
"geo": {
"type": "geo_point"
}
}
}
}
}
}'
And then you can access it in your queries using client_location.geo.
Also note that you have to re-index your data in order to populate that new sub-field... which means you might just as well delete your index and re-create it properly.
UPDATE
After installing Packetbeat you need to make sure to install the packetbeat template yourself as described here (i.e. it is not done automatically):
https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-getting-started.html#packetbeat-template
curl -XPUT 'http://localhost:9200/_template/packetbeat' -d#/etc/packetbeat/packetbeat.template.json
Say I have a document like this:
{
"events" : [
{
"event_id" : 123,
"props" : {
"version": "33"
},
{
"event_id" : 124,
"props" : {
"version": "44a"
}
]
}
Is it possible to specify that the events.props.version be mapped to some type?
I've tried:
{
"template" : "logstash-*",
...
"mappings" : {
"_default_" : {
"properties" : {
"events.props.version" : { "type" : "string" }
}
}
}
}
But that doesn't seem to work.
Please have a look at mapping API in elasticsearch Mapping API.
To set any analyzer in the inner element we need to consider each and every inner field as a separate properties set. try the following
{
"mappings": {
"properties": {
"events": {
"properties": {
"event_id": {
"type": "string",
"analyzer": "keyword"
},
"props": {
"properties": {
"version": {
"type": "string"
}
}
}
}
}
}
}
}
if this not works please provide me you mapping.
Sure, but you need to use the "object" type:
From the doc ( https://www.elastic.co/guide/en/elasticsearch/reference/1.5/mapping-object-type.html ) if you want to map
{
"tweet" : {
"person" : {
"name" : {
"first_name" : "Shay",
"last_name" : "Banon"
},
"sid" : "12345"
},
"message" : "This is a tweet!"
}
}
you can write:
{
"tweet" : {
"properties" : {
"person" : {
"type" : "object",
"properties" : {
"name" : {
"type" : "object",
"properties" : {
"first_name" : {"type" : "string"},
"last_name" : {"type" : "string"}
}
},
"sid" : {"type" : "string", "index" : "not_analyzed"}
}
},
"message" : {"type" : "string"}
}
}
}