I had created a copy_to field as of an existing field by altering its mapping below:
{
"properties": {
"ExistingField": {
"type": "date",
"copy_to": "CopiedField"
},
"CopiedField": {
"type": "date",
"store": true
}
}
}
I had used "store": "true" since I wanted this new fields value to be retrieved when I do a search. Aggregations with the "CopiedField" work fine but when I try to search for a value in this new CopiedField I cannot see anything being retrieved:
{
"stored_fields": [
"CopiedField"
],
"query": {
"match_all": {}
}
}
How do I retrieve the value of this "CopiedField" in a simple search?
Mapping cannot be changed for already existing fields.
You will need to create a new index(with correct mapping) and move documents from old index to new index. You can delete old index and use alias so that index name is not changed
(Mapping)[https://www.elastic.co/blog/changing-mapping-with-zero-downtime]
(Reindex)[https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html]
(Alias)[https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-aliases.html]
ex.
Old index index35 with below mapping
PUT index35
{
"mappings": {
"properties": {
"ExistingField": {
"type": "date"
}
}
}
}
Query: below query will not return anything
GET index35/_search
{
"stored_fields": [
"CopiedField"
],
"query": {
"match_all": {}
}
}
Create New index
PUT index36
{
"mappings": {
"properties": {
"ExistingField": {
"type": "date",
"copy_to": "CopiedField"
},
"CopiedField": {
"type": "date",
"store": true
}
}
}
}
Move old documents to new documents
POST _reindex
{
"source": {
"index": "index35"
},
"dest": {
"index": "index36" ----> must be created before reindex
}
}
Make sure document count is same in both old and new index(to prevent data loss)
Delete old index :-DELETE index35
Create an alias for new index(give old name) so that search queries are not affected
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "index36", "alias" : "index35" } }
]
}
Old query will now return results
Related
How can I define mapping in Elasticsearch 7 to index a document with a field value from another index? For example, if I have a users index which has a mapping for name, email and account_number but the account_number value is actually in another index called accounts in field number.
I've tried something like this without much success (I only see "name", "email" and "account_id" in the results):
PUT users/_mapping
{
"properties": {
"name": {
"type": "text"
},
"email": {
"type": "text"
},
"account_id": {
"type": "integer"
},
"accounts": {
"properties": {
"number": {
"type": "text"
}
}
}
}
}
The accounts index has the following mapping:
{
"properties": {
"name": {
"type": "text"
},
"number": {
"type": "text"
}
}
}
As I understand it, you want to implement field joining as is usually done in relational databases. In elasticsearch, this is possible only if the documents are in the same index. (Link to doc). But it seems to me that in your case you need to work differently, I think your Account object needs to be nested for User.
PUT /users/_mapping
{
"mappings": {
"properties": {
"account": {
"type": "nested"
}
}
}
}
You can further search as if it were a separate document.
GET /users/_search
{
"query": {
"nested": {
"path": "account",
"query": {
"bool": {
"must": [
{ "match": { "account.number": 1 } }
]
}
}
}
}
}
I like to run a nested terms aggregation on string field which is inside an object.
Usually, I use this query
"terms": {
"field": "fieldname.keyword"
}
to enable fielddata
But I am unable to do that for a nested document like this
{
"nested": {
"path": "objectField"
},
"aggs": {
"allmyaggs": {
"terms": {
"field": "objectField.fieldName.keyword"
}
}
}
}
The above query is just returning an empty buckets array
Is there a way this can be done without enabling field-data by default during index mapping.
Since that will take a large heap memory and I have already loaded a huge data without it
document mapping
{
"mappings": {
"properties": {
"productname": {
"type": "nested",
"properties": {
"productlineseqno": {
"type": "text"
},
"invoiceitemname": {
"type": "text"
},
"productlinename": {
"type": "text"
},
"productlinedescription": {
"type": "text"
},
"isprescribable": {
"type": "boolean"
},
"iscontrolleddrug": {
"type": "boolean"
}
}
}
sample document
{
"productname": [
{
"productlineseqno": "1.58",
"iscontrolleddrug": "false",
"productlinename": "Consultations",
"productlinedescription": "Consultations",
"isprescribable": "false",
"invoiceitemname": "invoice name"
}
]
}
Fixed
By changing the mapping to enable field data
Nested query is used to access nested fields similarly nested aggregation is needed to aggregation on nested fields
{
"aggs": {
"fieldname": {
"nested": {
"path": "objectField"
},
"aggs": {
"fields": {
"terms": {
"field": "objectField.fieldname.keyword",
"size": 10
}
}
}
}
}
}
EDIT1:
If you are searching for productname.invoiceitemname.keyword then it will give empty bucket as no field exists with that name.
You need to define your mapping like below
{
"mappings": {
"properties": {
"productname": {
"type": "nested",
"properties": {
"productlineseqno": {
"type": "text"
},
"invoiceitemname": {
"type": "text",
"fields":{ --> note
"keyword":{
"type":"keyword"
}
}
},
"productlinename": {
"type": "text"
},
"productlinedescription": {
"type": "text"
},
"isprescribable": {
"type": "boolean"
},
"iscontrolleddrug": {
"type": "boolean"
}
}
}
}
}
}
Fields
It is often useful to index the same field in different ways for
different purposes. This is the purpose of multi-fields. For instance,
a string field could be mapped as a text field for full-text search,
and as a keyword field for sorting or aggregations:
When mapping is not explicitly provided, keyword fields are created by default. If you are creating your own mapping(which you need to do for nested type), you need to provide keyword fields in mapping, wherever you intend to use them
text or keyword data type allows numeric values to be inserted.
index template is as shown below:
{
"index_patterns" : [ "temp-index" ],
"mappings": {
"docs" : {
"properties": {
"username" : {
"type": "keyword"
}
}
}
}
}
document inserted :
{
"username" : 10
}
as the above document is inserted successfully, is it possible to reject such numeric values for text/keyword datatype.
You can use the coerce setting but I think it only works vice versa -- i.e. not allowing strings to be indexed as numbers:
PUT roddo
{
"mappings": {
"properties": {
"username": {
"type": "integer",
"coerce": false
}
}
}
}
This way, attempting
POST roddo/_doc
{
"username": "10"
}
will throw an Integer value passed as String error.
Alternatively, you could extend the username functionality:
PUT roddo2
{
"mappings": {
"properties": {
"username": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
},
"integer": {
"type": "integer",
"coerce": true,
"ignore_malformed": true
}
}
}
}
}
}
which is going to attempt to coerce but will not throw an exception when failed.
After syncing your documents you'll be able to find those that did not have integer usernames:
GET roddo2/_search
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "username.integer"
}
}
}
}
}
I have index that contains two fields: longitude and latitude that are stored as float. I want to create new index and copy data from the first one but with different mappings. I use reindex api with elastic processors which can rename fields and give them different data types. When i try to create field with type "geo_point" it fails.
"type": "parse_exception",
"reason": "[type] type [geo_point] not supported, cannot convert field.",
but when i create new index i am able to create field with "geo_point" type.
i tried different workarounds but documentation says that with geo queries you can only use "geo_point" type.
is there any solution?
{
"description": "test pipe",
"processors": [
{
"convert": {
"field": "location",
"type": "geo_point"
}
}
]
}
added pipe definition.
OK, let's say that your current index mapping looks like this:
PUT oldindex
{
"mappings": {
"doc": {
"properties": {
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
}
}
}
}
}
You need to create a new index with the proper mapping, as follows
PUT newindex
{
"mappings": {
"doc": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
And then, you can simply leverage the reindex API to copy the old index into the new one with some additional scripting to create the location field:
POST _reindex
{
"source": {
"index": "oldindex",
},
"dest": {
"index": "newindex"
},
"script": {
"source": "ctx._source.location = ['lat': ctx._source.latitude, 'lon': ctx._source.longitude]; ctx._source.remove('latitude'); ctx._source.remove('longitude'); "
}
}
And you're good to go with the location field in your new shiny index!
I am using this request when creating my index:
PUT some_name
{
"mappings": {
"_default_": {
"_timestamp" : {
"enabled": true,
"store": true
},
"properties": {
"properties": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
}
}
However, _timestamp field is not being returned, basically when I add a document (without any time field) and request it back. I am running Elasticsearch 1.5, and I have tried "store": "yes", "store": "true".
What am I doing wrong? Thanks.
You need to specifically ask for that field to be returned: "fields": ["_timestamp"] because it's a field that's not commonly returned and is not included in the _source (the default being returned):
GET /some_name/_search
{
"query": {
"match_all": {}
},
"fields": ["_timestamp"]
}