I have a problem about my index. Some document of this index cannot be deleted.
I can get the document with :
GET /my_index/my_type/FXN5gs9QRk-xS3ag8RxmZg?routing=12345
but I cannot delete it with:
DELETE /my_index/my_type/FXN5gs9QRk-xS3ag8RxmZg
and with this one too :
DELETE /my_index/my_type/FXN5gs9QRk-xS3ag8RxmZg?routing=12345
When I tried to delete it returns 504 Time Out.
And also I cannot add new document with the same routing.
PS:
ElasticSearch Version : 1.4
Doc Count in that type: More than 400,000,000
6 shards and 3 replicas
EDIT :
I cannot even add new document with this routing.
You need to specify the routing value in the delete too. Once you index documents with a custom routing value, you "take control" of where documents are placed. ES won't know where to find it unless you provide the routing every time:
DELETE /my_index/my_type/FXN5gs9QRk-xS3ag8RxmZg?routing=12345
Although the timeout might be a different issue, since you should simply get a "File Not Found" style exception (or worse, if you haven't made routing required, deleting an unrelated document).
Related
geoip.location is of geo_point datatype when an event is sent from logstash to elasticsearch with default indexName. As geoip.location has geo_point datatype, i can view the plotting of location in maps in kibana as kibana looks for geo_point datatype for maps.
geoip.location becomes geoip.location.lat, geoip.location.lon with number datatype, when an event is sent from logstash to elasticsearch with modified indexName. Due to this i'm not able to view the plotting of location in maps in kibana.
i don't understand why elasticsearch would behave differently when i try to add data to a modifiedIndexName. is this a bug with elasticsearch?
For my usecase i need to use modified indexname, as i need new index for each day. The plan is to store the logs of a particular day in a single index. so, if there are 7 days then i need to have 7 indexes that contains logs of each day (new index should be created based on currentdate).
i searched around for solution, but i'm not able to comprehend and make it to work for me. Kindly help me out on this
Update (what i did after reading xeraa's answer?)
In the devtools in kibana,
GET _template/logstash - showed the allowed patterns in index_patterns property along with other properties
i included my pattern (dave*) inside index_patterns and triggered the PUT request. You have to pass the entire existing body content (which you would receive in the GET request) inside PUT request along with your required index_patterns, otherwise the default setting will disappear as the PUT api will replace whatever data you pass in the PUT body
PUT _template/logstash
{
...
"index_patterns": [
"logstash-*","dave*"
],
...
}
I'd guess that there is a template set for the default name, which isn't happening if you rename it.
Check with GET _template if any match your old index name and update the setting so that it also gets applied to the new one.
I'm using Elastic Search 5.6.2 with Kibana and I'm currently facing a problem
My documents are indexed on the field timestamp which is normally an integer, however recently somebody has logged a document with a timestamp that is not an integer, and Kibana complains of conflicting type.
The discover panels display nothing and the following errors pop:
Saved "field" parameter is now invalid. Please select a new field.
Discover: "field" is a required parameter
How can I look for the document(s) causing these conflicts so that to find the service creating bad logs ?
The field type (either integer or text/keyword) is not defined on per document basis but rather on per index basis (in the mappings). I guess you are manipulating timeseries data, and you probably have un index per day (or per month or ...).
In Kibana Dev Tools:
List the created indices with GET _cat/indices
For each index (logstash-2017.09.28 in my example) do a GET logstash-2017.09.28/_mapping and check the type of the field in #timestamp
The field type is probably different between indices.
You won't be able to change the field type on created indices. Deleting document won't solve you're problem. The only solution is to drop the index or reindex the whole index with a new field type (in a specific mapping).
To avoid this problem on future indices, the solution is to create an index template with a mapping telling that the field #timestamp is of type date or whatever.
I'm quite sure that I want to be notified with the inserted document by each insertion in the couch db.
something like this:
http://localhost:5058/db-name/_chnages/_view/inserted-document
And I like the response to be something like the following:
{
"id":"0552065465",
"name":"james"
.
.
.
}
Reconnecting to the database for giving the actual document by each notification can cause performance issues.
Can I define a view that return the actual document by each change?
There are 3 possible way to define if a document was just added:
You add a status field to your document with a specific status for new documents.
If the revision starts with a 1- but it's not 100% accurate according to this if you do replication.
In the changes response, check if the number of revision of the document is equal to one. If so, it means it was just added(best solution IMO)
If you want to query the _changes endpoint and directly get the newly inserted documents, you can use the approach #1 and use a filter function that only returns documents with status="new".
Otherwise, you should go with approach #3 and filter the _changes responses locally. Eg: your application would receive all changes and only handle documents with revisions array count equal to 1.
And as you mentioned, you want to receive the document, not only the _id and the _rev. To do so, you can simply add the query parameter: include_docs=true
I'm using kibana-4. Following the documentation here I should be able to create an index by putting this in my elasticsearch.yaml file:
PUT .kibana
{
"index.mapper.dynamic": true
}
I'm not sure I understand how to do this, because a yaml file should not take values formatted like the above block, right?
I noticed that .kibana was a default index, so after inputting it into the kibana console, I was asked to input a time field for the default index. However, the input HTML element is a dropdown that contained no options. Without selecting a time-field option I am not allowed to create a default index. What am I supposed to do? Has anyone else run into a similar problem?
I understand the problem faced by you. Even i faced the same while using Kibana 4 for first time.
Here are 2 possible solutions to your problem:-
1. Input data into elasticsearch which contains a timestamped field. So upon inputting data that field will be directly recognized by Kibana & would be showed to you in the dropdown menu (where you are currently seeing empty).
It is empty because Kibana couldn't recognize the timestamped field from the data inserted by you in elasticsearch.
2. Untick the option of Index contains time-based events which will allow you to just enter your index name & access Kibana.
Note:- while using Option 2 & specifying index name as .kibana you would notice that it doesn't contain any field or data because .kibana doesnt store any data.
I would suggest you to create an index using curl command and insert data in it with or without timestamped field. If inserted data without timestamped field use Option 2 otherwise use Option 1.
I have just updated a website, the update adds new fields to elasticsearch.
In my dev environment, it all works fine. but on the live site, the new fields are not being found.
Eg. I have added a new field with the value : 1
However, when adding a filtered query of
{"field":1}
It does not find any matching results.
When I look in the documents, I can see docs with the field set to 1
Would the reason for this be that the new field was added after the mappings was set? I am not all that familiar with elasticsearch, So I am not really sure where to start looking to fix it.
Any help would be appreciated.
Update:
querying from URL shows nothing either
_search/?pretty=true&size=50&q=field1:*
however there is another field that was added at the same time which I can search on.
I can see field1 in the result set but it just wont allow me to search on it.
Only difference i see in the mapping is that the one that is working is set to type:long whereas the one not working is set as type:string
Is it a length issue on the ngram? what was your "min_gram" settings?
When you check on your index settings like this:
GET <host>/<index_name>/_settings
Does it work when you filter for a two digit field?
Are all the field values one digit?
It's OK to add a field after the mapping was set. ElasticSearch will guess the mapping for you. (in fact, it's one of their selling features --- no need to define the mapping, just throw the data at it)
There are a few things that can go wrong:
Verify that data is actually in the index. To do that, just navigate to the _search url with no parameters, you should see the field if it is indexed.
Look at your mapping. Could it be that the field is explicitly set not to be indexed?
Another possibility is that your query is wrong (but that is unlikely, since you're saying it works in the development environment)