I have a time-based Kibana and the time-field is a type [number] (e.g. 1599026948 ) called timestamp. The time field name is not set and I need to set it to timestamp. Because of the bad format (number) of timestamp, I created a scripted field called datetime to get a better date-format.
The code for the scripted field datetime:
doc['timestamp'].value * 1000;
It works as it should:
Samples
Input Output
1599027119802 Sep 2, 2020 # 08:11:59.802
1577833200000 Jan 1, 2020 # 00:00:00.000
1609455599999 Dec 31, 2020 # 23:59:59.999
When I look at my "Discover"-tab, I get the warning:
"1 of 2 shards failed The data you are seeing might be incomplete or wrong."
Warning-detail:
script_exception at shard 0 index MY-INDEX-PATTERN node JgWo2yNxRBeR9mEuYLpK2R
Type script_exeption
Reason runtime error
Script stack org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)
org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)
doc['timestamp'].value * 1000;
^---- HERE
Script doc['timestamp'].value * 1000;
Lang painless
Caused by type illegal_argument_exeption
Caused by reason No field found for [timestamp] in mapping with types []
I ignored the warning and moved on.
I needed to set the time field name to the new field datetime.
I tried this by following code in the console:
POST .kibana/_update/MY-INDEX-PATTERN
{
"doc": {
"index-pattern": {
"timeFieldName" : "datetime"
}
}
}
I get the following result:
{
"_index" : ".kibana",
"_type" : "_doc",
"_id" : "MY-INDEX-PATTERN",
"_version" : 8,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 887,
"_primary_term" : 4
}
Looks like it worked, but when i try to create a time-based component inside my dashboard or just change the time-filter, it does not work and i get the same warning as above (1 of 2 shards failed..)
I think the time field name was not set to datetime.
I feel like I tried everything but nothing workes. I really want to avoid reindexing or is this the only solution? I am new to Kibana and hope someone can help me.
-Tobi
Solved: Kibana-administrator has to reindex the index, and set the type of timestamp to date. Now timestamp is visible when creating a pattern and can be choosen for time field name.
Related
The output of the curl -X GET "localhost:9200/_cat/shards?v" is as follows:
index
shard
prirep
state
docs
store
ip
node
test_index
1
p
STARTED
0
283b
127.0.0.1
Deepaks-MacBook-Pro-2.local
test_index
1
r
UNASSIGNED
0
test_index
1
r
UNASSIGNED
0
test_index
0
p
STARTED
1
12.5kb
127.0.0.1
Deepaks-MacBook-Pro-2.local
test_index
0
r
UNASSIGNED
0
test_index
0
r
UNASSIGNED
0
And the output of the query curl -X GET "localhost:9200/test_index/_search?size=1000" | json_pp is as follows:
{
"_shards" : {
"failed" : 0,
"skipped" : 0,
"successful" : 2,
"total" : 2
},
"hits" : {
"hits" : [
{
"_id" : "101",
"_index" : "test_index",
"_score" : 1,
"_source" : {
"in_stock" : -4,
"name" : "pizza maker",
"prize" : 10
},
"_type" : "_doc"
}
],
"max_score" : 1,
"total" : {
"relation" : "eq",
"value" : 1
}
},
"timed_out" : false,
"took" : 2
}
MY QUESTION: As you can see, only text_index 0 primary shard has the data (from the output of first query), why successful key inside the _shards key has the value of 2?
Also, there is only 1 document, then why the value of total key inside _shards key is 2?
test_index has two primary shards and four unassigned replica shards (probably because you have a single node). Since a primary shard is a partition of your index, a document can only be stored in a single primary shard, in your case primary shard 0.
total: 2 means that the search was run over the two primary shards of your index (i.e. 100% of the data) and successful: 2 means that all primary shards responded. So you know you can trust the response to have searched over all your test_index data.
There's nothing wrong here.
[ {
"id" : 57592,
"code" : "village1023",
"created_by_id" : null,
"created_date" : "Tue Mar 31 23:08:47 IST 2020",
"l_village_name" : "village1023",
"modified_by_id" : 70806,
"modified_date" : "Tue Mar 31 23:08:47 IST 2020",
"name" : "village1023",
"status" : "{0}",
"taluk_id" : 386
} ]
Any suggestion on how could I convert the value received from JSON as "status" : "{0}" as "TRUE" for the MySQL value "1" in bit data type.
If you want to replace the JSON value content "{0}" in to true, we can update the content of the flow file using ReplaceText processor, so you can simply use the ReplaceText processor to achieve the reported scenario but still it have its own limitations. Please check and proceed further.
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ReplaceText/
I got the expected result, I made mistake in the expression. It should be ${field.value:equals("{0}"):ifElse("TRUE","FALSE")} in "UpdateRecord" processor.
Doumentation - Example 5 - Use Expression Language to Modify Value
I use this command, which should match all documents:
curl -XGET 'localhost:9200/users/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} }
}
'
I get this response:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
But I'm 99.9% sure I have documents on that index. If I am right, why isn't it showing the matches? If I am wrong, how can I confirm this?
You should be able to determine what's happening if you know (a) where all your documents are being stored and (b) what the server thinks the 'users' index actually is.
For the first question, you can hit the _cat/indices endpoint to see how many documents you have in each index (the "docs.count" column):
$curl -XGET 'http://localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open query 1 0 0 0 159b 159b
green open some_index 1 0 54 0 24.7kb 24.7kb
green open autocomplete 1 0 0 0 159b 159b
green open test_index 2 0 10065 4824 7.9mb 7.9mb
For the second question, check the aliases defined on your server. It's possible that "users" has been defined as an alias to an index that doesn't have any documents, or it's possible that a filtered alias has been defined on that index with a filter that is excluding all of your documents (many aliases have date-related filters that will exclude all documents outside of a very specific date range). To check for the presence of aliases you can use
$curl -XGET 'http://localhost:9200/_aliases?pretty=true'
Elasticsearch version (bin/elasticsearch --version):5.2.2
JVM version (java -version): 1.8.0_121
OS version (uname -a if on a Unix-like system): opensuse
Do search with " curl -XGET 'localhost:9200/_search?pretty&timeout=1ms' "
The part of response is :
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 208,
"successful" : 208,
"failed" : 0
},
"hits" : {
"total" : 104429,
"max_score" : 1.0,
"hits" :
...
The took time is 5ms, and timeout setting is 1ms. Why "timed_out" is false rather than true.
Thanks
The timeout is per searched shard (looks like 208 in your case), while the took is for the entire query. On a per shard level you are within the limit. The documentation has some additional information on when you will hit timed_out and more caveats.
Try with a more expensive query (leading wildcard, fuzziness,...) — I guess then you should hit the (shard) limit.
I uploaded the data to my single node cluster and named the index as 'gequest'.
When I GET from http://localhost:9200/_cluster/stats?human&pretty, I get:
"cluster_name" : "elasticsearch",
"status" : "yellow",
"indices" : {
"count" : 1,
"shards" : {
"total" : 5,
"primaries" : 5,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 5,
"max" : 5,
"avg" : 5.0
},
"primaries" : {
"min" : 5,
"max" : 5,
"avg" : 5.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
}
When I do GET on http://localhost:9200/_stats?pretty=true
"_shards" : {
"total" : 10,
"successful" : 5,
"failed" : 0
}
How come total number of shards not consistent in two reports? Why total shards are 10 from stats API. How to track the other 5?
From the results it is likely that you have a single elasticsearch node running and created a index with default values(which creates 5 shards and one replica). Since there is only one node running elasticsearch is unable to assign the replica shards anywhere(elasticsearch will never assign the primary and replica of the same shard in a single node).
The _cluster/stats API gives information about the cluster including the current state. From your result it is seen that the cluster state is "yellow" indicating that all the primary shards are allocated but not all replicas have been allocated/initialized. So it is showing only the allocated shards as 5.
The _stats API gives information about your indices in the cluster. It will give information about how many shards the index will have and how many replicas. Since your index needs a total of 10 shards (5 primary and 5 replica as specified when you create the index) the stats contain information as total 10, successful 5 and failed 5(failed because unable to allocate in any node).
Use http://localhost:9200/_cat/shards to see the overall shard status