I am new to DevOps side of things for elastic search and have a few questions regarding effective monitoring of a elastic search cluster using Graphana
What I tried
run elasticsearch locally
curl http://localhost:9200/
{
"name" : "hnsKXlb",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "IsSAzHcZTDSA40Lfy0PKcw",
"version" : {
"number" : "5.5.2",
"build_hash" : "b2f0c09",
"build_date" : "2017-08-14T12:33:14.154Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}
run graphana locally
docker run -p 3000:3000 --net network_name \
-e "GF_SECURITY_ADMIN_PASSWORD=xxx" \
grafana/grafana
added an ElasticSearch datasource
Imported graphana dashboard
https://grafana.com/grafana/dashboards/878
Question
I don't seem to get any metrics
I suspect that the datasource is only allowing grafana to that specific index. How can I make it more generic ?
You'll need an elasticsearch exporter (eexporter) to export metric to prometheus then use prometheus as datasource in Grafana
Take a look into the tools like Prometheus/Graphite/Logstash/Beats which will collect the metrics from Elasticsearch and add it into ES. First, we need to collect the metrics and store it into Elasticsearch. Then we can have a tool like Grafana to visualize the data. Kibana has a built-in dashboard to visualize the cluster health. You can check here.
Currently I am getting these alerts:
Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above.
Can someone tell me if there is a way I can find the exact installed version of ELS?
from the Chrome Rest client make a GET request or
curl -XGET 'http://localhost:9200' in console
rest client: http://localhost:9200
{
"name": "node",
"cluster_name": "elasticsearch-cluster",
"version": {
"number": "2.3.4",
"build_hash": "dcxbgvzdfbbhfxbhx",
"build_timestamp": "2016-06-30T11:24:31Z",
"build_snapshot": false,
"lucene_version": "5.5.0"
},
"tagline": "You Know, for Search"
}
where number field denotes the elasticsearch version. Here elasticsearch version is 2.3.4
I would like to add which isn't mentioned in above answers.
From your kibana's dev console, hit following command:
GET /
This is similar to accessing localhost:9200 from browser.
Hope this will help someone.
You can check version of ElasticSearch by the following command. It returns some other information also:
curl -XGET 'localhost:9200'
{
"name" : "Forgotten One",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.3.4",
"build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
"build_timestamp" : "2016-06-30T11:24:31Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
Here you can see the version number: 2.3.4
Typically Kibana is installed in /opt/logstash/bin/kibana . So you can get the kibana version as follows
/opt/kibana/bin/kibana --version
navigate to the folder where you have installed your kibana
if you have used yum to install kibana it will be placed in following location by default
/usr/share/kibana
then use the following command
bin/kibana --version
To check Version of Your Running Kibana,Try this:
Step1. Start your Kibana Service.
Step2. Open Browser and Type below line,
localhost:5601
Step3. Go to settings->About
You can See Version of Your Running kibana.
Another way to do it on Ubuntu 18.0.4
sudo /usr/share/kibana/bin/kibana --version
You can use the Dev Tools console in Kibana to obtain version information about Elasticsearch.
You click "Dev Tools" to navigate into console.
In the Dev Tools Console, you do a below query
GET /
You will see version and number like below with other details also.
{
"version" : {
"number" : "6.5.1",
...
}
}
You can Try this,
After starting Service of elasticsearch Type below line in your browser.
localhost:9200
It will give Output Something like that,
{
"status" : 200,
"name" : "Hypnotia",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.7.1",
"build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
"build_timestamp" : "2015-07-29T09:54:16Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
If you have installed x-pack to secure elasticseach, the request should contains the valid credential details.
curl -XGET -u "elastic:passwordForElasticUser" 'localhost:9200'
Infact, if the security enabled all the subsequent requests should follow the same pattern (inline credentials should be provided).
If you are logged into your Kibana, you can click on the Management tab and that will show your Kibana version. Alternatively, you can click on the small tube-like icon and that will show the version number.
From Kibana host, a request to http://localhost:9200/ will not be answered, unless ElasticSearch is also running on the same node. Kibana listens on port 5601 not 9200.
In most cases, except for DEV, ElasticSearch will not be on the same node as Kibana, for a number of reasons.
Therefore, to get information about your ElasticSearch from Kibana, you should select the "Dev Tools" tab on the left and in the console issue the command: GET /
If you looking for version in kibana ui
I wanted to try Elasticsearch with Polish language support, but I have some problems with it.
I installed Stempel Analysis Plugin, I'm trying to create an index that uses Polish analyzer:
curl -XPUT localhost:9200/polisz -d '{
"mappings" : {
"_default_" : {
"properties" : {
"text_entry" : { "type": "string", "analyzer": "polish" }
}
}
}
}
'
But I get an error about not recognized analyzer:
{
"status" : 400,
"error" : "MapperParsingException[mapping [_default_]]; nested: MapperParsingException[Analyzer [polish] not found for field [text_entry]]; "
}
Should I do anything after installing the plugin and rebooting ES?
I can't find any specific instructions about using the plugin so maybe I'm just doing something obviously wrong?
Some more details on how I set up my environment:
I installed and run docker image with ES and kibana by commands:
docker pull minimum2scp/es-kibana
docker run -d -p 8080:80 -p 9200:9200 --name es minimum2scp/es-kibana
I installed the Stempel plugin by command:
host$ docker exec -it es bash
root#docker-es:/# /usr/share/elasticsearch/bin/plugin install elasticsearch/elasticsearch-analysis-stempel/2.4.2
Then I rebooted elasticsearch, by:
root#docker-es:/# service elasticsearch restart
I'll be grateful for any help!
Krzysztof
OK, I got it. It seems that my plugin didn't install correctly. Even that plugin install command doesn't return any errors, neither elasticsearch restart command, there was a Lucene version mismatch in Elasticsearch( I don't remember, but below 4.10.2) and the plugin (4.10.3).
It was enough to look into elasticsearch.log file to find it out...My bad.
BUT there is more to it: I switched to the most popular (by stars) elasticsearch docker image, which is: dockerfile/elasticsearch. It has the ES version 1.4.2 that is based on Lucene 4.10.2, still mismatching the plugin Lucene 4.10.3. That causes an error even though authors of plugin states it plugin in 2.4.2 (current stable) support 1.4 ES version(s).
Citing an error for future web searching the problem:
[2015-02-13 10:57:11,850][INFO ][node ] [Necromantra] version[1.4.2], pid[1], build[927caff/2014-12-16T14:11:12Z]
[2015-02-13 10:57:11,851][INFO ][node ] [Necromantra] initializing ...
[2015-02-13 10:57:11,884][ERROR][plugins ] [Necromantra] cannot start plugin due to incorrect Lucene version: plugin [4.10.3], node [4.10.2].
[2015-02-13 10:57:11,884][WARN ][plugins ] [Necromantra] failed to load plugin from [jar:file:/data/plugins/analysis-stempel/elasticsearch-analysi
s-stempel-2.4.2.jar!/es-plugin.properties]
Now I chose a path to downgrade the plugin to 2.4.1, which agreed with my ES 1.4.2. Although in the long term I would look for docker image that has 1.4.3 ES which, hopefully,upgraded Lucene version as well.
Dadoonet, thank you for having a closer look on my problem.
I'm trying to make the new relic elastic search plugin work, the following is the website of the author:
New Relic Plug in Github
I've followed all the instructions and even see that the plugin has been installed succesfuly when I run in my server the command
'curl -XGET http://localhost:9200/_newrelic?pretty'
, I even get the response that should confirm everything has installed just fine:
' {
"configuration" : {
"agents" : {
"http" : true,
"pool" : true,
"transport" : true,
"fs" : true,
"indices" : true,
"network" : true
},
"refreshInterval" : 10
}
}'
However, when I log to my newrelic account not a single statistic is shown, either for JVM or whatsoever, nor I see the indices being monitored.
Has anyone encountered this problem before? PS: ElasticSearch is not running as a service and I DO have the new relic java agent properly installed. Also I have configured bin/elasticsearch.in.sh as it should be.
Thanks in advance!
JM.
When I'm trying to connect to ElasticSearch (elasticsearch-0.90.3) installed on EC2 from a none local machine using play2-elastic plugin it throws the following exception (the plugin works fine when connecting locally)
error] application - ElasticSearch : No ElasticSearch node is available. Please check that your configuration is correct, that you ES server is up and reachable from the network. Index has not been created and prepared.
org.elasticsearch.client.transport.NoNodeAvailableException: No node available
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:205) ~[elasticsearch-0.90.3.jar:na]
at org.elasticsearch.client.transport.support.InternalTransportIndicesAdminClient.execute(InternalTransportIndicesAdminClient.java:85) ~[elasticsearch-0.90.3.jar:na]
at org.elasticsearch.client.support.AbstractIndicesAdminClient.exists(AbstractIndicesAdminClient.java:147) ~[elasticsearch-0.90.3.jar:na]
at org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequestBuilder.doExecute(IndicesExistsRequestBuilder.java:43) ~[elasticsearch-0.90.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85) ~[elasticsearch-0.90.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) ~[elasticsearch-0.90.3.jar:na]
I have used different methods to test the elasticsearch server is up and running, examples:
curl -XGET '184.72.55.204:9300/_analyze?analyzer=standard' -d 'this is a test'
curl: (52) Empty reply from server
telnet 184.72.55.204 9300
Trying 184.72.55.204...
Connected to ec2-184-72-55-204.us-west-1.compute.amazonaws.com.
Escape character is '^]'.
In some google groups I also saw other people having similar problem, they seem to be able to fix the problem with turning sniffing to off, so I have this in my application.conf
elasticsearch.client="184.72.55.204:9300"
elasticsearch.sniff=false # I ADDED THIS BUT DID NOT HELP
elasticsearch.index.name="phonotags"
elasticsearch.index.settings="{ analysis: { analyzer: { my_analyzer: { type: \"custom\", tokenizer: \"standard\" } } } }"
elasticsearch.index.clazzs="indexing.*"
elasticsearch.index.show_request=true
my build.scala file contains these:
"com.clever-age" % "play2-elasticsearch" % "0.7-SNAPSHOT"
resolvers += Resolver.url("play-plugin-releases", new URL("http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/"))(Resolver.ivyStylePatterns),
resolvers += Resolver.url("play-plugin-snapshots", new URL("http://repo.scala-sbt.org/scalasbt/sbt-plugin-snapshots/"))(Resolver.ivyStylePatterns)
I appreciate your help.
thanks
It seems your node is not available
curl -XPUT '184.72.55.204:9200/twitter/tweet/1' -d '{ "user": "kimchy", "post_date" : "2011-08-18T16:20:00", "message" : "trying out Elastic Search" }'
Can you check this ?