Why are there two modules in metric beat for ES
Elasticsearch
Elasticsearch-pack
Both has same configurations in the modules.d directory.
Kibana page for Elasticsearch module suggests to use Elasticsearch module.
But documentation of Elasticsearch modules suggests the later one. Reference
Alternatively, run metricbeat modules disable elasticsearch and metricbeat modules enable elasticsearch-xpack.
It's so confusing. I think that if I need to use ES-wtih-Xpack, then the later module. But from 6.7.0 onwards, ES ships basic features of x-pack with open source one.
Thanks.
The configuration are almost the same, the elasticsearch-xpack has the option xpack.enabled: true, which is not present in the elasticsearch module, and in the elasticsearch-xpack modulo you also do not specify any metricsets.
If you are using the monitoring UI in Kibana, then you should use the elasticsearch-xpack module, which will collect the metrics that kibana needs.
If you are not using the monitoring UI in Kibana, or are not even using Kibana and just want to collect the metrics, then you need to use the elasticsearch module and specify the metricsets that you want to collect.
The elasticsearch-xpack is just the elasticsearch module without any metricsets configured and with the option xpack.enabled: true.
Related
I am trying to enable search with Elasticsearch enabled for my self-hosted Gitlab instance.
My question is this:
I have an Elasticsearch instance that is external to my Gitlab instance i.e. my Gitlab instance is hosted on one server and my Elasticsearch instance is hosted on another server.
Am I able to use the external Elasticsearch instance to provide the Elasticsearch functionality that Gitlab requires in order to enable Advanced Search functionality?
Thanks,
Sean
As long as you have network connectivity between your gitlab instance and your Elasticsearch instance, you can use the Elasticsearch functionality.
Elasticsearch provides all the features and API in form of REST request and designed for distributed systems where several component of an application are hosted separately.
Note: you can just use the curl command curl :9200 from your gitlab instance to see if you have network connectivity and if not, using security setting you can enable them , like in AWS ports settings can be changed using the security group.
Note: GitLab 15.0 (May 2022) offers new options.
Advanced Search is compatible with Elasticsearch 8
Elasticsearch 8 is the current version of Elasticsearch by Elastic. Previously, you could not use Elasticsearch 8 for Advanced Search. You had to use older versions instead. Starting in 15.0, you can use Elasticsearch 8 for Advanced Search.
If you use Elasticsearch 7.x, you must upgrade to GitLab 15.0 before upgrading to Elasticsearch 8.
If you use Elasticsearch 6.8, upgrade to any Elasticsearch 7.x version before upgrading to GitLab 15.0.
See Documentation and Epic.
And even, still with GitLab 15.0 (May 2022)
Advanced Search is compatible with OpenSearch
OpenSearch is an open source Elasticsearch fork. Prior to GitLab 15.0, Advanced Search was not compatible with OpenSearch. If you used AWS-managed services, you had to use older versions of Elasticsearch.
You can now take full advantage of OpenSearch for Advanced Search.
See Documentation and Epic.
Objective is to create a Dashboard in Kibana that include visualizations based on some special queries to monitor Elasticsearch health and status, like GET /_cluster/settings?include_defaults=true&filter_path=defaults. the problem is this query is based on no index. how can i go thru it?
Please install the free version of xpack , cluster monitoring is free.
I am using that already.
I'm trying to add a role based access control over Kibana to have different privileges and dashboards for different types of users and I found those projects:
Elastic Shield Plugin
Search Guard Kibana Multitenancy Module
Elasticsearch Read Only Rest Plugin
Kibana plugin Own Home
What I found is that:
Elastic Shield (now called X-Pack) is the best, but unfortunately it's only 30 days free trial.
search guard module is a little complicated and it does the job, but it's not for free for commercial use which isn't suitable for my case.
ReadonlyREST plugin allows you to define which user reads which data, and their Enterprise Kibana plugin (non-free) also allows you to assign kibana indices to groups or users.
Kibana Plugin Own Home is a workaround solution as it defines different kibana index for different user which isn't practical.
If you know any other projects or any workaround for the mentioned projects above that satisfies my requirements, I'd be so grateful.
Disclaimer: I work for floragunn, makers of Search Guard.
The only real way to separate dashboards and visualizations based on users/tenants is to place them in separate Kibana indices. This is what both Search Guard and Own Home does.
The reason for this is how Kibana works internally: Out of the box, Kibana will place all of these objects in one global Kibana index, shared by all users. Since Kibana itself has no notion of users, roles or tenants, the only way to do it is to intercept the calls to the Kibana index, and rewrite the index name based on the provided user/role information.
Search Guard does this directly on Elasticsearch level, so the tenant
dashboards are separated even if you access Elasticsearch directly.
Own Home does that on Kibana level, but the effect/outcome is
basically the same.
X-Pack does not have multi-tenancy for Kibana, so
there's no separation of dashboards, only access control on Elasticsearch. Same is true for ReadOnly Rest.
One other option you have is to use the Red Hat OpenShift platform. They have an Elasticsearch plugin that is integrated with the Community Edition of Search Guard, and their own Kibana multi-tenancy module. As far as I know, this plugin is completely free.
This use case is now completely supported by ReadonlyREST via the Enterprise Kibana plugin.
And it comes with a lot more multitenancy/multiuser related features:
Remove certain Kibana apps to certain users
RBAC for indices, tenancies
Read only users won't see "save", "delete", "add" buttons in the Kibana UI
I made a video months ago that shows how this looks like.
Disclaimer: I work at ReadonlyREST, happy to take questions! :-)
Open Distro for Elasticsearch is a very good opensource project for multitenancy configuration in kibana. But the problem is proper documentation is not available easily for the older versions.
Set up Multi-Tenant Kibana Access in Open Distro for Elasticsearch
Kibana multi-tenancy
I've set up an Elastic Stack 5.3 to aggregate logs from a bunch of servers, with Filebeat in each of the servers scraping the logs and sending them to a centralised Logstash, Elasticsearch and Kibana.
I've set up my Logstash configuration to extract some custom string fields but I wish to change the index template to change their type from "text" to "keyword". I've found the configuration directives to specify my own template, but where can I find Logstash's default template so I can use it as a starting point? I've searched under /etc/logstash and /usr/share/logstash (I've installed a vanilla Logstash 5.3 RPM on RHEL 7) but couldn't find anything.
Any good example of how to create a non-standard index template on logstash 5.x would be really handy; most of the examples I have found predate Beats and the new string types in 5.x. The documentation leaves something to be desired.
The default elasticsearch index template can be found in the logstash-output-elasticsearch plugin repository at https://github.com/logstash-plugins/logstash-output-elasticsearch/tree/master/lib/logstash/outputs/elasticsearch
You'll find different templates in there, for ES 2.x, 5.x and 6.x, the one you're looking for is probably the 5.x one.
All,
I've been playing around with CrateData, and was wondering if you can utilize existing Elasticsearch tools such as drivers and add-ons like Logstash. For example, can you use an Elasticsearch river (http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/) for data ingest, then use the CrateData query engine, etc. against that data? Can incoming JSON objects be mapped to a table? Are there plans to have, or maintain a coexistence?
Thanks!
You can use existing tools for Elasticsearch with Crate if those tools use the REST API. In order to do so you'll have to enable the es rest api in the crate.yml file. There is aa setting to do so:
es.api.enabled: true
Elasticsearch Plugins won't work without minor modifications as Crate and Elasticsearch aren't binary compatible. Elasticsearch has a shading step in their maven configuration so the elasticsearch jar contains different namespaces then Crate does as Crate doesn't use shading.
So if you wanted to use a plugin you'd have to adjust the namespaces/imports and compile it against crate.