We are using Elasticsearch in a Kubernetes cluster (not exposed publicly) without X-Pack security, and had it working in 5.x with elastic/changeme, but after trying to get it set up with 6.x, it's now requiring a password, and the default of elastic/changeme no longer works.
We didn't explicitly configure it to require authentication, since it's not publicly exposed and only accessible internally, so not sure why it's requiring the password, or more importantly, how we can find out what it is or how to set/change it without using X-Pack security.
Will we end up needing to subscribe to X-Pack since we're trying to us it within a Kubernetes cluster?
Not sure how you are deploying Elasticseach in Kubernetes but we had a similar issue an ended passing this:
xpack.security.enabled=false
through the environment to the container.
If you don't use XPack at all you should use oss flavor of Elasticsearch. It includes only open source components of Elasticsearch:
docker pull docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.2
The interesting thins is, Elastic have removed any mention of it in documentation since 6.3.
See:
Docker 6.2
Docker current
Related
I have an application up and running in set of docker containers (deployed using docker stack and not kubernetes). I want to do performance monitoring for this application. I am confused about whether I should go for beats or Elastic Agent.
This page says:
When Elastic Agent runs inside of a container, it cannot be upgraded through Fleet as it expects that the container itself is upgraded.
This page says:
Standalone mode — All policies are applied to the Elastic Agent manually as a YAML file.
Q1. Does this mean that in standalone mode Elastic Agent is not "at all" managed by Fleet? Or some part of management
This page says:
Standalone Elastic Agents are manually configured and managed locally on the systems where they are installed. They are useful when you are not interested in centrally managing agents in Fleet, either due to your company’s security requirements, or because you prefer to use another configuration management system.
This page says:
To run an Elastic Agent in standalone mode, install the agent on each host you want to monitor and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents.
Q2. Does this mean that for monitoring docker containers (deployed using docker stack and not kubernetes), there is no difference between between beats and Elastic agents in terms of "central" management? Only difference would be that I have to configure different beats separately, which is avoided with Elastic Agent?
Q3. What is preferrable in this case? Beats or Elastic Agent?
Standalone is not managed by fleet, the "it cannot be upgraded through Fleet" refers to upgrading the actual version of the elastic-agent, you can still update the agent polices.
Yes you would need to configure beats via a config file on the container where elastic agent can be setup with some env variables to enroll it in a policy, then that policy is centrally managed and updated via Kibana.
Both are valid but elastic agent allows updating the policy after the container is running via a central location and therefore would be my choice.
I am trying to enable search with Elasticsearch enabled for my self-hosted Gitlab instance.
My question is this:
I have an Elasticsearch instance that is external to my Gitlab instance i.e. my Gitlab instance is hosted on one server and my Elasticsearch instance is hosted on another server.
Am I able to use the external Elasticsearch instance to provide the Elasticsearch functionality that Gitlab requires in order to enable Advanced Search functionality?
Thanks,
Sean
As long as you have network connectivity between your gitlab instance and your Elasticsearch instance, you can use the Elasticsearch functionality.
Elasticsearch provides all the features and API in form of REST request and designed for distributed systems where several component of an application are hosted separately.
Note: you can just use the curl command curl :9200 from your gitlab instance to see if you have network connectivity and if not, using security setting you can enable them , like in AWS ports settings can be changed using the security group.
Note: GitLab 15.0 (May 2022) offers new options.
Advanced Search is compatible with Elasticsearch 8
Elasticsearch 8 is the current version of Elasticsearch by Elastic. Previously, you could not use Elasticsearch 8 for Advanced Search. You had to use older versions instead. Starting in 15.0, you can use Elasticsearch 8 for Advanced Search.
If you use Elasticsearch 7.x, you must upgrade to GitLab 15.0 before upgrading to Elasticsearch 8.
If you use Elasticsearch 6.8, upgrade to any Elasticsearch 7.x version before upgrading to GitLab 15.0.
See Documentation and Epic.
And even, still with GitLab 15.0 (May 2022)
Advanced Search is compatible with OpenSearch
OpenSearch is an open source Elasticsearch fork. Prior to GitLab 15.0, Advanced Search was not compatible with OpenSearch. If you used AWS-managed services, you had to use older versions of Elasticsearch.
You can now take full advantage of OpenSearch for Advanced Search.
See Documentation and Epic.
I create a Kibana dashboard using HDFS data.(Note: I connect Elasticsearch with Hotonworks using Elasticsearch-Hadoop).I want to secure Kibana Dashboard.Is it possible to do that Security function with Ranger.
I don't think it is possible. You can check documentation which shows what projects does Apache Ranger support.
I your case you can use:
X-Pack - great security plugin for Elasticsearch and Kibana, but it is paid.
Search Guard - free Elasticsearch and Kibana security plugin. I think it is very good tool.
But if you don't want to deploy a real security plugin (which contains SSL etc.) or something like that and you only need dashboard security you can use Kibana Own Home plugin. It is multitenancy plugin for Kibana which creates separate .kibana index for every user. It works with Apache httpd or nginx. Read more here.
How to configure Netflix conductor to run standalone elastic search rather than embedded elastic search ?
if you have a conductor-config.properties just make sure you have these pointing to the valid elasticsearch you have up and running:
workflow.elasticsearch.instanceType=EXTERNAL
workflow.elasticsearch.url=http://elasticsearch:9200
Then should be able to run conductor up with that config:
java conductor-server-2.15.0-SNAPSHOT-all.jar conductor-config.properties
https://github.com/Netflix/conductor/blob/master/es5-persistence/src/main/java/com/netflix/conductor/dao/es5/index/ElasticSearchRestDAOV5.java
You can inspect this as an example, swapping the elastic container by your own, modifying the conductor-config.properties. It will be copied in when you run:
check out https://github.com/s50600822/conductor-cheat
inside the repo just do
docker-compose up
Check out https://github.com/Netflix/conductor/blob/master/es5-persistence/src/main/java/com/netflix/conductor/dao/es5/index/ElasticSearchRestDAOV5.java for other options.
To add external elastic search we need to follow code changes as mentioned in
below link.
https://github.com/Netflix/conductor/tree/master/es5-persistence.
And rebuild jar and run conductor server again with properties.
if you still get errors , I suggest to follow below link
https://github.com/Netflix/conductor/issues/489.
You can use the standalone installation of elasticsearch2 or elasticsearch5 because the associated support classes are already provided with Netflix Conductor binary.
To configure externally you have to do the following
Install and configure standalone elasticsearch. By default the
installation would expose 2 ports 9200/http or 9300/tcp.
Update server.properties file with the host and port so that the
communication will start happening with the standalone instance of
elasticsearch.
Hope this helps.
I’m trying to install grafana to work with OpenTSDB datasource. I’d like to know, what should I do to install it without elasticsearch?
I'm using grafana with Influxdb and I'm not using elasticsearch.
Grafana 2 is out in beta and I've been using that in production for a while. Grafana 2 now has its own data store, which either uses MySQL or SQLite. But you can always use Elasticsearch as well. You can read more about it here
Update: Stable version of Grafana 2 is now out, and it just works.
Grafana is a frontend, you will need some kind of database to store values and configuration in. I just grabbed the .tar.gz file from grafana's downloads page, created a config.js and pointed it at my influxdb server. No elasticsearch here, either.
You might want to take a look at gofana which will allow you to run Grafana without Elasticsearch. It's a self-contained binary that allows you to store dashboards on the filesystem and not in Elasticsearch or InfluxDB. It also supports HTTPS and basic authentication.
Note: I'm the author of gofana.