We have an elasticsearch cluster deployed to the Elastic Cloud and would like to send monitoring/health metrics to Datadog. What is the best way to do that?
It seems like our options are:
Installing the datadog agent binary via the plugins upload
Using metric beat -> logstash -> datadog_metrics output
You can deploy the Datadog agent in a container / instance that you manage and the configure it according to these instructions to gather metrics from the remote ElasticSearch cluster that is hosted on Elastic Cloud. You need to create a conf.yaml file in the elastic.d/ directory and provide the required information (Elasticsearch endpoint/URL, username, password, port, etc) for the agent to be able to connect to the cluster. You may find a sample configuration file here.
As George Tseres mentioned above, the way I had to get this working was to set up collection on a separate instance (through docker) and then to configure it to read the specific Elastic Cloud instances.
I ended up making this: https://github.com/crwang/datadog-elasticsearch, building that docker image, and then pushing it up to AWS ECR.
Then, I spun up a Fargate service / task to run the container.
I also set it to run locally with docker-compose as a test.
Related
I'm trying to run only the Elastic agent as a deployment in a Kubernetes cluster. The reason I'm doing this is maybe an atypical usage of the Elastic agent: I only want to deploy the HTTP log endpoint integration and have other pods send logs to this Elastic agent. I'm not using it to collect cluster metrics (so the manifest they supply is not relevant to me).
I'm using the image docker.elastic.co/beats/elastic-agent:8.4.2. Apparently, this image needs to write files and directories to /usr/share/elastic-agent/, which at first was leading to errors along the lines of failed: mkdir /usr/share/elastic-agent/state: read-only file system. So, I created an emptyDir volume and mounted it at /usr/share/elastic-agent. Now, that error disappears, but is replaced with a new error:
/usr/local/bin/docker-entrypoint: line 14: exec: elastic-agent: not found
The entrypoint of the image is
ENTRYPOINT ["/usr/bin/tini" "--" "/usr/local/bin/docker-entrypoint"]
and it is apparently unable to find /usr/local/bin/docker-entrypoint.
A couple questions:
Why is it not finding the elastic-agent executable? It is definitely at that path.
More broadly: I am new to Elasticsearch -- this is only to set up a QA environment meant to test a product feature where we forward data from certain of our services to customers' Elastic Cloud deployments. I thought deploying the agent as a service in the same cluster where these services run would be the least painful way to do this. Is this not a good way to achieve what I describe in the first paragraph?
Assuming I can get the deployment to actually work, is this the way the next steps would go?
Create the "Custom HTTP Endpoint Logs" integration on the agent policy, listening on a given port and on all interfaces.
Map that port to an external port for the pod.
Send data to the pod at that external port.
The issue is that mounting the emptyDir volume to /usr/share overwrites the elastic-agent binary. Remove this volume and set readOnlyRootFilesystem: false.
I have an application up and running in set of docker containers (deployed using docker stack and not kubernetes). I want to do performance monitoring for this application. I am confused about whether I should go for beats or Elastic Agent.
This page says:
When Elastic Agent runs inside of a container, it cannot be upgraded through Fleet as it expects that the container itself is upgraded.
This page says:
Standalone mode — All policies are applied to the Elastic Agent manually as a YAML file.
Q1. Does this mean that in standalone mode Elastic Agent is not "at all" managed by Fleet? Or some part of management
This page says:
Standalone Elastic Agents are manually configured and managed locally on the systems where they are installed. They are useful when you are not interested in centrally managing agents in Fleet, either due to your company’s security requirements, or because you prefer to use another configuration management system.
This page says:
To run an Elastic Agent in standalone mode, install the agent on each host you want to monitor and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents.
Q2. Does this mean that for monitoring docker containers (deployed using docker stack and not kubernetes), there is no difference between between beats and Elastic agents in terms of "central" management? Only difference would be that I have to configure different beats separately, which is avoided with Elastic Agent?
Q3. What is preferrable in this case? Beats or Elastic Agent?
Standalone is not managed by fleet, the "it cannot be upgraded through Fleet" refers to upgrading the actual version of the elastic-agent, you can still update the agent polices.
Yes you would need to configure beats via a config file on the container where elastic agent can be setup with some env variables to enroll it in a policy, then that policy is centrally managed and updated via Kibana.
Both are valid but elastic agent allows updating the policy after the container is running via a central location and therefore would be my choice.
How should I go about deploying Jaeger on AWS ECS with Elasticsearch as backend? Is it a good idea to use the Jaeger all in one image or should I use separate images?
While I didn’t find any official jaeger reference to this, I think the jaeger all in one image is not intended for use in production. It makes one container a single point of failure, making it better to use separate containers for each jaeger component(if one is down from some reason - others can continue to operate).
I have recently written a blog post about hosting jaeger on AWS with AWS Elasticsearch (OpenSearch) service. While it is done with all-in-one, it is still useful to get the general idea of how to go about this.
Just to generally outline the process (described in detail in the post):
Create AWS Elasticsearch cluster
Create an ECS Cluster (running on ec2)
Create an ECS Task Definition, configured with a jaeger all-in-one image with the elasticsearch url from the step 1
Create an ECS Service that runs the created task definition
Make sure security groups on your EC2 allow access to jaeger ports as described here
Send spans to your jaeger endpoint via OpenTelemetry SDK
View your spans via the hosted jaeger UI (your-ec2-url:16686)
The all in one is a useful tool in development to test your work locally.
For deployment it is very limiting. Ideally to handle a potentially large volume of traffic you will want to scale parts of your infrastructure.
I would recommend deploying multiple jaeger-collectors, configured to write to the ES cluster. Then you can configure jaeger-agents running as a sidecar to each app or service broadcasting telemetry info. These agents can be configured to forward to one of a list of collectors adding some extra resilience.
I am new in elasticsearch and we want to write daily backup job in our production environment ,In our production environment we are using openshift and kubernetes, and our elasticsearch deploy as a docker container in kubernetes environment,I have some idea about elasticsearch snapshot strategies but how
to implement for daily elasticsearch backup job in containerized environment.
As per documentation and #Tim Wong post, you can use for this task "Scheduled Backup using curator".
More information how to combine snapshot with cronjob here and here.
Please share with your results and findings.
you need to write a cronjob and in that cronjob, you should use the host as service name it will take backup. (as per kubernets view)
I have configured a small instance of google kubernetes cluster with one node. I want to deploy elasticsearch service in this cluster. How do I set up that? I need the necessary steps.
In Google Cloud Marketplacep there are different categorie of elasticsearch if you want container images:
You need just to use the gcloud pull command
Elasticsearch 5:
gcloud auth configure-docker && docker pull marketplace.gcr.io/google/elasticsearch5:latest
Elasticsearch 6:
gcloud auth configure-docker && docker pull marketplace.gcr.io/google/elasticsearch6:latest
For kubernetes app (like depluing directly to your cluster)
As well you can deply using HELM as suggested by #Luiz
You can use the elasticsearch helm chart and manually tune it's resources limits and requests to fit into one node.