i have a web hook in slack and i have to configure it with watcher. the doc of elastic says this -
To configure a Slack account in Watcher, you set the watcher.actions.slack.service property in elasticsearch.yml. You must set the url to your incoming webhook integration URL.
but found doesn't give access to yml.
For example, on the local server, the following snippet configures an account called notify-monitoring and sets the default sender name to Watcher.
watcher.actions.slack.service:
account:
monitoring:
url: https://hooks.slack.com/services/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
message_default:
from: Watcher
how do i configure it on found??
It's been a while, but for anyone else who comes across this, the Elastic Cloud (previously branded as Found.io), now does support setting some (and only some!) elasticsearch.yml configuration through their web interface.
It's accessible under:
Clusters => Cluster Name => Configuration => User Settings
I was able to insert the Slack notification settings there successfully!
IMPORTANT: this change does not propagate immediately. It took some time for me, so you might have to go get a cup of coffee while it's applied.
Also, another tip: You might find it easier to add a default_account for Slack, as follows:
watcher.actions.slack.service:
default_account: monitoring
account:
monitoring:
url: <your_slack_hook_url>
message_defaults:
from: watcher
Related
I was trying to configure my application to not report my health endpoint in datadog APM./ I checked the documentation here: https://docs.datadoghq.com/tracing/guide/ignoring_apm_resources/?tab=kuberneteshelm&code-lang=java
And tried adding the config in my helm deployment.yaml file:
- name: DD_APM_IGNORE_RESOURCES
value: GET /actuator/health
This had no effect. Traces were still showing up in datadog. The method and path are correct. I changed the value a few times with different combinations (tried a few regex options). No go.
The I tried the DD_APM_FILTER_TAGS_REJECT environment variable, trying to ignore http.route:/actuator/health. Also without success.
I even ran the agent and application locally to see if there was anything to do with the environment, but the configs were not applied.
What are more options to try in this scenario?
This is the span detail:
As of a few weeks ago we added filebeat, metricbeat and apm to our dotnet core application ran on our kubernetes cluster.
It works all nice and recently we discovered filebeat and metricbeat are able to write a different index upon several rules.
We wanted to do the same for APM, however searching the documentation we can't find any option to set the name of the index to write to.
Is this even possible, and if yes how is it configured?
I also tried finding the current name apm-* within the codebase but couldn't find any matches upon configuring it.
The problem which we'd like to fix is that every space in kibana gets to see the apm metrics of every application. Certain applications shouldn't be within this space so therefore i thought a new apm-application-* index would do the trick...
Edit
Since it shouldn't be configured on the agent but instead in the cloud service console. I'm having troubles to 'user-override' the settings to my likings.
The rules i want to have:
When an application does not live inside the kubernetes namespace default OR kube-system write to an index called apm-7.8.0-application-type-2020-07
All other applications in other namespaces should remain in the default indices
I see you can add output.elasticsearch.indices to make this happen: Array of index selector rules supporting conditionals and formatted string.
I tried this by copying the same i had for metricbeat and updated it to use the apm syntax and came to the following 'user-override'
output.elasticsearch.indices:
- index: 'apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}'
when:
not:
or:
- equals:
kubernetes.namespace: default
- equals:
kubernetes.namespace: kube-system
but when i use this setup it tells me:
Your changes cannot be applied
'output.elasticsearch.indices.when': is not allowed
Set output.elasticsearch.indices.0.index to apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}
Set output.elasticsearch.indices.0.when.not.or.0.equals.kubernetes.namespace to default
Set output.elasticsearch.indices.0.when.not.or.1.equals.kubernetes.namespace to kube-system
Then i updated the example but came to the same conclusion as it was not valid either..
In your ES Cloud console, you need to Edit the cluster configuration, scroll to the APM section and then click "User override settings". In there you can override the target index by adding the following property:
output.elasticsearch.index: "apm-application-%{[observer.version]}-{type}-%{+yyyy.MM.dd}"
Note that if you change this setting, you also need to modify the corresponding index template to match the new index name.
I have a snippet of code that I'm invoking as a Lambda Function in AWS. Inside the lambda function, I'm making a call to an EKS cluster using the fabric8 client. All I'm doing at this point is having the client retrieve unbound PVCs:
KubernetesClient client = new DefaultKubernetesClient();
MixedOperation<PersistentVolumeClaim, PersistentVolumeClaimList,
DoneablePersistentVolumeClaim, Resource<PersistentVolumeClaim,
DoneablePersistentVolumeClaim>> operation = client.persistentVolumeClaims();
PersistentVolumeClaimList pvcList = operation.inAnyNamespace().list();
java.util.List<PersistentVolumeClaim> unboundItems = pvcList.getItems()
.stream()
.filter(pvc -> !"bound".equalsIgnoreCase(pvc.getStatus().getPhase()))
.collect(Collectors.toList());
One of the DevOps guys at work added my user ARN to a ConfigMap, so if I run this locally and just passing the AWS credentials through System Properties, I get results back. He has also experimented with adding the Lambda role ARN, but it doesn't work. If I deploy the code in my lambda, I get a 403 error:
"errorMessage": "Failure executing: GET at: https://XYZ.us-west-2.eks.amazonaws.com/api/v1/persistentvolumeclaims. Message: persistentvolumeclaims is forbidden: User \"system:anonymous\" cannot list resource \"persistentvolumeclaims\" in API group \"\" at the cluster scope. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=persistentvolumeclaims, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=persistentvolumeclaims is forbidden: User \"system:anonymous\" cannot list resource \"persistentvolumeclaims\" in API group \"\" at the cluster scope, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).",
"errorType": "io.fabric8.kubernetes.client.KubernetesClientException"
I'm pretty new to k8s, fabric8, and EKS, so apologies if I'm doing something obviously dumb here. I do have a ~/.kube/config file, which I include in the Lambda ZIP file (I've been told that this is insecure). The config file seems to work fine when I run the code locally. I'm setting the kubeconfig system property to point to the file. I understand that there needs to be some external mechanism like aws-iam-authenticator available to the k8s client to properly authenticate with EKS. What I don't know is if I should use some other mechanism when this runs in a Lambda?
I found an article that describes the general process of having a Lambda talk to EKS, and I've played around with the IAM role configuration as suggested, as well as adding the lambda role ARN to a ConfigMap, but so far nothing has helped. I'm really blocked on this, and if anyone has any tips, I'm all ears. Thanks.
I'm trying to create an OpenNMS alert when a certain folder ISN'T empty but can't seem to find a way of doing it. Any ideas?
I assume you have a service which goes down if your folder is empty. See the short video. By default notifications are turned off. Every service down event will be notified by default. You can be more granular by filtering on nodes and services. The default setting will send a mail to the admin user. You set a mail address in the user of the admin. To configure the access to your mail server, configure the javamail-configuration.properties. I just tried to figure out where you stuck exactly.
One approach could be to poll the certain directory for the empty condition with an agent on your host system and expose the status, e.g. Net-SNMP. You can create a service by using the SNMP Monitor to poll the status of the exposed OID and create a mail notification for this particular service.
Yes, this can be done. I have performed similar tasks using simple perl and bash scripts on Linux.
OpenNMS allows you to create polling configurations based on scripts. Your script is expected to output "0" or "1", with 0 representing "OK" and 1 representing "Not OK".
You could use the GeneralPurposePoller:
https://wiki.opennms.org/wiki/GeneralPurposePoller
However, it seems that you should instead use the SystemExecuteMonitor:
https://wiki.opennms.org/wiki/SystemExecuteMonitor
I am new to Bluemix as part creating an app following the steps below:
Log in to Bluemix with your Bluemix account.
Open the Catalog menu.
From the Runtimes section, click Liberty for Java.
In the App field, specify the name of your app [entered name]
Click the Create button
I am getting the error below:
BXNUI2032E: An error occurred when contacting Cloud Foundry to create a resource. Resource: routes. Cloud Foundry message: "The host is taken: vTest." See the Troubleshooting topics in the IBM Bluemix Documentation to check service status, review troubleshooting information, or for information about getting help.
Thanks in Advance
Bluemix will auto create the hostname for your app based on what you type into the name of the app. The above error means someone else has that hostname for their app.
The error message will probably look like the following.
cf push watson-um-demo
Creating app watson-um-demo in org jbsloyer#us.ibm.com / space demos as jbsloyer#us.ibm.com...
OK
Creating route watson-um-demo.mybluemix.net...
FAILED
Server error, status code: 400, error code: 210003, message: The host is taken: watson-um-demo
To resolve the issue you can modify the hostname field.
Bluemix assigns your app with a domain of mybluemix.net if you are running in the US-South region. If you are running in the EU-GB region Bluemix will assign your app with a domain of eu-gb.mybluemix.net You can map your own domains to your app as well. See https://www.ng.bluemix.net/docs/#manageapps/index-gentopic3.html#domain.
The guys before me already gave you the answer you were looking for, I just want share some good practices, if you don't mind. While it might not work for you, it has worked like a charm for me over past months
What has worked best for me is to start every App Name with my initials and than the app name. Colleagues of mine are doing the same just adding their name at the end.
So whenever I create a new app I follow my little schema in head:
Initials in front
no spaces just underscores
Descriptive and easy to remember so that I can quickly type it in the command line
The host you input will be the part of your url before mybluemix.net (yours would be vTest.mybluemix.net), so it needs to be globally unique across all of Bluemix. Try adding some random characters to the end of your host.
Don't stress out :) The reality is you likely will not want to use mybluemix.net for a customer facing URL/Route. Simply replace the name that you initially used that is already used by someone else with a compliant one see Restrictions on Valid Hostnames. It really doesn't matter which one.
Then get yourself a DNS provider like Domainmaster.com register your own custom domain. Like: www.mygreatsite.com
Then modify Bluemix to use your own custom domain you created. Read the documentation on Creating and using a custom domain with Bluemix.
You may want to setup other settings in your DNS like Cname, AAAA records and A records. Work with the DNS provider to figure out which ones you want to change.