I am publishing metrics from local services I control to an OpenTelemetry Prometheus exporter.
$otel = Invoke-WebRequest http://localhost:8889/metrics
$otel.RawContent -split "`n" | Select-String Global_Foo
# HELP Global_Foo_x
# TYPE Global_Foo_x gauge
Global_Foo_x{LogLevel="Information"} 0.5
# HELP Global_Foo_y
# TYPE Global_Foo_y gauge
Global_Foo_z{LogLevel="Information"} 0.4
Global_Foo_z{LogLevel="Warning"} 0.4
# HELP Global_Foo_z
# TYPE Global_Foo_z gauge
Global_Foo_z{LogLevel="Information"} 0.1
I would like to exclude Global_Foo_y because reasons*, but have been unable to do so to my satisfaction. I have a clunky workaround but I'm hoping there's a more elegant implementation.
Using a delete_label_value metricstransform. This successfully deletes the label as long as a static label exists against which I can filter. I force this label to exist by also applying an add_label transform before the delete_label_value element.
processors:
metricstransform:
transforms:
- include: Global.Foo.y
match_type: strict
action: update
operations:
- action: add_label
new_label: deleteme
new_value: true
- include: Global.Foo.y
match_type: strict
action: update
operations:
- action: delete_label_value
label: deleteme
label_value: true
Is there a better way to do this?
* There are times I want to exclude metrics from export but it will be a while before I am able to change the running application config to prevent export.
Related
I transfer logfiles with filebeat to elasticsearch.
The data are analyzed with kibana.
Now to my problem:
Kibana shows not the timestamp from the logfile.
Kibana shows the time of the transmission in #timestamp.
I want to show the timestamp from the logfile in kibana.
But the timestamp in the logfile is overwritten.
Where is my fault?
Has anyone a solution for my problem?
Here a example from my logfile and the my filebeat config.
{"#timestamp":"2022-06-23T10:40:25.852+02:00","#version":1,"message":"Could not refresh JMS Connection]","logger_name":"org.springframework.jms.listener.DefaultMessageListenerContainer","level":"ERROR","level_value":40000}
## Filebeat configuration
## https://github.com/elastic/beats/blob/master/deploy/docker/filebeat.docker.yml
#
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
# The Docker autodiscover provider automatically retrieves logs from Docker
# containers as they start and stop.
- type: docker
hints.enabled: true
filebeat.inputs:
- type: filestream
id: pls-logs
paths:
- /usr/share/filebeat/logs/*.log
parsers:
- ndjson:
processors:
- add_cloud_metadata: ~
output.elasticsearch:
hosts: ['http://elasticsearch:9200']
username: elastic
password:
## HTTP endpoint for health checking
## https://www.elastic.co/guide/en/beats/filebeat/current/http-endpoint.html
#
http.enabled: true
http.host: 0.0.0.0
Thanks for any support!
Based upon the question, this could be one potential option, which would be to use filebeat processors. What you could do is write that initial #timestamp value to another field, like event.ingested, using the following script below:
#Script to move the timestamp to the event.ingested field
- script:
lang: javascript
id: init_format
source: >
function process(event) {
var fieldTest = event.Get("#timestamp");
event.Put("event.ingested", fieldTest);
}
And then the last processor you write could move that event.ingested field to #timestamp again using the following processor:
#setting the timestamp field to the Date/time when the event originated, which would be the event.created field
- timestamp:
field: event.created
layouts:
- '2006-01-02T15:04:05Z'
- '2006-01-02T15:04:05.999Z'
- '2006-01-02T15:04:05.999-07:00'
test:
- '2019-06-22T16:33:51Z'
- '2019-11-18T04:59:51.123Z'
- '2020-08-03T07:10:20.123456+02:00'
I need rundeck to parse an option based on the value selected in another option. I have an option ${option.env} and other options like ${option.id_dev}, ${option.id_qa}
I want to achieve something like below for extra-vars, so that "env" option value determines which id(dev or qa) to read.
ansible-playbook /build.yml -e id=${option.id_${option.env.value}}
Is this possible or Could I pass extra-vars like a conditional case based on env value ?. I'm using rundeck 3.0.X
Update :
To give clear info, If I select 'dev' for the option 'env', I need to use its value like ${option.id_${option.env.value}} , so it translates to ${option.id_dev} to get other option in the command line
You can use cascade remote options in a tricky way. The explanation is at the end of this answer.
I made a little example to see how to achieve this:
The branches.json file (referenced on the job options as "branches"):
[
{"name":"branch1", "value":"branch1.json"},
{"name":"branch2", "value":"branch2.json"},
{"name":"branch3", "value":"branch3.json"}
]
The branch1.json is the first tentative value of the branches option:
[
{"name":"v1", "value":"1"},
{"name":"v2", "value":"2"},
{"name":"v3", "value":"3"}
]
The branch2.json is the second tentative value of the branches option:
[
{"name":"v4", "value":"4"},
{"name":"v5", "value":"5"},
{"name":"v6", "value":"6"}
]
The branch3.json is the third tentative value of the branches option:
[
{"name":"v7", "value":"7"},
{"name":"v8", "value":"8"},
{"name":"v9", "value":"9"}
]
Full job definition to test:
- defaultTab: summary
description: ''
executionEnabled: true
id: ed0d84fe-135b-41ee-95b6-6daeaa94894b
loglevel: INFO
name: CascadeTEST
nodeFilterEditable: false
options:
- enforced: true
name: branches
valuesUrl: file:/Users/myuser/branches.json
- enforced: true
name: level2
valuesUrl: file:/Users/myuser/${option.branches.value}
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- fileExtension: .sh
interpreterArgsQuoted: false
script: |+
#!/bin/sh
# getting the options
first=#option.branches#
second=#option.level2#
# this just an example
echo "this an example: ${first%%.*}.$second"
scriptInterpreter: /bin/bash
keepgoing: false
strategy: node-first
uuid: ed0d84fe-135b-41ee-95b6-6daeaa94894b
Explanation
As you see, the value of the first option is always a file name, if you take the value directly you always obtain an "option1.json" like string, so, the trick here is to cut the file extension and take only the name as the value, for that, I used this approach.
So, with the first selection value and the second one, you can do anything later in a script, for example, launch the ansible-playbook command in the inline script.
Check the example result.
UPDATED ANSWER:
The closest way is just using two options and concatenating them in the command step but isn't possible to get the option value inside the other one as you say.
Just like this:
- defaultTab: nodes
description: ''
executionEnabled: true
id: 4e8df698-c7ca-4a10-9f70-bc68c1007a10
loglevel: INFO
name: NewJob
nodeFilterEditable: false
options:
- enforced: true
name: env
value: qa
values:
- qa
- prod
- stage
valuesListDelimiter: ','
- enforced: true
name: id_dev
value: '1'
values:
- '1'
- '2'
- '3'
valuesListDelimiter: ','
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: echo ${option.env}_${option.id_dev}
keepgoing: false
strategy: node-first
uuid: 4e8df698-c7ca-4a10-9f70-bc68c1007a10
Result.
(the cascade option is another approach)
Using Elasticsearch curator, how do I delete all indices matching a pattern, except for the newest?
I tried using filtertype: age but it does not seem to do what I need.
Here is an example code which you can use to delete the indices which
are older than 14 days assuming your index name have the date in it. You can get more information on the below link
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/curator.html
import os
import sys
import json, io, boto3
import time, datetime
import curator
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3
esEndPoint = ES_HOST # Add the ElasticSearch host.
region = REGION # Region where the ElasticSearch is present.
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
def lambda_handler(event, context):
esClient = connectES(esEndPoint)
index_list = curator.IndexList(esClient)
index_list.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=14)
print(index_list.indices)
if index_list.indices:
curator.DeleteIndices(index_list).do_action() # Delete the indices
def connectES(esEndPoint):
# Function used to connect to ES
try:
es = Elasticsearch(
hosts=[{'host': esEndPoint, 'port': 443}],
http_auth=awsauth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
return es
except Exception as E:
print("Unable to connect to {0}".format(esEndPoint))
print(E)
You need two filters: pattern (to match the indexes you want to delete) and age (to specify the age of the indexes to delete).
For instance the Curator configuration below is configured to delete
indexes named example_dev_*
and which are older than 10 days
Configuration:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 10 days (based on index name), for example_dev_
prefixed indices.
options:
ignore_empty_list: True
disable_action: True
filters:
- filtertype: pattern
kind: prefix
value: example_dev_
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 10
- filtertype: count
count: 1
You need to adapt both filter conditions to your needs, but that would achieve what you expect.
I suggest using the count filter after the pattern filter. Be sure to play with exclude true/false and dry-runs until it does what you expect.
I need to remove the tags field from each of the methods in my OpenAPI spec.
The spec must be in YAML format, as converting to JSON causes issues later on when publishing.
I couldn't find a ready tool for that, and my programming skills are insufficient. I tried Python with ruamel.yaml, but could not achieve anything.
I'm open to any suggestions how to approach this - a repo with a ready tool somewhere, a hint on what to try in Python... I'm out of my own ideas.
Maybe a regex that catches all cases all instances of tags so I can do a search and replace with Python, replacing them with nothing? Empty lines don't seem to break the publishing engine.
Here's a sample YAML piece (I know this is not a proper spec, just want to show where in the YAML tags sits)
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint
get:
tags:
-
tag1
-
tag3
#rest of GET definition
post:
tags:
- tag2
/anotherEndpoint
post:
tags:
- tag1
I need to get rid of all tags arrays entirely (not just make them empty)
I am not sure why you couldn't achieve anything with Python + ruamel.yaml. Assuing your spec
is in a file input.yaml:
import sys
from pathlib import Path
import ruamel.yaml
in_file = Path('input.yaml')
out_file = Path('output.yaml')
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(in_file)
# if you only have the three instances of 'tags', you can hard-code them:
# del data['paths']['/endpoint']['get']['tags']
# del data['paths']['/endpoint']['post']['tags']
# del data['paths']['/anotherEndpoint']['post']['tags']
# generic recursive removal of any key names 'tags' in the datastructure:
def rm_tags(d):
if isinstance(d, dict):
if 'tags' in d:
del d['tags']
for k in d:
rm_tags(d[k])
elif isinstance(d, list):
for item in d:
rm_tags(item)
rm_tags(data)
yaml.dump(data, out_file)
which gives as output.yaml:
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint:
get: {}
post: {}
/anotherEndpoint:
post: {}
You can write back data to input.yaml if you need that.
Please note that normally the comment #rest of GET definition would be preserved, but
not now as it is associated during loading with the key before it and that key gets deleted.
In my ansible coding i want to know the status of the service like service httpd status (service is runngin or not) the result would be store in to variable. Using that status i will use some other code in ansible.
I am using ansible service module there is no option for status. If i use the shell module i got this warning
[WARNING]: Consider using service module rather than running service
so is it any other module doing to get service status?
No, there is no standard module to get services' statuses.
But you can suppress warning for specific command task if you know what are you doing:
- command: service httpd status
args:
warn: false
I've posted a quick note about this trick a while ago.
You can use the service_facts module.
For example, say I want to see the status of Apache.
- name: Check for apache status
service_facts:
- debug:
var: ansible_facts.services.apache2.state
The output is:
ok: [192.168.blah.blah] => {
"ansible_facts.services.apache2.state": "running"
}
If you would like to see all of them, you can do that by just going two levels up in the array:
var: ansible_facts.services
The output will list all the services, and will look like this (truncated for the sake of brevity):
"apache2": {
"name": "apache2",
"source": "sysv",
"state": "running"
},
"apache2.service": {
"name": "apache2.service",
"source": "systemd",
"state": "running"
},
"apparmor": {
"name": "apparmor",
"source": "sysv",
"state": "running"
},
etc,
etc
I am using Ansible 2.7. Here are the docs for that module: Click here
here is an example of starting a service and then checking status using service facts, in my example you have to register the variable then output it using debug var and pointing to the correct format in the json chain resulting output:
## perform start service for alertmanager
- name: Start service alertmanager if not started
become: yes
service:
name: alertmanager
state: started
## check to see the state of the alertmanager service status
- name: Check status of alertmanager service
service_facts:
register: service_state
- debug:
var: service_state.ansible_facts.services["alertmanager.service"].state
Hopefully service: allow user to query service status #3316 will be merged into the core module soon.
You can patch it by hand using this diff to system/service.py
Here's my diff using ansible 2.2.0.0. I've run this on my mac/homebrew install and it works for me.
This is the file that I edited: /usr/local/Cellar/ansible/2.2.0.0_2/libexec/lib/python2.7/site-packages/ansible/modules/core/system/service.py
## -36,11 +36,12 ##
- Name of the service.
state:
required: false
- choices: [ started, stopped, restarted, reloaded ]
+ choices: [ started, stopped, status, restarted, reloaded ]
description:
- C(started)/C(stopped) are idempotent actions that will not run
- commands unless necessary. C(restarted) will always bounce the
- service. C(reloaded) will always reload. B(At least one of state
+ commands unless necessary. C(status) would report the status of
+ the service C(restarted) will always bounce the service.
+ C(reloaded) will always reload. B(At least one of state
and enabled are required.)
sleep:
required: false
## -1455,7 +1456,7 ##
module = AnsibleModule(
argument_spec = dict(
name = dict(required=True),
- state = dict(choices=['running', 'started', 'stopped', 'restarted', 'reloaded']),
+ state = dict(choices=['running', 'started', 'stopped', 'status', 'restarted', 'reloaded']),
sleep = dict(required=False, type='int', default=None),
pattern = dict(required=False, default=None),
enabled = dict(type='bool'),
## -1501,6 +1502,9 ##
else:
service.get_service_status()
+ if module.params['state'] == 'status':
+ module.exit_json(state=service.running)
+
# Calculate if request will change service state
service.check_service_changed()