OpenTelemetry dotnet otel configuration - open-telemetry

I'm writing simple project for study OpenTelemetry.
I want to use otel for configure many exporter.
Sample
Can anyone suggest me how can I must configure agent and collector?
I tried github sample but don't work properly.
My docker-compose:
otel-collector:
container_name: otel-collector
image: otel/opentelemetry-collector
volumes:
- ./../config/otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "1888:1888" # pprof extension
- "8888:8888" # Prometheus metrics exposed by the collector
- "8889:8889" # Prometheus exporter metrics
- "13133:13133" # health_check extension
- "4317" # OTLP gRPC receiver
- "55670:55679" # zpages extension
depends_on:
- zipkin
otel-agent:
container_name: otel-agent
image: otel/opentelemetry-collector
command: ["--config=/etc/otel-agent-config.yaml", "${OTELCOL_ARGS}"]
volumes:
- ./../config/otel-agent-config.yaml:/etc/otel-agent-config.yaml
ports:
- "8887:8888" # Prometheus metrics exposed by the agent
- "14250" # Jaeger grpc receiver
- "14268" # Jaeger http thrift receiver
- "55678" # OpenCensus receiver
- "4317" # OTLP gRPC receiver
- "9411" # Zipkin receiver
- "1777:1777" # pprof extension
- "55679:55679" # zpages extension
- "13133" # health_check
depends_on:
- otel-collector
My Agent config:
I want to receive data from .net application and redirect it to otel-collector.
receivers:
otlp:
protocols:
grpc:
http:
zipkin:
exporters:
otlp:
endpoint: otel-collector:4317
insecure: true
logging:
loglevel: debug
processors:
batch:
extensions:
health_check:
pprof:
endpoint: 0.0.0.0:1777
zpages:
endpoint: 0.0.0.0:55679
service:
extensions: [health_check, pprof, zpages]
pipelines:
traces:
receivers: [otlp, zipkin]
processors: [batch]
exporters: [otlp, logging]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp, logging]
My collector config:
I want to receive data from agent and redirect to exporter:
zipkin, newrelic, logger.
receivers:
otlp:
protocols:
grpc:
http:
exporters:
logging:
loglevel: debug
zipkin:
endpoint: "http://zipkin:9411/api/v2/spans"
format: proto
newrelic:
apikey: NRAK-abc
timeout: 30s
traces:
host_override: trace-api.eu.newrelic.com
timeout: 20s
metrics:
host_override: metric-api.eu.newrelic.com
logs:
host_override: log-api.eu.newrelic.com
processors:
batch:
extensions:
health_check:
pprof:
endpoint: :1888
zpages:
endpoint: :55679
service:
extensions: [pprof, zpages, health_check]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, zipkin, newrelic]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, zipkin, newrelic]
My dotnet application startup:
services.AddOpenTelemetryTracing((builder) => builder
.AddSource("Sample")
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("SampleApi").AddTelemetrySdk())
.AddAspNetCoreInstrumentation(options =>
{
options.Filter = (req) => !req.Request.Path.ToUriComponent().Contains("swagger", StringComparison.OrdinalIgnoreCase);
})
.AddOtlpExporter(otlpOptions =>
{
otlpOptions.Endpoint = new Uri("http://otel-collector:4317");
})

Related

how to fix swagger-ui path

There was a problem with the Swagger document path.
build.gradle
implementation("org.springdoc:springdoc-openapi-ui:1.6.8")
application.properties
server.servlet.context-path: /poo
REQ : http://localhost:port/poo/swagger-ui/index.html
-> Failed to load remote configuration.
REQ : http://localhost:port/poo//swagger-ui/index.html
-> OK
Can be Ingress, try this
Change this
spec:
rules:
- http:
paths:
- path: /my-context/my-path/swagger
pathType: ImplementationSpecific
backend:
service:
name: my-service #.ns-config.svc.cluster.local
port:
number: 8080
For this
spec:
rules:
- http:
paths:
- backend:
service:
name: my-service #.ns-config.svc.cluster.local
port:
number: 8080
path: /my-context/my-path/swagger
pathType: ImplementationSpecific
Might check this comment describing the best approach to using spring-doc behind custom context-path or proxy.

ECK Filebeat Daemonset Forwarding To Remote Cluster

I wish to forward logs from remote EKS clusters to a centralised EKS cluster hosting ECK.
Versions in use:
EKS v1.20.7
Elasticsearch v7.7.0
Kibana v7.7.0
Filebeat v7.10.0
The setup is using a AWS NLB to forward requests to Nginx ingress, using host based routing.
When the DNS lookup (filebeat test output) for the Elasticsearch is tested on Filebeat, it validates the request.
But the logs for Filebeat are telling a different story.
2021-10-05T10:39:00.202Z ERROR [publisher_pipeline_output]
pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)):
Get "https://elasticsearch.dev.example.com:9200": Bad Request
The Filebeat agents can connect to the remote Elasticsearch via the NLB, when using a curl request.
The config is below. NB: dev.example.com is the remote cluster hosing ECK.
app:
name: "filebeat"
configmap:
enabled: true
filebeatConfig:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- config:
- type: container
paths:
- /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- drop_event.when.not.or:
- contains.kubernetes.namespace: "apps-"
- equals.kubernetes.namespace: "cicd"
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
overwrite_keys: true
- add_fields:
fields:
kubernetes.cluster.name: dev-eks-cluster
target: ""
processors:
- add_cloud_metadata: ~
- add_host_metadata: ~
cloud:
id: '${ELASTIC_CLOUD_ID}'
cloud:
auth: '${ELASTIC_CLOUD_AUTH}'
output:
elasticsearch:
enabled: true
hosts: "elasticsearch.dev.example.com"
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl:
verification_mode: "none"
headers:
Host: "elasticsearch.dev.example.com"
proxy_url: "https://example.elb.eu-west-2.amazonaws.com"
proxy_disable: false
daemonset:
enabled: true
version: 7.10.0
image:
repository: "docker.elastic.co/beats/filebeat"
tag: "7.10.0"
pullPolicy: Always
extraenvs:
- name: ELASTICSEARCH_HOST
value: "https://elasticsearch.dev.example.com"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
value: "remote-cluster-elasticsearch-es-elastic-user-password"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
clusterrolebinding:
enabled: true
namespace: monitoring
clusterrole:
enabled: true
serviceaccount:
enabled: true
namespace: monitoring
deployment:
enabled: false
configmap:
enabled: false
Any tips or suggestions on how to enable Filebeat forwarding, would be much appreciated :-)
#1 Missing ports:
Even with the ports added in as suggested. Filebeat is erroring with:
2021-10-06T08:34:41.355Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)): Get "https://elasticsearch.dev.example.com:9200": Bad Request
...using a AWS NLB to forward requests to Nginx ingress, using host based routing
How about unset proxy_url and proxy_disable, then set hosts: ["<nlb url>:<nlb listener port>"]
The final working config:
app:
name: "filebeat"
configmap:
enabled: true
filebeatConfig:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- config:
- type: container
paths:
- /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- drop_event.when.not.or:
- contains.kubernetes.namespace: "apps-"
- equals.kubernetes.namespace: "cicd"
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
overwrite_keys: true
- add_fields:
fields:
kubernetes.cluster.name: qa-eks-cluster
target: ""
processors:
- add_cloud_metadata: ~
- add_host_metadata: ~
cloud:
id: '${ELASTIC_CLOUD_ID}'
cloud:
auth: '${ELASTIC_CLOUD_AUTH}'
output:
elasticsearch:
enabled: true
hosts: ["elasticsearch.dev.example.com:9200"]
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl:
verification_mode: "none"
daemonset:
enabled: true
version: 7.10.0
image:
repository: "docker.elastic.co/beats/filebeat"
tag: "7.10.0"
pullPolicy: Always
extraenvs:
- name: ELASTICSEARCH_HOST
value: "https://elasticsearch.dev.example.com"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
value: "remote-cluster-elasticsearch-es-elastic-user-password"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
clusterrolebinding:
enabled: true
namespace: monitoring
clusterrole:
enabled: true
serviceaccount:
enabled: true
namespace: monitoring
deployment:
enabled: false
configmap:
enabled: false
In addition the following changes were needed:
NBL:
Add listener for 9200 forwarding to the Ingress Controller for HTTPS
SG:
Opened up port 9200 on the EKS worker nodes

POST https://apm.<acme>.com/intake/v2/rum/events net::ERR_BLOCKED_BY_CLIENT

I have an elastic stack on kubernetes (k8s) using ECK.
Kibana version:
7.13.2
Elasticsearch version:
7.13.2
APM Server version:
7.13.2
APM Agent language and version:
https://www.npmjs.com/package/#elastic/apm-rum - 5.9.1
Browser version:
Chrome latest
Description of the problem
Frontend apm-run agent fails to send messages to apm server. if i disable cors on the browser it works - google-chrome --disable-web-security --user-data-dir=temp then navigate to my frontend http://localhost:4201/
[Elastic APM] Failed sending events! Error: https://apm.<redacted>.com/intake/v2/rum/events HTTP status: 0
at ApmServer._constructError (apm-server.js:120)
at eval (apm-server.js:48)
POST https://apm.<acme>.com/intake/v2/rum/events net::ERR_BLOCKED_BY_CLIENT
Code:
apm.yml
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-server-prod
namespace: elastic-system
spec:
version: 7.13.2
count: 1
elasticsearchRef:
name: "elasticsearch-prod"
kibanaRef:
name: "kibana-prod"
http:
service:
spec:
type: NodePort
config:
apm-server:
rum.enabled: true
ilm.enabled: true
elastic.ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: elastic-ingress
namespace: elastic-system
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/group.name: "<redacted>"
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
alb.ingress.kubernetes.io/backend-protocol: 'HTTPS'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:<rd>:certificate/0250a551-8971-468d-a483-cad28f890463
alb.ingress.kubernetes.io/tags: Environment=prod,Team=dev
alb.ingress.kubernetes.io/healthcheck-path: /health
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '300'
alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=<redacted>-aws-ingress-logs,access_logs.s3.prefix=dev-ingress
spec:
rules:
- host: elasticsearch.<redacted>.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: elasticsearch-prod-es-http
port:
number: 9200
- host: kibana.<redacted>.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kibana-prod-kb-http
port:
number: 5601
- host: apm.<redacted>.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apm-server-prod-apm-http
port:
number: 8200
frontend.js
import { init as initApm } from "#elastic/apm-rum";
import config from "<redacted>-web/config/environment";
export const apm = initApm({
serviceName: "frontend",
serverUrl: "https://apm.<redacted>.com",
environment: config.environment,
logLevel: "debug",
});
Errors in browser console:
apm server:
apm-server pod does not apear to display any errors in this case, i assume the client never reaches the server .
I was running into the same problem. Check your ad blocker. I found that UBlock was blocking requests to */rum/events.
I'm guessing that they consider this as a type of user "tracker" and that's why they're blocked, really no way around it though unless you change the endpoint path I guess.

Filebeat: How to export logs of specific pods

This is my filebeat config map.
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: $${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
setup.ilm.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
This sends logs from every pod to AWS ElasticSearch. How I can restrict it to send logs from specific pods by name and/or by the label?

The Kubernetes ingress doesn't allow Websocket communication over HTTPS

I am trying to host my WebAPI service in kubernetes over HTTPS. The service has inbound communications over port 80 and outbound websocket communication over port 81. I have configured AKS Ingress controller with a static IP assigned to it.
The application is working just fine in HTTP protocol but when I enabled TLS certificate to enable HTTPS, the websocket communication is not working any more.
Ingress.yaml
apiVersion: extensions/v1beta1
metadata:
name: mywebapiservice
namespace: mywebapi
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
ingress.kubernetes.io/rewrite-target: /
ingress.kubernetes.io/ssl-redirect: 'true'
kubernetes.io/ingress.class: mywebapi_customclass
kubernetes.io/tls-acme: 'true'
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/proxy-connect-timeout: '7200'
nginx.ingress.kubernetes.io/proxy-read-timeout: '7200'
nginx.ingress.kubernetes.io/proxy-send-timeout: '7200'
nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
tls:
- hosts:
- mywebapi.eastus.cloudapp.azure.com
secretName: my-tls-secret
rules:
- host: mywebapi.eastus.cloudapp.azure.com
http:
paths:
- path: >-
/app/api/ServiceA/subscription
backend:
serviceName: ServiceA
servicePort: 81
- path: >-
/app/api/ServiceA/(.+)
backend:
serviceName: ServiceA
servicePort: 80
- path: >-
/app/api/ServiceB/subscription
backend:
serviceName: ServiceB
servicePort: 81
- path: >-
/app/api/ServiceB/(.+)
backend:
serviceName: ServiceB
servicePort: 80
status:
loadBalancer:
ingress:
- ip: xx.xx.xxx.xxx
The c# code await listener.GetContextAsync() is never executed.
listener = new HttpListener();
listener.Prefixes.Add("wss://*:81/app/api/ServiceB/subscription/");
var listenerContext = await listener.GetContextAsync();
The error that I get at client side : Error during WebSocket handshake: Unexpected response code: 503
Please suggest if you find something wrong here.

Resources