Multiple paths in http module - metricbeats - elasticsearch

I am using http module of metricbeats to monitor jmx. I am using http module instead of the jolokia module because it lacks wildcard support at this point. The example configuration in the documents is as follows.
- module: http
metricsets: ["json"]
period: 10s
hosts: ["localhost:80"]
namespace: "json_namespace"
path: "/jolokia/"
body: '{"type" : "read", "mbean" : "kafka.consumer:type=*,client-id=*", "attribute" : "count"}'
method: "POST"
This works fine and I am able to get data to kibana. I see errors when I configure it as follows to call multiple paths.
- module: http
metricsets: ["json"]
enabled: true
period: 10s
hosts: ["localhost:80"]
namespace: "metrics"
method: POST
paths:
- path: "/jolokia/"
body: '{"type" : "read", "mbean" : "kafka.consumer:type=*,client-id=*", "attribute" : "bytes-consumed-rate"}'
- path: "/jolokia/"
body: '{"type" : "read", "mbean" : "kafka.consumer:type=*,client-id=*", "attribute" : "commit-latency-avg"}'
This does not seem to be the right config and I see that the http events have had failures.
2018/02/26 19:53:18.315740 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=4767600 beat.memstats.memory_alloc=4016168 beat.memstats.memory_total=47474256 libbeat.config.module.running=3 libbeat.output.read.bytes=4186 libbeat.output.write.bytes=16907 libbeat.pipeline.clients=7 libbeat.pipeline.events.active=0 libbeat.pipeline.events.published=18 libbeat.pipeline.events.total=18 libbeat.pipeline.queue.acked=18 metricbeat.http.json.events=3 metricbeat.http.json.failures=3
Documentation on how to setup http module: Example configuration

I had to query multiple URL's of my REST API and I could achieve that by having multiple modules of "http" with different host URL's following is the example:
- module: http
metricsets: ["json"]
period: 3600s
hosts: ["http://localhost/Projects/method1/"]
namespace: "testmethods"
method: "GET"
enabled: true
- module: http
metricsets: ["json"]
period: 3600s
hosts: ["http://localhost/Projects/method2/"]
namespace: "testmethods"
method: "GET"
enabled: true
This made me achieve have multiple paths for the same module

Multiple paths are not supported by the http module's json metricset.
What you found in the config example is for the http module's server metricset. This metricset does not query URLs. Instead it opens an http server on the specified port, and can receive input on multiple paths which are used to separate data into different namespaces.

Related

Elastic HeartBeat HTTP

I am working on a monitoring solution, and I am facing some issues. I need to create a request towards a URL, and parse the response. I have tried the following, but cannot find the message in grafana :
- type: http
urls: ["https://securpharm-status.de/status"]
schedule: '#every 600s'
proxy_url: http://user#1.1.1.1
tags: ["name"]
name: "status"
check.request:
method: GET
check.response:
json:
- description: check status
condition:
equals:
status: ok
also tried many other options, but still only the response code has been returned.

Kong lambda plugin - Correlation id being dropped

so I'm using kong as an API gateway. I have a simple lambda function on aws returning headers. I'm able to see a response from the aws lambda function, however there is no correlation id. I have set the correlation-id plugin global. Would like some clarification on why the lambda plugin is not adding the correlation-id in the headers of the original response.
exports.handler = async (event) => {
// TODO implement
const response = {
statusCode: 200,
body: JSON.stringify(event.request_headers),
};
return response;
};
My kong.yaml
_format_version: "1.1"
routes:
- name: lambda1
paths:
- /lambda1
path_handling: v0
preserve_host: false
protocols:
- http
- https
regex_priority: 0
strip_path: true
https_redirect_status_code: 426
plugins:
- name: aws-lambda
config:
aws_key: XXXXXXXXXXXXXXXXXXXXXXX
aws_region: us-east-1
aws_secret: XXXXXXXXXXXXXXXXXXXXXXXXx
awsgateway_compatible: false
forward_request_body: true
forward_request_headers: true
forward_request_method: false
forward_request_uri: false
function_name: kong-lambda-plugin
host: null
invocation_type: RequestResponse
is_proxy_integration: false
keepalive: 60000
log_type: Tail
port: 443
proxy_scheme: null
proxy_url: null
qualifier: null
skip_large_bodies: true
timeout: 60000
unhandled_status: null
enabled: true
protocols:
- grpc
- grpcs
- http
- https
plugins:
- name: correlation-id
config:
echo_downstream: false
generator: uuid#counter
header_name: correlation-id
enabled: true
protocols:
- grpc
- grpcs
- http
- https
So what worked for me was
https://github.com/Kong/priority-updater
I created the .rock file generated by the script above.
Since I'm using a dockerized version of Kong, I coppied the file over to my Kong docker host.
Installed the plugin in my Kong docker host using luarocks command.
Added the plugin to my 'kong.conf' file.
Added it to the lambda route and worked beautifully.
You are right the problem comes from the plugin priority
CorrelationIdHandler.PRIORITY = 1
AWSLambdaHandler.PRIORITY = 750
Aws Lambda plugin is breaking the chain of plugins as in the handler phase it is doing
return kong.response.exit(status, content, headers)
So the other plugins cannot been used.
You can create a custom correlationId plugin and change the PRIORITY there is a tool to handle that for you https://github.com/Kong/priority-updater
You can also create a route to add the correlation id calling another route doing the lambda call

Spring Cloud Gateway - RewriteLocationResponseHeader

I have the following route inside my Spring Cloud Gateway configuration:
- id: pgadmin
uri: lb://pg-admin-service
predicates:
- Path=/pgadmin/**
filters:
- RewritePath=/pgadmin(?<segment>/?.*), $\{segment}
- RewriteLocationResponseHeader=AS_IN_REQUEST, Location, , <---- this line is incorrect
The request comes in on http://10.0.0.100/pgadmin/ as an unauthenticated user. The application returns a response containing a location header with the value:
http://10.0.0.100/login
The browser tries to rediect to that URL. Instead, it should redirect to:
http://10.0.0.100/pgadmin/login
What is the correct value for the RewriteLocationResponseHeader in the route definition?
Many thanks
JT
Try something this if you want to rewrite your path:
- id: docs
uri: lb://auth-service
predicates:
- Path=/my-auth/**
filters:
- RewritePath=/my-auth(?<segment>/?.*), $\{segment}
or to simply forward the request onto your service:
- id: docs
uri: lb://auth-service
predicates:
- Path=/my-auth/**

Envoy lua filter - http call cluster invalid. Must be configured

I am trying to create an envoy filter for all of my microservices deployed on a GKE cluster which is running istio.
In the filter I want to read 2 header values and send a request to an external service on web to get list of all groups that the user has authority of.
When I send the request I get "http call cluster invalid. Must be configured" error.
As per my understanding we need to register the host as cluster so that envoy understand where to send the request and I have added a value in "clusters" section of my filter.yaml file.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: authorization-filter
namespace: default
spec:
filters:
- listenerMatch:
listenerType: SIDECAR_INBOUND
listenerProtocol: HTTP
filterName: envoy.lua
filterType: HTTP
filterConfig:
inlineCode: |
function envoy_on_request(request_handle)
request_handle:logWarn("Inside filter")
local token = request_handle:headers():get("Authorization")
local partition = request_handle:headers():get("partition-id")
if token == nil or partition == nil then
request_handle:logErr("Token or Partition is empty")
request_handle:respond({[":status"] = "400",["Content-Type"] = "application/json"}, "{\"ErrorMessage\":\"Bad Request: Authorization or Partition-Id missing in headers\"}")
end
request_handle:logWarn("Sending request")
local headers, body = request_handle:httpCall(
"lua_cluster",
{
[":method"] = "GET",
[":authority"] = "example.com",
[":path"] = "/api/authorization/groups",
["Authorization"] = token,
["partition-id"] = partition
},
nil,
5000)
request_handle:headers():add("authorization-response-headers", headers)
end
clusters:
- name: lua_cluster
connect_timeout: 0.5s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: "example.com"
port_value: 8888
What am i missing here?

Spring Cloud Gateway : disable default routes

I'm using spring cloud to manage my microservices.
For security reasons, for one specific microservice (name it ms_secure), I want to use custom route choose a specific microservice version depending on client IP.
My gateway config looks like this:
spring:
cloud:
gateway:
discovery:
locator:
enabled: true
routes:
- id: ms_secure_v1
uri: lb://ms_secure_v1
predicates:
- Path=/ms_secure/**
filters:
- RewritePath=/ms_secure/(?<segment>.*), /$\{segment}
- name: <my filter>
args:
xForwardedForHeaderName: X-Forwarded-For
hosts:
- <IP1>
- <IP2>
- id: ms_secure
uri: lb://ms_secure_v2
predicates:
- Path=/ms_secure/**
filters:
- RewritePath=/ms_secure/(?<segment>.*), /$\{segment}
- name: <my filter>
args:
xForwardedForHeaderName: X-Forwarded-For
hosts:
- <IP3>
- <IP4>
When when requesting /ms_secure:
IP1 and IP2 will be redirected to ms_secure_v1
IP3 and IP4 will be redirected to ms_secure_v2
My problem is that all my clients will also be able to access directly ms_secure_v1 or ms_secure_v2 by using the default routes:
http:///ms_secure_v1/...
http:///ms_secure_v2/...
I tried to disable these routes by using SetStatus GatewayFilter:
- id: setstatusstring_route
uri: lb://ms-gateway
predicates:
- Path=/ms_secure_v**
filters:
- SetStatus=403
But this route is not matched.
Is there a way to disable these default routes in spring gateway?
The following creates routes in gateway based on services registered:
spring:
cloud:
gateway:
discovery:
locator:
enabled: true
Set it to false (which is the default), if you don't want this.

Resources