I am playing around with Spring Boot v2 at the moment. So far, my set up looks like this:
Spring -> Telegraf -> Kafka -> Telegraf -> influx
I am wondering whether or not it's possible to take out the the first telegraf inbetween Spring and Kafka, so something like this:
Spring -> Kafka -> Telegraf -> Influx
I've looked at the configurations of micrometer and there is no config for Kafka. Also, telegraf was pulling data from Spring.. and as Kafka is a push model (i.e. you are pushing data into Kafka), would Spring be able to push data to Kafka? If yes, how? Through the use of HTTP POST methods?
New to the whole concept.
would Spring be able to push data to Kafka? If yes, how? Through the use of HTTP POST methods?
Kafka uses its own TCP protocol, not HTTP so no. At least not without using the Kafka REST Proxy.
You would basically be embedding the same thing that Telegraf does into your Spring code.
It's possible, sure, but built into Micrometer? Not that I'm aware of.
Plus, it would be overhead on your app having an internal producer thread, and you'd be required to include kafka clients with each of your monitored apps, plus have some control preventing your app from failing if Kafka connection isn't possible...
I would suggest keeping Telegraf installed on each host machine, or at the very least, Prometheus JMX exporter or Jolokia for your individual Java apps. From this, JMX metrics can be collected and pushed to downstream monitoring systems
Or, as commented, you could skip Kafka, but I'm guessing you want to keep it there as a buffer.
On the other side, you can use Kafka Connect Influxdb sink to get optimal performance of consumer scaling
Related
I cannot find any reference on implementation of getting metrics.
Can Someone help with an example and references?
As stats_example says here, You can get stats listed in STATISTICS.md. But clearly mentioned in the example comments, You need to implement metrics
Stats events are emitted as JSON (as string). Either directly forward
the JSON to your statistics collector, or convert it to a map to
extract fields of interest.
So in this case, In your application, you need to implement metrics collector something like prometheus
And if you want full broker side metrics, You can implement Kafka monitoring As Kafka Documentation explained here
Kafka uses Yammer Metrics for metrics reporting in the server. The
Java clients use Kafka Metrics, a built-in metrics registry that
minimizes transitive dependencies pulled into client applications.
Both expose metrics via JMX and can be configured to report stats
using pluggable stats reporters to hook up to your monitoring system.
In my corporate project I am using Spring Boot and Apache ActiveMQ 5.x Spring Boot starter. I am a totally beginner in this.
My goal is to expose Prometheus endpoint with some JMS queue metrics:
number of messages in queue
number of messages in error queue
What are dedicated tools for retrieving such metrics? Up to now I have found two possible ways. Can anyone confirm which of these two tools can solve my problem?
https://docs.spring.io/spring-integration/docs/5.1.7.RELEASE/reference/html/#system-management-chapter
https://activemq.apache.org/components/artemis/documentation/latest/metrics.html (here the example is not very helpful)
I don't think the Spring stuff will work because that will provide Spring-related metrics from the application itself, not the ActiveMQ broker.
Also, the documentation for ActiveMQ you cited is for ActiveMQ Artemis. However, the dependency you're using is for ActiveMQ 5.x. Therefore, the documentation is not applicable. However, if you choose to use ActiveMQ Artemis it is very simple to expose a Prometheus endpoint using this Prometheus metrics plugin implementation. It's worth noting that Artemis is ActiveMQ's next generation message broker. If you're starting a new project I would recommend you use it rather than 5.x. Artemis is planned to replace 5.x and become ActiveMQ 6.0 in the future.
I think your best bet would be to configure the Prometheus JMX exporter. It even has a sample configuration for ActiveMQ 5.x.
ActiveMQ comes with Jolokia bundled by default for extracting JMX Beans for the JVM, queues and a bunch of other metrics using HTTP. That way we can easily export using a software like Telegraf, which comes with a simple input plugin for ActiveMQ and a simple output plugin for Prometheus.
I am running a Kafka Stream app with Springboot 2.
I would like to have my kafka stream metrics available in the prometheus format at host:8080/actuator/prometheus
I don't manage to have this. I am not sure I understand how kafka stream metrics are exported.
Can actuator get these JMX metrics ?
Is there a way to get these metrics and expose them in Prometheus format ?
PS: didn't worked with java jmx_prometheus_agent neither
Does someone has a solution or an example ?
Thank you
You could produce all available Kafka-Streams metrics (the same as from KafkaStreams.metrics()) into Prometheus using micrometer-core and spring-kafka libraries. For integrating Kafka-Streams with micrometer, you could have KafkaStreamsMicrometerListener bean:
#Bean
KafkaStreamsMicrometerListener kafkaStreamsMicrometerListener(MeterRegistry meterRegistry) {
return new KafkaStreamsMicrometerListener(meterRegistry);
}
where MeterRegistry is from micrometer-core dependency.
If you create Kafka Streams using StreamsBuilderFactoryBean from spring-kafka, then you need to add listener into it:
streamsBuilderFactoryBean.addListener(kafkaStreamsMicrometerListener);
And if you create KafkaStreams objects directly, then on each KafkaStreams object you need to invoke
kafkaStreamsMicrometerListener.streamsAdded(beanId, kafkaStreams);
where beanId is any unique identifier per KafkaStreams object.
As a result, Kafka Streams provides multiple useful Prometheus metrics, like kafka_consumer_coordinator_rebalance_latency_avg, kafka_stream_thread_task_closed_rate, etc. KafkaStreamsMicrometerListener under the hood uses KafkaStreamsMetrics.
If you need to have Grafana Prometheus graphs with these metrics, you need to add them as Gauge metric type.
I don't have a complete example, but metrics are well accessible and documented in Confluent documentation on Monitoring Kafka Streams.
Maybe dismiss actuator and use #RestController from Spring Web along with KafkaStreams#metrics() to publish exactly what you need.
I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?
Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distributed mode, as a cluster, not embedded within other (single-instance) applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself
The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector. So, if you want to use Kafka Connect, run it by itself, then just write the consumer in the Spring app.
The alternatives to Kafka Connect itself would be to use Apache Camel within a Spring application (Spring Integration) or Spring Cloud Dataflow and interfacing with those Kafka "components" (which aren't using the Connect API, AFAIK)
Another option, specific for listening to MySQL, is to use Debezium Engine within your code.
I'm looking to consume from Kafka and save data into Hadoop and Elasticsearch.
I've seen 2 ways of doing this currently: using Filebeat to consume from Kafka and send it to ES and using Kafka-Connect framework. There is a Kafka-Connect-HDFS and Kafka-Connect-Elasticsearch module.
I'm not sure which one to use to send streaming data. Though I think that if I want at some point to take data from Kafka and place it into Cassandra I can use a Kafka-Connect module for that but no such feature exists for Filebeat.
Kafka Connect can handle streaming data and is a bit more flexible. If you are just going to elastic, Filebeat is a clean integration for log sources. However, if you are going from Kafka to a number of different sinks, Kafka Connect is probably what you want. I'd recommend checking out the connector hub to see some examples of open source connectors at your disposal currently http://www.confluent.io/product/connectors/