Closed. This question is opinion-based. It is not currently accepting answers.
Closed 3 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
I am new to kafka. We want to monitor and manage kafka topics. We tried different open source monitoring tools like
kafka-monitor
kafka-manager
Both tools are good. But we are unable to make a decision which should be included in our deployment stack. Which one is better and why, and in which scenario?
'kafka manager' from yahoo looks the older one and 'kafka monitor' from LinkedIn is newer one
Kafka Monitor-
Lenses
Lenses (ex Landoop) enhances Kafka with User Interface, streaming SQL engine and cluster monitoring. It enables faster monitoring of Kafka data pipelines.
They provide a free all-in-one docker (Lenses Box) which can serve a single broker for up to 25M messages. Note that this is recommended for development environments.
Cloudera SMM
Streams Messaging Manager is the solution for monitoring and managing clusters running Cloudera or Hortonworks kafka. It also comes with replication capability.
Confluent
Another option is Confluent Enterprise which is a Kafka distribution for production environments. It also includes Control Centre, which is a management system for Apache Kafka that enables cluster monitoring and management from a User Interface.
Yahoo CMAK (Cluster Manager for Apache Kafka, previously known as Kafka Manager)
Kafka Manager or CMAK is a tool for monitoring Kafka offering less functionality compared to the aforementioned tools.
KafDrop
KafDrop is a UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a lightweight application that runs on Spring Boot and requires very little configuration.
LinkedIn Burrow
Burrow is a monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service.
Kafka Tool
Kafka Tool is a GUI application for managing and using Apache Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators.
If you cannot afford licenses, then go for Yahoo Kafka Manager, LinkedIn Burrow or KafDrop. Confluent's and Landoop's products are the best out there, but unfortunately, they require licensing.
For more details, you can refer to my blog post Overview of UI Monitoring tools for Apache Kafka Clusters.
If you want to pay for licensing and Kafka cluster support, then you can use Confluent Control Center
Alternatively, the free route would be to use JMX exporters from Datadog and/or Prometheus/Influxdb (with Grafana dashboards) to see overall system health checks (CPU, network, memory, etc)... Much more information than what you get only by monitoring Kafka processes with Kafka tools
At my company, we used the Yahoo product, we investigated the LinkedIn product, and several others mentioned. My company ultimately chose to use Prometheus+Grafana. Everyone loves it and I'd highly recommend it.
There are two big advantages to Prometheus+Grafana. The first is it does full featured Kafka metrics ingestion+visualization+alerting but it's not limited to Kafka. While our initial needs were just to monitor Kafka, we also wanted metrics on HTTP servers+traffic, server utilization (cpu/ram/disk), and custom application level metrics. Prometheus handles all of the above. Secondly, Prometheus + Grafana are very high quality, well designed, and easy to use. A lot of other products in this space are old and complicated to work with. Prometheus + Grafana are both excellent to work with, they are very customizable, polished, and easy to use. Grafana has a very flashy + functional JavaScript interface that lets you make exactly the customized dashboards that you want. Prometheus has a very polished metric collection engine, storage engine, query language, and alerting system. Something like Yahoo Kafka Manager has much more limited functionality in all of these categories.
If you want to try Prometheus, you need to do two things:
1) install+configure the JMX->Prometheus exporter on your Kafka brokers:
https://github.com/prometheus/jmx_exporter
2) Setup a Prometheus server to collect metrics + and setup a Grafana dashboard to display the graphs that you want.
I'd also say that this is just for monitoring+dashboards+alerting. For management functions, you still need other tools.
The kafka-monitor is (despite the name) a load generation and reporting tool. Yahoo's kafka-manager is an overall monitoring tool.
Related
Context: it is my first use of FireStore. I want to use it to push notification status to our Mobile Application. I can see that there is Google Firestore Dashboard under Analytics umbrella. In our company we use mainly three tools for monitoring our applications: Zabbix, Dynatrace and certain internal solution based on Elasticsearch. I need to ntegrate our internal monitoring systems with metrics resulted from our first Firestore project.
What I am looking for: based on personal assumptions:
1) Maybe there might exist either some GET endpoints that a I can connect and poll for information let's say each minute
2) Maybe, following the idea of Database Realtime pushing events accross a long time connection, I can code a Spring Boot application that import Firebase SDK and every day I connect to some specific Firestore endpoint which will push any interested events (eg. delay based on custom logic or dead service)
3) Maybe some plugin I can connect straight to a Kafka hosted in our internal Datacent
4) Some plugin to connect from Firestore/Firebase to either third tools (eg. Zabbix or Dynatrace or Elasticsearch)
5) Some dependency I could import in google-cloud-funtions thiggered from Firestore Healcheck engine in orther to consume some internal end-point posting data
Perhaps there is already some approach universally used for a scenario when you have to connect Firestore to internal monitoring system. I will be highly appreciated if tell me that than I can narrow my googling searchs because I am not finding anything usefull.
Please, it is not part of this question comparing Monitoring approach. It is a very solid fact in our company use internal Dashboards and some custom alerts trigger. I just mentioned the names above to clarify what I mean by internal monitoring tools. The focus on this question is HOW IMPORT/INTEGRATE/OBSERVE/CONSUME Firestore monitoring data. Our internal stack is beyond this question.
Here is the Official Documentation for Cloud Monitoring using which you can collect metrics, events, and metadata from Google Cloud Platform products that you can use to create dashboards, charts, and alerts.
Please let me know if you have further questions.
Is there a tool available to manage the microservice dependency.
For eg:- If there are service like Inventory service, Catalog service and identity service which together constitute product service.
Is there a visual tool which can map all the dependency and if any of the service is getting changed it should show what all other service is going to be effected by this.
While this questions was posted some years ago, there is now an open source tool called Ortelius.io that does microservice dependency mapping across clusters. It tracks and versions 'logical' views of the application, and shows what apps are dependent upon what services. Tracks this across all clusters with a full versioning engine.
https://github.com/ortelius/ortelius
I think your requirement is closely satisfied by Service map feature of New Relic which is an Application Performance Monitoring platform
Check out https://docs.newrelic.com/docs/using-new-relic/service-maps/get-started/introduction-service-maps
Service maps are visual, customizable representations of your application architecture.
Maps automatically show you your app's connections and dependencies, including databases and external services.
Health indicators and performance metrics show you the current operational status for every part of your architecture.
Well not exactly a dependency manager, if at all there is anything like that, but we made use of a tool called Pinpoint. Amongst it many features is one which shows all the services which are configured with pinpoint and how they interact with other services and databases.
It may help you find how services are linked, and you can infer what all services be impacted if you alter a given service.
It may be long shot, to get a whole apm set up just to find these dependecies, but if you starting from scratch, you may think around it.
I am creating a system where a CEP engine looks like a good fit (I need detect time based patterns between events in a number of streams). The events detected need to be displayed to end users in a WPF application but not all users are entitled to see all events.
How is this typically handled when using a CEP engine? Do I need an additional service layer between the UI application and the CEP engine or can it be done in the CEP engine itself?
I'm not sure there's a "usual" answer to this question across all CEP engines. CEP engines are really not at all uniform in the services they provide.
You tagged this question with "streambase", so I can talk about that some. The StreamBase CEP runtime has support for user authentication when creating a client connection to a server to dequeue tuples from output streams. People typically hook this up to their local LDAP or AD server.
That said, there's no per-tuple dequeue authorization/entitlements enforcement mechanism in the StreamBase CEP runtime, at least as of the current release (7.3).
So, when StreamBase users want per-tuple entitlements enforcement, we usuallygude them toward building that in at the application layer, or use some other piece of available infrastructure. For example, rather than using native StreamBase Client connections, use an adapter to something like TIBCO EMS or IBM MQSeries and use the per topic/queue entitlements there to restrict access. Or as another example, have the StreamBase server send back a token to an authenticated user after connection, and use that token as a key for entitlements checks on the client side against LDAP using a dequeue interceptor.
Or, as you say, use your own services layer.
There's another example of "Poor Man's Entitlement Service" on the StreamBase Component Exchange. Search for "StreamBase Entitlements". This uses an in-memory or JDBC database to store users, roles, and entitlements and applies them on the server side with an efficient runtime entitlements checking mechanism. But there's no strict runtime restriction on the connections with this; it's just an example of implementing entitlements functionality with a nice UI. The clients voluntarily respect the restrictions, which is probably not what you were asking for.
Disclosure/Disclaimer: I am an employee of TIBCO Software, Inc. Opinions expressed here are my own and not TIBCO's.
I have some hardware devices that send some data that need to be stored on the cloud server and also I need to do some real time processing on them.
The data they send need to be preserved for months in some custom binary files. These files related to each device can grow in size up to 10GB over time.
There will client programs (mobile / web) that will be looking at the processed data at real time.
My prefered choice of language is C/C++/C#, since there is time sensitive number crunching involved.
Goal is write scalable application that can have thousands of such devices monitored on the cloud.
Do I have to upfront write the code for running on the cloud ( undestand Azure / amazon EC2) ? Can I write multi threaded desktop application and later migrate to cloud ?
I have used Message passing interface (MPI) in the past for clusters. Can I still use MPI ?
If I use microsoft azure API can I still host my software on Amazon cloud ?
For mobile devices to talk to the server, I understand that I need to have a webservice running. how can I convert a desktop program writeen in C++ / C# to act as a web service talking to client?
Are there any 3rd part frame works or tools taht can help me with my work ?
With most cloud compute services you can deploy an off-the-shelf server and install your own software on it. So, yes, you can write and test you application locally then migrate to the cloud once you get all the bugs worked out. Here are the available EC2 server configurations.
I have not tried MPI but you should be able to run just about anything you want on the servers in the cloud. However, Amazon does offer the Simple Queue Service which provides message passing in the cloud. Your software does not need to run in the cloud to use this service.
I have not used Azure. I doubt there are any restrictions regarding which external servers you use for storage and/or compute. However, keeping your cloud storage and compute resources within a single provider will reduce costs, improve performance and provide you with a unified management interface and billing system.
Web servers are fairly simple things. See this post. That took me about 10 seconds to find.
There is plenty of third party software out there. Figure out what you need in more detail and ask more specific questions
Can anyone give me some suggestions of a business scenarios where I can implement Java Messaging Services (JMS). The message can be sent either by queue(point-to-point) or topic (regular/durable subscription).
I will be using JMS (enabled through TIBCO Enterprise Messaging Services).
The business scenarios must involve atleast 3 IT systems/applications.
The classic use case is that of an Enterprise Service Bus with JMS as one of the available transports. In this case any number of IT systems can request a service invocation by placing a message on a well-known queue. The service provider listening on that queue dynamically determines the reply based on the JMS message's Reply-To fields. An example of a typical service is to inquire on or update customer demographic information. For purposes of inquiry, this definitely meets your requirement of involving at least 3 IT systems since pretty much everything dealing with customers would need to request this service.
Another example with broad application is logging. I have several customers using JMS messages to capture log records from across the network and forward them to a hub of central servers. Because it is JMS, the central hub can be highly available by using redundant servers and can scale horizontally to absorb seasonal loads.
For pub/sub an example I really liked is from an insurance company. They publish events on topics that are subscribed in various call centers, internal news tickers and to business partners. During a hurricane a few years back, these events included updates on landfall predictions and then after the storm passed the updates included locations of mobile claims adjusters and other support services. Pub/Sub was a great way to coordinate this massive mobilization of personnel and communicate back to ground support back at headquarters.
A more mundane pub/sub use case with broad applicability is systems management. Instrumented applications can publish their status and interested parties can receive those notifications. If something is acting weird in Production, the administrator can dynamically enable a subscription to a stream of diagnostics. Ordinarily with no subscribers, the diagnostics are not produced. However, without any interruption in the running system, simply by subscribing, diagnostic messages from the app are produced on demand.
It's actually harder to find examples where JMS messaging should not be used. The most common contraindications are truly synchronous messaging and a requirement to process messages in strict sequence. All JMS providers I'm aware of make allowances for these requirements to varying degrees and I'm aware of many deployments of systems with these requirements. However the ideal use cases for JMS messaging are truly asynchronous or pseudo-synchronous communication and messages that are atomic (that is to say messages have no dependencies on each other or to specific broker instances).
Here are some of the scenarios where we (food retailer) use messaging:
-connection systems between remote locations, in our case POS and inventory management systems in stores, and central ERP and forecast systems: master data changes are sent as XML messages from the central ERP system to the store systems. the store systems send changes in inventory, orders and sales to the central systems. This is completely PTP based, as the master data is unique for each store.
-usage as a central messaging backbone, either directly for systems that are capable to do messaging, or via some adapter functionality for databases, files, SAP systems or HTTP. Here the messaging system builds the base for our ESB.