I've a 3 node Apache Nifi cluster and now I would like to put monitoring on top of it. Apache Ambari will be a good monitoring tool for it. Will I be able to integrate Ambari with Nifi? Or I need to installed Ambari first and then use Ambari feature to install Nifi. Note : I'm using open source software and not HortonWork.
As far as I know, I don't think it is possible to add an existing NiFi cluster as a new service to Ambari. You need to have Ambari then create and install the NiFi service.
Once installed, to monitor your NiFi service, you can use AmbariReportingTask available in NiFi to report statistics to your Ambari server for monitoring.
Useful Links
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ambari-nar/1.8.0/org.apache.nifi.reporting.ambari.AmbariReportingTask/index.html
https://pierrevillard.com/2017/05/16/monitoring-nifi-ambari-grafana/
Related
I have a 6-node-cluster running Hortonworks HDP 2.5.3 and Ambari 2.4.2.0
I want to install Apache NiFi on this cluster. When looking in the documentation, the following line jumps to my eyes:
1.1. Interoperability Requirements
You cannot install HDF on a system where HDP is already installed.
I wonder how I can install NiFi on my cluster. I would like to manage it with Ambari too, if possible.
Should I just go ahead and install the standalone version of NiFi and changing the port to something else than 8080, which is in use by Ambari? The problem is that I'd have to install it on every node and this process is not automated.
Currently you can only install one stack into a given Ambari instance, and there is an HDP stack which does not include NiFi, and an HDF stack which includes NiFi, Kafka, Storm, and Ranger. So you need a second Ambari instance where you can install the HDF stack. You also can't share nodes between two Ambaris because there can only be one Ambari agent running on a node.
There might be enhancements in future Ambari releases to improve this situation, but for now if you are limited to using your 6 HDP nodes then you would have to install/manage NiFi manually using the RPM or TAR.
As of HDP 2.6.1 it is possible to install HDF components on an HDP cluster. See https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_installing-hdf-and-hdp/content/ch_install-ambari.html
Since the latest HDP 3.0, it can add HDF 3.2 and work together with NiFi
I'm going to apply AMBARI to my system.
But my system already has hadoop.
How do I add existing Hadoop clusters to my new AMBARI environment
Sorry for my English.
Ambari can only manage clusters that it provisioned. Your pre-existing hadoop cluster was not provisioned with Ambari so it cannot be managed by Ambari.
Ambari is designed around a Stack concept where each stack consists of several services. A stack definition is what allows Ambari to install, manage and monitor the services in the cluster.
You can not do right now because already hadoop is installed in the system and you want to apply AMBARI over that for managing the hadoop cluster that's not possible.
Detailed description about the Apache Ambari :---
The Apache Ambari project is aimed at making Hadoop management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs.
Ambari enables System Administrators to:
Provision a Hadoop Cluster
Ambari provides a step-by-step wizard for installing Hadoop services across any number of hosts.
Ambari handles configuration of Hadoop services for the cluster.
Manage a Hadoop Cluster
Ambari provides central management for starting, stopping, and reconfiguring Hadoop services across the entire cluster.
Monitor a Hadoop Cluster
Ambari provides a dashboard for monitoring health and status of the Hadoop cluster.
Ambari leverages Ambari Metrics System for metrics collection.
Ambari leverages Ambari Alert Framework for system alerting and will notify you when your attention is needed (e.g., a node goes down, remaining disk space is low, etc).
I have a Elasticsearch cluster which is already installed and running.
Now I want to add Elasticsearch as a service in Ambari so that I can manage Elasticsearch's action(for example,start or stop) through Ambari Web UI.
How can I do this? Any help? Thanks!
Ambari cannot manage services that it did not provision.
HDP 2.4 does not include Elasticsearch as part of its stack. If you want to manage/monitor Elasticsearch you will have to create a custom service for the ambari stack. Details on creating a custom stack can be found on the Ambari Wiki
I have a hadoop cluster with apache hadoop 2.0.7.
I want to know how to integrate Ambari with the apache hadoop without the HDP(HortonWorks).
Actually, If I use HDP the solution is easy. but , I don't want to use the in my situation.
Do you have an any Idea?
Ambari relies on 'Stack' definitions to describe what services the Hadoop cluster consists of. Hortonworks defined a custom Ambari stack, its called HDP.
You could define your own stack and use any services and respective versions that you wanted. See the ambari wiki for more information about defining stacks and services.
That being said, I don't think it's possible to use your pre-existing installation of Hadoop with Ambari. Ambari is used to provision and manage hadoop clusters. It keeps track of the state of each of its stacks services, and the states of each services components. Since your cluster is already provisioned it would be difficult (maybe impossible) to add it to an Ambari instance.
I have a 10 node existing cluster in RHEL 6.6 which was prepared by plain apache Hadoop configuration XMLs. Now I wanted to check the cluster status by Ambari. Would it be possible to install Hortonworks Ambari just to monitor only not to install Hadoop.
No, Ambari must provision the cluster it's monitoring.
Ambari is designed around a Stack concept where each stack consists of several services. A stack definition is what allows Ambari to install, manage and monitor the services in the cluster.
In order for you to use Ambari with the hadoop core that you built you would have to provide your own Ambari stack definition.
Specifically in your case your existing Hadoop installation would not have the necessary alert.json descriptors used by Ambari to provide alerts for any given service.