This is in regard to a big data analytics engine put out by http://hamrtech.com
Does Zookeeper have to be on its own server with HAMR?
No it does not have to be part of the hamr cluster but every node within the cluster must have access to it.
Related
After rebooting the server, NiFi does not start. Before the server reboot, I was able to shutdown/start NiFi without any issues.
I ensured that the 3 config files (flow.xml.gz, authorizations.xml, and users.xml) are identical on all the nodes.
2019-12-08 14:36:10,085 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1026)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1009)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations: Proposed fingerprint is not inheritable because the current access policies is not empty.
Also, ruled out any zookeeper corruption issue by deleting the znode for NiFi in the zookeeper cluster.
I am on NiFi 1.9.1
Any help is highly appreciated.
This means there is a difference in authorizations.xml or users.xml, most likely authorizations.xml. I would try copying those two files from one of the other nodes over to the node that is having the problem, this will ensure they are exactly the same.
A suggestion, if there is case we can't copy flow.xml.gz (like it was with me) for various process restrictions. We can stop Nifi service on problematic node, rename existing flow.xml.gz to a backup (just to be sure we don't loose it) and restart Nifi service.
Nifi would automatically generate a flow.xml.gz and connect the node to the cluster. It worked for me, hence, sharing.
Thanks
I am trying to setup HDFS on minikube (for now) and later on a DEV kubernetes cluster so I can use it with Spark. I want Spark to run locally on my machine so I can run in debug mode during development so it should have access to my HDFS on K8s.
I have already set up 1 namenode deployment and a datanode statefulset (3 replicas) and those work fine when I am using HDFS from within the cluster. I am using a headless service for the datanodes and a cluster-ip service for the namenode.
The problem starts when I am trying to expose hdfs. I was thinking of using an ingress for that but that only exposes port 80 outside of the cluster and maps paths to different services inside the cluster which is not what I'm looking for. As far as I understand, my local spark jobs (or hdfs client) talk to the namenode which replies with an address for each block of data. That address though is something like 172.17.0.x:50010 and of course my local machine can't see those.
Is there any way I make this work? Thanks in advance!
I know this question is about just getting it to run in a dev environment, but HDFS is very much a work in progress on K8s, so I wouldn't by any means run it in production (as of this writing). It's quite tricky to get it working on a container orchestration system because:
You are talking about a lot of data and a lot of nodes (namenodes/datanodes) that are not meant to start/stop in different places in your cluster.
You have the risk of having a constantly unbalanced cluster if you are not pinning your namenodes/datanodes to a K8s node (which defeats the purpose of having a container orchestration system)
If you run your namenodes in HA mode and it for any reason your namenodes die and restart you run the risk of corrupting the namenode metadata which would make you lose all your data. It's also risky if you have a single node and you don't pin it to a K8s node.
You can't scale up and down easily without running in an unbalanced cluster. Running an unbalanced cluster defeats one of the main purposes of HDFS.
If you look at DC/OS they were able to make it work on their platform, so that may give you some guidance.
In K8s you basically need to create services for all your namenode ports and all your datanode ports. Your client needs to be able to find every namenode and datanode so that it can read/write from them. Also the some ports cannot go through an Ingress because they are layer 4 ports (TCP) for example the IPC port 8020 on the namenode and 50020 on the datanodes.
Hope it helps!
I'm using a ZooKeeper cluster (3 mchines) for my Storm cluster (4 machines). The problem is that -because of the topologies deployed on the storm cluster- the zookeeper transactional logs grow to be extremly large making the zookeeper desk to be full and what is really strange that those logs are not devided into multiple files instead I'm having one big transactional file in every zookeeper machine! making the autopurge in my zookeeper configuration not to have any affect on those files.
Is there a way to solve this problem from zookeeper side, or can I change the way storm uses zookeeper to minimize the size of those logs?
Note: I'm using zookeeper 3.6.4 and Storm 0.9.6 .
I was able to resolve this problem by using Pacemarker to process heartbeats from workers instead of zookeeper; That allowed me to avoid writting to zookeeper disk in order to maintain consistency and use in-memory store instead. In order to be able to use Pacemaker I upgraded to Storm-1.0.2.
I have an application running on two servers. There is a mariadb running on each one of them and a galera cluster that takes care of the replication.
When upgrading the app I need to stop the replication, so I wanted some guidelines on how I can start up the db outside the cluster for one of the servers and then what's the best way to reconnect it.
All ideas are appreciated
Thanks in advance
You can remove the node from the cluster by setting 'wsrep_cluster_address=gcomm://' in the config file of the node.Then restart the server,this time the node will be running outside the cluster,hence,no replication.Similarly,to reconnect change the wsrep_cluster_address to whatever it was earlier and restart the node.
Hope it helps.
I am setting up a secure cluster with NiFi 1.7.1 using an external Zookeeper
While I do plan to have 3 nodes in the cluster, at the moment I am only going to start one node.
Is that a valid cluster from NiFi standpoint.
The reason for my question is I get this SSL Handshake error during startup and I want to rule out that having single not is not the cause of this problem,
Thanks
Vijay
Yes, a single node can be run as a cluster.