We created a storm topology and tested in local mode and everything works great. We then did a build and submitted to nimbus and one supervisor with 4 slots. The topology appears on the storm UI and is shown as active, with 4 slots used on the cluster. But when the topology is clicked, there are no spouts, no bolts, no statistics information. Nothing is written to our Redis database either. So we are wondering if there is something we are not doing.
Storm version: 2.0.0
OS: Linux Mint 19.1 Cinnamon
Related
My topology is configured to use 14 workers and i current have 16 workers available in the cluster.
But when i submit the topology, the "Worker Resources" section on the Storm UI shows that all the 14 workers are up but components (Topology Components) are assigned only to 7. The rest shows N/A (see snapshot below).
The workers/supervisors log files shows no meaningful error(s) as of why some workers are not been assigned components.
I have been googling for the past 6hrs+ to no avail
My environment:
Apache Storm 2.1.0
Zookeeper 3.4.9
One master node (Ubuntu 18.0.4 LTS)
Three supervisor nodes (Ubuntu 18.0.4 LTS)
What are parallelism hints for each of the components (spouts and bolts) in your topology? If they do not add up to 14 or more then you will not have entries on all worker processes.
Remember you can also set the parallelism of system components such as the Ackers and (in metrics V1) the metrics consumers.
I set-up a new Hadoop Cluster with Hortonworks Data Platform 2.5. In the "old" cluster (installed HDP 2.4) I was able to see the information about running Spark jobs via the History Server UI by clicking the link show incomplete applications:
Within the new installation this link opens the page, but it always sais No incomplete applications found! (when there's still an application running).
I just saw, that the YARN ResourceManager UI shows two different kind of links in the "Tracking UI" column, dependent on the status of the Spark application:
application running: Application Master
this link opens http://master_url:8088/proxy/application_1480327991583_0010/
application finished: History
this link opens http://master_url:18080/history/application_1480327991583_0009/jobs/
Via the YARN RM link I can see the running Spark app infos, but why can't I access them via Spark History Server UI? Was there somethings changed from HDP 2.4 to 2.5?
I solved it, it was a network problem: Some of the cluster hosts (Spark slaves) couldn't reach each other due to a incorrect switch configuration. Found it out, as I tried to ping each host from each other.
Since all hosts can ping each other hosts the problem is gone and I can see active and finished jobs in my Spark History server UI again!
I didn't noticed the problem, because the ambari-agents worked on each host, and the ambari-server was also reachable from each cluster host! However, since ALL hosts can reach each other the problem is solved!
If so, from which version, does storm support this? I want to know this because I want to upgrade my storm version(now my storm version is 0.10.0).
High availability for Nimbus was introduced in Storm 1.0.0 (see https://storm.apache.org/2016/04/12/storm100-released.html)
However, even for prior Storm versions, missing HA for Nimbus was not a critical issue, because a failing Nimbus does not affect running topologies. The only problem if Nimbus is down is, that no interaction with the cluster is possible from outside (eg, submitting new topologies etc.).
Workers are HA too, ie, supervisors can restart failing workers. Supervisors are not HA -- however, the task they host will be redistributed automatically to other supervisors if one supervisor fails.
This time, we build a storm cluster and don't have a tool like cloudera manager to monitor the status of cluster except the storm ui, and send alert notice when the cluster is in bad status.
Please write linux scripts with some basic storm commands and zookeeper commands to ensure the health of your cluster.
I am developing a storm topology locally. I am using the Storm 0.9.2-incubating and have developed a simple Topology. When I deploy it using the LocalCluster() option, it works fine, but it will not show up in my Storm UI it just executes.
When I deploy it regularly, it will show the Topology in my Storm UI, but no spouts or bolts will be visible when I click it.
I have also tried this with example WordCountTopology that comes in many storm starter projects. The same behavior happens.
My question is really, why are the spouts and bolts not showing up? If you deploy a topology locally without using LocalCluser() option will that cause problems? Is it possible to deploy a topology on my local box and see it in Storm UI with all the spouts and bolts and not have it execute immediately, but wait for something such as a kafka message?
Are you running Storm Supervisor? If you deploy a new topology and Supervisor isn't running the topology will show up in the UI but since its never initialized it doesn't show any stats when you click into it.