apache storm throws Exception "java.net.BindException: Address already in use" - apache-storm

Question

Apache storm throws exception when I submit a new topology in storm cluster which has 100+ free slots. The storm cluster has 35 nodes.
org.apache.storm.shade.org.jboss.netty.channel.ChannelException: Failed to bind to: 0.0.0.0/0.0.0.0:6734
at org.apache.storm.shade.org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.messaging.netty.Server.<init>(Server.java:101) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.messaging.netty.Context.bind(Context.java:67) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.daemon.worker$worker_data$fn__5173.invoke(worker.clj:272) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.util$assoc_apply_self.invoke(util.clj:931) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.daemon.worker$worker_data.invoke(worker.clj:269) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.daemon.worker$fn__5471$exec_fn__1371__auto__$reify__5473.run(worker.clj:613) ~[storm-core-1.0.3.jar:1.0.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
at org.apache.storm.daemon.worker$fn__5471$exec_fn__1371__auto____5472.invoke(worker.clj:611) ~[storm-core-1.0.3.jar:1.0.3]
at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__5471$mk_worker__5562.doInvoke(worker.clj:585) [storm-core-1.0.3.jar:1.0.3]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$_main.invoke(worker.clj:769) [storm-core-1.0.3.jar:1.0.3]
at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.3.jar:1.0.3]
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_121]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_121]
at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_121]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:1.8.0_121]
2019-11-02 18:39
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[?:1.8.0_121]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[storm-core-1.0.3.jar:1.0.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_121]
2019-11-01 17:57:42.487 o.a.s.util main [ERROR] Halting process: ("Error on initialization")java.lang.RuntimeException: ("Error on initialization")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.3.jar:1.0.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__5471$mk_worker__5562.doInvoke(worker.clj:585) [storm-core-1.0.3.jar:1.0.3]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$_main.invoke(worker.clj:769) [storm-core-1.0.3.jar:1.0.3]
at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.3.jar:1.0.3]
When I use lsof command to find out which process uses the port, it shows
$ lsof -i:6734
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 21812 orc 292u IPv6 2397383060 0t0 TCP bj4191:6734->10.15.70.5:boe-resssvr2 (ESTABLISHED)
java 30636 orc 58u IPv6 2394755694 0t0 TCP bj4191:28367->10.12.178.13:6734 (ESTABLISHED)
java 30636 orc 90u IPv6 2394760492 0t0 TCP bj4191:23217->10.12.174.3:6734 (ESTABLISHED)
java 30636 orc 1191u IPv6 2397398591 0t0 TCP bj4191:6734->10.12.172.46:6307 (ESTABLISHED)
java 30645 orc 88u IPv6 2394657692 0t0 TCP bj4191:22997->10.12.174.3:6734 (ESTABLISHED)
java 30645 orc 99u IPv6 2394657702 0t0 TCP bj4191:28257->10.12.178.13:6734 (ESTABLISHED)
java 30645 orc 3016u IPv6 2394770505 0t0 TCP bj4191:6734->10.32.210.18:boks_clntd (ESTABLISHED)
I can't understand why the second process use an address which has been used. How can I solve it? Thx.