Spark shell connect to Mesos hangs: No credentials provided. Attempting to register without authentication - mesos

Question

I installed Mesos in an OpenStack environment using these instructions from Mesosphere: https://open.mesosphere.com/getting-started/datacenter/install/. I ran the verification test as described and it was successful. UI for both Mesos and Marathon are working as expected.
When I run the Spark shell from my laptop I cannot connect. The shell hangs with the output below. I don't see anything in the Mesos master or slave logs that would indicate an error, so am not sure what to investigate next.
Any help would be appreciated.
TOMWATER-M-60SN:bin tomwater$ ./spark-shell --master mesos://zk://10.93.193.78:2181,10.93.193.79:2181,10.93.193.80:2181/mesos
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/06 15:39:02 INFO SecurityManager: Changing view acls to: tomwater
15/08/06 15:39:02 INFO SecurityManager: Changing modify acls to: tomwater
15/08/06 15:39:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tomwater); users with modify permissions: Set(tomwater)
15/08/06 15:39:02 INFO HttpServer: Starting HTTP Server
15/08/06 15:39:02 INFO Utils: Successfully started service 'HTTP class server' on port 63056.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.1
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_51)
Type in expressions to have them evaluated.
Type :help for more information.
15/08/06 15:39:05 INFO SparkContext: Running Spark version 1.4.1
15/08/06 15:39:05 INFO SecurityManager: Changing view acls to: tomwater
15/08/06 15:39:05 INFO SecurityManager: Changing modify acls to: tomwater
15/08/06 15:39:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tomwater); users with modify permissions: Set(tomwater)
15/08/06 15:39:05 INFO Slf4jLogger: Slf4jLogger started
15/08/06 15:39:05 INFO Remoting: Starting remoting
15/08/06 15:39:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver#10.93.235.120:63057]
15/08/06 15:39:05 INFO Utils: Successfully started service 'sparkDriver' on port 63057.
15/08/06 15:39:05 INFO SparkEnv: Registering MapOutputTracker
15/08/06 15:39:05 INFO SparkEnv: Registering BlockManagerMaster
15/08/06 15:39:05 INFO DiskBlockManager: Created local directory at /private/var/folders/7g/p1nw5zg94yx5cck_6c4jgwh80000gp/T/spark-74145a91-396f-4989-b2c0-5902e32e9e16/blockmgr-511d3fdf-f84a-40dc-b6e5-daace4d3f786
15/08/06 15:39:05 INFO MemoryStore: MemoryStore started with capacity 265.1 MB
15/08/06 15:39:05 INFO HttpFileServer: HTTP File server directory is /private/var/folders/7g/p1nw5zg94yx5cck_6c4jgwh80000gp/T/spark-74145a91-396f-4989-b2c0-5902e32e9e16/httpd-4ce76073-5636-4656-9fba-633fbc1c16f4
15/08/06 15:39:05 INFO HttpServer: Starting HTTP Server
15/08/06 15:39:05 INFO Utils: Successfully started service 'HTTP file server' on port 63058.
15/08/06 15:39:05 INFO SparkEnv: Registering OutputCommitCoordinator
15/08/06 15:39:05 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/08/06 15:39:05 INFO SparkUI: Started SparkUI at http://10.93.235.120:4040
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#716: Client environment:host.name=TOMWATER-M-60SN
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#723: Client environment:os.name=Darwin
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#724: Client environment:os.arch=14.4.0
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#725: Client environment:os.version=Darwin Kernel Version 14.4.0: Thu May 28 11:35:04 PDT 2015; root:xnu-2782.30.5~1/RELEASE_X86_64
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#733: Client environment:user.name=tomwater
I0806 15:39:06.235976 547205120 sched.cpp:157] Version: 0.23.0
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#741: Client environment:user.home=/Users/tomwater
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#log_env#753: Client environment:user.dir=/Users/tomwater/development/tools/spark-1.4.1-bin-hadoop2.6/bin
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO#zookeeper_init#786: Initiating client connection, host=10.93.193.78:2181,10.93.193.79:2181,10.93.193.80:2181 sessionTimeout=10000 watcher=0x11eca0d00 sessionId=0 sessionPasswd=<null> context=0x7f8f7cffbaf0 flags=0
2015-08-06 15:39:06,333:30782(0x12147c000):ZOO_INFO#check_events#1703: initiated connection to server [10.93.193.78:2181]
2015-08-06 15:39:06,705:30782(0x12147c000):ZOO_INFO#check_events#1750: session establishment complete on server [10.93.193.78:2181], sessionId=0x14f0502209a0006, negotiated timeout=10000
I0806 15:39:06.707475 544960512 group.cpp:313] Group process (group(1)#10.93.235.120:63059) connected to ZooKeeper
I0806 15:39:06.707785 544960512 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0806 15:39:06.707952 544960512 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0806 15:39:06.712241 547741696 detector.cpp:138] Detected a new leader: (id='126')
I0806 15:39:06.712530 555130880 group.cpp:656] Trying to get '/mesos/info_0000000126' in ZooKeeper
W0806 15:39:06.714071 544960512 detector.cpp:444] Leading master master#192.168.1.69:5050 is using a Protobuf binary format when registering with ZooKeeper (info): this will be deprecated as of Mesos 0.24 (see MESOS-2340)
I0806 15:39:06.714269 544960512 detector.cpp:481] A new leading master (UPID=master#192.168.1.69:5050) is detected
I0806 15:39:06.714498 544960512 sched.cpp:254] New master detected at master#192.168.1.69:5050
I0806 15:39:06.714643 544960512 sched.cpp:264] No credentials provided. Attempting to register without authentication

Answer 1

I've just had this - obviously check that you can talk to the master mesos node (on port 5050 normally). However you also need to allow the mesos master to talk back to your client (it's an emphemeral port annoyingly).
If you strace it you can see what's going on.
strace -e trace=network -f -s 16384 -o /tmp/strace.log pyspark
Looking at the strace.log - first we ask for a random socket and listen on it:
28462 socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 254
28462 setsockopt(254, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
28462 bind(254, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
28462 getsockname(254, {sa_family=AF_INET, sin_port=htons(46975), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
28462 listen(254, 500000) = 0
Now's the interesting part - we talk to the mesos master (10.1.201.191:5050) and we tell it our IP and that port we opened (10.1.200.212:46975)
It then talks back to us (the accept()):
28507 connect(258, {sa_family=AF_INET, sin_port=htons(5050), sin_addr=inet_addr("10.1.201.191")}, 16) = -1 EINPROGRESS (Operation now in progress)
28510 getsockopt(258, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
28510 sendto(258, "POST /master/mesos.scheduler.Call HTTP/1.1\r\nUser-Agent: libprocess/scheduler-52db362e-d5dd-4109-97d3-e28e80f2391b#10.1.200.212:46975\r\nLibproce
ss-From: scheduler-52db362e-d5dd-4109-97d3-e28e80f2391b#10.1.200.212:46975\r\nConnection: Keep-Alive\r\nHost: \r\nTransfer-Encoding: chunked\r\n\r\n54\r\n\20\1\32P\n
N\n\6ubuntu\22\fPySparkShell:\34ip-10-1-200-212.ec2.internalJ\30http://10.1.200.212:4040\r\n0\r\n\r\n", 375, MSG_NOSIGNAL, NULL, 0) = 375
28510 accept(254, {sa_family=AF_INET, sin_port=htons(33743), sin_addr=inet_addr("10.1.201.191")}, [16]) = 259