Changing Hostname on Mesos-Slave not working - mesos

Question

I followed the tutorial from https://open.mesosphere.com/getting-started/install/ to setup mesos and marathon.
I am using Vagrant to create 2 nodes, a master and a slave.
At the end of the tutorial I have marathon and mesos functioning.
First problem: Only the slave on the master machine is visible to mesos. The "independent" slave on the second vagrant node machine is not visible on mesos even though I have put the same settings in /etc/mesos/zk for both the nodes. From what I understand this is the file that gives the master node addresses.
Second Problem: When I change the hostname for the slave on the master machine to the ip address, the slave does not run.
When I remove the file /etc/mesos-slave/hostname, and restart the slave start running again.
I get the following logs in /var/log/mesos:
Log file created at: 2016/09/13 19:38:57
Running on machine: vagrant-ubuntu-trusty-64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0913 19:38:57.316082 2870 logging.cpp:194] INFO level logging started!
I0913 19:38:57.319680 2870 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0913 19:38:57.321099 2870 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0913 19:38:57.322904 2870 main.cpp:434] Starting Mesos agent
I0913 19:38:57.323637 2887 slave.cpp:198] Agent started on 1)#10.0.2.15:5051
I0913 19:38:57.323648 2887 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="192.168.33.20" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://192.168.33.20:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
I0913 19:38:57.323942 2887 slave.cpp:519] Agent resources: cpus(*):1; mem(*):244; disk(*):35164; ports(*):[31000-32000]
I0913 19:38:57.323969 2887 slave.cpp:527] Agent attributes: [ ]
I0913 19:38:57.323974 2887 slave.cpp:532] Agent hostname: 192.168.33.20
I0913 19:38:57.326578 2886 state.cpp:57] Recovering state from '/var/lib/mesos/meta'
After this when I do "sudo service mesos-slave status" it says stop/waiting.
I am not sure how to go about dealing with these two problems. Any help appreciated.
Update
On the "Independent Slave Machine" I am getting the following logs:
file: mesos-slave.vagrant-ubuntu-trusty-64.invalid-user.log.ERROR.20160914-141226.1197
Log file created at: 2016/09/14 14:12:26
Running on machine: vagrant-ubuntu-trusty-64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0914 14:12:26.699146 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:26.700430 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:27.634099 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:28.784499 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:34.914746 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:36.906472 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:37.242663 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:40.442214 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:42.033504 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:47.239245 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:12:50.712105 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0914 14:13:03.200935 1234 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
file: mesos-slave.vagrant-ubuntu-trusty-64.invalid-user.log.INFO.20160914-141502.4788
Log file created at: 2016/09/14 14:15:02
Running on machine: vagrant-ubuntu-trusty-64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0914 14:15:02.491973 4788 logging.cpp:194] INFO level logging started!
I0914 14:15:02.495968 4788 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0914 14:15:02.497270 4788 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0914 14:15:02.498855 4788 main.cpp:434] Starting Mesos agent
I0914 14:15:02.499091 4788 slave.cpp:198] Agent started on 1)#10.0.2.15:5051
I0914 14:15:02.499195 4788 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="192.168.33.31" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://192.168.33.20:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
I0914 14:15:02.499560 4788 slave.cpp:519] Agent resources: cpus(*):1; mem(*):244; disk(*):35164; ports(*):[31000-32000]
I0914 14:15:02.499620 4788 slave.cpp:527] Agent attributes: [ ]
I0914 14:15:02.499650 4788 slave.cpp:532] Agent hostname: 192.168.33.31
I0914 14:15:02.502511 4803 state.cpp:57] Recovering state from '/var/lib/mesos/meta'
I0914 14:15:02.502554 4803 state.cpp:697] No checkpointed resources found at '/var/lib/mesos/meta/resources/resources.info'
I0914 14:15:02.502630 4803 state.cpp:100] Failed to find the latest agent from '/var/lib/mesos/meta'
I0914 14:15:02.510077 4807 status_update_manager.cpp:200] Recovering status update manager
I0914 14:15:02.510150 4807 containerizer.cpp:522] Recovering containerizer
I0914 14:15:02.510758 4807 provisioner.cpp:253] Provisioner recovery complete
I0914 14:15:02.510815 4807 slave.cpp:4782] Finished recovery
I0914 14:15:02.511342 4804 group.cpp:349] Group process (group(1)#10.0.2.15:5051) connected to ZooKeeper
I0914 14:15:02.511368 4804 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0914 14:15:02.511376 4804 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
I0914 14:15:02.513720 4804 detector.cpp:152] Detected a new leader: (id='4')
I0914 14:15:02.513813 4804 group.cpp:706] Trying to get '/mesos/json.info_0000000004' in ZooKeeper
I0914 14:15:02.514854 4804 zookeeper.cpp:259] A new leading master (UPID=master#10.0.2.15:5050) is detected
I0914 14:15:02.514928 4804 slave.cpp:895] New master detected at master#10.0.2.15:5050
I0914 14:15:02.514940 4804 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0914 14:15:02.514961 4804 slave.cpp:927] Detecting new master
I0914 14:15:02.514976 4804 status_update_manager.cpp:174] Pausing sending status updates
E0914 14:15:03.228878 4811 process.cpp:2105] Failed to shutdown socket with fd 11: Transport endpoint is not connected
I0914 14:15:03.229086 4806 slave.cpp:3732] master#10.0.2.15:5050 exited
W0914 14:15:03.229099 4806 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0914 14:15:03.342586 4811 process.cpp:2105] Failed to shutdown socket with fd 11: Transport endpoint is not connected
I0914 14:15:03.342675 4806 slave.cpp:3732] master#10.0.2.15:5050 exited
W0914 14:15:03.342685 4806 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0914 14:15:06.773352 4811 process.cpp:2105] Failed to shutdown socket with fd 11: Transport endpoint is not connected
I0914 14:15:06.773438 4806 slave.cpp:3732] master#10.0.2.15:5050 exited
W0914 14:15:06.773448 4806 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0914 14:15:09.190912 4811 process.cpp:2105] Failed to shutdown socket with fd 11: Transport endpoint is not connected
I0914 14:15:09.191007 4806 slave.cpp:3732] master#10.0.2.15:5050 exited
W0914 14:15:09.191017 4806 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0914 14:15:16.597836 4811 process.cpp:2105] Failed to shutdown socket with fd 11: Transport endpoint is not connected
I0914 14:15:16.597929 4806 slave.cpp:3732] master#10.0.2.15:5050 exited
W0914 14:15:16.597940 4806 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0914 14:15:33.944555 4809 slave.cpp:3732] master#10.0.2.15:5050 exited
W0914 14:15:33.944607 4809 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0914 14:15:33.944682 4811 process.cpp:2105] Failed to shutdown socket with fd 11: Transport endpoint is not connected
I0914 14:16:02.515676 4804 slave.cpp:4591] Current disk usage 4.72%. Max allowed age: 5.969647788608773days
E0914 14:16:11.307096 4811 process.cpp:2105] Failed to shutdown socket with fd 11: Transport endpoint is not connected
I0914 14:16:11.307189 4806 slave.cpp:3732] master#10.0.2.15:5050 exited
W0914 14:16:11.307199 4806 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
Update 2
The configurations for both the machines seem to be the same (I say seem because I have verified, they are the same but still I cannot seem to connect the remote slave, so there must be something going wrong).
The logs for the machine slave1 are as following:
mesos-slave.slave1.invalid-user.log.WARNING
Log file created at: 2016/09/17 20:28:34
Running on machine: slave1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0917 20:28:34.018565 17112 slave.cpp:202]
**************************************************
Agent bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
E0917 20:28:34.797722 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0917 20:28:34.797917 17129 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:35.612090 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0917 20:28:35.612185 17133 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:37.841622 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0917 20:28:37.841723 17128 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:38.358543 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0917 20:28:38.358711 17128 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:51.705592 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0917 20:28:51.705704 17128 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
mesos-slave.slave1.invalid-user.log.INFO
Log file created at: 2016/09/17 20:28:34
Running on machine: slave1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0917 20:28:34.011777 17112 logging.cpp:194] INFO level logging started!
I0917 20:28:34.014294 17112 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0917 20:28:34.016263 17112 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0917 20:28:34.017916 17112 main.cpp:434] Starting Mesos agent
I0917 20:28:34.018307 17112 slave.cpp:198] Agent started on 1)#127.0.0.1:5051
I0917 20:28:34.018381 17112 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="192.168.33.31" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://192.168.33.20:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
W0917 20:28:34.018565 17112 slave.cpp:202]
**************************************************
Agent bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0917 20:28:34.018896 17112 slave.cpp:519] Agent resources: cpus(*):1; mem(*):244; disk(*):35164; ports(*):[31000-32000]
I0917 20:28:34.018959 17112 slave.cpp:527] Agent attributes: [ ]
I0917 20:28:34.018987 17112 slave.cpp:532] Agent hostname: 192.168.33.31
I0917 20:28:34.022061 17127 state.cpp:57] Recovering state from '/var/lib/mesos/meta'
I0917 20:28:34.022337 17127 state.cpp:697] No checkpointed resources found at '/var/lib/mesos/meta/resources/resources.info'
I0917 20:28:34.022431 17127 state.cpp:100] Failed to find the latest agent from '/var/lib/mesos/meta'
I0917 20:28:34.028128 17133 group.cpp:349] Group process (group(1)#127.0.0.1:5051) connected to ZooKeeper
I0917 20:28:34.028177 17133 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0917 20:28:34.028187 17133 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
I0917 20:28:34.028659 17130 status_update_manager.cpp:200] Recovering status update manager
I0917 20:28:34.028875 17129 containerizer.cpp:522] Recovering containerizer
I0917 20:28:34.029595 17129 provisioner.cpp:253] Provisioner recovery complete
I0917 20:28:34.029912 17112 slave.cpp:4782] Finished recovery
I0917 20:28:34.030637 17133 detector.cpp:152] Detected a new leader: (id='6')
I0917 20:28:34.030733 17133 group.cpp:706] Trying to get '/mesos/json.info_0000000006' in ZooKeeper
I0917 20:28:34.032158 17133 zookeeper.cpp:259] A new leading master (UPID=master#127.0.0.1:5050) is detected
I0917 20:28:34.032232 17133 slave.cpp:895] New master detected at master#127.0.0.1:5050
I0917 20:28:34.032245 17133 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0917 20:28:34.032263 17133 slave.cpp:927] Detecting new master
I0917 20:28:34.032281 17133 status_update_manager.cpp:174] Pausing sending status updates
E0917 20:28:34.797722 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0917 20:28:34.797904 17129 slave.cpp:3732] master#127.0.0.1:5050 exited
W0917 20:28:34.797917 17129 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:35.612090 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0917 20:28:35.612174 17133 slave.cpp:3732] master#127.0.0.1:5050 exited
W0917 20:28:35.612185 17133 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:37.841622 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0917 20:28:37.841713 17128 slave.cpp:3732] master#127.0.0.1:5050 exited
W0917 20:28:37.841723 17128 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:38.358543 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0917 20:28:38.358700 17128 slave.cpp:3732] master#127.0.0.1:5050 exited
W0917 20:28:38.358711 17128 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0917 20:28:51.705592 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0917 20:28:51.705665 17128 slave.cpp:3732] master#127.0.0.1:5050 exited
W0917 20:28:51.705704 17128 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
mesos-slave.slave1.invalid-user.log.ERROR
Log file created at: 2016/09/17 20:28:34
Running on machine: slave1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0917 20:28:34.797722 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:35.612090 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:37.841622 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:38.358543 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:51.705592 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
mesos-slave.INFO
Log file created at: 2016/09/17 20:28:34
Running on machine: slave1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0917 20:28:34.011777 17112 logging.cpp:194] INFO level logging started!
I0917 20:28:34.014294 17112 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0917 20:28:34.016263 17112 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0917 20:28:34.017916 17112 main.cpp:434] Starting Mesos agent
I0917 20:28:34.018307 17112 slave.cpp:198] Agent started on 1)#127.0.0.1:5051
I0917 20:28:34.018381 17112 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="192.168.33.31" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://192.168.33.20:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
W0917 20:28:34.018565 17112 slave.cpp:202]
**************************************************
Agent bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0917 20:28:34.018896 17112 slave.cpp:519] Agent resources: cpus(*):1; mem(*):244; disk(*):35164; ports(*):[31000-32000]
I0917 20:28:34.018959 17112 slave.cpp:527] Agent attributes: [ ]
I0917 20:28:34.018987 17112 slave.cpp:532] Agent hostname: 192.168.33.31
I0917 20:28:34.022061 17127 state.cpp:57] Recovering state from '/var/lib/mesos/meta'
mesos-slave.ERROR
Log file created at: 2016/09/17 20:28:34
Running on machine: slave1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0917 20:28:34.797722 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:35.612090 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:37.841622 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:38.358543 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
E0917 20:28:51.705592 17135 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
Logs for the master machine are as following:
mesos-slave.master.invalid-user.log.WARNING
Log file created at: 2016/09/17 20:28:30
Running on machine: master
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0917 20:28:30.118418 21439 slave.cpp:202]
**************************************************
Agent bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
mesos-slave.master.invalid-user.log.INFO
Log file created at: 2016/09/17 20:28:30
Running on machine: master
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0917 20:28:30.107797 21423 logging.cpp:194] INFO level logging started!
I0917 20:28:30.112454 21423 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0917 20:28:30.113862 21423 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0917 20:28:30.114965 21423 main.cpp:434] Starting Mesos agent
I0917 20:28:30.118180 21439 slave.cpp:198] Agent started on 1)#127.0.0.1:5051
I0917 20:28:30.118201 21439 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="192.168.33.20" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://192.168.33.20:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
W0917 20:28:30.118418 21439 slave.cpp:202]
**************************************************
Agent bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0917 20:28:30.118688 21439 slave.cpp:519] Agent resources: cpus(*):1; mem(*):244; disk(*):35164; ports(*):[31000-32000]
I0917 20:28:30.118716 21439 slave.cpp:527] Agent attributes: [ ]
I0917 20:28:30.118719 21439 slave.cpp:532] Agent hostname: 192.168.33.20
I0917 20:28:30.121039 21440 state.cpp:57] Recovering state from '/var/lib/mesos/meta'