How to handle dynamic resource in mesos without a lofs of agents? - mesos

Question

What's my scenario?
I have e.g. two external cards which can be plug/unplug without power down of pc.
And these cards are the resources I want to managed with mesos.
Currently, I use attributes to manage them: the attributes nodeKey:card1_key and nodeKey:card2_key are registered to master to distinguish two different cards. Then if card1 was used, I directly flag all cpu, mem was used for mesos-agent1, then master will not offer mesos-agent1 to framework.
Also, with this if I need to unplug card1, I can directly shutdown mesos-agent1 without affect for mesos-agent2 which is used for card2.
Above is my scenario, every works fine except if I have a lots of cards, I had to setup a lots of mesos-agent for every card. This will somewhat memory consume.
Current solution command:
Card1:
docker run -d --net=host --name=mesos-agent1 --privileged \
-e MESOS_IP=$PC_IP \
-e MESOS_HOSTNAME=$PC_IP \
-e MESOS_PORT=$node_port \
-e MESOS_MASTER=zk://$SERVER_IP:2181/mesos \
-e MESOS_ATTRIBUTES="nodeKey:card1_key" \
-e MESOS_SWITCH_USER=0 \
-e MESOS_CONTAINERIZERS=docker,mesos \
-e MESOS_LOG_DIR=/var/log/mesos \
-e MESOS_WORK_DIR=/var/tmp/mesos \
-v "$(echo ~)/.dp/mesos-slave/log/mesos-$nodeKey:/var/log/mesos" \
-v "$(echo ~)/.dp/mesos-slave/tmp/mesos-$nodeKey:/var/tmp/mesos" \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /cgroup:/cgroup \
-v /sys:/sys \
-v $(which docker):/usr/bin/docker \
mesosphere/mesos-slave:1.3.0
Card2:
docker run -d --net=host --name=mesos-agent2 --privileged \
-e MESOS_IP=$PC_IP \
-e MESOS_HOSTNAME=$PC_IP \
-e MESOS_PORT=$node_port \
-e MESOS_MASTER=zk://$SERVER_IP:2181/mesos \
-e MESOS_ATTRIBUTES="nodeKey:card2_key" \
-e MESOS_SWITCH_USER=0 \
-e MESOS_CONTAINERIZERS=docker,mesos \
-e MESOS_LOG_DIR=/var/log/mesos \
-e MESOS_WORK_DIR=/var/tmp/mesos \
-v "$(echo ~)/.dp/mesos-slave/log/mesos-$nodeKey:/var/log/mesos" \
-v "$(echo ~)/.dp/mesos-slave/tmp/mesos-$nodeKey:/var/tmp/mesos" \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /cgroup:/cgroup \
-v /sys:/sys \
-v $(which docker):/usr/bin/docker \
mesosphere/mesos-slave:1.3.0
My question:
So if possible I can just register one mesos agent to mesos master, meanwhile can support my scenario:
a) Card1 was used, the scheduler framework can tag it as used, then next resource offer will have no card1 just card2 offerd? Seems something like --resources='cpus:24;gpus:2;mem:24576;disk:409600;ports:[21000-24000,30000-34000]', if one task used 4 cpus, next time master will just offer 20 cpus, but this cannot be done with --attributes. But mesos seems just can afford interface to programmer customize for --attributes not --resources?
b) If we need to unplug card1 or add a new card3, we could change some parameters of mesos-agent without restart the agent, then currently used e.g. card2 will not be impacted?
Any solution possible, or I had to bear with my current solution?

Answer 1

The simple answer is NO.
You cannot just start one mesos agent for multiple resources. Mesos is a solution of virtualization which indicating multiple(resources) to one.
But I think your requirements would be supported well by an external tool — marathon, one of scheduler frameworks based on mesos.
Marathon would maintain every container’s status scheduled by itself. In your case, if you unplug card1 without any other operations, marathon would know(of course there is an internal gap) the containers on card1(mesos-agent1) dead already. Then marathon will re-schedule these containers, which would request resource from mesos(master). Mesos master offer resources for re-scheduled containers, DONE!
See? No extra operations, you may unplug any cards if you wish to — without any impact of running containers or mesos agents. But you must register new cards to mesos master by starting a new mesos agent on them.
Hope this helps.