How to allocate resource to container while using Spring Yarn? - spring

Question

I am tring the Spring Yarn example on [github][1] which is build by gradle. And I sucessful run the custom-amservice example on yarn.
But I don't know how to allocate special resource to the containers. I tried to override the onContainerAllocated and onContainerLaunched method in class StaticEventingAppmaster at my CustomAppmaster and allocate the resource, just like below.
#Override
protected void onContainerAllocated(Container container) {
//==allocate resource
Resource resource = new ResourcePBImpl();
resource.setMemory(1300);
resource.setVirtualCores(7);
container.setResource(resource);
//====
if (getMonitor() instanceof ContainerAware) {
((ContainerAware)getMonitor()).onContainer(Arrays.asList(container));
}
getLauncher().launchContainer(container, getCommands());
}
#Override
protected void onContainerLaunched(Container container) {
//==allocate resource
Resource resource = new ResourcePBImpl();
resource.setMemory(1300);
resource.setVirtualCores(7);
container.setResource(resource);
//====
if (getMonitor() instanceof ContainerAware) {
((ContainerAware)getMonitor()).onContainer(Arrays.asList(container));
}
}
and in the log it seems works:
2014-12-30 20:06:35,524 DEBUG [AbstractPollingAllocator] - response has 1 new containers
2014-12-30 20:06:35,525 DEBUG [AbstractPollingAllocator] - new container: container_1419934738198_0004_01_000003
//// this line shows the memory is 1300 and cpu core is 7
2014-12-30 20:06:35,525 DEBUG [DefaultContainerMonitor] - Reporting container=Container: [ContainerId: container_1419934738198_0004_01_000003, NodeId: yarn-master1:57799, NodeHttpAddress: yarn-master1:8042, Resource: <memory:1300, vCores:7>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.170:57799 }, ]
2014-12-30 20:06:35,526 DEBUG [DefaultContainerMonitor] - State after reportContainer: DefaultContainerMonitor [allocated=[container_1419934738198_0004_01_000003,], running=[container_1419934738198_0004_01_000002,], completed=[], failed=[]]
//// this line shows the memory is 1300 and cpu core is 7
2014-12-30 20:06:35,526 DEBUG [DefaultContainerLauncher] - Launching container: Container: [ContainerId: container_1419934738198_0004_01_000003, NodeId: yarn-master1:57799, NodeHttpAddress: yarn-master1:8042, Resource: <memory:1300, vCores:7>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.170:57799 }, ] with commands $JAVA_HOME/bin/java,org.springframework.yarn.container.CommandLineContainerRunner,container-context.xml,yarnContainer,1><LOG_DIR>/Container.stdout,2><LOG_DIR>/Container.stderr
However, when I try to run an application which resource out of the limit, its log shows that the memory remains 1GB instead of 1300, just see below:
2014-12-30 20:07:05,929 DEBUG [AbstractPollingAllocator] - response has 1 completed containers
//The Same container was stopped because it beyond the limits.
2014-12-30 20:07:05,932 DEBUG [AbstractPollingAllocator] - completed container: container_1419934738198_0004_01_000003 with status=ContainerStatus: [ContainerId: container_1419934738198_0004_01_000003, State: COMPLETE, Diagnostics: Container [pid=10587,containerID=container_1419934738198_0004_01_000003] is running beyond virtual memory limits. Current usage: 86.6 MB of 1 GB physical memory used; 31.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1419934738198_0004_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 10587 32315 10587 10587 (bash) 2 3 12652544 353 /bin/bash -c /home/novelbio/software/jdk//bin/java org.springframework.yarn.container.CommandLineContainerRunner container-context.xml yarnContainer 1>/home/novelbio/software/hadoop/logs/userlogs/application_1419934738198_0004/container_1419934738198_0004_01_000003/Container.stdout 2>/home/novelbio/software/hadoop/logs/userlogs/application_1419934738198_0004/container_1419934738198_0004_01_000003/Container.stderr
|- 10761 10587 10587 10587 (java) 108 10 34135896064 21811 /home/novelbio/software/jdk//bin/java org.springframework.yarn.container.CommandLineContainerRunner container-context.xml yarnContainer
, ExitStatus: 0, ]
The key point is, in the log :. Current usage: 86.6 MB of 1 GB physical memory used instead of 1.3GB.
So I think my method didn't take effect. Could any body tells me how to allocate resource correctly?

Answer 1

This is one of a problematic areas in YARN which I believe will eventually get better when more and more non-MR apps are used in YARN. I believe your settings are correctly applied but a little strange behaviour from YARN is causing these problems. Currently there is very little we can do from an application point of view because most of the mem settings are enforced in YARN itself and requests from apps are just "requests".
Spring XD on YARN rely in this same stuff and it's worth to check what we wrote into its docs: https://github.com/spring-projects/spring-xd/wiki/Running-on-YARN. (see section Configuring YARN memory reservations).
I'll try to make sure that this same info also goes to our Spring Hadoop and Spring YARN ref docs.