How do I change root volume size of AWS Batch at runtime - amazon-ec2

Question

I have an application which makes requests to AWS to start batch jobs. The jobs vary, and therefore resource requirements change for each job.
It is clear how to change CPUs and memory, however I cannot figure out how to specify root volume size, or if it is even possible
Here is an example of the code I am running:
import boto3
client = boto3.client('batch')
JOB_QUEUE = "job-queue"
JOB_DEFINITION="job-definition"
container_overrides = {
'vcpus': 1,
'memory': 1024,
'command': ['echo', 'Hello World'],
# 'volume_size': 50 # this is not valid
'environment': [ # this just creates env variables
{
'name': 'volume_size',
'value': '50'
}
]
}
response = client.submit_job(
jobName="volume-size-test",
jobQueue=JOB_QUEUE,
jobDefinition=JOB_DEFINITION,
containerOverrides=container_overrides)
My question is similar to this However I am specifically asking if this is possible at runtime. I can change the launch template however that doesn't solve the issue of being able to specify required resources when making the request. Unless the solution is to create multiple launch templates and then select that at run time, though that seems unnecessarily complicated

Answer 1

You can use AWS Elastic File System for this. EFS volumes can be mounted to the containers created for your job definition. EFS doesn't require you to provide a specific volume size because it automatically grows and shrinks depending on the usage.
You need to specify an Amazon EFS file system in your job definition through the efsVolumeConfiguration property
{
"containerProperties": [
{
"image": "amazonlinux:2",
"command": [
"ls",
"-la",
"/mount/efs"
],
"mountPoints": [
{
"sourceVolume": "myEfsVolume",
"containerPath": "/mount/efs",
"readOnly": true
}
],
"volumes": [
{
"name": "myEfsVolume",
"efsVolumeConfiguration": {
"fileSystemId": "fs-12345678",
"rootDirectory": "/path/to/my/data",
"transitEncryption": "ENABLED",
"transitEncryptionPort": integer,
"authorizationConfig": {
"accessPointId": "fsap-1234567890abcdef1",
"iam": "ENABLED"
}
}
}
]
}
]
}
Reference: https://docs.aws.amazon.com/batch/latest/userguide/efs-volumes.html