How do I pass arguments to a Glue job in CloudFormation YAML? - yaml

Question

You can pass arguments to an AWS Glue job via the --arguments parameter (see here).
The CloudFormation documentation says DefaultArguments are "UTF-8 string–to–UTF-8 string key-value pairs" and that their type is "JSON object". Since YAML is a super set of JSON, I was expecting to be able to pass arguments like this in a (YAML) CloudFormation template:
DefaultArguments:
"--arguments": {"--test_argument": "foo"}
However, it raises this error during CloudFormation deployment:
Property validation failure: [Value of property {/DefaultArguments/--arguments=} does not match type {String}]
How do I specify the values correctly?

Answer 1

The correct way of passing multiple parameters is
DefaultArguments:
"--argument1": value1
"--argument2": value2
and then accessing them in the job (e.g. in Python) like this:
from awsglue.utils import getResolvedOptions
args = getResolvedOptions(sys.argv, ['argument1', 'argument2'])
print args['argument1']
print args['argument2']
What confused me was that for passing parameters with the AWS CLI, you use an explicit --arguments='--argument1="value1"' structure but in CloudFormation, you specify arguments one by one.

Answer 2

The value for the key --arguments needs to be a string, but you actually give it a mapping (or in JSON-speak an object), because it starts witha {. You should quote the value, and since you have double quotes in the value, you best do that with single quotes:
DefaultArguments:
"--arguments": '{"--test_argument": "foo"}'
(any existing single quotes in the value you would need to escape by putting two single quotes)
If your JSON is more complex it can be benificial to use folded-style scalars. Within those the { has no special meaning either and (single) newlines followed by spaces are replaced by a single space. So the following loads to same data as the solution above:
DefaultArguments:
"--arguments": >
{"--test_argument":
"foo"}
Of course with YAML (1.2) being a superset of JSON, glue could easily assume that a value is already parsed if it is not a string, but it doesn't seem to be that smart and always expects the JSON in string form.