Which role needs getSecrets permissions to be able to deploy this CDK stack? - aws-lambda

Question

I'm getting a deploy-time permissions error related to a Secret referenced with Secret.from_secret_attributes.
# Both param 2 and secret_complete_arn need to populate from env vars.
secret = Secret.from_secret_attributes(self, "AmazonMSK_AdminSecret", secret_complete_arn="somearn" )
# MSK, the one, the only.
# Defaults to the sandbox_ideas2IT MKS instance Sampath set up while prototyping.
mks = os.getenv('MKS_ARN', 'somearn')
# Lambda layer.
lambdaLayer = lambda_.LayerVersion(self, 'lambda-layer',
code = lambda_.AssetCode('utils/lambda-deployment-packages/lambda-layer.zip'),
compatible_runtimes = [lambda_.Runtime.PYTHON_3_7],
)
# Source for the lambda.
with open("src/Blah/BlahIngest.py", encoding="utf8") as fp:
Blah_code = fp.read()
# Config for it.
lambdaFn = lambda_.Function(
self, "QFDS",
code=lambda_.InlineCode(Blah_code),
# role=QFDSrole,
handler="lambda_handler",
timeout=Duration.seconds(300),
runtime=lambda_.Runtime.PYTHON_3_7,
layers=[lambdaLayer],
)
# Set up the event (managed Kafka).
lambdaFn.add_event_source(ManagedKafkaEventSource(
cluster_arn=mks,
topic="foreign.blah.availabilty.notify",
secret=secret,
batch_size=100, # default
starting_position=lambda_.StartingPosition.TRIM_HORIZON
))
This fails to deploy with errors similar to:
Resource handler returned message: "Invalid request provided: Cannot access secret manager value arn:aws:secretsmanager:arn_stuff. Please ensure the role can perform the 'secretsmanager:GetSecretValue' action on your broker in IAM.
However, I am unsure what role would be needed for this. CDK seems to be creating several roles all on its own to perform tasks and to set up a role for the stack itself. I've attempted manually creating a role and trying to assign permissions to that, but I get no end of weird error messages with that, and I suspect that it's some misunderstanding on my part. If the secret is being included into the lambda event source by CDK, is it the CDK role itself that needs permission to access this secret?
Using fedonev's answer seems to get farther than I used to, but I still receive similar errors:
Do you wish to deploy these changes (y/n)? y
ZZZ: deploying... [1/1]
[10:14:11] Retrieved account ID 4### from disk cache
[10:14:11] Assuming role 'some_arn'.
[10:14:11] Waiting for stack CDKToolkit to finish creating or updating...
[0%] start: Publishing 55ce025aa43e61714e9a9eed823a61b7008f6994a8aee67cd1bed9689e2342ad:current_account-current_region
[0%] start: Publishing 8cb1c33ae6d9c870bb9fdd5c36a8d8fa9655d03004f85a79994a33c41690efce:current_account-current_region
[10:14:12] Retrieved account ID 4### from disk cache
[10:14:12] Assuming role 'some_arn'.
[10:14:12] Retrieved account ID 4### from disk cache
[10:14:12] Assuming role 'some_arn'.
[10:14:12] [0%] check: Check s3://cdk-hnb659fds-assets-445841279194-us-east-1/8cb1c33ae6d9c870bb9fdd5c36a8d8fa9655d03004f85a79994a33c41690efce.json
[10:14:12] [0%] check: Check s3://cdk-hnb659fds-assets-445841279194-us-east-1/55ce025aa43e61714e9a9eed823a61b7008f6994a8aee67cd1bed9689e2342ad.zip
[10:14:13] [0%] found: Found s3://cdk-hnb659fds-assets-445841279194-us-east-1/55ce025aa43e61714e9a9eed823a61b7008f6994a8aee67cd1bed9689e2342ad.zip
[50%] success: Published 55ce025aa43e61714e9a9eed823a61b7008f6994a8aee67cd1bed9689e2342ad:current_account-current_region
[10:14:13] [50%] found: Found s3://cdk-hnb659fds-assets-445841279194-us-east-1/8cb1c33ae6d9c870bb9fdd5c36a8d8fa9655d03004f85a79994a33c41690efce.json
[100%] success: Published 8cb1c33ae6d9c870bb9fdd5c36a8d8fa9655d03004f85a79994a33c41690efce:current_account-current_region
[10:14:13] Found existing stack ZZZ that had previously failed creation. Deleting it before attempting to re-create it.
[10:14:14] Waiting for stack ZZZ to finish creating or updating...
[10:14:14] Stack ZZZ has an ongoing operation in progress and is not stable (DELETE_IN_PROGRESS (User Initiated))
[10:14:20] Call failed: describeStacks({"StackName":"ZZZ"}) => Stack with id ZZZ does not exist (code=ValidationError)
[10:14:20] Stack ZZZ does not exist
[10:14:20] ZZZ: checking if we can skip deploy
[10:14:20] ZZZ: no existing stack
[10:14:20] ZZZ: deploying...
[10:14:20] Attempting to create ChangeSet with name cdk-deploy-change-set to create stack ZZZ
ZZZ: creating CloudFormation changeset...
[10:14:21] Initiated creation of changeset: some_arn; waiting for it to finish creating...
[10:14:21] Waiting for changeset cdk-deploy-change-set on stack ZZZ to finish creating...
[10:14:21] Changeset cdk-deploy-change-set on stack ZZZ is still creating
[10:14:26] Initiating execution of changeset some_arn on stack ZZZ
[10:14:27] Execution of changeset some_arn on stack ZZZ has started; waiting for the update to complete...
[10:14:27] Waiting for stack ZZZ to finish creating or updating...
[10:14:27] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS (User Initiated))
ZZZ | 0/7 | 10:14:20 AM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | ZZZ User Initiated
ZZZ | 0/7 | 10:14:27 AM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | ZZZ User Initiated
ZZZ | 0/7 | 10:14:32 AM | CREATE_IN_PROGRESS | AWS::Lambda::LayerVersion | lambda-layer (lambdalayerAEB0E7A7)
ZZZ | 0/7 | 10:14:32 AM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
[10:14:33] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 0/7 | 10:14:33 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | QFDS/ServiceRole (QFDSServiceRole9EF8ABD9)
ZZZ | 0/7 | 10:14:33 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | QFDS/ServiceRole (QFDSServiceRole9EF8ABD9) Resource creation Initiated
ZZZ | 0/7 | 10:14:35 AM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated
ZZZ | 1/7 | 10:14:35 AM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
[10:14:38] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 1/7 | 10:14:40 AM | CREATE_IN_PROGRESS | AWS::Lambda::LayerVersion | lambda-layer (lambdalayerAEB0E7A7) Resource creation Initiated
ZZZ | 2/7 | 10:14:40 AM | CREATE_COMPLETE | AWS::Lambda::LayerVersion | lambda-layer (lambdalayerAEB0E7A7)
[10:14:44] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 3/7 | 10:14:47 AM | CREATE_COMPLETE | AWS::IAM::Role | QFDS/ServiceRole (QFDSServiceRole9EF8ABD9)
[10:14:49] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 3/7 | 10:14:50 AM | CREATE_IN_PROGRESS | AWS::IAM::Policy | QFDS/ServiceRole/DefaultPolicy (QFDSServiceRoleDefaultPolicyDDA95578)
ZZZ | 3/7 | 10:14:51 AM | CREATE_IN_PROGRESS | AWS::IAM::Policy | QFDS/ServiceRole/DefaultPolicy (QFDSServiceRoleDefaultPolicyDDA95578) Resource creation Initiated
[10:14:54] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[10:15:00] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 4/7 | 10:15:04 AM | CREATE_COMPLETE | AWS::IAM::Policy | QFDS/ServiceRole/DefaultPolicy (QFDSServiceRoleDefaultPolicyDDA95578)
[10:15:05] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 4/7 | 10:15:08 AM | CREATE_IN_PROGRESS | AWS::Lambda::Function | QFDS (QFDSA3E8E14E)
[10:15:10] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 4/7 | 10:15:15 AM | CREATE_IN_PROGRESS | AWS::Lambda::Function | QFDS (QFDSA3E8E14E) Resource creation Initiated
[10:15:16] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[10:15:21] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 5/7 | 10:15:21 AM | CREATE_COMPLETE | AWS::Lambda::Function | QFDS (QFDSA3E8E14E)
ZZZ | 5/7 | 10:15:23 AM | CREATE_IN_PROGRESS | AWS::Lambda::EventSourceMapping | QFDS/KafkaEventSource:ZZZQFDSD3CBC52Eforeign.engrain.availabilty.notify (QFDSKafkaEventSourceZZZQFDSD3CBC52EforeignengrainavailabiltynotifyEFE73996)
ZZZ | 5/7 | 10:15:26 AM | CREATE_FAILED | AWS::Lambda::EventSourceMapping | QFDS/KafkaEventSource:ZZZQFDSD3CBC52Eforeign.engrain.availabilty.notify (QFDSKafkaEventSourceZZZQFDSD3CBC52EforeignengrainavailabiltynotifyEFE73996) Resource handler returned message: "Invalid request provided: Cannot access secret manager value some_arn. Please ensure the role can perform the 'secretsmanager:GetSecretValue' action on your broker in IAM. (Service: Lambda, Status Code: 400, Request ID: 54dd811e-ede9-4de2-aace-839344aab72f)" (RequestToken: 7c072ae3-b566-3b19-f0b0-d6ff51960f29, HandlerErrorCode: InvalidRequest)
[10:15:27] Stack ZZZ has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
ZZZ | 5/7 | 10:15:27 AM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | ZZZ The following resource(s) failed to create: [QFDSKafkaEventSourceZZZQFDSD3CBC52EforeignengrainavailabiltynotifyEFE73996]. Rollback requested by user.
[10:15:32] Stack ZZZ has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS)
ZZZ | 5/7 | 10:15:32 AM | DELETE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
ZZZ | 4/7 | 10:15:33 AM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
[10:15:37] Stack ZZZ has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS)

Answer 1

This github issue mentions the same error in the same context. It suggests the problem is due to missing Lambda permissions on the (customer-managed) KMS key used to encrypt the MKS cluster secret.
If so, the solution -- shamelessly stolen from the linked issue -- is something like:
# get a reference to the MSK cluster key
cluster_key = aws_kms.Key.from_key_arn(this, 'ClusterKey', CLUSTER_ENCRYPTION_KEY_ARN)
# add the key reference to the secret reference
secret = Secret.from_secret_attributes(self, "AmazonMSK_AdminSecret",
secret_complete_arn="somearn",
encryption_key=cluster_key
)
# give the Lambda permission to decrypt the Cluster Key
cluster_key.grant_decrypt(lambdaFn.role)