I have a Lambda that polls the status of a DataBrew job using a boto3 client. The code - as written here - works fine in my local environment. When I put it into a Lambda function, I get the error:
[ERROR] AttributeError: 'GlueDataBrew' object has no attribute 'describe_job_run'
This is the syntax found in the Boto3 documentation:
client.describe_job_run(
Name='string',
RunId='string')
This is my code:
import boto3
def get_brewjob_status(jobName, jobRunId):
brew = boto3.client('databrew')
try:
jobResponse = brew.describe_job_run(Name=jobName, RunId=jobRunId)
status = jobResponse['State']
except Exception as e:
status='FAILED'
print('Unable to get job status')
raise(e)
return {
'jobStatus':status
}
def lambda_handler(event, context):
jobName=event['jobName']
jobRunId=event['jobRunId']
response=get_brewjob_status(jobName, jobRunId)
return response
I am using the Lambda runtime version of boto3. The jobName and jobRunId variables are strings passed from a Step Function, but I've also tried to hard code them into the Lambda to check the error and I get the same result. I have tried running it on both the runtime Python3.7 and Python3.8 versions. I'm also confident (and have double checked) that the IAMs permissions allow the Lambda access to DataBrew. Thanks for any ideas!
Fixed my own problem. There must be some kind of conflict with the boto3 runtime and databrew - maybe not updated to include databrew yet? I created a .zip deployment package and it worked fine. Should have done that two days ago...
Related
I have code that seems to "almost" deploy. It will fail with the following error:
10:55:25 AM | CREATE_FAILED | AWS::Lambda::EventSourceMapping | QFDSKafkaEventSour...iltynotifyEFE73996
Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster some-valid-an. Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: some-uuid )" (RequestToken: some-uuid, HandlerErrorCode: InvalidRequest)
I've cobbled together the cdk stack from multiple tutorials, trying to learn CDK. I've gotten it to the point that I can deploy a lambda, specify one (or more) layers for the lambda, and even specify any of several different sources for triggers. But our production Kafka requires credentials... and I can't figure out for the life of me how to supply those so that this will deploy correctly.
Obviously, those credentials shouldn't be included in the git repo of my codebase. I assume I will have to set up a Secrets Manager secret with part or all of the values. We're using scram-sha-512, and it includes a user/pass pair. The 'secret_name' value to Secret() is probably the name/path of the Secrets Manager secret. I have no idea what the second, unnamed param is for, and I'm having trouble figuring that out. Can anyone point me in the right direction?
Stack code follows:
#!/usr/bin/env python3
from aws_cdk import (
aws_lambda as lambda_,
App, Duration, Stack
)
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource
from aws_cdk.aws_secretsmanager import Secret
class ExternalRestEndpoint(Stack):
def __init__(self, app: App, id: str) -> None:
super().__init__(app, id)
secret = Secret(self, "Secret", secret_name="integrations/msk/creds")
msk_arn = "some valid and confirmed arn"
# Lambda layer.
lambdaLayer = lambda_.LayerVersion(self, 'lambda-layer',
code = lambda_.AssetCode('utils/lambda-deployment-packages/lambda-layer.zip'),
compatible_runtimes = [lambda_.Runtime.PYTHON_3_7],
)
# Source for the lambda.
with open("src/path/to/sourcefile.py", encoding="utf8") as fp:
mysource_code = fp.read()
# Config for it.
lambdaFn = lambda_.Function(
self, "QFDS",
code=lambda_.InlineCode(mysource_code),
handler="lambda_handler",
timeout=Duration.seconds(300),
runtime=lambda_.Runtime.PYTHON_3_7,
layers=[lambdaLayer],
)
# Set up the event (managed Kafka).
lambdaFn.add_event_source(ManagedKafkaEventSource(
cluster_arn=prototype_mks,
topic="foreign.endpoint.availabilty.notify",
secret=secret,
batch_size=100, # default
starting_position=lambda_.StartingPosition.TRIM_HORIZON
))
Looking into a code sample, I understand that you are working with Amazon MSK as an event source, and not just self-managed (cross-account) Kafka.
I assume I will have to set up a Secrets Manager secret with part or all of the values
You don't need to setup credentials. If you use MSK with SALS_SCRAM, you already have credentials, which must be associated with MSK cluster.
As you can see from the doc, you secret name should start with AmazonMSK_, for example AmazonMSK_LambdaSecret.
So, in the code above, you will need to fix this line:
secret = Secret(self, "Secret", secret_name="AmazonMSK_LambdaSecret")
I assume you already aware of the CDK python doc, but will just add here for reference.
I'm trying to create a EventBridgeScheduler client in my Python 3.9.15 Lambda, but the service responds with UnknownServiceError: Unknown service: 'scheduler'. Isn't this service supported even though it's included in boto3's official documentation? The list included in the error message doesn't include scheduler among Valid service names.
import boto3
eb_client = boto3.client('scheduler')
def lambda_handler(event, context):
print(json.dumps(event))
schedule_name = event['resources']
delete_schedule(schedule_name)
def delete_schedule(name):
try:
response = eb_client.delete_schedule(
Name=name
)
except Exception as e:
print(str(e))
The scheduler feature was added to boto3 in v1.26.7. The Lambda Python runtimes currently come with v1.20.32.
Package up the latest SDK version with your Lambda to use the EventBridge scheduler client.
I am automating Dialogflow CX using Python client libraries. That includes agent/intent/entity etc. creation/updation/deletion.
But for the first time run, I am encountering the below error from python.
If I login to console and set the location from there and rerun the code, it is working fine. I am able to create agent.
Followed this URL of GCP -
https://cloud.google.com/dialogflow/cx/docs/concept/region
I am looking for code to automate the region & location setting before running the python code. Kindly provide me with the code.
Below is the code I am using to create agent.
Error -
google.api_core.exceptions.FailedPrecondition: 400 com.google.apps.framework.request.FailedPreconditionException: Location settings have to be initialized before creating the agent in location: us-east1. Code: FAILED_PRECONDITION
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.FAILED_PRECONDITION
details = "com.google.apps.framework.request.FailedPreconditionException: Location settings have to be initialized before creating the agent in location: us-east1. Code: FAILED_PRECONDITION"
debug_error_string = "{"created":"#1622183899.891000000","description":"Error received from peer ipv4:142.250.195.170:443","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"com.google.apps.framework.request.FailedPreconditionException: Location settings have to be initialized before creating the agent in location: us-east1. Code: FAILED_PRECONDITION","grpc_status":9}"
main.py -
# Import Libraries
import google.auth
import google.auth.transport.requests
from google.cloud import dialogflowcx as df
from google.protobuf.field_mask_pb2 import FieldMask
import os, time
import pandas as pd
# Function - Authentication
def gcp_auth():
cred, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
auth_req = google.auth.transport.requests.Request()
cred.refresh(auth_req)
# Function - Create Agent
def create_agent(agent_name, agent_description, language_code, location_id, location_path):
if location_id == "global":
agentsClient = df.AgentsClient()
else:
agentsClient = df.AgentsClient(client_options={"api_endpoint": f"{location_id}-dialogflow.googleapis.com:443"})
agent = df.Agent(display_name=agent_name, description=agent_description, default_language_code=language_code, time_zone=time_zone, enable_stackdriver_logging=True)
createAgentRequest = df.CreateAgentRequest(agent=agent, parent=location_path)
agent = agentsClient.create_agent(request=createAgentRequest)
return agent```
Currently, Dialogflow does not support configuring the location settings through the API, thus you can not initialise location settings through it. You can only set the location through the Console.
As an alternative, since the location setting has to be initialised only once for each region per project you could set the location and automate the agent creation process, some useful links: 1 and 2.
On the other hand, if you would find this feature useful, you can file a Feature Request, here. It will be evaluated by the Google's product team.
Many thanks Alexandre Moraes. I have raised a feature request for the same.
I am writing a lambda to update all the services in all the ecs clusters based on their tags. For that I need to extract the tags from the description of the service but the corresponding function gives error.
import boto3
import botocore
client = boto3.client('ecs')
def lambda_handler(event, context):
responseToListClusters = client.list_clusters() #gets list of clusters
clusterArnsList=responseToListClusters['clusterArns'] #extracts list of clusterArns
for CLUSTER in clusterArnsList:
responseToListServices = client.list_services(cluster= CLUSTER) #gets list of services
serviceArnsList=responseToListServices['serviceArns'] #extracts list of serviceArns
for SERVICE in serviceArnsList:
responseToDescribeServices= client.describe_services(cluster=CLUSTER,services=[SERVICE,],include=['TAGS',])
print(responseToDescribeServices)
#client.update_service(cluster=CLUSTER,service=SERVICE,desiredCount=1) #updates all services
You are encountering this error because AWS lambda by-default runs with older version of boto3.
Currently AWS lamda has following versions :
python3.7
boto3-1.9.42
botocore-1.12.42
python3.6
boto3-1.7.74
botocore-1.10.74
python2.7
N/A
Reference : Lambda Runtimes
To upgrade the boto3 version you can refer to following articles :
AWS Lambda Console - Upgrade boto3 version
https://www.mandsconsulting.com/lambda-functions-with-newer-version-of-boto3-than-available-by-default/
I am trying to attach volume to instance using boto3 but its failed to attach with below error
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IncorrectState) when calling the AttachVolume operation: vol-xxxxxxxxxxxxxxx is not 'available'.
I can see volumme exists in aws console but somehow boto3 is not able to attach volume.
os.environ['AWS_DEFAULT_REGION'] = "us-west-1"
client = boto3.client('ec2', aws_access_key_id=access_key, aws_secret_access_key=secret_key,
region_name='us-west-1')
response1 = client.attach_volume(
VolumeId=volume_id,
InstanceId=instance_id,
Device='/dev/sdg',
)
I tried using aws cli for attaching the same and its working fine after exporting AWS_DEFAULT_REGION="us-west-1"
Also tried exporting the same in python script using os.environ['AWS_DEFAULT_REGION'] = "us-west-1" but python script is failing with the same error as mentioned above.
I figured it out. I am not giving enough time after creating ebs volume. I am able to attach now after adding sleep