Bubble.io API Connector with Lambda + AWS Rekognition Problem - aws-lambda

Hi I was trying to use AWS Rekognition -> Lambda -> AWS API -> Bubble.io
I want to pass a image base64 code to process the image recognition.
Here is the lambda code:
import json
import boto3
import base64
def lambda_handler(event, context):
def detect_labels():
# 1. create a client object, to connect to rekognition
client=boto3.client('rekognition')
image = base64.b64decode(event['face'])
# 4. call Rekognition, store result in 'response'
response = client.detect_labels(
Image={
'Bytes': image
},
MaxLabels=20,
)
#6. Return response from function
return response
# Call detect_labels
response = detect_labels()
# Return results to API gateway
return {
'statusCode': 200,
'body': json.dumps(response)
}
and if I test it within the lambda test with config:
{
"face": "<imgbase64codehere>"
}
It works fine and return success message.
However I set the same thing in Bubble.io
Setting img. It wont work.
I check the Cloudwatch, the error is:
[ERROR] KeyError: 'face'
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 30, in lambda_handler
response = detect_labels()
File "/var/task/lambda_function.py", line 14, in detect_labels
image = base64.b64decode(event['face'])
[ERROR] KeyError: 'face' Traceback (most recent call last): File "/var/task/lambda_function.py", line 30, in lambda_handler response = detect_labels() File "/var/task/lambda_function.py", line 14, in detect_labels image = base64.b64decode(event['face'])
I am sure there is not connection problem (if I dont use JSON it works fine)
So what do I do wrong in the setting?
Thanks for your help in advance

In your test you are sending an input of just a JSON request with face.
It looks like the implementation you've setup uses Amazon API Gateway. If you are doing that your Lambda function might receive the event in a different format depending on how you've implemented the API Gateway integration.
The Lambda developer guide has an example of the JSON format. You will probably want to access the body attribute.

Related

StatusCode.UNIMPLEMENTED when making Vertex AI API call

I have a simple Python app that invokes a Vertex AI API that fails when it runs and I can't understand why. The application is as follows:
from google.cloud import aiplatform_v1
def sample_list_datasets():
client = aiplatform_v1.DatasetServiceClient()
request = aiplatform_v1.ListDatasetsRequest(
parent="projects/MYPROJECT/locations/us-central1",
)
page_result = client.list_datasets(request=request)
for response in page_result:
print(response)
sample_list_datasets()
when run, it fails with:
E0126 03:52:04.146970105 22462 hpack_parser.cc:1218] Error parsing metadata: error=invalid value key=content-type value=text/html; charset=UTF-8
Traceback (most recent call last):
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "Received http2 header with status: 404"
debug_error_string = "UNKNOWN:Error received from peer ipv4:108.177.120.95:443 {created_time:"2023-01-26T03:52:04.147076255+00:00", grpc_status:12, grpc_message:"Received http2 header with status: 404"}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "run.py", line 25, in <module>
sample_list_datasets()
File "run.py", line 19, in sample_list_datasets
page_result = client.list_datasets(request=request)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/dataset_service/client.py", line 1007, in list_datasets
metadata=metadata,
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
return wrapped_func(*args, **kwargs)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.MethodNotImplemented: 501 Received http2 header with status: 404
What might I be doing wrong?
Changing the code to the following caused it to work:
from google.cloud import aiplatform_v1
from google.api_core.client_options import ClientOptions
def sample_list_datasets():
service_base_path='aiplatform.googleapis.com'
region='us-central1'
client_options = ClientOptions(api_endpoint=f"{region}-{service_base_path}")
client = aiplatform_v1.DatasetServiceClient(client_options=client_options)
request = aiplatform_v1.ListDatasetsRequest(
parent="projects/MYPROJECT/locations/us-central1",
)
# Make the request
page_result = client.list_datasets(request=request)
# Handle the response
for response in page_result:
print(response)
sample_list_datasets()
The resolution was hinted at in the documentation for the API request found here. At that article there is a code sample and in the code sample there are some comments and in the comments the following is written:
It may require specifying regional endpoints when creating the service
client as shown in:
https://googleapis.dev/python/google-api-core/latest/client_options.html
And this was the core clue. When we make Vertex AI calls we must specify where the request is to be sent. We do this by setting the api_endpoint option to a URL of the form [REGION]-aiplatform.googleapis.com.

boto3 and lambda: Invalid type for parameter KeyConditionExpression when using DynamoDB resource and localstack

I'm having very inconsistent results when trying to use boto3 dynamodb resources from my local machine vs from within a lambda function in localstack. I have the following simple lambda handler, that just queries a table based on the Hash Key:
import boto3
from boto3.dynamodb.conditions import Key
def handler(event, context):
dynamodb = boto3.resource(
"dynamodb", endpoint_url=os.environ["AWS_EP"]
)
table = dynamodb.Table("precalculated_scores")
items = table.query(
KeyConditionExpression=Key("customer_id").eq(event["customer_id"])
)
return items
The environment variable "AWS_EP" is set to my localstack DNS when protyping (http://localstack:4566).
When I call this lamdba I get the following error:
{
"errorMessage": "Parameter validation failed:\nInvalid type for parameter KeyConditionExpression, value: <boto3.dynamodb.conditions.Equals object at 0x7f7440201960>, type: <class 'boto3.dynamodb.conditions.Equals'>, valid types: <class 'str'>",
"errorType": "ParamValidationError",
"stackTrace": [
" File \"/opt/code/localstack/localstack/services/awslambda/lambda_executors.py\", line 1423, in do_execute\n execute_result = lambda_function_callable(inv_context.event, context)\n",
" File \"/opt/code/localstack/localstack/services/awslambda/lambda_api.py\", line 782, in exec_local_python\n return inner_handler(event, context)\n",
" File \"/var/lib/localstack/tmp/lambda_script_l_dbef16b3.py\", line 29, in handler\n items = table.query(\n",
" File \"/opt/code/localstack/.venv/lib/python3.10/site-packages/boto3/resources/factory.py\", line 580, in do_action\n response = action(self, *args, **kwargs)\n",
" File \"/opt/code/localstack/.venv/lib/python3.10/site-packages/boto3/resources/action.py\", line 88, in __call__\n response = getattr(parent.meta.client, operation_name)(*args, **params)\n",
" File \"/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py\", line 514, in _api_call\n return self._make_api_call(operation_name, kwargs)\n",
" File \"/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py\", line 901, in _make_api_call\n request_dict = self._convert_to_request_dict(\n",
" File \"/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py\", line 962, in _convert_to_request_dict\n request_dict = self._serializer.serialize_to_request(\n",
" File \"/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/validate.py\", line 381, in serialize_to_request\n raise ParamValidationError(report=report.generate_report())\n"
]
}
Which is a weird error - From what I researched on other question it usually happens when using the boto3 client, but I am using boto3 resources. Furthermore, when I run the code locally in my machine it runs fine.
At first I thought that it might be due to different versions for boto3 (My local machine is using version 1.24.96, while the version inside the lambda runtime is 1.16.31). However I downgraded my local version to the same as the one in the runtime, and I keep getting the same results.
After some answers on this question I managed to get the code working against actual AWS services, but it still won't work when running against localstack.
Am I doing anything wrong? Os might this be a bug with localstack?
--- Update 1 ---
Changing the return didn't solve the problem:
return {"statusCode": 200, "body": json.dumps(items)}
--- Update 2 ---
The code works when running against actual AWS services instead of running against localstack. Updating the question with this information.
This works fine from both my local machine and Lambda:
import json, boto3
from boto3.dynamodb.conditions import Key
def lambda_handler(event, context):
dynamodb = boto3.resource(
"dynamodb",
endpoint_url="https://dynamodb.eu-west-1.amazonaws.com"
)
table = dynamodb.Table("test")
items = table.query(
KeyConditionExpression=Key("pk").eq("1")
)
print(items)
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
Also be sure that event["customer_id"] is in fact a string value as expected by the eq function.
I would check to ensure you have the endpoint setup correctly and that you have the current version deployed.
It may also be the fact you are trying to return the results of your API call via the handler, instead of a proper JSON response as expected:
return {
'statusCode': 200,
'body': json.dumps(items)
}

Lambda function splunk-firehose-flowlogs-processor

I try to use Lambda function splunk-firehose-flowlogs-processor to transform data from vpc flowlogs before sending them to splunk. I do it according to amazon blogpost: https://aws.amazon.com/blogs/big-data/ingest-vpc-flow-logs-into-splunk-using-amazon-kinesis-data-firehose/
I used test data and it worked but suddenly I started to get an Lambda function error:
[ERROR] KeyError: 'message'
Traceback (most recent call last):
File "/var/task/app.py", line 127, in lambda_handler
records = list(processRecords(event['records']))
File "/var/task/app.py", line 74, in processRecords
joinedData = ''.join(transformLogEvent(data))
File "/var/task/app.py", line 66, in transformLogEvent
return log_event['message'] + '\n'
I'm really confused and I think maybe there is someone who knows why this Lambda function gives errors.

Get metadata for a column with google Sheets API

I have a Google spreadsheet that I am connecting to and interacting with using the google-python-api-client package. Following this description on metadata search, and the links in it for the request body, I have written a function to get metadata for a range:
def get_metadata_by_range(range_: Union[dict, str]) -> dict:
if isinstance(range_, str):
print("String range: ", range_)
request_body = {"dataFilters": \
{"a1Range": range_}}
elif isinstance(range_, dict):
print("Dict range: ", range_)
request_body = {"dataFilters": \
[{"gridRange": range_}]}
else:
return None
request = service.spreadsheets().developerMetadata().\
search(spreadsheetId=SPREADSHEET_ID, body=request_body)
return request.execute()
Calling this with a range, either A1 notation or a gridRange will cause an error to occur though. For example, calling it with this line get_metadata_by_range("Metadata!A:A") will cause the following traceback.
String range: Metadata!A:A
Traceback (most recent call last):
File "oqc_server/fab/gapc.py", line 82, in <module>
get_metadata_by_range("Metadata!A:A")
File "oqc_server/fab/gapc.py", line 69, in get_metadata_by_range
return request.execute()
File "/media/kajsa/Storage/Projects/oqc_server/venv/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/media/kajsa/Storage/Projects/oqc_server/venv/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 500 when requesting https://sheets.googleapis.com/v4/spreadsheets/1RhheCsI3kHrm8yK2Yio2kAOU4VOzYdz-eK0vjiMY7co/developerMetadata:search?alt=json returned "Internal error encountered."
Any ideas on what is causing this and how to solve it?
You want to search and retrieve the developer metadata from the range using the method of Method: spreadsheets.developerMetadata.search of Sheets API.
You want to achieve this using google-api-python-client with python.
You have already been able to get and put values for Spreadsheet with Sheets API.
If my understanding is correct, how about this answer? Please think of this as just one of several possible answers.
Modification points:
When you want to search the developer metadata with the range, please set the gridrange to dataFilters[].developerMetadataLookup.metadataLocation.dimensionRange.
When the range is set to dataFilters[].a1Range and dataFilters[].gridRange, I could confirm that the same error occurs.
Sample script:
The sample script for retrieving the developer metadata from the range is as follows. Before you use this, please set the variables of spreadsheet_id and sheet_id.
service = build('sheets', 'v4', credentials=creds)
spreadsheet_id = '###' # Please set the Spreadsheet ID.
sheet_id = ### # Please set the sheet ID.
search_developer_metadata_request_body = {
"dataFilters": [
{
"developerMetadataLookup": {
"metadataLocation": {
"dimensionRange": {
"sheetId": sheet_id,
"dimension": "COLUMNS",
"startIndex": 0,
"endIndex": 1
}
}
}
}
]
}
request = service.spreadsheets().developerMetadata().search(
spreadsheetId=spreadsheet_id, body=search_developer_metadata_request_body)
response = request.execute()
print(response)
Above script retrieves the developer metadata from the column "A" of sheet_id.
Note:
Please modify above script for your actual script.
In the current stage, the Developer Metadata can be added to the Spreadsheet, each sheet in the Spreadsheet and row and column. Please be careful this. Ref
References:
Method: spreadsheets.developerMetadata.search
Adding Developer Metadata- DeveloperMetadataLookup
If I misunderstood your question and this was not the direction you want, I apologize.
There is a bug with developerMetadata related to a1Range objects being passed as filters.
Edit
I've checked the bug again and a fix has been implemented.

MicroPython urequests.post failures

I'm trying to receive notifications on my (Android) mobile device from an ESP8266 MCU running MicroPython. For this reason I subscribed to a couple of online services exposing some APIs for this task, Pushbullet, and Pushed, and I installed the matching apps on my device.
This is what I'm trying:
Pushbullet:
import json
import urequests
body = "Test Notification"
title = "Pushbullet"
data_sent = {"type": "note", "title": title, "body": body}
API_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxx'
pb_headers = {
'Authorization': 'Bearer ' + API_KEY,
'Content-Type': 'application/json'
}
r = urequests.post(
'https://api.pushbullet.com/v2/pushes',
data=json.dumps(data_sent),
headers=pb_headers
)
print(r)
Error:
ssl_handshake_status: -256
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
File "urequests.py", line 104, in post
File "urequests.py", line 56, in request
OSError: [Errno 5] EIO
Pushed:
import json
import urequests
payload = {
"app_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxx",
"app_secret": "xxxxxxxxxxxxxxxxxxxxxxxxxxx",
"target_type": "app",
"content": "Remote Mic MCU test from ESP8266"
}
r = urequests.post("https://api.pushed.co/1/push", data=payload)
print(r)
Error:
Traceback (most recent call last):
File "<stdin>", line 8, in <module>
File "urequests.py", line 104, in post
File "urequests.py", line 74, in request
TypeError: object with buffer protocol required
Searching for these errors, doesn't get me anywhere useful.
The exact same code snippets work OK on my Linux box (using requests instead of urequests), but I understand that urequests may have some limitations.
Do you have any hint on how to fix this?
The exception message suggests that you pass the type of data which urequests doesn't expect. From my knowledge of how HTTP POST works (see HTTP standard), I know that it accepts octet stream, which in Python would be represented by str or bytes type. Whereas you pass a dictionary.
`
problem is with urequests library, if you can switchback to usocket and close the socket before timeout then works fine. For urequests you need to have a board with spiram.

Resources