CloudWatch log multiple custom metric filters to trigger lambda function - aws-lambda

I just start learning on aws, and I have created couple of cloudwatch log custom metric filter and subscribe to the same lambda function do to some stuff, but I want to get my lambda function to perform difference action depends on which metric filter trigger it. In the lambda function(in python) how can I know which metric filter trigger it?

In Lambda function handlers, you'll have an event object, this object will usually contain information about the request. According to the documentation, this is base64 encoded and zipped. I imagine this is because the logs are expected to get fairly large.
Once unzipped the structure is:
{ messageType: 'DATA_MESSAGE',
owner: '123456789123',
logGroup: 'testLogGroup',
logStream: 'testLogStream',
subscriptionFilters: [ 'testFilter' ],
logEvents:
[ { id: 'eventId1',
timestamp: 1440442987000,
message: '[ERROR] First test message' },
{ id: 'eventId2',
timestamp: 1440442987001,
message: '[ERROR] Second test message' } ] }
This can be found in the AWS docs under CloudWatch Logs.
You can check on the subscriptionFilters field in the event data to check which filter triggered the Lambda.
If your Lambda was in NodeJS, you could write something like the following:
const zlib = require('zlib');
const YOUR_FILTER = 'filter1';
exports.handler = async (event) => {
const zippedInput = new Buffer(event.awslogs.data, 'base64');
await new Promise((resolve, reject) => {
zlib.gunzip(zippedInput, function (e, buffer) {
const awslogsData = JSON.parse(buffer.toString('utf-8'));
// Conditions on filters to do something
if (awslogsData.subscriptionFilters.includes(YOUR_FILTER)) {
// do something
} else if (awslogsData.subscriptionFilters.includes('your_other_filter')) {
// do something else
}
resolve();
});
})
return 'Hello from Lambda!'
};

Related

NodeJS: AWS SDK V3: Not receiving any response data from lambda function

I'm trying to use the v3 javascript sdk to invoke a AWS Lambda function, and I'm having problems getting any meaningful response.
My code looks like so...
const { Lambda } = require("#aws-sdk/client-lambda");
const client = new Lambda();
const params = {
FunctionName: "MyLamdaFuncton",
Payload: JSON.stringify({ "action": "do_something" }),
InvocationType: "Event"
};
client.invoke(params)
.then((response) => {
console.log(JSON.stringify(response,null,4));
})
.catch((err) => {
console.error(err);
})
I can confirm from checking the CloudWatch logs that the lambda function works as exepcted. However this is the response I get in my NodeJS code...
{
"$metadata": {
"httpStatusCode": 202,
"requestId": "d6ba189d-9156-4f01-bd51-efe34a66fe34",
"attempts": 1,
"totalRetryDelay": 0
},
"Payload": {}
}
How do I get the actual response and status from the Lambda function?
If I change the payload above to intentionally throw an exception in my Lambda, the response in the console is still exactly the same.
update:
The Lambda function is written in Ruby. The response is returned like so...
{ statusCode: 200, body: JSON.generate(response.success?) }
where "response" is from another service it calls internally.
I've figured out what I was doing wrong. The issue was the "InvocationType". I got it working by changing to...
InvocationType: "RequestResponse"
Then I had to extract the response data like so...
const response_data = JSON.parse(new TextDecoder("utf-8").decode(response.Payload))

How do you use a nodejs Lambda in AWS as a producer to send messages to MSK topic without creating EC2 client server?

I am trying to create a Lambda in AWS that serves as a producer to an MSK topic. All the AWS docs say to create a new EC2 instance, but as my Lambda is in the same VPC I feel like this should work. I am very new to this and I notice my log statement never hits in my producer.on function.
I am using nodejs and the kafka-node module. The code can be found below.
Essentially, I am just wondering if anyone knows how to do this and why the producer.on function is never hit when I run test through the Lambda? This is just some test code to see if I can get it to send, but if any more data is needed oy help please let me know and thanks in advance.
exports.handler = async (event, context,callback) => {
const kafka = require('kafka-node');
const bp = require('body-parser');
const kafka_topic = 'MyTopic';
const Producer = kafka.Producer;
var KeyedMessage = kafka.KeyedMessage;
const Client = kafka.Client;
const client = new kafka.KafkaClient({kafkaHost: 'myhost:9094'});
console.log('client :: '+JSON.stringify(client));
const producer = new Producer(client);
console.log('about to hit producer code');
producer.on('ready', function() {
console.log('Hello there!')
let message = 'my message';
let keyedMessage = new KeyedMessage('keyed', 'me keyed message');
producer.send([
{ topic: kafka_topic, partition: 0, messages: [message, keyedMessage], attributes: 0 }
], function (err, result) {
console.log(err || result);
process.exit();
});
});
producer.on('error', function (err) {
console.log('error', err);
});
}
return "success";
What you need is to be able to produce messages on your MSK cluster using REST API. Why not setup a REST proxy for MSK as detailed here and then call this API to produce your messages to MSK.

Serverless Cloudwatch events appearing encrypted

I'm testing out Cloudwatch logs as a trigger for serverless functions. However when my serverless function is triggered, it's just outputting jibberish, which I figured was some form of encryption. Unfortunately I don't know what that encryption is.
Here's the output:
{"awslogs":{"data":"H4sIAAAAAAAAAGVR...4BmF05wEAAA=="}}
And here's the function's code:
const handler = async (event) => {
console.log('*********RECEIVED EVENT FROM CLOUDWATCH**********')
console.log(JSON.stringify(event));
return {
statusCode: 200,
body: JSON.stringify(
event,
null,
2
)
}
};
export { handler };
The "data" section that you get back is base64 encoded and compressed. To get the information out from the event, you just need to decode the base64 information and unzip the data.
Here's a code snippet that shows basically what needs to be done in order to read the log data.
...
const payload = Buffer.from(event.awslogs.data, 'base64');
zlib.gunzip(payload, (err, res) => {
if (err) {
...
}
const parsed = JSON.parse(res.toString('utf8'));
...
});

Apollo server subscription not recognizing Async Iterable

I'm having an issue with Apollo GraphQL's subscription. When attempting to start the subscription I'm getting this in return:
"Subscription field must return Async Iterable. Received: { pubsub: { ee: [EventEmitter], subscriptions: {}, subIdCounter: 0 }, pullQueue: [], pushQueue: [], running: true, allSubscribed: null, eventsArray: [\"H-f_mUvS\"], return: [function return] }"
I have other subscriptions setup and are completely functional - so I can confirm the webserver is setup correctly.
I'm just curious if anyone else has ever ran onto this issue before.
Source code in PR diff (it's an open source project):
https://github.com/astronomer/houston-api/pull/165/files
error in playground
I don't think this is an issue specific to the PR you posted. I'd be surprised if any of the subscriptions were working as is.
Your subscribe function should return an AsyncIterable, as the error states. Since it returns a call to createPoller, createPoller should return an AsyncIterable. But here's what that function looks like:
export default function createPoller(
func,
pubsub,
interval = 5000, // Poll every 5 seconds
timeout = 3600000 // Kill after 1 hour
) {
// Gernate a random internal topic.
const topic = shortid.generate();
// Create an async iterator. This is what a subscription resolver expects to be returned.
const iterator = pubsub.asyncIterator(topic);
// Wrap the publish function on the pubsub object, pre-populating the topic.
const publish = bind(curry(pubsub.publish, 2)(topic), pubsub);
// Call the function once to get initial dataset.
func(publish);
// Then set up a timer to call the passed function. This is the poller.
const poll = setInterval(partial(func, publish), interval);
// If we are passed a timeout, kill subscription after that interval has passed.
const kill = setTimeout(iterator.return, timeout);
// Create a typical async iterator, but overwrite the return function
// and cancel the timer. The return function gets called by the apollo server
// when a subscription is cancelled.
return {
...iterator,
return: () => {
log.info(`Disconnecting subscription ${topic}`);
clearInterval(poll);
clearTimeout(kill);
return iterator.return();
}
};
}
So createPoller creates an AsyncIterable, but then creates a shallow copy of it and returns that. graphql-subscriptions uses iterall's isAsyncIterable for the check that's producing the error you're seeing. Because of the way isAsyncIterable works, a shallow copy won't fly. You can see this for yourself:
const { PubSub } = require('graphql-subscriptions')
const { isAsyncIterable } = require('iterall')
const pubSub = new PubSub()
const iterable = pubSub.asyncIterator('test')
const copy = { ...iterable }
console.log(isAsyncIterable(iterable)) // true
console.log(isAsyncIterable(copy)) // false
So, instead of returning a shallow copy, createPoller should just mutate the return method directly:
export default function createPoller(...) {
...
iterator.return = () => { ... }
return iterator
}

AWS lambda function failing in Amazon Connect

I have written an AWS lambda function in node.js to send an email, which is invoked in an Amazon Connect contact flow. In the error branch it plays a prompt saying "lambda function failed." I verified the IAM role has permission to send email with SES, the sender/receiver emails are verified in SES, and also the lambda function has permissions for Amazon Connect.
The email does actually get sent out, but oddly I still hear the prompt "lambda function failed." Here is the code:
"use strict";
const aws = require('aws-sdk');
const sender = "Sender Name <sender#email.com>";
const recipient = "recipient#email.com";
const subject = "ALERT: no agents are logged in";
const body_text = "There are no agents logged in";
const body_html =
`<html>
<head></head>
<body>
<h1>ALERT</h1>
<p>Ther are no agents logged in to take calls in the queue.</p>
</body>
</html>`;
const charset = "UTF-8";
let params = {
Source: sender,
Destination: {
ToAddresses: [
recipient
],
},
Message: {
Subject: {
Data: subject,
Charset: charset
},
Body: {
Text: {
Data: body_text,
Charset: charset
},
Html: {
Data: body_html,
Charset: charset
}
}
},
};
const ses = new aws.SES({apiVersion: '2010-12-01'});
exports.handler = function(event, context, callback) {
ses.sendEmail(params, function(err, data) {
if(err) {
console.log("fail");
callback(err, err.message);
}
else {
console.log("success");
callback(null);
}
});
};
I checked the cloudwatch logs and don't see any error:
00:17:24
START RequestId: 17f1e239-990e-11e8-96bb-a1980f44db91 Version: $LATEST
00:17:24
2018-08-06T00:17:24.723Z 17f1e239-990e-11e8-96bb-a1980f44db91 success
00:17:24
END RequestId: 17f1e239-990e-11e8-96bb-a1980f44db91
00:17:24
REPORT RequestId: 17f1e239-990e-11e8-96bb-a1980f44db91 Duration: 226.51 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 32 MB
How do I troubleshoot this?
EDIT:
I enabled contact flow logs. In CloudWatch I noticed this:
{
"Parameters": { "FunctionArn": "arn:aws:lambda:us-west-2:769182588423:function:noAgentEmail", "TimeLimit": "8000" },
"Timestamp": "2018-08-06T06:36:31.786Z",
"ContactFlowModuleType": "InvokeExternalResource",
"Results": "The Lambda Function Returned An Error.",
"ExternalResults": { "forceClose": "false" },
"ContactId": "458027b0-d895-439e-bc06-114500dce64a",
"ContactFlowId": "arn:aws:connect:us-west-2:769182588423:instance/1e2ddedd-8335-42fe-89de-1e986fc016ef/contact-flow/2329af39-682c-4dc8-b3a2-5e7fe64de5d2"
}
What's confusing is it indicates the lambda function returned something:
"ExternalResults": { "forceClose": "false" }
But this is clearly not the case given the code. What's going on?
There's a couple of things to ensure your working Lambda function can be used for Connect.
As mentioned in the above answer, the return value should be a flat JSON object. As the guide says:
Nested and complex objects are not supported.
So, it should look something like this when you execute the lambda directly:
Response:
{
"statusPop": "success",
"id": "1",
"name": "me",
}
The property names and values need to be
alphanumeric, dash, and underscore characters.
Oh, and here's what tripped me up:
Amazon Connect sends paramters as part of a hierarchical structure, so any parameters you're expecting need to be referenced as
event['Details']['Parameters']['statusPop'];
instead of
event['statusPop'];
And don't forget to add permissions using:
aws lambda add-permission --function-name function:my-lambda-function --statement-id 1 \
--principal connect.amazonaws.com --action lambda:InvokeFunction --source-account 123456789012 \
--source-arn arn:aws:connect:us-east-1:123456789012:instance/def1a4fc-ac9d-11e6-b582-06a0be38cccf
More info and in depth details of what I mentioned here: https://docs.aws.amazon.com/connect/latest/adminguide/connect-lambda-functions.html
Your lambda needs to return a flat filed JSON object. It looks like on success your lambda callbacks with null.
Update your exports to this:
exports.handler = function(event, context, callback) {
ses.sendEmail(params, function(err, data) {
if(err) {
console.log("fail");
//callback(err, err.message);
callback(null, {"status":"error"});
}
else {
console.log("success");
callback(null, {"status":"success"});
}
});
});
More information on lambda/connect requirements here: https://docs.aws.amazon.com/connect/latest/adminguide/connect-lambda-functions.html
Note:
The output returned from the function must be a flat object of key/value pairs, with values that include only alphanumeric, dash, and underscore characters. Nested and complex objects are not supported. The size of the returned data must be less than 32 Kb of UTF-8 data.
you must return a valid JSON object such as below example
export const handler = async(event) => {
console.log("event" + JSON.stringify(event));
let amount = event['Details']['Parameters']['amount'];
console.log("amount:" + amount);
let resultMap = {
"amountValid": "valid"
};
console.log("resultMap" + JSON.stringify(resultMap));
return resultMap;
};

Resources