I created an application with a AWS::SERVERLESS::FUNCTION, which has an HttpApi Event attached to it. I thought it would be a good idea to create one lambda per resource, so e.g. Post, Get and Put on /customer are all handled by a single lambda, which decides which action to take using
switch (input.getHttpMethod()) {
case "GET": ...
case "POST: ...
}
So now coming to my problem: When starting an application using sam local start-api my lambda gets called correctly, but neither input.getHttpMethod() nor input.getRequestContext().getHttpMethod() is set.
Given that SAM supports multiple HttpApi events, failing to provide the http method when running the application locally mitigates local development virtually completely. Am I doing something wrong, or is this really not working? I'm using Java btw, I can't tell if this problem exists using other languages as well.
Just in Case:
Is my "one lambda per resource" approach wrong, should every single action have its own lambda?
I found the problem. HttpApi uses "PayloadFormatVersion: 2.0" as default. input was of type com.amazonaws.services.lambda.runtime.events.APIGatewayProxyRequestEvent, but this represents format 1.0, I had to use com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent to get it to work.
This fails silently because instances are just constructed from json input, and several fields are the same among both classes.
Related
I have deployed an AWS Lambda function, written in Python, and AWS API Gateway structure to cause POST requests to an API endpoint to be redirected to my function. I want to upload a PDF document to my function and have it store the document in a S3 bucket. The problem I have is that the payload of any POST request to my API is being UTF-8 encoded. I don't want that but can't figure out the magic mojo to disable encoding of the request payload.
I am testing using curl, with the following command line:
curl -XPOST https://xxxxxxxxxx.execute-api.us-west-1.amazonaws.com/test -H 'content-type: application/pdf' --data-binary #document.pdf
UPDATE: I just found the following article describing how API Gateway and Lambda support uploading binary data:
https://aws.amazon.com/blogs/compute/handling-binary-data-using-amazon-api-gateway-http-apis/
This article suggests that all of the complexities that I discussed in the initial formation of my question (still provided below) should not be necessary. All I should need to do to upload binary content to my Lambda function is insure that my request includes an appropriate Content-Type header. I was already doing that, but I massaged my Curl command a bit (modified above) to define my request in exactly the way that is done in this article. I still get UTF-8 encoded data and NOT base-64 encoded data. I tried uploading a jpeg file rather than a PDF so I was doing exactly what was done in the article. Still no love. I don't get it. This article demonstrates exactly what I'm doing. But I don't get the result it suggests I should. Ggggrrrr.
ORIGINAL POST:
I am using Terraform to define my deployment. I want to cause the PDF to not be encoded/mangled at all. This is my first time using API Gateway, and I'm obviously missing some bit of config. The one thing I'm doing specifically right now to say that I want incoming payloads to be treated as binary is via the binary_media_types argument to my API definition in Terraform:
resource aws_api_gateway_rest_api proxy {
...
binary_media_types = [
"application/pdf",
"application/octet-stream",
"*/*"
]
This sets the Binary Media Types configuration associated with the API I've defined. I've confirmed via the AWS Console that this setting is having the desired effect...I can see these types in the console. I should need just the first item in the list, but I've added the others while I try to figure out the problem here. By adding that wildcard item, I believe that it shouldn't matter what the incoming Content-Type is...all payloads should be being treated as binary.
The other bit of config that I know about that might be important is the "integration contentHandling property". Here is the key bit of AWS docs that seems to explain all this:
I think the case that applies to me here is the one I've highlighted, per what I say above. This says to me that I shouldn't need to do anything else, per the "unspecified" value in the table for "contentHandling. I've tried setting the "contentHandling" argument on the integration record of my Terraform config, like this:
resource aws_api_gateway_integration proxy {
...
passthrough_behavior = "WHEN_NO_MATCH"
content_handling = "CONVERT_TO_BINARY"
}
I first tried only specifying the content_handling value. I've also tried setting that value to "CONVERT_TO_TEXT", hoping to then get base64-encoded data. Neither of these has any effect. I've tried adding the passthrough_behavior value as shown. I've also tried replacing "WHEN_NO_MATCH" with "WHEN_NO_TEMPLATES". Nothing I do changes the behavior. I haven't been able to figure out where these settings would show up in the AWS console. If I knew they were necessary, I'd explore this further. But I don't think I need to set these.
What am I missing? How can I POST a PDF document to my AWS Lambda function through API Gateway and have the payload of the request not be converted in any way? TIA!
NOTE: I am aware of this Q/A: PDF Uploaded via AWS API Gateway getting corrupted. The answer there doesn't apply to me, as I need to avoid having to form-encode the upload. The client code that will eventually be doing the upload is set in stone and sends a POST request with a payload that is just the bytes of the PDF.
I've recently updated my serverless project, and I've found that many things have changed in the last few updates.
https://serverless.com/
I don't fully understand whats the correct way to have multiple lambda functions and api gateway endpoints related to the same project. With the old serverless I have every lambda and endpoint as a completely seperate function, this worked pretty well for me.
I can't seem to do this anymore, if I try my second lambda function overrides my first, presumably because my "service name" for both is the same. My service name is the same because I want both rest endpoints in the same API in API Gateway. Since serverless creates the API name based on the service name.
So then I tried to add both functions to the same "Service". this worked for the most part, except that now I need to include my custom role statement for all my functions into the same role (because this one role is now being linked to all my functions). Effectively giving more permissions to each individual function than it should have. The other issue is that all my handler files for the different functions are being put into each functions deployment bundle.
So basically, I'm not sure what is the correct approach to have multiple functions that relate to the same project but are separate in functionality. It used to make sense, now doesn't.
If anybody can give me some pointers please
Thanks
I understand your frustration. I had the same feeling until I looked deeper into the new version and formed a better understanding. One thing to note though, is the new version is not completely finished yet. So if something is completely missing, you can file an issue and have it prioritized before 1.0 is out.
You are supposed to define multiple functions under the same service under the functions: section of serverless.yml. To package these functions individually (exclude code for other functions) you will have to set individually: true under package: section. You can then use include and exclude options at the root level and at the function level as well. There's an upcoming change that will let you use glob syntax in your include and exclude options (example **/*-fn.js). You can find more about packaging here https://www.serverless.com/framework/docs/providers/aws/guide/deploying.
Not sure how to use different roles for different functions under the same service.. How did you do it with 0.5?
I was trying to find a solution for individual iam roles per function as well. I couldn't find a way to do it, but while I was looking through the documentation I found the line: "Support for separate IAM Roles per function is coming soon." on this page, so at least we know they are working on it.
The "IAM Roles Per Function" plugin for Serverless allows you to do exactly what it says on the tin: specify roles for each function. You can still use the provider-level roles as well:
By default, function level iamRoleStatements override the provider level definition. It is also possible to inherit the provider level definition by specifying the option iamRoleStatementsInherit: true
EDIT: You can also apply a predefined AWS role at both the provider and function level.
If I have two services, 'Users' and 'Products', each with several functions with endpoints defined for each one (as any traditional API would), is it possible for them to be organised separately in a code base (for clarity) but once deployed share the same API base URL? For example, consider I have the following structure:
/src
-- /users
---- event.json
---- handler.js
---- serverless.yml
-- /products
---- event.json
---- handler.js
---- serverless.yml
and my src/users/serverless.yml has the following defined:
functions:
create:
handler: handler.create
events:
- http: POST user
read:
handler: handler.read
events:
- http: GET user
and my src/products/serverless.yml has basically the same thing, just swap 'user' for 'products'.
Currently, both of those services will be deployed to distinctly different API endpoints, one with a URL https://fghijklmnop.execute-api... and another with a URL https://abcdevwxyz.execute-api....
My question is, would it be possible to have these services be deployed but remain under a single API with a single URL (so both would be served under URL https://abcdevwxyz.execute-api....)?
I'm assuming the answer to be, 'No, because Cloud Formation...', but I thought I would post the question here simply for the sake of discussion and to aid my own understanding of building serverless applications.
I'm aware of using Custom Domains, as per the answer here, but for a quicker development cycle this is not really an ideal solution.
My only solution so far would be to simply create a service called 'api' which would contain all the endpoints my API would need which would simply invoke my other services' Lambda functions directly rather than via previously-configured endpoints. It would be an abstraction layer, really, but add potentially unnecessary layers to my application. Again, curious to see what the community feels on this.
You can put multiple functions in one serverless.yml
/src
-- event.json
-- users.handler.js
-- products.handler.js
-- serverless.yml
I can't speak directly to Serverless framework support, but this is certainly possible in API Gateway.
You can maintain multiple Swagger files for each "sub API", and use import?mode=merge to import both definitions into the same API.
See
http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-import-api.html
Thanks,
Ryan
What I've done with my own code is to pull all code out of the handler.js files and put it inside modules. These modules would be required into the handler.js files and a simple function would then be called.
usersModule.js:
export const doSomething = () => {
// Do something here.
};
users/handler.js:
import {doSomething} from '../.../usersModule.js';
export const handler = ( event, context, callback ) => {
doSomething();
// Do other stuff...
callback( null, "Success" );
};
This way, you can place the meat of your code wherever you would like to have it, organized in whatever way makes sense to you.
You would, however, still have to have a single API defined. Or use the answer from RyanG-AWS to merge the APIs.
If you'd still like to keep code and API definitions separate, you could create the users API and the products API separately. You would then have another combined API that would call either of these APIs. So this way, you would have a single service with a single base URL that you would call. You can do this with integration type HTTP. I have not tried this, so I don't know how well it would work.
You can use Custom Domain Names : http://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-custom-domains.html
setup the custom domains name (you'll need a SSL certificate) http://myapi.com/
then map your apis :
http://myapi.com/users
http://myapi.com/products
Just call your functions like this :
http://myapi.com/users/create
http://myapi.com/users/read
http://myapi.com/products/whaterver
I came up with my own solution to this problem. I abstracted the integration points of my application so that I have specific Integration services (API, S3, SNS, etc.) which respond to events and then process those events and delegate them to separate microservices. I wrote an article on it, with code examples.
I created an ASP.NET MVC4 Web API service (REST) with a single GET action. The action currently needs 11 input values, so rather than passing all of those values in the URL, I opted to encapsulate those values into a single class type and have it passed as Content-Body. When I test in Fiddler, I specify the verb as GET, and enter the JSON text in the "Request Body" input box. This works great!
The problem is when I attempt to perform Load Testing in Visual Studio 2010 Ultimate. I am able to specify the GET action and the JSON Content-Body just fine. But when I run the Load test, VS reports exceptions of type ProtocolViolationException (Cannot send a content-body with this verb-type) in the test results. The test executes in 1ms so I suspect the exceptions are causing the test to immediately abort. What can I do to avoid those exceptions? I'd prefer to not change my API to use URL arguments just to work-around the test tooling. If I should change the API for other reasons, let me know. Thanks!
I found it easier to put this answer rather than carry on the discussions.
Sending content with GET is not defined in RFC 2616 yet it has not been prohibited. So as far as the spec is concerned we are in a territory that we have to make our judgement.
GET is canonically used to get a resource. So you are retrieving this resource using this verb with the parameters you are sending. Since GET is both safe and idempotent, it is ideal for caching. Caching usually takes place based on the resource URI - and sometimes based on various headers. The point is cache implementations - AFAIK - would not use the GET content (and to be honest I have not seen any GET with content in real world). And it would not make sense to include the content in the key generation since it reduces the scalability of the caches.
If you have parameters to send, they must be in the URI since this is part of what defines that URI. As such, I strongly believe sending content with GET is wrong.
Even when you look at implementations such as OData, they put the criteria in the URI. I cannot imagine your (or any) applications requirements is beyond OData query requirements.
I have resigned myself to the fact that many of the features that EC2 users are accustomed to (in particular, tagging) do not exist in OpenStack. There is, however, one piece of functionality whose absence is driving me crazy.
Although OpenStack doesn't have full support for instance tags (like EC2 does), it does have the notion of an instance name. This name is exposed by the Web UI, which even allows you to set it:
This name is also exposed through the nova list command line utility.
However (and this is my problem) this field is not exposed through the nova-ec2 API layer. The cleanest way for them to integrate this with existing EC2 platform tools would be to simulate an instance Tag with name "Name", but they don't do this. What's more, I can't figure out which Nova API endpoint I can use to read and write the name (it doesn't seem to be documented on the API reference); but of course it must be somehow possible since the web client and nova-client can both somehow do it.
At the moment, I'm forced to change it manually from the website every time I launch a new instance. (I can't do it during instance creation because I use the nova-ec2 API, not the nova command line client).
My question is:
Is there a way to read/write the instance name through the EC2 API layer?
Failing that, what is the REST endpoint to set it programmatically?
(BONUS): What is Nova's progress on supporting general instance tagging?
The Python novaclient.v1_1 package has a method on the server object:
def update(self, server, name=None):
"""
Update the name or the password for a server.
:param server: The :class:`Server` (or its ID) to update.
:param name: Update the server's name.
"""
if name is None:
return
body = {
"server": {
"name": name,
},
}
self._update("/servers/%s" % base.getid(server), body)
This indicates that you can update the name of a server by POST-ing
the following JSON to http://nova-api:port/v2.0/servers/{server-id}:
{
"server": {
"name": "new_name"
}
}
Of course, all of the usual authentication headers (namely X-Auth-Token
from your Keystone server) are still required, so it is probably easier to
use a client library for whatever language you prefer to manage all that.