SAM build - does it also build layers? - aws-lambda

Question

I'm new to both lambda's and also SAM - so if I've screwed anything simple up don't yell :D.
Summary: I can't get sam build to build a layer specified in template.yaml, it only builds the lambda function.
Background: I'm trying to build a lambda function in python3.7 that uses the skimage (scikit-image) module. To do that, I'm trying to use SAM to build and deploy it all. ...this is working
I'm trying to deploy the scikit-image module as a layer (and also build with SAM), rather than having it included in the lambda function direction ...this isn't working
As a start, I'm simply extending the standard SAM Hello World app.
I've got skimage working by simply add it to requirements.txt , then using sam build -u, then manually removing the numpy/scipy dependencies from the built package directory (I've got the AWS scipy/numpy layer included).
(I added import numpy, scipy.ndimage and skimage.draw to the standard hello world app, and included some test function calls to each)
requirements.txt:
requests
scikit-image
After that, everything works fine (running locally and/or on AWS).
However, I'd now like to move the skimage module out of my app and into a new custom layer (I'd like to have skimage in a layer to re-use for a few functions)
To set that up, I've created a dependencies directory and moved requirements.txt into there (leaving empty requirements.txt in the app directory).
I then updated template.yaml to also specify the new layer:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.7
Layers:
- arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python37-SciPy1x:2
- !Ref SkimageLayer
Events:
HelloWorld:
Type: Api
Properties:
Path: /hello
Method: get
SkimageLayer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: Skimage
Description: Skimage module layer
ContentUri: dependencies/
CompatibleRuntimes:
- python3.7
RetentionPolicy: Retain
DependsOn:
- Skimage
directory structure:
▾ dependencies/
requirements.txt (responses and scikit-image)
▸ events/
▾ hello_world/
__init__.py
app.py
requirements.txt (now empty)
▸ tests/
README.md
template.yaml
However, when I run sam build -u with that template file, nothing gets built for the layer specified in ./dependencies: SkimageLayer in the template.yml file. However the HelloWorldFunction still gets built correctly (now of course without any included modules)

Answer 1

Since SAM Cli version v0.50.0, it is building layers as part of sam build.
Design document could be a good starting point to understand how it works.
Basically you have to set a custom BuildMethod with your lambda's target runtime:
MyLayer:
Type: AWS::Serverless::LayerVersion
Properties:
ContentUri: my_layer
CompatibleRuntimes:
- python3.8
Metadata:
BuildMethod: python3.8 (or nodejs8.10 etc..)
Warning: For compiled language as Java, it has a issue which it tries to build layers before functions. It's expected to have it fixed on the next release (PR opened already).

Answer 2

Quick answer - No, currently SAM does not build layers you define in a SAM template.yaml file.
It will only build any functions you define.
However (curiously) it will package (upload to S3) and deploy (setup within AWS, assign ARN so it can be used etc) any layers you define.
There is a feature request on the SAM github issues to implement layer building with SAM.
This can actually be hacked right now to get SAM to build your layers as well, by creating a dummy function in your SAM template file, as well as a layer entry, and having the layer ContentUri entry point to the .aws-sam build directory that gets created for the function.
See my post here on that.
That approach actually seems to work pretty well for twisting SAM right now to build your layers for you.

Answer 3

I'm not sure if something changed recently but I'm able to do this without issue. My template file and structure is very similar to the OP except I've put all my common code into...
/dependencies/python/lib/python3.7/site-packages/
I didn't include a requirements.txt file in that directory... just the __init__.py file and various .py files that I need to import into my functions.
SAM then finds the code and builds the layer. You don't even need to zip the contents of the directory as some tutorials tell you to do.
The best part is Layers: is able to be put into the Globals: section of the template file so that the layer is available to all of your functions!
Globals:
Function:
Handler: main.lambda_handler
Timeout: 10
Runtime: python3.7
Layers:
- !Ref HelperFunctions
Resources:
HelperFunctions:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: MyHelperFunctions
Description: My Lambda Layer with Helper Functions for accessing RDS, Logging, and other utilities.
ContentUri: dependencies/
CompatibleRuntimes:
- python3.6
- python3.7
LicenseInfo: MIT
RetentionPolicy: Delete

Answer 4

The AWS team must have made things easier, relative to these older answers. From the current docs, all you do is list a layer as a property in your template (Nov 2020):
ServerlessFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: .
Handler: my_handler
Runtime: Python3.7
Layers:
- arn:aws:lambda:us-west-2:111111111111:layer:myLayer:1
- arn:aws:lambda:us-west-2:111111111111:layer:mySecondLayer:1
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-layers.html

Answer 5

I got it to work with the following script. Tested with Ubuntu 18 and CodeBuild
It pip install's the layer's requirements to .aws-sam/build/layername/python/. Then you can run sam package and sam deploy as normal
build-layers.py:
import yaml
import subprocess
import sys
import shutil
SAM_BUILD_PATH = ".aws-sam/build"
with open("template.yaml", "r") as f:
template = yaml.safe_load(f)
for key, resource in template["Resources"].items():
if resource["Type"] not in ["AWS::Serverless::LayerVersion", "AWS::Lambda::LayerVersion"]:
continue
properties = resource["Properties"]
content_uri = properties["ContentUri"]
layer_name = properties["LayerName"]
requirements_path = f"{content_uri}/requirements.txt"
subprocess.check_call([sys.executable, "-m", "pip", "install", "-r", requirements_path, "-t", f"{SAM_BUILD_PATH}/{layer_name}/python"])
shutil.copyfile("template.yaml", f"{SAM_BUILD_PATH}/template.yaml")
template.yaml:
Transform: AWS::Serverless-2016-10-31
Resources:
pandas:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: pandas
ContentUri: pandas
CompatibleRuntimes:
- python3.6
- python3.7
- python3.8
sqlparse:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: sqlparse
ContentUri: sqlparse
CompatibleRuntimes:
- python3.6
- python3.7
- python3.8
so call python build-layers.py first, then sam package then sam deploy
my directories look like this:
lambda
layers
pandas
requirements.txt (content = pandas)
sqlparse
requirements.txt (content = sqlparse)
template.yaml
build-layers.py
buildspec.yml:
--- # build spec for AWS CodeBuild
version: 0.2
phases:
install:
runtime-versions:
python: 3.8
commands:
- pip install aws-sam-cli
build:
commands:
- cd lambda/layers
- python build-layers.py
- sam package --s3-bucket foo --s3-prefix sam/lambda/layers | sam deploy --capabilities CAPABILITY_IAM -t /dev/stdin --stack-name LAYERS