AWS SAM Include gem with native dependencies - ruby

Question

I want to run my lambda function locally (I am currently using ruby 2.7.1), but when I require a gem that needs native dependencies it fails because it doesn't find them.
I tried to use the 'pg' gem to connect to a postgresql database. Then I proceeded to run sam build and sam local invoke HelloWorldFunction --event events/event.json, which failed with the next error:
Invoking app.lambda_handler (ruby2.7)
Failed to download a new amazon/aws-sam-cli-emulation-image-ruby2.7:rapid-1.0.0 image.
Invoking with the already downloaded image.
Mounting /home/user/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated
inside runtime container
Init error when loading handler app.lambda_handler
{
"errorMessage": "libpq.so.5: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg_ext.so",
"errorType": "Init<LoadError>",
"stackTrace": [
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg.rb:5:in `<top (required)>'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
"/var/task/app.rb:3:in `<top (required)>'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'"
]
}
Next thing I tried was to execute sam build --use-container and got a Gem::Ext::BuildError with the message ERROR: Failed to build gem native extension.
It seems that the external libraries that the gem needs are not being included.
My questions are:
How can I make this possible using AWS SAM? And what would be the best approach for this?
How can I solve this problem for testing the lambda locally and also leave it working in the productive environment?
I read something about using Layers for this but I don't fully understand, as it's my first time working with lambda functions.
It also seems that the sam proyect has some open issues about solving this, but it won't be in the near future.
Any help will be very much appreciated!
Thank you!

Answer 1

Layers in Lambda basically give you a way of installing dependencies that aren't related directly to your function but are used by it. It is a really great way to support re-use and to speed up build and deploy times. The idea here is that you would install common or shared dependencies like the postgresql libraries into the layer, then just reference them from your function.
In short, they give you a way to place things on the filesystem that the lambda can access at runtime.
However, there are a few interlinked issues in this case:
the pg gem requires packages to be installed on the OS and files to be available at build-time
the pg gem does not install the dependencies itself
This means that using layers is a little less simple than would be ideal.
For your issue, try the following:
Create a new directory called pg_layer
In that directory, create a file called Makefile (the case is important) with the following content:
LIB_DIR = $(ARTIFACTS_DIR)/lib
LIB_FILES = \
/usr/lib64/libpq.so.* \
/usr/lib64/libldap_r-2.4.so.2* \
/usr/lib64/liblber-2.4.so.* \
/usr/lib64/libsasl2.so.* \
/usr/lib64/libssl3.so \
/usr/lib64/libsmime3.so \
/usr/lib64/libnss3.so
build-PGLayer:
yum install -y amazon-linux-extras
yum install -y postgresql-devel postgresql-libs
mkdir -p $(ARTIFACTS_DIR)/vendor/bundle
mkdir -p $(LIB_DIR)
for f in $(LIB_FILES); do cp $$f $(LIB_DIR); done
cp -r . $(ARTIFACTS_DIR)
mkdir -p $(ARTIFACTS_DIR)/ruby/gems
gem install pg --install-dir $(ARTIFACTS_DIR)/ruby/gems/2.7.0
Alter your template.yaml file, and add the below to your Resources section:
PGLayer:
Type: AWS::Serverless::LayerVersion
Properties:
CompatibleRuntimes:
- ruby2.7
ContentUri: 'pg_layer'
Metadata:
BuildMethod: makefile
In the template.yaml file, alter your function definition to add the below to the Properties:
Layers:
- !Ref PGLayer
Remove pg from your function's Gemfile (it will be available automatically because of the layer and leaving it there will prevent you from building the function)
When you're ready to build, use the command sam build --use-container.
To run your lambda function locally with the layer, use sam local start-api.
When you're ready to deploy sam deploy will push up your entire application, including the layer, and configure the function to use the layer.