Package files into specific folder of application bundle when deploying to AWS Lambda via Serverless Framework - aws-lambda

Question

Context
I am using the aws-node-typescript example of the Serverless Framework. My goal is to integrate Prisma into it.
So far, I have:
Created the project locally using serverless create
Set up a PostgreSQL database on Railway
Installed prisma, ran prisma init, created a basic User model and ran prisma migrate dev successfully
Created a second users function by copying the existing hello function
Deployed the function using serverless deploy
Now in my function, when I instantiate PrismaClient, I get an internal server error and the function logs this error: "ENOENT: no such file or directory, open '/var/task/src/functions/users/schema.prisma'"
My project structure looks as follows:
.
├── README.md
├── package-lock.json
├── package.json
├── prisma
│ ├── migrations
│ │ ├── 20221006113352_init
│ │ │ └── migration.sql
│ │ └── migration_lock.toml
│ └── schema.prisma
├── serverless.ts
├── src
│ ├── functions
│ │ ├── hello
│ │ │ ├── handler.ts
│ │ │ ├── index.ts
│ │ │ ├── mock.json
│ │ │ └── schema.ts
│ │ ├── index.ts
│ │ └── users
│ │ ├── handler.ts
│ │ └── index.ts
│ └── libs
│ ├── api-gateway.ts
│ ├── handler-resolver.ts
│ └── lambda.ts
├── tsconfig.json
└── tsconfig.paths.json
Also, here's the handler for the users function:
ts
import { formatJSONResponse } from '#libs/api-gateway';
import { middyfy } from '#libs/lambda';
import { PrismaClient } from '#prisma/client'
const users = async (event) => {
console.log(`Instantiating PrismaClient inside handler ...`)
const prisma = new PrismaClient()
return formatJSONResponse({
message: `Hello, ${event.queryStringParameters.name || 'there'} welcome to the exciting Serverless world!`,
event,
});
};
export const main = middyfy(users);
The problem arises because in order to instantiate PrismaClient, the schema.prisma file needs to be part of the application bundle. Specifically, it needs to be in /var/task/src/functions/users/ as indicated by the error message.
I already adjusted the package.patterns option in my serverless.ts file to look as follows:
package: { individually: true, patterns: ["**/*.prisma"] },
Question
This way, the bundle that's uploaded to AWS Lambda includes the prisma directory in its root, here's the .serverless folder after I ran sls package (I've unzipped users.zip here so that you can see its contents):
.
├── cloudformation-template-create-stack.json
├── cloudformation-template-update-stack.json
├── hello.zip
├── serverless-state.json
├── users
│ ├── prisma
│ │ └── schema.prisma
│ └── src
│ └── functions
│ └── users
│ ├── handler.js
│ └── handler.js.map
└── users.zip
I can also confirm that the deployed version of my AWS Lambda has the same folder structure.
How can I move the users/prisma/schema.prisma file into users/src/functions/users using the patterns in my serverless.ts file?

Answer 1

I found a (pretty ugly) solution. If anyone can think of a more elegant one, I'm still very open to it and happy to give you the points for a correct answer.
Solving the "ENOENT: no such file or directory, open '/var/task/src/functions/users/schema.prisma'" error
To solve this, I just took a very naive approach and manually copied over the schema.prisma file from the prisma directory into the src/functions/users. Here's the file structure I now had:
.
├── README.md
├── package-lock.json
├── package.json
├── prisma
│ ├── migrations
│ │ ├── 20221006113352_init
│ │ │ └── migration.sql
│ │ └── migration_lock.toml
│ └── schema.prisma
├── serverless.ts
├── src
│ ├── functions
│ │ ├── hello
│ │ │ ├── handler.ts
│ │ │ ├── index.ts
│ │ │ ├── mock.json
│ │ │ └── schema.ts
│ │ ├── index.ts
│ │ └── users
│ │ ├── schema.prisma
│ │ ├── handler.ts
│ │ └── index.ts
│ └── libs
│ ├── api-gateway.ts
│ ├── handler-resolver.ts
│ └── lambda.ts
├── tsconfig.json
└── tsconfig.paths.json
This is obviously a horrible way to solve this, because I now have two Prisma schema files in different locations and have to make sure I always update the one in src/functions/users/schema.prisma after changing the original one in prisma/schema.prisma to keep them in sync.
Once I copied this file and redeployed, the schema.prisma file was in place in the right location in the AWS Lambda and the error went away and PrismaClient could be instantiated.
I then added a simple Prisma Client query into the handler:
const users = async (event) => {
console.log(`Instantiating PrismaClient inside handler ...`)
const prisma = new PrismaClient()
const userCount = await prisma.user.count()
console.log(`There are ${userCount} users in the database`)
return formatJSONResponse({
message: `Hello, ${event.queryStringParameters.name || 'there'} welcome to the exciting Serverless world!`,
event,
});
};
export const main = middyfy(users);
... and encountered a new error, this time, about the query engine:
Invalid `prisma.user.count()` invocation:
Query engine library for current platform \"rhel-openssl-1.0.x\" could not be found.
You incorrectly pinned it to rhel-openssl-1.0.x
This probably happens, because you built Prisma Client on a different platform.
(Prisma Client looked in \"/var/task/src/functions/users/libquery_engine-rhel-openssl-1.0.x.so.node\")
Searched Locations:
/var/task/.prisma/client
/Users/nikolasburk/prisma/talks/2022/serverless-conf-berlin/aws-node-typescript/node_modules/#prisma/client
/var/task/src/functions
/var/task/src/functions/users
/var/task/prisma
/tmp/prisma-engines
/var/task/src/functions/users
To solve this problem, add the platform \"rhel-openssl-1.0.x\" to the \"binaryTargets\" attribute in the \"generator\" block in the \"schema.prisma\" file:
generator client {
provider = \"prisma-client-js\"
binaryTargets = [\"native\"]
}
Then run \"prisma generate\" for your changes to take effect.
Read more about deploying Prisma Client: https://pris.ly/d/client-generator
Solving the Query engine library for current platform \"rhel-openssl-1.0.x\" could not be found. error
I'm familiar enough with Prisma to know that Prisma Client depends on a query engine binary that has to be built specifically for the platform Prisma Client will be running on. This can be configured via the binaryTargets field on the generator block in my Prisma schema. The target for AWS Lamda is rhel-openssl-1.0.x.
So I adjusted the schema.prisma file (in both locations) accordingly:
generator client {
provider = "prisma-client-js"
binaryTargets = ["native", "rhel-openssl-1.0.x"]
}
After that, I ran npx prisma generate to update the generated Prisma Client in node_modules.
However, this hadn't resolved the error yet, the problem still was the Prisma Client couldn't find the query engine binary.
So I followed the same approach as for the schema.prisma file when it was missing:
I manually copied it into src/functions/users (this time from its location inside node_modules/.prisma/libquery_engine-rhel-openssl-1.0.x.so.node)
I added the new path to the package.patterns property in my serverless.ts:
package: {
individually: true,
patterns: ["**/*.prisma", "**/libquery_engine-rhel-openssl-1.0.x.so.node"],
},
After I redeployed and tested the function, another error occured:
Invalid `prisma.user.count()` invocation:
error: Environment variable not found: DATABASE_URL.
--> schema.prisma:11
|
10 | provider = \"postgresql\"
11 | url = env(\"DATABASE_URL\")
|
Validation Error Count: 1
Solving the Environment variable not found: DATABASE_URL. error
This time, it was pretty straightforward and I went into the AWS Console at https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/aws-node-typescript-dev-users?tab=configure and added a DATABASE_URL env var via the Console UI, pointing to my Postgres instance on Railway:

Answer 2

I usually lurk but the answer above lead me to come up with a reasonably elegant solution I felt I should share for the next poor sap to come along and try to integrate serverless and prisma in Typescript (though, I bet this solution and process would work in other build systems).
I was using the example aws-nodejs-typescript template, which is plagued with a bug which required me to apply the fix here by patching the local node_modules serverless package.
I then had to essentially walk through #nburk's answer to get myself up and running which is, as stated, inelegant.
In my travels trying to understand prisma's behavior, requirement of a platform-specific binary, and how to fix it, figured that if I could manually side-load the binary into the build folder post-compile I could get the serverless bundler to zip it up.
I came across the 'serverless-plugin-scripts' plugin, which allows us to do exactly this via serverless lifecycle hooks.
I put this in my serverless.ts:
plugins: ['serverless-esbuild', 'serverless-plugin-scripts'],
I put the following in package.json:
"scripts": {
"test": "echo 'Error: no test specified' && exit 1",
"postbuild": "yarn fix-scrape-scheduler",
"fix-scrape-scheduler": "cp ../../node_modules/.prisma/client/schema.prisma .esbuild/.build/src/functions/schedule-scrapes/. && cp ../../node_modules/.prisma/client/libquery_engine-rhel-openssl-1.0.x.so.node .esbuild/.build/src/functions/schedule-scrapes/."
},
and this also in my serverless.ts:
scripts: {
hooks: {
'before:package:createDeploymentArtifacts': 'yarn run postbuild'
}
}
This causes the 'serverless-plugin-scripts' plugin to call my post-build yarn script and fixup the .build folder that esbuild creates. I imagine that if your build system (such as webpack or something) creates the build dir under a different name (such as lib), this process could be modified accordingly.
I will have to create a yarn script to do this for each function that is individually packaged, however this is dynamic, precludes the need to keep multiple copies of schema.prisma in source, and copies the files from the dynamically generated .prisma folder in node_modules.
Note, I am using yarn workspaces here, so the location of your node_modules folder will vary based on your repo setup.
Also, I did run into this error Please make sure your database server is running at which was remedied by making sure the proper security groups were whitelisted outbound for the lambda function, and inbound for RDS. Also make sure to check your subnet ACLs and route-tables.

Answer 3

We ran into the same issue recently. But our context is slightly different: the path to the schema.prisma file was /var/task/node_modules/.prisma/client/schema.prisma.
We solved this issue by using Serverless Package Configuration.
serverless.yml
service: 'your-service-name'
plugins:
- serverless-esbuild
provider:
# ...
package:
include:
- 'node_modules/.prisma/client/schema.prisma' # <-------- this line
- 'node_modules/.prisma/client/libquery_engine-rhel-*'
This way only the src folder containing the lambda functions and the node_modules folder containing these two Prisma files were packaged and uploaded to AWS.
Although the use of serverless.package.include and serverless.package.exclude is deprecated in favor of serverless.package.patterns, this was the only way to get it to work.

Answer 4

An option is to use webpack with the copy-webpack-plugin and change the structure of your application, put all handlers inside the handlers folder.
Folders structure:
.
├── handlers/
│ ├── hello.ts
│ └── ...
└── services/
├── hello.ts
└── ...
webpack.config.js:
/* eslint-disable #typescript-eslint/no-var-requires */
const path = require("path");
// const nodeExternals = require("webpack-node-externals");
const CopyPlugin = require("copy-webpack-plugin");
const slsw = require("serverless-webpack");
const { isLocal } = slsw.lib.webpack;
module.exports = {
target: "node",
stats: "normal",
entry: slsw.lib.entries,
// externals: [nodeExternals()],
mode: isLocal ? "development" : "production",
optimization: { concatenateModules: false },
resolve: { extensions: [".js", ".ts"] },
output: {
libraryTarget: "commonjs",
filename: "[name].js",
path: path.resolve(__dirname, ".webpack"),
},
module: {
rules: [
{
test: /\.tsx?$/,
loader: "ts-loader",
exclude: /node_modules/,
},
],
},
plugins: [
new CopyPlugin({
patterns: [
{
from: "./prisma/schema.prisma",
to: "handlers/schema.prisma",
},
{
from: "./node_modules/.prisma/client/libquery_engine-rhel-openssl-1.0.x.so.node",
to: "handlers/libquery_engine-rhel-openssl-1.0.x.so.node",
},
],
}),
],
};
If you need to run the npx prisma generate before assembling the package you can use the plugin serverless-scriptable-plugin (put before webpack):
plugins:
- serverless-scriptable-plugin
- serverless-webpack
custom:
scriptable:
hooks:
before:package:createDeploymentArtifacts: npx prisma generate
webpack:
includeModules: false
Dependences:
npm install -D webpack serverless-webpack webpack-node-externals copy-webpack-plugin serverless-scriptable-plugin