Context
I am using the aws-node-typescript example of the Serverless Framework. My goal is to integrate Prisma into it.
So far, I have:
Created the project locally using serverless create
Set up a PostgreSQL database on Railway
Installed prisma, ran prisma init, created a basic User model and ran prisma migrate dev successfully
Created a second users function by copying the existing hello function
Deployed the function using serverless deploy
Now in my function, when I instantiate PrismaClient, I get an internal server error and the function logs this error: "ENOENT: no such file or directory, open '/var/task/src/functions/users/schema.prisma'"
My project structure looks as follows:
.
├── README.md
├── package-lock.json
├── package.json
├── prisma
│ ├── migrations
│ │ ├── 20221006113352_init
│ │ │ └── migration.sql
│ │ └── migration_lock.toml
│ └── schema.prisma
├── serverless.ts
├── src
│ ├── functions
│ │ ├── hello
│ │ │ ├── handler.ts
│ │ │ ├── index.ts
│ │ │ ├── mock.json
│ │ │ └── schema.ts
│ │ ├── index.ts
│ │ └── users
│ │ ├── handler.ts
│ │ └── index.ts
│ └── libs
│ ├── api-gateway.ts
│ ├── handler-resolver.ts
│ └── lambda.ts
├── tsconfig.json
└── tsconfig.paths.json
Also, here's the handler for the users function:
ts
import { formatJSONResponse } from '#libs/api-gateway';
import { middyfy } from '#libs/lambda';
import { PrismaClient } from '#prisma/client'
const users = async (event) => {
console.log(`Instantiating PrismaClient inside handler ...`)
const prisma = new PrismaClient()
return formatJSONResponse({
message: `Hello, ${event.queryStringParameters.name || 'there'} welcome to the exciting Serverless world!`,
event,
});
};
export const main = middyfy(users);
The problem arises because in order to instantiate PrismaClient, the schema.prisma file needs to be part of the application bundle. Specifically, it needs to be in /var/task/src/functions/users/ as indicated by the error message.
I already adjusted the package.patterns option in my serverless.ts file to look as follows:
package: { individually: true, patterns: ["**/*.prisma"] },
Question
This way, the bundle that's uploaded to AWS Lambda includes the prisma directory in its root, here's the .serverless folder after I ran sls package (I've unzipped users.zip here so that you can see its contents):
.
├── cloudformation-template-create-stack.json
├── cloudformation-template-update-stack.json
├── hello.zip
├── serverless-state.json
├── users
│ ├── prisma
│ │ └── schema.prisma
│ └── src
│ └── functions
│ └── users
│ ├── handler.js
│ └── handler.js.map
└── users.zip
I can also confirm that the deployed version of my AWS Lambda has the same folder structure.
How can I move the users/prisma/schema.prisma file into users/src/functions/users using the patterns in my serverless.ts file?
I found a (pretty ugly) solution. If anyone can think of a more elegant one, I'm still very open to it and happy to give you the points for a correct answer.
Solving the "ENOENT: no such file or directory, open '/var/task/src/functions/users/schema.prisma'" error
To solve this, I just took a very naive approach and manually copied over the schema.prisma file from the prisma directory into the src/functions/users. Here's the file structure I now had:
.
├── README.md
├── package-lock.json
├── package.json
├── prisma
│ ├── migrations
│ │ ├── 20221006113352_init
│ │ │ └── migration.sql
│ │ └── migration_lock.toml
│ └── schema.prisma
├── serverless.ts
├── src
│ ├── functions
│ │ ├── hello
│ │ │ ├── handler.ts
│ │ │ ├── index.ts
│ │ │ ├── mock.json
│ │ │ └── schema.ts
│ │ ├── index.ts
│ │ └── users
│ │ ├── schema.prisma
│ │ ├── handler.ts
│ │ └── index.ts
│ └── libs
│ ├── api-gateway.ts
│ ├── handler-resolver.ts
│ └── lambda.ts
├── tsconfig.json
└── tsconfig.paths.json
This is obviously a horrible way to solve this, because I now have two Prisma schema files in different locations and have to make sure I always update the one in src/functions/users/schema.prisma after changing the original one in prisma/schema.prisma to keep them in sync.
Once I copied this file and redeployed, the schema.prisma file was in place in the right location in the AWS Lambda and the error went away and PrismaClient could be instantiated.
I then added a simple Prisma Client query into the handler:
const users = async (event) => {
console.log(`Instantiating PrismaClient inside handler ...`)
const prisma = new PrismaClient()
const userCount = await prisma.user.count()
console.log(`There are ${userCount} users in the database`)
return formatJSONResponse({
message: `Hello, ${event.queryStringParameters.name || 'there'} welcome to the exciting Serverless world!`,
event,
});
};
export const main = middyfy(users);
... and encountered a new error, this time, about the query engine:
Invalid `prisma.user.count()` invocation:
Query engine library for current platform \"rhel-openssl-1.0.x\" could not be found.
You incorrectly pinned it to rhel-openssl-1.0.x
This probably happens, because you built Prisma Client on a different platform.
(Prisma Client looked in \"/var/task/src/functions/users/libquery_engine-rhel-openssl-1.0.x.so.node\")
Searched Locations:
/var/task/.prisma/client
/Users/nikolasburk/prisma/talks/2022/serverless-conf-berlin/aws-node-typescript/node_modules/#prisma/client
/var/task/src/functions
/var/task/src/functions/users
/var/task/prisma
/tmp/prisma-engines
/var/task/src/functions/users
To solve this problem, add the platform \"rhel-openssl-1.0.x\" to the \"binaryTargets\" attribute in the \"generator\" block in the \"schema.prisma\" file:
generator client {
provider = \"prisma-client-js\"
binaryTargets = [\"native\"]
}
Then run \"prisma generate\" for your changes to take effect.
Read more about deploying Prisma Client: https://pris.ly/d/client-generator
Solving the Query engine library for current platform \"rhel-openssl-1.0.x\" could not be found. error
I'm familiar enough with Prisma to know that Prisma Client depends on a query engine binary that has to be built specifically for the platform Prisma Client will be running on. This can be configured via the binaryTargets field on the generator block in my Prisma schema. The target for AWS Lamda is rhel-openssl-1.0.x.
So I adjusted the schema.prisma file (in both locations) accordingly:
generator client {
provider = "prisma-client-js"
binaryTargets = ["native", "rhel-openssl-1.0.x"]
}
After that, I ran npx prisma generate to update the generated Prisma Client in node_modules.
However, this hadn't resolved the error yet, the problem still was the Prisma Client couldn't find the query engine binary.
So I followed the same approach as for the schema.prisma file when it was missing:
I manually copied it into src/functions/users (this time from its location inside node_modules/.prisma/libquery_engine-rhel-openssl-1.0.x.so.node)
I added the new path to the package.patterns property in my serverless.ts:
package: {
individually: true,
patterns: ["**/*.prisma", "**/libquery_engine-rhel-openssl-1.0.x.so.node"],
},
After I redeployed and tested the function, another error occured:
Invalid `prisma.user.count()` invocation:
error: Environment variable not found: DATABASE_URL.
--> schema.prisma:11
|
10 | provider = \"postgresql\"
11 | url = env(\"DATABASE_URL\")
|
Validation Error Count: 1
Solving the Environment variable not found: DATABASE_URL. error
This time, it was pretty straightforward and I went into the AWS Console at https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/aws-node-typescript-dev-users?tab=configure and added a DATABASE_URL env var via the Console UI, pointing to my Postgres instance on Railway:
I usually lurk but the answer above lead me to come up with a reasonably elegant solution I felt I should share for the next poor sap to come along and try to integrate serverless and prisma in Typescript (though, I bet this solution and process would work in other build systems).
I was using the example aws-nodejs-typescript template, which is plagued with a bug which required me to apply the fix here by patching the local node_modules serverless package.
I then had to essentially walk through #nburk's answer to get myself up and running which is, as stated, inelegant.
In my travels trying to understand prisma's behavior, requirement of a platform-specific binary, and how to fix it, figured that if I could manually side-load the binary into the build folder post-compile I could get the serverless bundler to zip it up.
I came across the 'serverless-plugin-scripts' plugin, which allows us to do exactly this via serverless lifecycle hooks.
I put this in my serverless.ts:
plugins: ['serverless-esbuild', 'serverless-plugin-scripts'],
I put the following in package.json:
"scripts": {
"test": "echo 'Error: no test specified' && exit 1",
"postbuild": "yarn fix-scrape-scheduler",
"fix-scrape-scheduler": "cp ../../node_modules/.prisma/client/schema.prisma .esbuild/.build/src/functions/schedule-scrapes/. && cp ../../node_modules/.prisma/client/libquery_engine-rhel-openssl-1.0.x.so.node .esbuild/.build/src/functions/schedule-scrapes/."
},
and this also in my serverless.ts:
scripts: {
hooks: {
'before:package:createDeploymentArtifacts': 'yarn run postbuild'
}
}
This causes the 'serverless-plugin-scripts' plugin to call my post-build yarn script and fixup the .build folder that esbuild creates. I imagine that if your build system (such as webpack or something) creates the build dir under a different name (such as lib), this process could be modified accordingly.
I will have to create a yarn script to do this for each function that is individually packaged, however this is dynamic, precludes the need to keep multiple copies of schema.prisma in source, and copies the files from the dynamically generated .prisma folder in node_modules.
Note, I am using yarn workspaces here, so the location of your node_modules folder will vary based on your repo setup.
Also, I did run into this error Please make sure your database server is running at which was remedied by making sure the proper security groups were whitelisted outbound for the lambda function, and inbound for RDS. Also make sure to check your subnet ACLs and route-tables.
We ran into the same issue recently. But our context is slightly different: the path to the schema.prisma file was /var/task/node_modules/.prisma/client/schema.prisma.
We solved this issue by using Serverless Package Configuration.
serverless.yml
service: 'your-service-name'
plugins:
- serverless-esbuild
provider:
# ...
package:
include:
- 'node_modules/.prisma/client/schema.prisma' # <-------- this line
- 'node_modules/.prisma/client/libquery_engine-rhel-*'
This way only the src folder containing the lambda functions and the node_modules folder containing these two Prisma files were packaged and uploaded to AWS.
Although the use of serverless.package.include and serverless.package.exclude is deprecated in favor of serverless.package.patterns, this was the only way to get it to work.
An option is to use webpack with the copy-webpack-plugin and change the structure of your application, put all handlers inside the handlers folder.
Folders structure:
.
├── handlers/
│ ├── hello.ts
│ └── ...
└── services/
├── hello.ts
└── ...
webpack.config.js:
/* eslint-disable #typescript-eslint/no-var-requires */
const path = require("path");
// const nodeExternals = require("webpack-node-externals");
const CopyPlugin = require("copy-webpack-plugin");
const slsw = require("serverless-webpack");
const { isLocal } = slsw.lib.webpack;
module.exports = {
target: "node",
stats: "normal",
entry: slsw.lib.entries,
// externals: [nodeExternals()],
mode: isLocal ? "development" : "production",
optimization: { concatenateModules: false },
resolve: { extensions: [".js", ".ts"] },
output: {
libraryTarget: "commonjs",
filename: "[name].js",
path: path.resolve(__dirname, ".webpack"),
},
module: {
rules: [
{
test: /\.tsx?$/,
loader: "ts-loader",
exclude: /node_modules/,
},
],
},
plugins: [
new CopyPlugin({
patterns: [
{
from: "./prisma/schema.prisma",
to: "handlers/schema.prisma",
},
{
from: "./node_modules/.prisma/client/libquery_engine-rhel-openssl-1.0.x.so.node",
to: "handlers/libquery_engine-rhel-openssl-1.0.x.so.node",
},
],
}),
],
};
If you need to run the npx prisma generate before assembling the package you can use the plugin serverless-scriptable-plugin (put before webpack):
plugins:
- serverless-scriptable-plugin
- serverless-webpack
custom:
scriptable:
hooks:
before:package:createDeploymentArtifacts: npx prisma generate
webpack:
includeModules: false
Dependences:
npm install -D webpack serverless-webpack webpack-node-externals copy-webpack-plugin serverless-scriptable-plugin
Gist
I have a monorepo and I am using yarn workspaces and lerna to manage it. I had no issues with it until now. I need to know all the changed packages since the last release.
Issue
So I run lerna changed (docs), but this is what it returns:
info cli using local version of lerna
lerna notice cli v3.16.4
lerna info Looking for changed packages since v0.3.0
lerna info No changed packages found
Similarly, lerna doesn't find any packages when running lerna list (docs):
info cli using local version of lerna
lerna notice cli v3.16.4
lerna success found 0 packages
It seems like something is broken. But I can't find any issues in my setup.
Setup
File tree
├── lerna.json
├── package.json
├── packages
│ ├── enums
│ ├── event-sourcing
│ ├── models
│ └── utils
└── services
├── consumer
├── frontend
├── gateway
└── ideas
lerna.json
{
"packages": [
"packages/*",
"services/*"
],
"version": "0.3.0",
"useWorkspaces": "true"
}
package.json
{
"name": "cents-ideas",
"version": "0.0.0",
"workspaces": [
"packages/*",
"services/*"
],
"private": true,
"devDependencies": {
"lerna": "^3.16.4",
"npm-run-all": "^4.1.5",
"rimraf": "^3.0.0",
"typescript": "^3.6.2"
}
}
The entire repository is on GitHub, if you want to take a closer look at it.
The solution is very simple. As all my packages have a "private": true in their package.json files, I need to add the --all flag.
lerna changed --all
lerna list -all
I'm getting the following errors trying to run a multi-module gradle build:
/Users/ashley/Personal/juggernaut/display/src/main/java/module-info.java:2: error: module not found: org.lwjgl.natives
requires org.lwjgl.natives;
^
/Users/ashley/Personal/juggernaut/display/src/main/java/module-info.java:3: error: module not found: org.lwjgl.glfw.natives
requires org.lwjgl.glfw.natives;
^
/Users/ashley/Personal/juggernaut/display/src/main/java/module-info.java:4: error: module not found: org.lwjgl.opengl.natives
requires org.lwjgl.opengl.natives;
^
/Users/ashley/Personal/juggernaut/display/src/main/java/module-info.java:5: error: module not found: org.lwjgl.stb.natives
requires org.lwjgl.stb.natives;
^
/Users/ashley/Personal/juggernaut/display/src/main/java/module-info.java:6: error: module not found: org.lwjgl.assimp.natives
requires org.lwjgl.assimp.natives;
My project structure is as follows:
├── build.gradle
├── display
│ ├── build.gradle
│ └── src
│ └── main
│ ├── java
│ │ ├── module-info.java
├── engine
│ ├── build.gradle
│ └── src
│ └── main
│ ├── java
│ │ ├── module-info.java
└── settings.gradle
the module-info.java for display requires the necessary LWJGL3 modules:
module uk.ashleybye.juggernaut.display {
requires org.lwjgl.natives;
requires org.lwjgl.glfw.natives;
requires org.lwjgl.opengl.natives;
requires org.lwjgl.stb.natives;
requires org.lwjgl.assimp.natives;
exports uk.ashleybye.juggernaut.display;
}
The relevant dependencies are included in the build.gradle:
project.ext.lwjglVersion = "3.2.2"
project.ext.lwjglNatives = "natives-macos"
dependencies {
implementation "org.lwjgl:lwjgl:$lwjglVersion"
implementation "org.lwjgl:lwjgl-assimp:$lwjglVersion"
implementation "org.lwjgl:lwjgl-glfw:$lwjglVersion"
implementation "org.lwjgl:lwjgl-openal:$lwjglVersion"
implementation "org.lwjgl:lwjgl-opengl:$lwjglVersion"
implementation "org.lwjgl:lwjgl-stb:$lwjglVersion"
runtimeOnly "org.lwjgl:lwjgl:$lwjglVersion:$lwjglNatives"
runtimeOnly "org.lwjgl:lwjgl-assimp:$lwjglVersion:$lwjglNatives"
runtimeOnly "org.lwjgl:lwjgl-glfw:$lwjglVersion:$lwjglNatives"
runtimeOnly "org.lwjgl:lwjgl-openal:$lwjglVersion:$lwjglNatives"
runtimeOnly "org.lwjgl:lwjgl-opengl:$lwjglVersion:$lwjglNatives"
runtimeOnly "org.lwjgl:lwjgl-stb:$lwjglVersion:$lwjglNatives"
}
Engine requires uk.ashleybye.juggernaut.display; and has an implementation dependency: implementation project(":display").
My version info:
./gradlew --version Sat 17 Aug 09:31:44 2019
------------------------------------------------------------
Gradle 5.5.1
------------------------------------------------------------
Build time: 2019-07-10 20:38:12 UTC
Revision: 3245f748c7061472da4dc184991919810f7935a5
Kotlin: 1.3.31
Groovy: 2.5.4
Ant: Apache Ant(TM) version 1.9.14 compiled on March 12 2019
JVM: 12 (Oracle Corporation 12+33)
OS: Mac OS X 10.14.6 x86_64
Why can gradle not find the modules?
Your module-info.class doesn't need to include any natives at all.
Remove all the .nativess from each requires.
Also, you can compact your build.gradle:
["", "-assimp", "-glfw", "-openal", "-opengl", "-stb"].each {
String base = "org.lwjgl:lwjgl$it:$lwjgl_version"
implementation base
runtimeOnly = "$base:natives-$lwjgl_natives"
}
You may refer to this project of ours for reference, it's a multi-module project, which relies heavily on lwjgl and uses JPMS too.
I am using Yocto 2.3 to build my device image.
My image includes packagegroup-core-boot that, in turn, includes busybox.
IMAGE_INSTALL = "\
....
packagegroup-core-boot \
Busybox is configured to include syslogd:
CONFIG_SYSLOGD=y
CONFIG_FEATURE_ROTATE_LOGFILE=y
CONFIG_FEATURE_REMOTE_LOG=y
CONFIG_FEATURE_SYSLOGD_DUP=y
CONFIG_FEATURE_SYSLOGD_CFG=y
CONFIG_FEATURE_SYSLOGD_READ_BUFFER_SIZE=256
CONFIG_FEATURE_IPC_SYSLOG=y
CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=64
CONFIG_LOGREAD=y
CONFIG_FEATURE_LOGREAD_REDUCED_LOCKING=y
CONFIG_FEATURE_KMSG_SYSLOG=y
CONFIG_KLOGD=y
It is built and installed correctly.
Relevant syslog files do appear in busybox image directory:
tmp/work/armv5e-poky-linux-gnueabi/busybox/1.24.1-r0/image$ tree etc/
etc/
├── default
├── init.d
│ └── syslog.busybox
├── syslog.conf.busybox
├── syslog-startup.conf.busybox
These files don't appear in my main image rootfs, though. Only the syslogd command is included. See output on target device:
# ls -l $( which syslogd )
lrwxrwxrwx 1 root root 19 Jan 10 12:31 /sbin/syslogd -> /bin/busybox.nosuid
What can be happening to make this files not to be included in the final image?
Additional question:
As shown in the tree output, the init script for syslog is included in busybox but no link to /etc/rc?.d/ is created.
I understand that is should be created by a do_install() hook, shouldn't?
Thanks in advance.
EDIT
Contents of packages-split, as #Anders says, seems ok:
poky/build-idprint/tmp/work/armv5e-poky-linux-gnueabi/busybox/1.24.1-r0$ tree packages-split/busybox-syslog/
packages-split/busybox-syslog/
└── etc
├── init.d
│ ├── syslog
│ └── syslog.busybox
├── syslog.conf
├── syslog.conf.busybox
├── syslog-startup.conf
└── syslog-startup.conf.busybox
I just can't figure out what is stripping this files out of the final image.
Check tmp/work/armv5e-poky-linux-gnueabi/busybox/1.24.1-r0/packages-split. This is where all files are split into the packages that will be generated. If you search that directory, you'll find eg syslog.conf in the busybox-syslog package.
Thus, in order to get those files into your image, you'll need to add busybox-syslog to your image. I.e. IMAGE_INSTALL += "busybox-syslog".
I'm building a react application with SASS for the styles and serving it with webpack-dev-server, a couple of components and some extract of the structure look like that:
├── App.js
├── components
│ ├── Layout.jsx
│ ├── Layout.scss
│ ├── header
│ ├── pages
│ └── footer
└── css
├── _vendors.scss
├── _variables.scss
├── _mixins.scss
└── utils.scss
Inside main.scss, I have all the #imports, also inside the vendors I have the imports in node_modules dependencies, etc...
Per each component in react have his own scss file.
But, I want the main.scss loaded in the Layout.jsx and use the mixins or imports or placeholder classes inside the scss file per component.
I was trying playing with css-loader and style-loader with includePath's and nothing successful.
It's this possible? It's related on the -loaders of webpack?
ps: Isn't viable have an .html file, this is a library of components.
A good way I found to keep track of styles for various react components is to have a style file for each corresponding component. So Header.react.jsx would have a corresponding Header.sass. Then you just include the style file on top of the react component, like so (using the ES6 syntax):
import './Header.sass';
For this to work with webpack, you need the appropriate loaders:
loaders: [
{
test: /\.sass/,
loader: 'style!css!autoprefixer!sass'
}
]
This is nice for development, because the styles are loaded in javascript and get updated automatically on any change if you're using the hot loader, but in production, you'd want to separate out the css file with all the styles. You can do this with a separate webpack config file for production, which can look something like this:
plugins: [
new ExtractTextPlugin('bundle.css', { allChunks: false })
],
loaders: [
{
test: /\.scss/,
loader: ExtractTextPlugin.extract(
'style-loader',
'css!autoprefixer!sass'
)
}
]
When doing things this way, it's good to include a sass config file and a mixin file, for the variables or mixins you use globally (with the regular sass #import). It's important not to define any actual styles in the files you import, since the styles would be included into all files you include it in, resulting in loads of duplication.
This is good for component libraries (as you put it), since webpack will only include the react components you use in your application, and in turn, it will only pull in the sass files you use.
I found this approach very handy, and used it in a smaller application here. Take a look if I haven't explained it well enough.