I'm using a Google library called DialogFlow. And in the last 6 or 7 days all the lambda functions that import this library, started to give initialization error.
I noticed that it does pretty much the same time the serverless framework has been upgraded from version 1.31.0 to 1.32.0. In my serverless.yml file I put: frameworkVersion: ">=1.0.0 <2.0.0"
If I compile a simple code like this:
import dialogflow
def hi(event, context):
return {
"statusCode": 200,
"body": "ahhh hiiii"
}
The error generated in lambda is as follows:
START RequestId: 907fe23d-c2b1-11e8-b745-27859211eefc Version: $LATEST
module initialization error: The 'google-api-core' distribution was
not found and is required by the application
END RequestId: 907fe23d-c2b1-11e8-b745-27859211eefc REPORT RequestId:
907fe23d-c2b1-11e8-b745-27859211eefc Duration: 47.02 ms Billed
Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 32 MB module
initialization error The 'google-api-core' distribution was not found
and is required by the application
The problem is libraries that use low-level language (usually in C). And when serverless generates packets, those languages are not sent.
The solution: Enable docker packaging, through the serverless-python-requirements plugin.
custom:
pythonRequirements:
dockerizePip: true
Related
I started studying Quarkus.
https://quarkus.io/guides/amazon-lambda
There was a problem while I was handling the tutorial on the site above.
If make a native work locally using AWS SAM or AWS Lambda the same error occurs during invoke.
(If it is not native, it works normally.)
MacBook M1 is in use, both graalvm and Java are arm64.
**
ahahah#bcd0745cd453 ahahaha % sh target/manage.sh native invoke
**
Invoking function
++ aws lambda invoke response.txt --cli-binary-format raw-in-base64-out --function-name HoiriasNative --payload file://payload.json --log-type Tail --query LogResult --output text
++ base64 --decode
START RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543 Version: $LATEST
RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543
Error: fork/exec /var/task/bootstrap: exec format error
Runtime.InvalidEntrypoint
END RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543
REPORT RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543 Duration: 2.94 ms Billed Duration: 3 ms Memory Size: 256 MB Max Memory Used: 3 MB
{"errorMessage":"RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543
Error: fork/exec /var/task/bootstrap: exec format error","errorType":"Runtime.InvalidEntrypoint"}%
It's my bash_profile.
GRAALVM_HOME=/Library/JAVA/JavaVirtualMachines/graalvm-ce-java17-22.3.0/Contents/Home
JAVA_HOME=/Library/JAVA/JavaVirtualMachines/graalvm-ce-java17-22.3.0/Contents/Home
PATH=$PATH:$JAVA_HOME
export JAVA_HOME
export PATH=${GRAALVM_HOME}/bin:$PATH
When I uploaded the native to Lambda, it was distributed as x86 so could this be a problem?
What should I do?
please Help me.
I tried to build it in another way, and the image was created normally.
quarkus build --native --no-tests -Dquarkus.native.container-build=true
I Tried this yesterday as well.
It's currently not possible to build native images on Mac M1 (arm64) to run on Amazon Lambda. Probably with some virtual machine it might be possible.
Even on linux x86_64 you should build with:
quarkus build --native -Dquarkus.native.container-build=true to maximize compatibility.
For some reason I can not create a native image using the quarkus-rest-client-jackson extension in Quarkus version 2.9.2. (I haven't tested out in a different one)
If I package it in a non-native image, everything works as expected and the image is creates successfully.
Thank you in advance for any detail.
GraalVM Native Image: Generating 'was-fits-scanner-1.0.0-SNAPSHOT-runner'...
========================================================================================================================
[1/7] Initializing... (17.5s # 0.13GB)
Version info: 'GraalVM 22.0.0.2 Java 11 CE'
3 user-provided feature(s)
- io.quarkus.runner.AutoFeature
- io.quarkus.runtime.graal.DisableLoggingAutoFeature
- io.quarkus.runtime.graal.ResourcesFeature
[2/7] Performing analysis... [] (14.1s # 0.74GB)
To see how the classes got initialized, use --trace-class-initialization=org.apache.http.conn.ssl.SSLConnectionSocketFactory
3,941 (85.67%) of 4,600 classes reachable
4,948 (64.71%) of 7,646 fields reachable
16,245 (80.04%) of 20,297 methods reachable
123 classes, 10 fields, and 380 methods registered for reflection
Error: Classes that should be initialized at run time got initialized during image building:
org.apache.http.conn.ssl.SSLConnectionSocketFactory the class was requested to be initialized at run time (from feature io.quarkus.runner.AutoFeature.beforeAnalysis with 'SSLConnectionSocketFactory.class'). To see why org.apache.http.conn.ssl.SSLConnectionSocketFactory got initialized use --trace-class-initialization=org.apache.http.conn.ssl.SSLConnectionSocketFactory
com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
org.apache.http.conn.ssl.SSLConnectionSocketFactory the class was requested to be initialized at run time (from feature io.quarkus.runner.AutoFeature.beforeAnalysis with 'SSLConnectionSocketFactory.class'). To see why org.apache.http.conn.ssl.SSLConnectionSocketFactory got initialized use --trace-class-initialization=org.apache.http.conn.ssl.SSLConnectionSocketFactory
at com.oracle.svm.core.util.UserError.abort(UserError.java:73)
at com.oracle.svm.hosted.classinitialization.ConfigurableClassInitialization.checkDelayedInitialization(ConfigurableClassInitialization.java:555)
at com.oracle.svm.hosted.classinitialization.ClassInitializationFeature.duringAnalysis(ClassInitializationFeature.java:167)
at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$10(NativeImageGenerator.java:704)
at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:74)
at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$11(NativeImageGenerator.java:704)
at com.oracle.graal.pointsto.PointsToAnalysis.runAnalysis(PointsToAnalysis.java:755)
at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:702)
at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:537)
at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:494)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:426)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:587)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:126)
at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:617)
------------------------------------------------------------------------------------------------------------------------
0.6s (1.9% of total time) in 16 GCs | Peak RSS: 1.17GB | CPU load: 2.69
------------------------------------------------------------------------------------------------------------------------
I am deploying a Python 3.7 Lambda function via Chalice. Because the code with its environment requirements, is larger than 50 MB limit, I am using the "automatic_layer" feature of Chalice to generate the layer with the requirements, which is awswrangler.
Because the generated layer is > 50 MB, I am uploading the generated managed-layer-...-python3.7.zip manually to s3 and create a Lambda layer. Then I re-deploy with chalice, removing the automatic_layer option and setting the layers to the generated ARN of the layer I manually created.
The function deployed this way worked OK for a couple of times, then started failing occasionally with "Segmentation Fault". The error rate increased shortly and now it is failing 100%.
Traceback:
> OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
> START RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc Version: $LATEST
> OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
> END RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc
> REPORT RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc Duration: 7165.04 ms Billed Duration: 7166 ms Memory Size: 128 MB Max Memory Used: 41 MB
> RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc Error: Runtime exited with error: signal: segmentation fault (core dumped)
> Runtime.ExitError
As awswrangler itself requires boto3 & botocore, and they are already in the Lambda environment, I suspected that there might be a conflict of different versions of boto. I tried the same flow by explicitly including boto3 and botocore in the requirements but I am still receiving the same segmentation fault error.
Any help is much appreciated.
You could use AWS X-Ray to get more information on the problem : https://docs.aws.amazon.com/lambda/latest/dg/python-tracing.html
Moreover you might analyze the core dump generated executing your lambda function on a bash shell:
ulimit -c unlimited
cd /tmp
ececute your python ...
You should find a file named /tmp/core..... that you should analyze with gdb after download. The command "man core" could help you.
On my laptop with Fedora 30 I have Performance Co-Pilot (PCP) daemons installed and running, and Prometheus installed from the package golang-github-prometheus-prometheus-1.8.0-4.fc30.x86_64. In PCP collector's config I specified the following metric namespaces:
# Performance Metrics Domain Specifications
#
# This file is automatically generated during the build
# Name Id IPC IPC Params File/Cmd
#root 1 pipe binary /var/lib/pcp/pmdas/root/pmdaroot
#pmcd 2 dso pmcd_init /var/lib/pcp/pmdas/pmcd/pmda_pmcd.so
proc 3 pipe binary /var/lib/pcp/pmdas/proc/pmdaproc -d 3
#xfs 11 pipe binary /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux 60 pipe binary /var/lib/pcp/pmdas/linux/pmdalinux
#pmproxy 4 dso pmproxy_init /var/lib/pcp/pmdas/mmv/pmda_mmv.so
mmv 70 dso mmv_init /var/lib/pcp/pmdas/mmv/pmda_mmv.so
#jbd2 122 dso jbd2_init /var/lib/pcp/pmdas/jbd2/pmda_jbd2.so
#kvm 95 pipe binary /var/lib/pcp/pmdas/kvm/pmdakvm -d 95
[access]
disallow ".*" : store;
disallow ":*" : store;
allow "local:*" : all;
When I visit the URL localhost:44323/metrics the output is very rich and covers many namespaces, ie. mem, network, kernel, filesys, hotproc etc., but when I scrape it with Prometheus where the job is defined as:
scrape_configs:
- job_name: 'pcp'
scrape_interval: 10s
sample_limit: 0
static_configs:
- targets: ['127.0.0.1:44323']
I see the target status UP, but in the console only these two metric namespaces are available for querying: hinv and mem. I tried to copy other metric names from /metrics page, but queries result in the error: 'No datapoints found.' Initially I thought the problem could have been due to a limit on the number of samples per target or the sampling interval too small (I originally set it to 1s), but hinv and mem are not next to each other and there are other metrics (ie. filesys, kernel) in between them that are omitted. What could be the reason for that?
I have not found the exact cause of the problem, but it must have been version-specific, because after I downloaded and launched the latest version 2.19 the problem was gone and with exactly the same config Prometheus was reading all metrics from the target.
Adding another answer, because I have just seen this issue again in another environment where Prometheus v.2.19 was pulling metrics via PMAPI from PCP v.5 on CentOS 7 servers. In Prometheus config file the scrape was configured as a single job with multiple metric domains, ie.:
- job_name: 'pcp'
file_sd_configs:
- files: [...]
metric_path: '/metrics'
params:
target: ['kernel', 'mem', 'disk', 'network', 'mounts', 'lustre', 'infiniband']
When there was a problem with one metric domain, usually lustre or infiniband due to the lack of corresponding hardware on the host, only kernel metrics were collected and no other.
The issue was fixed by splitting the scrape job into multiple jobs with only one target each, ie.:
- job_name: 'pcp-kernel'
file_sd_configs:
- files: [...]
metric_path: '/metrics'
params:
target: ['kernel']
- job_name: 'pcp-mem'
file_sd_configs:
- files: [...]
metric_path: '/metrics'
params:
target: ['mem']
[...]
This way metrics from the core domains were always scraped successfully despite of one or all of the extra ones failing. Such setup seems to be more robust, however it makes the target status view busier, because there are more scrape jobs.
Context
I'm developing an AWS Lambda function using Go and one of my dependencies is gopkg.in/h2non/bimg.v1 which has a dependency: libvips 7.42+ or 8+ (8.4+ recommended).
Problem
The problem is that in my local machine the lambda handler is working, but when I deploy it this error occures:
START RequestId: b4becbd1-3fca-4aed-9574-8df0e3d13c9e Version: $LATEST
/var/task/main: error while loading shared libraries: libvips.so.42: cannot open shared object file: No such file or directory
END RequestId: b4becbd1-3fca-4aed-9574-8df0e3d13c9e
REPORT RequestId: b4becbd1-3fca-4aed-9574-8df0e3d13c9e Duration: 42.36 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 12 MB
RequestId: b4becbd1-3fca-4aed-9574-8df0e3d13c9e Process exited before completing request
My build command is:
GOOS=linux GOARCH=amd64 go build -o main main.go
What I tried
I tried to build it with c-shared option enabled:
GOOS=linux GOARCH=amd64 go build -buildmode=c-shared -o main main.go
But got an error too, a different one tho;
START RequestId: 9b90df21-1025-463b-89b1-1a4ee31f295c Version: $LATEST
fork/exec /var/task/main: permission denied: PathError
null
END RequestId: 9b90df21-1025-463b-89b1-1a4ee31f295c
REPORT RequestId: 9b90df21-1025-463b-89b1-1a4ee31f295c Duration: 0.77 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 30 MB Init Duration: 1.84 ms
I have two options (?):
Rewrite with a fully Go library
Generate a library with the libvips library packed into the Go binary.
There is another option - put all .so file into zip archive with your binary and upload zip as lambda. So, your achive content should looks like that
╰─ unzip -l function.zip
Archive: function.zip
Length Date Time Name
--------- ---------- ----- ----
6764336 10-08-2020 01:01 imgconvert
284008 06-19-2020 09:16 libexif.so.12
276072 08-22-2019 08:14 libjpeg.so.62
155824 12-10-2015 02:17 libpng12.so.0
468376 10-01-2019 03:37 libtiff.so.5
12261600 10-08-2020 00:48 libvips.so.42
3579016 10-08-2020 00:45 libwebp.so.7
85328 10-08-2020 00:45 libwebpdemux.so.2
205696 10-08-2020 00:45 libwebpmux.so.3