AWS Lambda Chalice Layers Segmentation Fault - aws-lambda

I am deploying a Python 3.7 Lambda function via Chalice. Because the code with its environment requirements, is larger than 50 MB limit, I am using the "automatic_layer" feature of Chalice to generate the layer with the requirements, which is awswrangler.
Because the generated layer is > 50 MB, I am uploading the generated managed-layer-...-python3.7.zip manually to s3 and create a Lambda layer. Then I re-deploy with chalice, removing the automatic_layer option and setting the layers to the generated ARN of the layer I manually created.
The function deployed this way worked OK for a couple of times, then started failing occasionally with "Segmentation Fault". The error rate increased shortly and now it is failing 100%.
Traceback:
> OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
> START RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc Version: $LATEST
> OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
> END RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc
> REPORT RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc Duration: 7165.04 ms Billed Duration: 7166 ms Memory Size: 128 MB Max Memory Used: 41 MB
> RequestId: 3b98bd4b-6cda-4d21-8090-1a49b17c06fc Error: Runtime exited with error: signal: segmentation fault (core dumped)
> Runtime.ExitError
As awswrangler itself requires boto3 & botocore, and they are already in the Lambda environment, I suspected that there might be a conflict of different versions of boto. I tried the same flow by explicitly including boto3 and botocore in the requirements but I am still receiving the same segmentation fault error.
Any help is much appreciated.

You could use AWS X-Ray to get more information on the problem : https://docs.aws.amazon.com/lambda/latest/dg/python-tracing.html
Moreover you might analyze the core dump generated executing your lambda function on a bash shell:
ulimit -c unlimited
cd /tmp
ececute your python ...
You should find a file named /tmp/core..... that you should analyze with gdb after download. The command "man core" could help you.

Related

Quarkus native run exec user process caused: exec format error

I started studying Quarkus.
https://quarkus.io/guides/amazon-lambda
There was a problem while I was handling the tutorial on the site above.
If make a native work locally using AWS SAM or AWS Lambda the same error occurs during invoke.
(If it is not native, it works normally.)
MacBook M1 is in use, both graalvm and Java are arm64.
**
ahahah#bcd0745cd453 ahahaha % sh target/manage.sh native invoke
**
Invoking function
++ aws lambda invoke response.txt --cli-binary-format raw-in-base64-out --function-name HoiriasNative --payload file://payload.json --log-type Tail --query LogResult --output text
++ base64 --decode
START RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543 Version: $LATEST
RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543
Error: fork/exec /var/task/bootstrap: exec format error
Runtime.InvalidEntrypoint
END RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543
REPORT RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543 Duration: 2.94 ms Billed Duration: 3 ms Memory Size: 256 MB Max Memory Used: 3 MB
{"errorMessage":"RequestId: a97a6513-afbf-4925-a0f0-1acab8dec543
Error: fork/exec /var/task/bootstrap: exec format error","errorType":"Runtime.InvalidEntrypoint"}%
It's my bash_profile.
GRAALVM_HOME=/Library/JAVA/JavaVirtualMachines/graalvm-ce-java17-22.3.0/Contents/Home
JAVA_HOME=/Library/JAVA/JavaVirtualMachines/graalvm-ce-java17-22.3.0/Contents/Home
PATH=$PATH:$JAVA_HOME
export JAVA_HOME
export PATH=${GRAALVM_HOME}/bin:$PATH
When I uploaded the native to Lambda, it was distributed as x86 so could this be a problem?
What should I do?
please Help me.
I tried to build it in another way, and the image was created normally.
quarkus build --native --no-tests -Dquarkus.native.container-build=true
I Tried this yesterday as well.
It's currently not possible to build native images on Mac M1 (arm64) to run on Amazon Lambda. Probably with some virtual machine it might be possible.
Even on linux x86_64 you should build with:
quarkus build --native -Dquarkus.native.container-build=true to maximize compatibility.

Sometimes when I run `npx hardhat compile` I get this error: FATAL ERROR: NewNativeModule Allocation failed - process out of memor

Sometimes when I run this command npx hardhat compile on my windows cli I get the error below:
Compiling 72 files with 0.7.0
contracts/libraries/ERC20.sol: Warning: SPDX license identifier not provided in source file. Before publishing, consider adding a comment containing "SPDX-License-Identifier: <SPDX-License>" to each source file. Use "SPDX-License-Identifier: UNLICENSED" for non-open-source code.
Please see https://spdx.org for more information.
contracts/libraries/ERC1155/EnumerableSet.sol:158:5: Warning: Variable is shadowed in inline assembly by an instruction of the same name
function add(Bytes32Set storage set, bytes32 value) internal returns (bool) {
^ (Relevant source part starts here and spans across multiple lines).
contracts/libraries/ERC1155/EnumerableSet.sol:224:5: Warning: Variable is shadowed in inline assembly by an instruction of the same name
function add(AddressSet storage set, address value) internal returns (bool) {
^ (Relevant source part starts here and spans across multiple lines).
Compiling 1 file with 0.8.0
<--- Last few GCs --->
[8432:042B0460] 263058 ms: Mark-sweep (reduce) 349.8 (356.3) -> 248.2 (262.4) MB, 434.4 / 0.2 ms (+ 70.9 ms in 3 steps since start of marking, biggest step 69.6 ms, walltime since start of marking 800 ms) (average mu = 0.989, current mu = 0.990) memory[8432:042B0460] 263627
ms: Mark-sweep (reduce) 248.2 (259.4) -> 248.2 (252.1) MB, 555.5 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 556 ms) (average mu = 0.969, current
mu = 0.023) memory p
<--- JS stacktrace --->
FATAL ERROR: NewNativeModule Allocation failed - process out of memory`
After some time the error just kind of goes away.
It goes away probably after I've restarted my system or created a new Hardhat project and imported the code there.
But this is happening too often, what could be the cause?
I've done quite some research and some answers suggested it might be a problem with Node and the application's memory allocation, but I don't know how I would apply the solutions to a Hardhat project.
Here is a link to one possible solution: https://medium.com/#vuongtran/how-to-solve-process-out-of-memory-in-node-js-5f0de8f8464c
OS: WINDOWS 10
CLI: WINDOWS CMD

Deploy does not generate all packages

I'm using a Google library called DialogFlow. And in the last 6 or 7 days all the lambda functions that import this library, started to give initialization error.
I noticed that it does pretty much the same time the serverless framework has been upgraded from version 1.31.0 to 1.32.0. In my serverless.yml file I put: frameworkVersion: ">=1.0.0 <2.0.0"
If I compile a simple code like this:
import dialogflow
def hi(event, context):
return {
"statusCode": 200,
"body": "ahhh hiiii"
}
The error generated in lambda is as follows:
START RequestId: 907fe23d-c2b1-11e8-b745-27859211eefc Version: $LATEST
module initialization error: The 'google-api-core' distribution was
not found and is required by the application
END RequestId: 907fe23d-c2b1-11e8-b745-27859211eefc REPORT RequestId:
907fe23d-c2b1-11e8-b745-27859211eefc Duration: 47.02 ms Billed
Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 32 MB module
initialization error The 'google-api-core' distribution was not found
and is required by the application
The problem is libraries that use low-level language (usually in C). And when serverless generates packets, those languages ​​are not sent.
The solution: Enable docker packaging, through the serverless-python-requirements plugin.
custom:
pythonRequirements:
dockerizePip: true

Error running Fast RCNN on CPU: Cannot use GPU

I am trying to run Fast RCNN on CPU. I am working with the code I cloned as follows:
git clone --recursive https://github.com/rbgirshick/py-faster-rcnn.git
When running py-faster-rcnn/tools/demo.py, I am getting following error:
Warning: Logging before InitGoogleLogging() is written to STDERR
F0625 01:37:25.908700 24397 common.cpp:66] Cannot use GPU in CPU-only Caffe: check mode.
* Check failure stack trace: *
Aborted
I have modified the Makefile to use CPU. Have set CPU_ONLY:= 1.
What I am missing here or doing wrong?
Are you passing --cpu flag to command line itself i.e. are you running demo.py as follows:
py-faster-rcnn/tools/demo.py --cpu?
If not so, run it with the flag. This error should be resolved with this.

Riak eating 100% CPU on OSX install

This question is related to:
Riak node not working, but using 100% cpu
but since the poster seems to have left I'm posting my case here.
Last night I installed erlang(R15B01) from source, using the config options from the Riak website:
http://docs.basho.com/riak/1.2.1/tutorials/installation/Installing-Erlang/#Installing-on-Mac-OS-X
and Riak(1.4.1) on my 2013 MacBook Pro (2.8GHz i7, 16GB ram, OSX 10.8.3). I did not change the ulimit, as I assumed it would be fine for a vanilla run.
Installation went fine; warnings but no errors, and I was able to run the toy examples no problem.
However the empty instance quickly ate through all 4 cores and my machine started whining and overheating.
Looking in the logs I see the following error repeated a jillion times:
2013-10-11 09:04:04.266 [error] CRASH REPORT ¥
Process with 0 neighbours exited with reason: ¥
call to undefined function eleveldb:o
also tons of crash reports:
2013-10-11 09:14:47 =CRASH REPORT====
crasher:
initial call: riak_kv_index_hashtree:init/1
pid:
registered_name: []
exception exit: {{undef,[{eleveldb,open,
["./data/anti_entropy/479555224749202520035584085735030365824602865664",
[{create_if_missing,true},{max_open_files,20},{write_buffer_size,12886952}]],[]},
{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,
[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,
[{file,"src/riak_kv_index_hashtree.erl"},{line,421}]},{lists,foldl,3,[{file,"lists.erl"},
{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},
{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},
{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]}]},
[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]}
ancestors: [,riak_core_vnode_sup,riak_core_sup,]
messages: []
links: []
dictionary: []
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 492
neighbours:
erlang.log says
=====
===== LOGGING STARTED Fri Oct 11 09:04:01 CEST 2013
=====
Node 'riak#127.0.0.1' not responding to pings.
config is OK
!!!!
!!!! WARNING: ulimit -n is 2560; 4096 is the recommended minimum.
!!!!
Exec: /tmp/riak-1.4.1/rel/riak/bin/../erts-5.9.1/bin/erlexec
-boot /tmp/riak-1.4.1/rel/riak/bin/../releases/1.4.1/riak
-config /tmp/riak-1.4.1/rel/riak/bin/../etc/app.config
-pa /tmp/riak-1.4.1/rel/riak/bin/../lib/basho-patches
-args_file /tmp/riak-1.4.1/rel/riak/bin/../etc/vm.args -- console
Root: /tmp/riak-1.4.1/rel/riak/bin/..
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:8:8] [async-threads:64]
[kernel-poll:true]
Eshell V5.9.1 (abort with ^G)
(riak#127.0.0.1)1>
After less than 10m there are already 144MB of logging files with variations of the above.
I had the same problem by building riak 1.4.6 from source.
I changed in the file etc/app.config the line to
{anti_entropy, {off, []}},
Leveldb is used by AAE. See the config parameter anti_entropy_leveldb_opts.
Use process of elimination:
It's hard to say without more information. Is the 200% being used by
beam.smp? Do you see anything in console.log, error.log or crash.log
that would indicate something odd is happening? Are there clients
communicating with the cluster at the time? If so what client/protocol are
they using and what types of operations are being performed (e.g.
get/put/map reduce/etc)?
References
Riak consuming too much CPU
Interesting sawtooth increasing CPU usage on lightly-used Riak
Inspecting a Node
Riak Performance Tuning
Open Files Limit
Configuration Files

Resources