I've written a small lambda Python function that makes use of pdf2image to convert a PDF file's pages into separate JPG files. This library is a wrapper around poppler-utils, in particular pdftoppm. Works fine on my Ubuntu system, but of course AWS Lambda is different.
So I went to look for a version of poppler that I could compile on an EC2 instance for AWS Lambda and found the slightly dated Poppler-build ansible playbook. With some tweaks, I got it to work, however the Poppler version this playbook builds does not include the essential -jpeg flag which turns ppms data straight into jpegs.
I tried compiling newer versions of Poppler but they all seem to involve libraries and C++ versions that I'd have to install separately and I worry that this will break AWS Lambda support.
Poppler 0.58 always balks at
make[3]: Entering directory
`/home/ec2-user/poppler3/poppler-0.58.0/poppler' CXX
libpoppler_la-SignatureInfo.lo In file included from
SignatureInfo.cc:22:0: /usr/include/nss3/hasht.h:48:29: error:
‘PRBool’ has not been declared
void (*destroy)(void *, PRBool);
Poppler 0.69 (latest) on the other hand gets antsy about:
/home/ec2-user/poppler2/poppler-0.69.0/poppler/Annot.cc: In
constructor ‘DefaultAppearance::DefaultAppearance(GooString*)’:
/home/ec2-user/poppler2/poppler-0.69.0/poppler/Annot.cc:826:23: error:
‘make_unique’ is not a member of ‘std’
fontColor = std::make_unique(gatof(( (GooString *)daToks->get(i-1) )->getCString()));
^ /home/ec2-user/poppler2/poppler-0.69.0/poppler/Annot.cc:826:50: error:
expected primary-expression before ‘>’ token
fontColor = std::make_unique(gatof(( (GooString *)daToks->get(i-1) )->getCString()));
This endless trying to compile libraries and configuring with various flags and compilers etc has taken me over two days now and I'm ready to give up. Is there a chance someone can tell me what I'm doing wrong, or suggest another efficient way to turn PDF pages into JPEG files on AWS Lambda?
Related
I'm trying to deploy an app built in dash to heroku. This app uses simulation code written in c++, which is imported as a python module using pybind 11. When I upload the compiled code I get the following error message when looking at the heroku logs
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /app/simulation_module_name.so)
I presume that this means that code compiled with pybind11 on my machine is not compatible with the machines I am trying to deploy to.
My next attempt was the build the module directly on the heroku servers using setup.py and cmake, but this requires functions from the boost libraries, which are >1GB and so I can't upload them over git. I also tried uploading only the relevant boost header files, with the relevant cmake, such that
set(include_dir "lib")
set(source_dir "src")
# Boost
list(APPEND include_dirs ${include_dir}/boost)
set(header_files
${include_dir}/boost/math/tools/minima.hpp
${include_dir}/boost/math/constants/constants.hpp
${include_dir}/boost/math/tools/roots.hpp
${include_dir}/boost/math/tools/tuple.hpp
)
# Pybind11
add_subdirectory(${include_dir}/pybind11)
include_directories(${source_dir} ${include_dirs})
pybind11_add_module(simulation_module_name ${header_files} "${source_dir}/simulation_module.cpp")
and
#include "minima.hpp"
#include "constants.hpp"
#include "roots.hpp"
But I received the error:
fatal error: minima.hpp: No such file or directory
remote: #include "minima.hpp"
remote: ^~~~~~~~~~~~
Are any of these three options possible?
Get the heroku dynos to run the pybind11 module I compiled on my machine
Upload the installed boost library (over the size limit) and compile on the heroku server
Use the header functions and cmake and compile on the heroku server (as I have tried but failed to do here)
Get the heroku dynos to run the pybind11 module I compiled on my machine
To do this, you will need to compile the shared library on a linux version with (at least) the same or older version of GLIBC. One possibility is to compile your binaries in a docker image matching or predating whatever linux distro is used by Heroku. Alternatively, I would also suggest looking at the manylinux10 or manylinux14 images here, which are designed specifically for solving this cross-distro build/sharing problem and include many of the necessary build tools.
I want to take the GCC Compiler that is on my machine and all its dependencies and zip them up in a deployment package that I can send off to AWS Lambda (That way I can use a Lambda to compile C code). Is there an easy way to package the whole thing in one go so I can deploy and use it from AWS Lambda?
This is what I have right now
However when I invoke the function I get
"gcc: error trying to exec 'cc1': execvp: No such file or directory\n"
as the response. Currently the way I compile gcc and those dependencies you see on the left panel was by spinning up a Amazon Linux docker container, installing gcc, and then zipping up gcc and its dependencies I found with the ldd command.
AWS Lambda runtime is described here. Basically, it's Amazon Linux. If I were you, I would try to grab the specified AMI and create a EC2 instance with it. Or just create an Amazon Linux 2 EC2 instance. Then I would log in to that instance and compile the binaries you need. Finally, I would export them in a ZIP file and ship with Lambda. This way chances are high that the binaries would work on Lambda.
Great question!
As you say, just packaging the binary doesn't help because you're missing the shared objects .so files, or some other dependencies. You can find your dependencies by running something like ldd, and this question helps. Projects like yumda, try to simplify this, and definitely worth a shot.
But under the hood, Lambda uses AmazonLinux and there's really no reason it can't be done. High level points we need are:
Build the binary in a AmazonLinux container
Determine the binary, and it's dependencies
Copy the binary and dependencies out of the container into a lambci container
Test out the lambci container (you'd typically need to set some env vars for this to work, e.g. $LD_LIBARY_PATH.
Once it runs, package that as a zip, and load it into your lambda, remembering to set the right env vars
As an option, I'd package gcc as a layer, so you can share it.
When i searched around, it looks like someone has done exactly this here. Hopefully it's exactly what you're looking for.
Configure, build and install gcc into a specific directory, specified by --prefix option to configure.
After installing, change gcc spec file, so that it hardcodes -rpath into executables and shared libraries, so that you do not need to tinker with LD_LIBARY_PATH (which is a wrong solution most of the time) to make the executables find the right libstdc++.so, libgcc_s.so and its friends.
rsync the directory onto another machine into the same place in filesystem.
Or archive the install directory and unpack it on your target machine.
However, the target must should have the same libc and system libraries gcc was built with, otherwise that may not work as intended.
Alternatively, build locally and deploy your executables with all their dependencies using Exodus.
I'm trying to install frama-c on my distro of manjaro 18.1.5, but no matter what I try, I always have an error.
First I tried to install via the AUR, and it seems to work, but when i try to open a file from the gui, it fails and says something like "invalid user input" even though I use files that works on a known good install.
Edit: here is the error output for this file:
[kernel] Parsing max.c (with preprocessing)
[kernel:annot-error] max.c:2: Warning:
unbound logic variable INT_MIN. Ignoring logic specification of function max
[kernel] User Error: warning annot-error treated as fatal error.
[kernel] User Error: stopping on file "max.c" that has errors. Add '-kernel-msg-key pp'
for preprocessing command.
Then I tried to install it using opam, so I first try to install dependencies with depext but it does not install anything then when I tried to install frama-c, it fails with the following error :
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] The compilation of frama-c failed at
"/home/benoit/.opam/opam-init/hooks/sandbox.sh build make -j7".
#=== ERROR while compiling frama-c.20.0 =======================================#
# context 2.0.5 | linux/x86_64 | ocaml-system.4.09.0 | https://opam.ocaml.org#2d21a0b6
# path ~/.opam/default/.opam-switch/build/frama-c.20.0
# command ~/.opam/opam-init/hooks/sandbox.sh build make -j7
# exit-code 2
# env-file ~/.opam/log/frama-c-4880-6d07ae.env
# output-file ~/.opam/log/frama-c-4880-6d07ae.out
### output ###
# [...]
# Ocamlopt src/plugins/value/legacy/eval_terms.cmx
# Ocamlopt src/plugins/value/domains/cvalue/cvalue_transfer.cmx
# Ocamlopt src/plugins/value/legacy/eval_annots.cmx
# Ocamlopt src/plugins/value/engine/transfer_logic.cmx
# Ocamlopt src/plugins/value/domains/cvalue/cvalue_domain.cmx
# /usr/bin/ld: cannot find -lgtksourceview-2.0
# collect2: error: ld returned 1 exit status
# Ocamlopt src/plugins/value/domains/cvalue/cvalue_specification.cmx
# File "_none_", line 1:
# Error: Error while building custom runtime system
# make: *** [Makefile:1294: bin/viewer.byte] Error 2
# make: *** Waiting for unfinished jobs....
My final attempt was to build it from source, but the make fails with the following error :
Ocamlc src/plugins/server/jbuffer.cmo
File "src/plugins/server/jbuffer.ml", line 23, characters 12-26:
23 | type json = Yojson.Basic.t
^^^^^^^^^^^^^^
Error: Unbound type constructor Yojson.Basic.t
make: *** [share/Makefile.generic:78: src/plugins/server/jbuffer.cmo] Error 2
My only remaining solution is to use a VM with debian but it's really ideal for me...
Does anyone have an idea of what i could do?
Thank you!
There are multiple issues in your question, and I believe some of them are worthy of investigation as possible bugs or issues. I'd recommend trying one of the official Frama-C support channels:
Creating an issue in the Frama-C Gitlab public repository;
Or creating an issue in the Frama-C Github snapshot repository (currently being deprecated in favor of the Gitlab repository, which is updated daily);
Or trying some interactive opam-related support in the IRC #frama-c channel on freenode.net (interactive support is often more efficient for installation issues related to opam, especially since it's hard to know in advance which information might be useful).
Dealing with "Invalid user input" errors in the GUI
As a general rule, when trying to open a file in the GUI fails, I'd recommend running the command-line version of Frama-C with the filename. Its output is much more detailed and can indicate if there are parsing issues with the file (which could be due to missing dependencies, non-C99 syntax, or configuration problems).
Otherwise, the Console tab in the GUI should contain detailed error messages, which you could then add to this SO question, in case they are relevant to understand your issue.
Issues with graphical libraries in non-major Linux distributions
The second issue I can see in your question, is the error related to -lgtksourceview-2.0. This means that one of the library dependencies for the graphical interface is missing. Maybe there's a way to fix it, but at the very least it should be possible to identify the error before compilation, and report the incompatibility earlier. The fact that depext did not work is unusual; maybe it's not currently well-supported for Manjaro, but it's worth investigating. Unfortunately, opam can sometimes be complex to handle for first-time users, so I understand it's a bit frustrating and may take more time than you'd want. But if you intend to use Frama-C or other OCaml packages for some reasonable amount of time, it is worth understanding some of the common errors in opam, since it is very useful for handling dependencies.
Handling issues with OCaml dependencies when compiling manually
Finally, the issue with Yojson is probably due to the version of the package that is installed. In particular, versions before 1.6.0 do not have the type Yojson.Basic.t. Frama-C used to support 1.4.1, but the latest version requires at least 1.6.0. Again, it's not obvious to spot in advance all possible issues with different package versions, but reporting them allows us to include checks to prevent it in the future, or at least to give a more precise error message.
Normally opam handles that part, hence why it's recommended even for manual compilation of Frama-C, since it helps manage its dependencies.
Overall, it seems that some underlying issue with the dependencies in opam depext is preventing the whole chain from working properly. Your report does contain much useful information but not all the details, so I believe the interactive route should allow to quickly converge towards a working installation.
I've been trying to install LuaJIT on Windows 10 for some time following the official guide, and I actually get to install it. For example, if I execute luajit I get into the prompt. Also, luajit -v returns the version of luajit (2.0.4). And I can also execute code with luajit -e <lua code>. However, whenever I try to save bytecode with luajit -b, I get the following message:
luajit: unknown luaJIT command or jit.* modules not installed
I tried to make all sort of installations: using Cygwin, luajit-rocks, MinGW, ... However, no matter what I try, I always get the same result, and I have no clue of what to do.
Could you point me to some potential problems I might be overlooking?
I have on my system Lua 5.1 and Luarocks.
Some extra LuaJIT features are implemented as separate Lua modules (e.g. jit.bcsave for bytecode saving), and LuaJIT depends on package.path to find those modules. The suggested install location for those modules is in the default package.path, but if you override it via the LUA_PATH environment variable, you have to make sure to include that location there. One easy way to do that is to put two consecutive semicolons into LUA_PATH: Double semicolons are replaced by the compile-time default value of package.path.
You need place modules to "jit" folder near with juajit.exe. That folder include some system modules (bcsave too). package.path can dont work, becouse it hardlinked, how i understand. That folders distributed with source code.
Download lua from official sice: https://luajit.org/download.html
You can see "jit" folder inside archive:
LuaJIT-2.0.5.zip\LuaJIT-2.0.5\src\jit\
I installed Ada and AWS on a Macbook for the first time tonight.
Everything seemed successful with the installation, but I'm getting the feeling that AWS didn't install to the right directory.
GNAT is in /usr/local/gnat.
The AWS makefile.conf said,
AWS will be installed under $(prefix). Update this variable to point to
the install directory. By default, it is set to the current GNAT root
directory to have AWS project files automatically available.
So I didn't change the target.
But when I try to compile hello_world from anywhere on the system except the AWS demos folder, I get this error:
~/projects/ada:gnatmake hello.adb
gcc -c hello.adb
hello.adb:1:06: file "aws.ads" not found
hello.adb:2:06: file "aws.ads" not found
hello.adb:3:06: file "aws.ads" not found
hello.adb:4:06: file "aws.ads" not found
gnatmake: "hello.adb" compilation error
How can I make sure AWS was installed properly, or move it if it wasn't?
The answer (well, more of a hint to those in the know) lies in the words "to have AWS project files automatically available".
In the GNAT context, "project file" means a file of type .gpr which tells the builder (gnatmake or gprbuild) where to find the source files, how to compile them, and where to find any libraries that "this" project depends on.
Create hello.gpr:
with "aws";
project Hello is
for Main use ("hello.adb");
end Hello;
(which is very minimal) then
with AWS;
with Ada.Text_IO;
procedure Hello is
begin
Ada.Text_IO.Put_Line ("AWS version is " & AWS.Version);
end Hello;
compiled with
gprbuild -P hello.gpr
outputs
AWS version is 2.10.0w
There is on-line information on gprbuild and gnatmake (gnatmake has been superseded by gprbuild); there is probably information in your installed GNAT at (subdirectories of) ${prefix}/share/doc.