Google Cloud Functions and shared libraries - wkhtmltopdf

I'm trying to use wkhtmltopdf on GCF for PDF generation.
When my function tries to spawn the child process I get the following error:
Error: ./services/wkhtmltopdf: error while loading shared libraries: libXrender.so.1: cannot open shared object file: No such file or director
The problem is clearly due to the fact that wkhtmltopdf binary depends on external shared libraries which are not installed in GCF environment.
Is there a way to solve this issue or should I give up and use other solutions (AWS Lambda o GAE)?
Thank you in advance

Indeed, I’ve found a way to solve this issue by copying all required libraries in the same folder (/bin for me) containing wkhtmltopdf binary. In order to let the binary file use uploaded libraries I added the following lines to wkhtmltopdf.js:
wkhtmltopdf.command = 'LD_LIBRARY_PATH='+path.resolve(__dirname, 'bin')+' ./bin/wkhtmltopdf';
wkhtmltopdf.shell = '/bin/bash';
module.exports = wkhtmltopdf;
Everything worked fine for a while. At a sudden I receive many connection errors from GCF or timeouts but I think it’s not related to my implementation but rather to Google.
I’ve ended up setting a dedicated server.

I have managed to get it working, there are 2 things needed to be done, as wkhtmltopdf won't work if:
libXrender.so.1 can't be loaded
you are using stdout to collect resulting pdf. Wkhtmltopdf has to write the result into a file
First you need to obtain correct version of libXrender.
I have found out, which docker image Cloud functions are using as base for nodejs functions. I've ran it locally, installed libxrender and copied the library into my function's directory.
docker run -it --rm=true -v /tmp/d:/tmp/d gcr.io/google-appengine/nodejs bash
Then, inside the runing container:
apt update
apt install libxrender1
cp /usr/lib/x86_64-linux-gnu/libXrender.so.1 /tmp/d
I have put this into my function's project directory and under lib sub directory. In my function's source file, I then set-up LD_LIBRARY_PATH to include the /user_code/lib directory (/user_code is the directory, where at last your function will end up being put by google):
process.env['LD_LIBRARY_PATH'] = '/user_code/lib'
This is enough for wkhtmltopdf to be able to execute. It will fail, as it won't be able to write to stdout and the function will eventually timeout and be killed (as Matteo experienced). I think this is because google runs the containers without a tty (just speculation), I can run my code in their container, if I run it with docker run -it flags. To solve this, I am invoking wkhtmltopdf so that it writes the output into a file under /tmp (this is in-memory tmpfs). I then read the file back and send it as my response body. Note that the tmpfs might be reused between function calls, so you need to use unique file every time.
This seems to do the trick and I am able to run wkhtmltopdf as Google CloudFunction.

Related

Can't save a temporary csv file to /home/app directory on heroku using R shiny app [duplicate]

I published my first simple app on Heroku with a free dyno. This app writes a simple .txt file, that seems to be correctly written because my API services are working fine.
But if I try to check this file by entering in the file system using "heroku run bash -a MYAPP", I can't see that file in the folder I thought to see. It is like the file is not existing. Can someone tell me why?
Thanks.
I found this on https://devcenter.heroku.com/articles/active-storage-on-heroku:
In addition, any files stored on disk will not be visible from one-off dynos such as a heroku run bash instance or a scheduler task because these commands use new dynos.
It is still not so clear to me, but at least I know it is a normal (but strange) behaviour of Heroku!

Is there any way to inspect the contents of the RocksDB instance used by NEAR Protocol?

Disclosure: I work with NEAR and am currently on-boarding.
When I start up a local node on a clean machine I see that a .near folder is created in my home directory with a few configuration files (exact files seem to depend on which start_ script I run). Another folder appears inside of the .near folder called data.
Running strings ~/.near/data/*.sst in the folder spits out a few lines starting with the string "rocksdb" which led me to this reference to RocksDB
Is there any way to inspect the contents of a node's RocksDB instance?
I found Keylord but it crashes when I try to configure a new connection to the database (by pointing the connection to ~/.near/data). I didn't pursue that thread.
PSA1: sometimes it's useful to backup the ~/.near folder between node restarts if you want to reset the environment or avoid reusing old data while troubleshooting
mv ~/.near ~/.near_`date +%Y-%m-%d.%s`
PSA2: on MacOS you can watch what happens to the contents of the ~/.near folder while the node boots up and runs. (brew install watch).
watch -d -c -n 0.5 find ~/.near
The content of RocksDB is serialized using our own binary serialization format (http://borsh.io/), so you won't be able to examine the content with general-purpose third-party tools

Ros Environment in root

I have a ros (kinetic) environment set up on a raspberry pi 3 and am trying to get ros to execute upon startup via a simple bash script which calls roslaunch. Ros works in the user domain but fails when called from root.
Here is my launch_ros.sh script:
#!/bin/bash
source /home/pi/ros_catkin_ws/devel/setup.bash
export PYTHONPATH=/opt/ros/kinetic/lib/python2.7/dist-packages
roslaunch my_pkg pkg_launch.launch
When I run sudo /home/pi/Desktop/ros_launch.sh the roscore crashes with
ERROR: cannot launch node of type [rosout/rosout]: can't locate node
[rosout] in package [rosout] failed to start core service [/rosout]
The traceback for the exception was written to the log file
But, if I comment out
source /home/pi/ros_catkin_ws/devel/setup.bash
and execute /home/pi/Desktop/ros_launch.sh, ros works fine.
Also worthy of noting is if I leave the above source line uncommented when running in the user domain I get the same error as I do in the root. I think this might be pointing me to the solution but I am still very new to ros.
Has anyone come across this issue and found a solution?
In order to run a node as root after changing your shell to root using commands like sudo -i, You can source your current bash profile thats located inside your normal user .bashrc and use it inside root shell.
Try the following code:
#!/bin/bash
source /opt/ros/kinetic/setup.bash
source /home/pi/ros_catkin_ws/devel/setup.bash
export PYTHONPATH=/opt/ros/kinetic/lib/python2.7/dist-packages
roslaunch my_pkg pkg_launch.launch
You need to source your workspace devel to be able to find your own package.
But, you need to source ROS devel to be able to use roscd, roslaunch, ...
In the code below I added:
source /opt/ros/kinetic/setup.bash
to source ROS and be able to use it.
PS: If it's still not working you should try a short delay before running roslaunch.
I was accidentally in a conda environment (base only) and it was messing up big-time. Try disabling any Python virtual environments.
It really worked and it inspired me to report that I didn't have rosnode as a program when I was planning to call ROS scripts in my own applications. Calling source .bashrc directly from the Raspberry Pi's system would refresh the terminal, but there was no way for my program to take over. The solution was to place the required ROS environment scripts in a separate script like name init_env.sh and then call source init_env.sh before any other ROS scripts were executed.

WKHTMLTOPDF and "Error: Unable to create temporery file"

I've written a piece of code in PHP to generate PDF using WKHTMLTOPDF binary file. It was working fine till I had to recompile my Apache. Now it fails with error Error: Unable to create temporery file (this is the exact wording).
The situation in which the error is reproducible is a little complicated. I managed to narrow down the error and now I'm pretty sure that the error happens because of the user that Apache runs as. It seems to me that when WKTHMLTOPDF is running as a user with no home folder, it's unable to access a temporary folder within the user's home folder.
Surely I can change the Apache's user but I would rather resolve this problem once and for all. To this end it would be great if I could somehow set the temp folder for WKHTMLTOPDF or at least print its current value to make it valid! Does anyone know how to do any of these two?
BTW, I'm using WKHTMLTOPDF 0.11.0 rc1.
I saw the same error today in Rails4 + pdfkit gem(0.8.2) + wkhtmltopdf(0.12.2.1) under CentOS 6.7.
This error came from wkhtmltopdf and the reason was it couldn't create temporary file. wkhtmltopdf depends on some temporary filename creation API (I'm not sure), but probably following shows some hints:
$ man tempfile
$ man tempnam
In my case, my TMPDIR environment variable showed wrong path (I had accidentaly deleted the directory!) so that wkhtmltopdf couldn't create work file.
When I unset TMPDIR, then it worked! Of course, setting correct existence directory to TMPDIR should be OK too.

Download large files in Heroku

I am facing some issues when downloading large files in Heroku. I have to download and parse files greater than 1Gb. What I am trying to do right now, is use curl to download them into /tmp folder (of a Rails application).
The curl command is: "curl --retry 999 -o #{destination} #{uri} 2> /dev/null" and destination is Rails.root.join("tmp", "file.example")
The problem is that after a few minutes downloading, the "curl" process that is downloading the file is finished, way far from the download is finished. Before being finished, the logs show lots of "Memory exceeded". This led me to the thinking that when I am saving to /tmp folder, it is storing the downloaded content in the memory and when it memory hit its limit, the process is killed.
I would like to know if any of you have already experienced a similar issue on Heroku and if saving to /tmp folder really works like this. If so, do you have any suggestions to get this working at Heroku?
thanks,
Elvio
You are probably better off saving the file in an external cloud provider like S3 using the fog gem. In any case, Heroku is a read only filesystem, so they won't allow you to curl, must less write to it.

Resources