I'm using Poetry to manage my Python dependencies.
However, the delay in adding a new dependency is unbearable (often 1-3 h spent in package "resolving dependencies")
This is partly because I work in data science where it's not uncommon to have many "large" packages. Eg. Pandas, Sklearn, Tensorflow, Torch, FastAI, RapidAPI, Jupyter, IPython, ...
It is also not uncommon to install or update a new dependency fairly often.
Is there any practical way to either disable or speed up the dependency resolution? I want to keep using poetry for it's support of pyproject.toml, but I can't justify spending 10h/week waiting for package installs that would take 20min using pip and requirements.txt.
There is a useful issue thread on the poetry Github repo on this point, which includes a number of suggestions on how to speed up dependency resolution: https://github.com/python-poetry/poetry/issues/2094
If you use Ubuntu disabling IPv6 often does the trick:
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
Related
I have a problem with poetry install after the virtual cache is cleared.
A couple of weeks ago the poetry install worked on a python image in a gitlab CICD pipeline.
The installation of the packages in the .toml and lock file were- after running it the first time- stored in a cache. The cache is however cleared now and after running the pipeline again the poetry install fails. It doesn't give any error messages about which dependencies fail, but suddenly stops.
A lot of dependencies are noted with =^version in the .toml file. So we are assuming that some of the higher version dependencies have conflicts with eachother. We were hoping that poetry would help with these types of dependency problems? What is the best practice to deal with this conflicts? And to find the conflicting dependencies, because there were no error messages about the conflicts.
I'm running a project that uses pip and a requirements.txt file to install and keep track of some dependencies. I want to write some sh scripts to run, build and test the application. For starters I would like a way to check if the current folder is in a pyenv and, if not, create one to enclose the application and not mess around other people's dependencies. Also, I would like an opinion of the best way to keep track of this kind of dependencies, if the requirements.txt is a good approach and if there's a way to keep track of installed versions just like happens with node packages.
Use Pipenv. It's a better way of tracking your depencies than requirements.txt and it uses Pyenv to automatically install your project's required Python version.
From the website:
The problems that Pipenv seeks to solve are multi-faceted:
You no longer need to use pip and virtualenv separately. They work together.
Managing a requirements.txt file can be problematic, so Pipenv uses Pipfile and Pipfile.lock to separate abstract dependency
declarations from the last tested combination.
Hashes are used everywhere, always. Security. Automatically expose security vulnerabilities.
Strongly encourage the use of the latest versions of dependencies to minimize security risks arising from outdated components.
Give you insight into your dependency graph (e.g. $ pipenv graph).
Streamline development workflow by loading .env files.
[...]
Pipenv Features
Enables truly deterministic builds, while easily specifying only what you want.
Generates and checks file hashes for locked dependencies.
Automatically install required Pythons, if pyenv is available.
Automatically finds your project home, recursively, by looking for a Pipfile.
Automatically generates a Pipfile, if one doesn’t exist.
Automatically creates a virtualenv in a standard location.
Automatically adds/removes packages to a Pipfile when they are installed or uninstalled.
Automatically loads .env files, if they exist.
How would I install things on Pepper, since I don't know what package manager it uses. I usually use apt on my Ubuntu machine and want to install some packages on Pepper. I'm not sure what package manager Pepper has (if any) and want to install some packages, but also only know the name of the package using apt (not sure if the package name is the same on other package managers). And if possible, would I be able to install apt on Pepper. Thanks.
Note: From the research I've done, Pepper is using NaoQi which is based off Gentoo which uses portage.
You don't have root access on Pepper, which limits what you can install (and apt isn't on the robot anyway).
Some possibilities:
Include your content in Choregraphe projects - when you install a package, the whole directory structure is installed (more exactly, what's listed in the .pml); so you can put arbitrary files on your robot, and you can usually include whatever dependencies your code needs.
Install python packages with pip.
In NAOqi 2.5, a slightly older version of pip is installed that will not always work out of the box; I recommend upgrading it:
pip install --user --upgrade pip
... you can then use the upgraded pip to install other packages, using the upgraded pip, and always --user:
/home/nao/.local/bin/pip install --user whatever-package-you-need
Note however that if you do this and use your packages in your code running on Pepper, that code won't work on other robots until you do pip on them, which is why I usually only do this for tests; for production code I prefer packaging all dependencies in my app's package.
As a workaround if you need to install software (or just newer versions of software) using Gentoo Prefix is an option.
Gentoo Prefix builds a Gentoo OS on any location (no need of root, can be any folder). It includes it's own portage (package manager) to install new software.
I maintain a few projects to work with Pepper and use "any" software I want. Note that they are built for 64b (amd64) and 32b (x86) even though for Pepper only the 32b matter.
gentoo_prefix_ci and gentoo_prefix_ci_32b Which builds nightly the bootstrap of the Gentoo Prefix system. This is a process that takes a while to compile (3-6h depending on your machine) and that breaks from time to time (as upstream packages are updated and bugs are found, Gentoo is a rolling release distribution). Every night updated binary images ready to use can be found in the Releases section.
For ROS users that want to run it on the robot, based on the previous work, I maintain also ros_overlay_on_gentoo_prefix and ros_overlay_on_gentoo_prefix_32b. They provide nightly builds with binary releases of ROS Kinetic and ROS Melodic over Gentoo Prefix using ros-overlay. You can find ready-to-use 'ros_base' and 'desktop' releases.
For purposes related to the RoboCup#Home Social Standard Platform League where the Pepper robot is used I also maintain a specific build that contains a lot of additional software. This project is called pepper_os and it builds 270+ ROS packages, a lot of Python packages (250+ including Theano, dlib, Tensorflow, numpy...) and all the necessary dependencies for these to build (750+ packages). Note that the base image (it's built with Docker) is the actual Pepper 2.5.5.5 image, so it can be used for debugging as it if it was in the real robot (although without sensors and such).
Maybe this approach, or these projects, are useful.
The package manager on pepper is disabled. But you can copy the files to the robot and write your own service that imports any package you might need.
As a supplement on importing:
http://www.about-robots.com/how-to-import-python-files-in-your-pepper-apps.html
To get rid of error :
" SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
".
If you use python and requests package, just add verify=False at the end of your parameters.
r=requests.get(URL,params,header,verify=False)
Works with my Pepper
To get rid of
InsecurePlatformWarning: A true SSLContext object is not available.
install
/home/nao/.local/bin/pip install --user requests[security]
To get rid of:
CryptographyDeprecationWarning: Support for your Python version is deprecated.
install
/home/nao/.local/bin/pip install --user cryptography==2.2.2
If it based on Gentoo maybe we could try to install portage with pip.
pip install portage
Just a thought.
I am using Coq (versions 8.5-6), installed w/ Nix. I want to install ssreflect, preferably also w/ Nix. The only information I found about this is here. However, this is not about installing ssreflect, merely trying it out. Nevertheless, I tried to try it out, but ended up w/ hundreds of warnings (about the contents of various .v and .ml4 files) and couldn't wait for the process to end. A fairly typical warning looked like this:
File "./algebra/ssralg.v", line 856, characters 0-39: Warning:
Implicit Arguments is deprecated; use Arguments instead
So the question: How on earth do I install ssreflect w/ Nix?
EDIT: After reading ejgallego's comments, it seems it may be impossible to install ssreflect w/ Nix -- esp. if one wants install only ssreflect w/out the other modules (fingroup, algebra, etc.). So I've also the following question:
Would the standard Opam or make install installation of ssreflect work w/ a Nix-installed Coq?
There are a few things that you need to be aware of:
Nix is a source-based package manager with a binary cache. A lot of packages are pre-built and available in the binary cache, thus their installation doesn't take long; some packages (in particular development libraries) are not pre-built and Nix, when installing them will take the time it needs to compile them. Please be patient: you will only need to wait for the full compilation the first time (and yes, math-comp emits lots of warning upon compilation); next times, the package will be already available in your local Nix store.
Since OPAM is also source-based, using OPAM instead of Nix won't make you save time. You can't mixup Nix-installed Coq with OPAM installed SSReflect because the latter will want the former as an OPAM dependency.
The Nix way to use libraries is not to install them but to load them with nix-shell instead. nix-shell will "install" the libraries and set some environment variables for you (e.g. $COQPATH in this case).
You can also compile the package from source yourself using a Nix-installed Coq but you cannot run make install because this would try to install SSReflect at the same place where Coq is installed but the Nix store is non-mutable. Instead you could skip this step, and set up $COQPATH manually.
Indeed, the compilation of the full math-comp takes very long. There is a Coq ssreflect package which is lighter. You can get it using:
nix-shell -p coqPackages_8_6.ssreflect
The rpm command provides three main operations for upgrading and installing packages:
Upgrade
An upgrade operation means installing a new version of a package and removing all previous versions of the same package. If you have not installed a package previously, the upgrade operation will install the package.
Freshen
A freshen operation means to install a new version of a package only if you have already installed another version of the package.
Install
An install operation installs a package for the first time. It also, through special command-line parameters, allows you to install multiple versions of a package, usually not what we want. So, in the vast majority of cases, you want to run the upgrade operation for all package installations.
Should normally install packages with rpm -U, not rpm -i. One of the main reasons is that rpm -i allows you to install multiple instances of the same (identical) package.
Is this the standard conclusion or
should I stop installing the second instance of the package along with the first instance by writing any wrapper script or by adding code in spec file section.
If 2 point is the answer how can achieve this. Please guide me about this confusion.
Assuming you only every want one version of an RPM installed at once, then yes use "rpm -U".
Creating an RPM that can have multiple versions installed requires that all common files between the versions are identical. This frequently happens, so you may get this behaviour "by default".
You can also prevent multiple versions with the following in you spec:
Conflicts : %{name} < %{version}