I am installing lots of pip packages on a debootstrap'd Debian which is understandably quite bare bones and lacks a ton of packages that you'd usually expect on a regular installation. Naturally, pip installations are failing because the OS doesn't have the libraries they depend on.
Instead of individually digging for dependencies I need to install, is there a way to let pip resolve apt dependencies?
No. pip is a portable tool designed to be run on many operating systems. But on every operating system pip works with just Python packages and doesn't touch system libraries. If you install Python packages with apt or any other distribution-specific package manager it handles all dependencies. When you add pip to the toolbox — you are on your own.
Related
I read that on OS X you can install Yarn either by
curl -o- -L https://yarnpkg.com/install.sh | bash
brew install yarn
npm i -g yarn
What functional difference is there between these three methods? Why would someone choose one over the others?
when using brew to install packages, you install them system wide. that is, you cannot have more than one version for the same package, which is usually problematic. for this reason, many other technologies have spawn, such as docker, snap.
moreover, each package manager has its own lifecycle and packs original package in a different manner for ease of use, distribution and maintenance. for instance, npm container is based on the release of npm package itself.
usually, you should stick to the package manager of the same ecosystem that you are using. specifically to your case, it will be recommended to use npm to install and update your package (using package.json). which will give each of your project to pin and lock the desired yarn version that you like, without any affecting your system wide.
speaking of npm, you might wish to look at this answer
curl downloads the installation script from yarnpkg.com, and installs yarn using that script
brew is a package manager for MacOS. It's meant to make things easier for people when installing commands for the terminal. When you install with brew, the package get's put into /usr/local/bin instead of /usr/bin so I believe that this is kind of like a virtual environment, and yarn wouldn't be installed into the core of your machine. You'll have to install homebrew before you can use it, and you install it using curl. I believe that there is less risk when using homebrew because of the reasons that it is kind of like a virtual environment
npm is a package manager for javascript, it's the same as yarn. It's meant for easy installation of javascript packages.
I use brew for all installs to the terminal, and npm for all installs of javascript packages.
Im trying to get something running in a lab, and I need to install a whole bunch of packages to compile the code:
sudo apt-get install build-essential libgtk2.0-dev libwxgtk3.0-dev libwx-perl libmodule-build-perl git cpanminus libextutils-cppguess-perl libboost-all-dev libxmu-dev liblocal-lib-perl wx-common libopengl-perl libwx-glcanvas-perl libtbb-dev libxmu-dev freeglut3-dev libwxgtk-media3.0-dev libboost-thread-dev libboost-system-dev libboost-filesystem-dev libcurl4-openssl-dev libextutils-makemaker-cpanfile-perl
The problem is, I'm working on a Mac; So I only have brew available - and the package names are not equivalent.
Am I trying to do something dumb and impossible, or is there a way to cross reference those apt-get packages to ones available in brew and install those so I can build the app?
For added context, its a modified "slicer" application for generating files that a 3d printer uses to make parts. I could fire up an ubuntu VM and use apt-get, but I'm on an M1 mac at the moment and recompiling experimental QEMU code just so I can fire up an ubuntu VM takes me way more out of my depth than I already am... I'm running brew on a duplicated, rosetta emulation forced terminal.
There is no straightforward way to automatically figure out whether a corresponding Homebrew package exists for each of these packages, no. But you can probably guess the majority, and manually figure out the rest. Each Debian package has a link to the upstream sources, and a home page if one exists, from which you can often find links to packages for other architectures, etc.
From the Debian package search page you can search e.g. for the libxmu-dev package, and discover the corresponding package page for Buster (the current stable Debian release), which in turn has links to the upstream repo, etc. But this is an X11 package, so it's not straightforwardly compatible with macOS, which uses an entirely different GUI architecture.
I know there are a lot of questions on the coexistence and interchangeability/non-interchangeability of pip and conda. That is not my question: I know I need both for my work, I use both, and for the most part, my conda envs are a manageable mess.
But here's the thing: there's many ways to install pip. I happened to get conda going first, so my pip is through anaconda/bin/pip. It is the only pip on my machine. Here are my questions:
Is this sensible? Do I want my pip to be usr/bin/pip and be independent of global conda? It feels not-sensible.
If I install a new pip through say brew or easy_install, should I start downloading packages through this new pip? Would that be awful and mess everything up?
Thanks!
Pip always requires a version of Python to be installed, and is associated with that specific Python installation. By default, pip installs packages for its own Python, into the related site-packages directory inside the Python library directory. The exact location of this directory depends on your operating system and how you installed conda.
If you install pip via Homebrew or with another installation of Python, you should not use that pip and expect it to install for conda. For that matter, if you create a new conda environment, you should not expect that the pip in that environment will install packages into another environment.
There is the --user option to pip, which installs packages into a directory in your user account (on *nix systems, this is ~/.local; I can't recall for Windows where this is). These packages will be able to be found by all Python versions with the same major and minor version number. However, it is not recommended to install packages with the intent of sharing them among several Pythons this way, because if the different Pythons were compiled with different compilers, you may run into trouble.
What is the difference between pip install and pip download?
$ pip --version
pip 9.0.1 from c:\users\nahawk\python\python27\lib\site-packages (python 2.7)
I did a diff on the help for the 2 commands and it wasn't very helpful. Unfortunately, the options are listed in a little different order, which makes them look more different that they actually are.
pip help download
pip help install
I found the answer in the online docs for pip download.
IMO, some of this should be stated in the help (i.e. in the description). Otherwise, how would one know how to use it, using pip help download?
pip download replaces the --download option to pip install, which is
now deprecated and will be removed in pip 10.
pip download does the same resolution and downloading as pip install,
but instead of installing the dependencies, it collects the downloaded
distributions into the directory provided (defaulting to the current
directory). This directory can later be passed as the value to pip
install --find-links to facilitate offline or locked down package
installation.
pip download with the --platform, --python-version, --implementation,
and --abi options provides the ability to fetch dependencies for an
interpreter and system other than the ones that pip is running on.
--only-binary=:all: is required when using any of these options. It is important to note that these options all default to the current
system/interpreter, and not to the most restrictive constraints (e.g.
platform any, abi none, etc). To avoid fetching dependencies that
happen to match the constraint of the current interpreter (but not
your target one), it is recommended to specify all of these options if
you are specifying one of them. Generic dependencies (e.g. universal
wheels, or dependencies with no platform, abi, or implementation
constraints) will still match an over- constrained download
requirement.
I am interested in getting TensorFlow to run on Windows, however at present I realize that this is not possible due to some of the dependencies not being usable with Windows, e.g. Bazel.
The need arises because as I currently understand it the only way to access the GPU from TensorFlow is via a non-virtual install of Linux. I do realize I can dual boot into a Linux install, but would prefer to avoid that route.
To resolve the problem I am in need of the entire dependency chain to build TensorFlow as was wondering if this already existed.
I also realize that I can capture the build output when building from source as a solid start, but would like to avoid that work if it is already known.
There is a beta of Bazel that runs on Windows - https://github.com/dslomov/bazel-windows
See related GitHub Issue to run TensorFlow on Windows. - https://github.com/tensorflow/tensorflow/issues/17
Another reason to run on Windows is the possibility to port to Xbox One.
I found a possible answer, still need to check it. This will generate a dependency graph as a dot file.
$ bazel query 'deps(//tensorflow/tools/pip_package:build_pip_package)' --output graph > tensorflow.dependency.dot
There are now three main options for building and/or running TensorFlow on Windows:
You can install a GPU-enabled PIP package of TensorFlow 0.12rc0 from PyPI: pip install tensorflow-gpu
You can build the GPU-enabled PIP package yourself using the experimental CMake build. This also gives you the ability to work with TensorFlow in Visual Studio. The documentation for this build can be found here.
There is preliminary support for building TensorFlow using Bazel for Windows. However, we are still ironing out some bugs with this build.
This may not be exactly what you want one way to run TensorFlow under Windows is to install a virtual machine (VMWare player v12 is free to use for non-commercial) and then install Ubuntu in that and finally TensorFlow in Ubuntu. Works well for me.
Since the begin of 2017, Tensorflow is now officially supported on Windows and can be installed via pip:
pip install --upgrade tensorflow
pip install --upgrade tensorflow-gpu
or by fetching packages directly (pick the one that matches your needs, e.g. x64/gpu)
# x86 / CPU
pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.0.0-cp35-cp35m-win_x86_64.whl
# x64 / CPU
pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.0.0-cp35-cp35m-win_amd64.whl
# x64 / GPU
pip install --upgrade https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-1.0.0-cp35-cp35m-win_amd64.whl