How can I install GRPCIO on an Apple M1 Silicon laptop? - pip

Question

Every time I try to install a pip package with GRPCIO as a dependency inside a .direnv project I get a build failure due to my architecture being arm64. How can I work around this while waiting for the GRPCIO crew to release an update?
Using cached grpcio-1.34.0.tar.gz (21.0 MB)
ERROR: Command errored out with exit status 1:
command: /Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"'; __file__='"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-pip-egg-info-z41yqiw6
cwd: /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/
Complete output (10 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py", line 359, in <module>
if mac_target and (pkg_resources.parse_version(mac_target) <
File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 113, in parse_version
return packaging.version.Version(v)
File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 275, in __init__
match = self._regex.search(version)
TypeError: expected string or bytes-like object
ASM Builds for BoringSSL currently not supported on: macosx-11-arm64
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Answer 1

This seems to work well, do the following in the terminal
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
And then install whatever package that you wanted to install, in my case I was trying to install firebase admin and I had to run the following
pip install firebase-admin
Or just run the below command if you just want to install grpcio
pip install grpcio
I found this thread helpful to understand this issue in detail
https://github.com/grpc/grpc/issues/25082

Answer 2

CFLAGS="-I /opt/homebrew/opt/openssl/include" LDFLAGS="-L /opt/homebrew/opt/openssl/lib" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt
this works for me in my mac book pro M1 bigsur.
Thanks for the solution

Answer 3

I found a solution that works taking from https://github.com/grpc/grpc/issues/25082 with more detail and the removal of certain unnecessary steps. Note that this only works on python version 3.9 as of this post date.
First, install pyenv and pyenv-virtualenv.
brew install pyenv pyenv-virtualenv
Note that you might need to futz with the shell evaluation variable as simply adding eval "$(pyenv init -)" does not seem to work on the Apple M1 computers. Instead, run the following
pyenv init - >> .zshrc
pyenv virtualenv-init - >> .zshrc
Then go to your python project directory and create an .envrc and register it with direnv
echo 'layout pyenv 3.9.1' > .envrc
direnv allow
Next there install your requirements.txt via
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt
Collecting google-cloud-pubsub
Using cached google_cloud_pubsub-2.3.0-py2.py3-none-any.whl (210 kB)
Collecting proto-plus>=1.7.1
Using cached proto_plus-1.17.0-py3-none-any.whl (42 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
Collecting libcst>=0.3.10
Using cached libcst-0.3.17-py3-none-any.whl (507 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.22.2
Using cached google_api_core-1.26.1-py2.py3-none-any.whl (92 kB)
Collecting protobuf>=3.12.0
Using cached protobuf-3.15.6-py2.py3-none-any.whl (173 kB)
Collecting grpcio<2.0.0dev,>=1.0.0
Using cached grpcio-1.36.1.tar.gz (21.5 MB)
Collecting googleapis-common-protos[grpc]<2.0.0dev,>=1.5.2
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting typing-inspect>=0.4.0
Using cached typing_inspect-0.6.0-py3-none-any.whl (8.1 kB)
Processing /Users/rkmacmini/Library/Caches/pip/wheels/b7/a5/c4/504d913c2a55bb09c607541578ec5f844d1ff33467abe93ba5/PyYAML-5.4.1-cp39-cp39-macosx_11_0_arm64.whl
Collecting typing-extensions>=3.7.4.2
Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting six>=1.13.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting google-auth<2.0dev,>=1.21.1
Using cached google_auth-1.27.1-py2.py3-none-any.whl (136 kB)
Collecting pytz
Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Requirement already satisfied: setuptools>=40.3.0 in ./.direnv/python-3.9.1/lib/python3.9/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-pubsub->-r requirements.txt (line 1)) (49.2.1)
Collecting packaging>=14.3
Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting requests<3.0.0dev,>=2.18.0
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting mypy-extensions>=0.3.0
Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<5.0,>=2.0.0
Using cached cachetools-4.2.1-py3-none-any.whl (12 kB)
Collecting rsa<5,>=3.1.4; python_version >= "3.6"
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyparsing>=2.0.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
|████████████████████████████████| 153 kB 1.1 MB/s
Collecting certifi>=2017.4.17
Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting pyasn1<0.5.0,>=0.4.6
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Using legacy 'setup.py install' for grpc-google-iam-v1, since package 'wheel' is not installed.
Using legacy 'setup.py install' for grpcio, since package 'wheel' is not installed.
Installing collected packages: six, protobuf, proto-plus, grpcio, googleapis-common-protos, grpc-google-iam-v1, mypy-extensions, typing-extensions, typing-inspect, pyyaml, libcst, pyasn1, pyasn1-modules, cachetools, rsa, google-auth, pytz, pyparsing, packaging, idna, urllib3, certifi, chardet, requests, google-api-core, google-cloud-pubsub
Running setup.py install for grpcio ... done
Running setup.py install for grpc-google-iam-v1 ... done
Successfully installed cachetools-4.2.1 certifi-2020.12.5 chardet-4.0.0 google-api-core-1.26.1 google-auth-1.27.1 google-cloud-pubsub-2.3.0 googleapis-common-protos-1.53.0 grpc-google-iam-v1-0.12.3 grpcio-1.36.1 idna-2.10 libcst-0.3.17 mypy-extensions-0.4.3 packaging-20.9 proto-plus-1.17.0 protobuf-3.15.6 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-2.4.7 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 rsa-4.7.2 six-1.15.0 typing-extensions-3.7.4.3 typing-inspect-0.6.0 urllib3-1.26.4
NOTE: Some references say you need to point your CFLAGS, etc, at the brew installed openssl locations but that has not seemed necessary for me. If you have issue you can try the command with the CFLAGS enabled via
CFLAGS="-I /opt/homebrew/opt/openssl/include" LDFLAGS="-L /opt/homebrew/opt/openssl/lib" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt

Answer 4

There's a much simpler and more reliable option:
pip uninstall grpcio
conda install grpcio

Answer 5

Unfortunately, Google does not provide grpcio wheels built for Apple Silicon Macs. As you see from this issue, users have come up with a variety of compiler flags to compile the library on M1, which don't always work.
Tired of these problems, I published the grpcio-mac-arm-build repository, which contains a daily GitHub Actions job that compiles the latest stable grpcio and grpcio-tools releases for Apple Silicon. The compiled wheels for Python 3.8, 3.9, 3.10, and 3.11 are available for download in the "Releases" page. Everything is open sourced and released with the same license of the grpcio library. This way, you don't need to compile the library at all.

Answer 6

After many attempts, I finally got it installed by running these commands:
xcode-select --install
brew install openssl re2
export LDFLAGS="-L$(/opt/homebrew/bin/brew --prefix openssl)/lib -L$(/opt/homebrew/bin/brew --prefix re2)/lib"
export CPPFLAGS="-I$(/opt/homebrew/bin/brew --prefix openssl)/include -I$(/opt/homebrew/bin/brew --prefix re2)/include"
GRPC_BUILD_WITH_BORING_SSL_ASM=""
GRPC_PYTHON_BUILD_SYSTEM_RE2=true
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true
pip install grpcio
Taken from here: https://candid.technology/error-command-usr-bin-clang-failed-with-exit-code-1/

Answer 7

I manage to get it work without changing any variable in my OS. This seems to be to do with python version issue.. i was using 3.7.12 failed. 3.8.12 failed. Finally, 3.10.2 got this package installed.

Answer 8

None of the suggestion worked for me. What worked is below (assuming you have homebrew installed):
Run brew install miniconda (follow through with - conda init "$(basename "${SHELL}")", as instructed.)
Either source your ~/.zshrc or kill your terminal and reopen.
Run conda create --name myenv python=3.10.
Run conda activate myenv.
Run pip install grpcio.
This has brew install the correct arm64 conda env in step 1, hence the rest is doable.

Answer 9

The following settings worked for me,
Used https://asdf-vm.com/ to install and manage my python
Used https://direnv.net and created $PWD/.envrc with just one line
layout python
Made sure I installed latest python 3.9 using asdf
asdf plugin-add python
asdf install python 3.9.15
Then did,
direnv allow .
That should create a python venv under $PWD/.direnv, then did
pip install -U pip wheel grpcio
IMPORTANT: Python versions other than 3.9.x did not work for my e.g. 3.10 fails with some clang issue