Have a new MacBook w/ macOS Mojave but cannot get spark/pyspark working. I have done the following:
Downloaded the latest Apache Spark from https://spark.apache.org/downloads.html
Unzipped it in my home directory (/Users/myname/)
Renamed the folder spark
Checked in the Finder window that spark was there, and the folder contained python, bin, and bin/spark-shell subfolders
Got the ability to pip install packages with: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >> python get-pip.py in the terminal >> Checked that it was installed by doing import pip in the python shell >> sudo easy_install pip in the regular terminal
pip install pyspark, which worked
Created both a .bashrc and .bash_profile files in the Users/myname/ path
I tried the following versions of the commands in these bash files:
export SPARK_HOME="/Users/myname/spark/python"
export PATH="$SPARK_HOME/bin:$PATH"
export SPARK_HOME=/Users/myname/spark/python
export PATH=$SPARK_HOME/bin:$PATH
export SPARK_HOME=/Users/myname/spark
export PATH=$SPARK_HOME/bin:$PATH
export SPARK_HOME=~/spark
export PATH=$SPARK_HOME/bin:$PATH
Different installation articles say I should just be able to do pyspark in the terminal, or spark/bin/spark-shell or just 'bin/spark-shell`, I just keep getting versions of the message
-bash: pyspark: command not found
Any advice?
If you've installed pyspark from pip, you don't need the standalone installation. You just need to fix you env vars. First, find the location of your site packages:
python -c "import site; print(site.getsitepackages())"
This is probably something like this /usr/lib/pythonX.Y/site-packages
Then, add the exports to your bash profile and source it.
echo "export SPARK_HOME=/usr/lib/pythonX.Y/site-packages/pyspark" >> ~/.bash_profile
echo "export PATH=$SPARK_HOME/bin:$PATH" >> ~/.bash_profile
source ~/.bash_profile
Related
I installed python 3.6 before and installed python 3.7.4 today. When I type python3 in command, it opens python 3.6, I want to change it to python 3.7.4.
which python3 shows /Library/Frameworks/Python.framework/Versions/3.6/bin/python3,
but the link in /usr/local/bin/ is :
python3 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3
so is the case of pip3. why? ?
How can I change which python3 to python 3.7.4? I don't want to use alias.
I use MacOS 10.14.2
Your OS uses the PATH environment variable to look for the commands you write into the shell, so if you type python3 it will go through the directories listed in this PATH and check if there's your programm. It takes the first matching program and executes it, so in your case the directory /Library/Frameworks/.../3.6/bin is before the directory usr/local/bin, which means that the python3 from /Library/Frameworks/.../3.6/bin will be used.
You need therefore to change this PATH variable:
export PATH="/Users/sky/Documents/software/Montage-master/bin:/usr/share/file/magic/mercurial:/Library/Frameworks/Python.framework/Versions/3.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/mysql/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin"
You can put that into your ~/.bash_profile so that it is permanent, and you don't need to set it every time you open a new terminal window.
Note that this will not automatically update your path for the remainder of the session. To do this, you should run:
source ~/.bash_profile
Are you sure have python3.7 intalled?
you can view the folder
cd /usr/bin
Next you search the python's file:
find /usr/bin/ python3
if don´t exist filename python3.7 install
sudo apt install python3.7
sudo apt update
I just followed the installation guide for golang (ubuntu 16).
I extracted the archive at /etc/usr
I added env variable in /home/user/.profile
I just tested a basic go build on the hello world code.
I get the following error:
The program 'go' is currently not installed. You can install it by typing: sudo apt install golang-go
Why does it ask me to install it (again?)?
open the go documentation download
https://go.dev/dl/
choice your os and go version
download then extract the file
extract the file
open the file and open the terminal
6.Add /usr/local/go/bin to the PATH environment variable.
export PATH=$PATH:/usr/local/go/bin
then check the go version
go version
The location of the binary go is not in your path. Ubuntu does not find it and suggests to install it. Add this line to your file /etc/profile, or better $HOME/.profile:
export PATH=$PATH:/usr/local/go/bin
This is documented in the docs: https://golang.org/doc/install#install
If you want to try this solution before editing any files, you can just execute the above command and try to execute the go command in the shell.
There are paths which needs to be set correctly for you go installation to work
GOROOT points to directory where go is installed
export GOROOT=/usr/lib/go
GOPATH points to you workspace directory
export GOPATH=$HOME/go
These paths need to be added in global path variable.
export PATH=$PATH:$GOROOT/bin
You need to put the go executable in your system path. which you can do by
export PATH=$PATH:/etc/usr/go/bin
You can put the same in /home/user/.profile
just use asdf for installation. You can have several version also :D
Docs: https://asdf-vm.com/#/core-manage-asdf
downlaod the installer form enter link description here, choose intaller for linux that suit your device and then you go to your CLI and use wget or curl :
$ wget https://storage.googleapis.com/golang/go1...
and then extract the file to /usr/local :
$ tar -C /usr/local -xzf go1...
add path binary Go to PATH environment variable :
$ echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.bashrc
$ source ~/.bashrc
and then use go version to check if the Go already installed
If You are using linux then open your terminal and run this command.
sudo apt install golang-go
This command will Install Go lang. in your system. ThankYou
Steps for Go installation:
sudo apt-get update && sudo apt-get -y upgrade
wget https://dl.google.com/go/go1.17.5.linux-amd64.tar.gz
sudo tar -xvf go1.17.5.linux-amd64.tar.gz
sudo mv go /usr/local/
export GOROOT=/usr/local/go
Add in .bashrc
vi .bashrc
export GOPATH="/root/go"
export GOROOT=/usr/local/go
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
Download latest version from https://golang.org/doc/install
tar -xzf go1.15.7.linux-amd64.tar.gz
move to /usr/lib/ to folder with version number
sudo mv go /usr/lib/go-1.15
create symkink link on /usr/bin/
ln -s /usr/lib/go-1.15/bin/go /usr/bin/go
I have installed SDK doing brew cask install android-ndk and NDK doing brew cask install android-ndk, everything was ok during the installation and inserted the PATH, like this:
export GRADLE_HOME=/usr/local/opt/gradle
export ANDROID_HOME=/usr/local/Caskroom/android-sdk/3859397
export ANDROID_NDK_HOME=/usr/local/Caskroom/android-ndk/16b/android-ndk-r16b
export ANDROID_SDK_ROOT=/usr/local/Caskroom/android-sdk/3859397
export PATH=$GRADLE_HOME/bin:$PATH
export PATH=$ANDROID_HOME/platform-tools:$PATH
export PATH=$ANDROID_HOME/tools:$PATH
Saved and committed doing source ~/.bash_profile.
After, I tried to run my gradle task, during start of compile of a C++ library, I got this:
* What went wrong:
A problem occurred configuring project ':some-library'.
> NDK not configured.
Download it with SDK manager.
Is there any way to solve it?
Thank you.
In macOS installed with brew use this the following paths
In file ~/.bash_profile
export ANDROID_NDK_HOME = /usr/local/share/android-ndk
export ANDROID_HOME = /usr/local/share/android-sdk
If you are using another bash script or zsh for example, you should put it begin of that script file, eg.
myortherscript.sh
#!/bin/bash
# THIS AT TOP
source ~/.bash_profile
Important: This sample I'm considering the same user for configuration bash_profile and script execution.
Well, looking around here there are tons of questions about how to correctly installing virtualenvwrapper on Mac. I really tried all of them but for no avail. I've noticed the previous questions don't have the configuration like my computer, so here it is:
Mac OS Sierra 10.12
Python3.5.2 installed from python.org installer for Mac OS
This installation puts Python3.5.2 in this path: /Library/Frameworks/Python.framework/Versions/3.5/bin/python3
I've run python3 get-pip.pycommand to install pip correctly, as I think
sudo pip3 virtualenvwrapper to be sure all settings will be done as they should
As the documentation says, edited ~/.bash_profile with the code below:
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/projects
source /usr/local/bin/virtualenvwrapper.sh
Then, whenever I open a new fresh Terminal, I got this:
/usr/bin/python: No module named virtualenvwrapper virtualenvwrapper.sh:
There was a problem running the initialization hooks.
If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is set properly.
I've tried the fixes from others questions, to no avail:
set VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3 on virtualenvwrapper.sh
set VIRTUALENVWRAPPER_PYTHON="$(command \which python3)" on virtualenvwrapper.sh
ln -fs /Library/Frameworks/Python.framework/Versions/3.5/bin/python3 /usr/local/bin/python3
alias python=/usr/local/bin/python3
Well, looking forward to anyone with a configuration like mine to see what I'm doing wrong.
This is pretty old, but posting this because I came here from google and finally figured out the right paths.
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
export PROJECT_HOME=$HOME/Devel
source /Library/Frameworks/Python.framework/Versions/3.6/bin/virtualenvwrapper.sh
You have to make sure to have the actual python executable. This can be found by just simply browsing in Finder through to /bin and seeing what is there.
Well, after a lot of effort, I've decided to install pip and virtualenvwrapper on Python 2.7 shipped in macOS Sierra.
After that, I wrote the lines below on .bash_profile. Maybe there's something redundant here but, it works.
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Documents/Projects/
ln -fs /Users/myusername/Library/Python/2.7/bin/virtualenvwrapper.sh /usr/local/bin/virtualenvwrapper.sh
ln -fs /Users/myusername/Library/Python/2.7/bin/virtualenv /usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh
After that, I've installed Python 3.5.2 from python.org and created a new enviroment pointing to it and everything is working just fine.
mkvirtualenv --python=python3.5 env_name
I am trying to install sphinx on a remote machine.
Since I don't have an access to the root, I did this:
$bash
$mkdir -p ~/local/lib/python2.7/site-packages
$export PYTHONPATH=$PYTHONPATH:~/local/lib/python2.7/site-packages
$export PATH=$PATH::~/local/lib/python2.7/site-packages
$easy_install -U --prefix=$HOME/local Sphinx
But apparently, $easy_install doesn't build sphinx-quickstart; when I type
$sphinx-quickstart
I get the following message:
bash: sphinx-quickstart: command not found
I tried
find $HOME -name sphinx-quickstart
and no result was found. However, I can import sphinx inside python:
$python
And then
>>import sphinx
works. Any idea why sphinx-quickstart doesn't work?
An alternative way to invoke sphinx-quickstart is to explicitly load Sphinx's quickstart module. For Sphinx v1.7+:
python -m sphinx.cmd.quickstart
For older versions of Sphinx:
python -m sphinx.quickstart
For example:
$ /c/Python35/python -m sphinx.quickstart
Welcome to the Sphinx 1.6.2 quickstart utility.
...
I found the solution in this webpage:
User (root/sudo free) installation of Python modules.
In section 3. Python 2.6+ he mentioned that the command line commands are in
~/local/bin
Although I had put ~/local/lib/python2.7/siste-packages in the path, the ~/local/bin directory was not in the path. So all I did
$export PYTHONPATH=$PYTHONPATH:~/local/bin
and now it works.
I don't know why find $HOME -name sphinx-quickstart did not find sphinx-quickstart
In MacOS (Catalina) and zsh:
I installed sphinx with brew and had the same problem as you.
Solution:
Installed it with conda as conda install sphinx and now it works for me
$ sphinx-quickstart
Welcome to the Sphinx 2.3.0 quickstart utility.
Installation Guide
Note, if you're in a virtual environment and using poetry to run
poetry run py -m sphinx.cmd.quickstart
where py is the python launcher for Windows.
Have added an answer for installation on macOS using brew here
For new users, installing with brew gives the following caveats:
==> sphinx-doc
sphinx-doc is keg-only, which means it was not symlinked into /usr/local,
because this formula is mainly used internally by other formulae.
Users are advised to use `pip` to install sphinx-doc.
If you need to have sphinx-doc first in your PATH, run:
echo 'export PATH="/usr/local/opt/sphinx-doc/bin:$PATH"' >> ~/.zshrc
Running the echo command:
echo 'export PATH="/usr/local/opt/sphinx-doc/bin:$PATH"' >> ~/.zshrc
should fix the issue.
Don't miss source-ing the .zshrc after the updating it:
source ~/.zshrc