I want to work on an aws machine for a kaggle competition. While working on my own pc i have Anaconda installed, pycharm.
How do i set it up on an AWS machine?
Do i need to install the tools each time i log in the AWS machine.
What is the recommended way for this cases?
I thought of running a jupyter notebook instance on the remote machine, and connect to that web page via my local machine.
What are the pitfalls to be expected?
You need to launch an EC2 instance in AWS and connect it through SSH to run your algorithms. The following link can help you further.
http://www.grant-mckinnon.com/?p=6
In these cases using Docker images helps the best in encapsulating the environment. Kaggle also uses docker to create environments for kernels. You can install Docker from here. You can pull the docker image using following command,
docker pull datmo/kaggle:python
After this, you can run containers using this image and mount necessary files in order to run the same environment.
docker run --rm -it -p 8888:8888 -v ~/.:/home/ datmo/kaggle:python 'jupyter notebook'
You can also use datmo CLI in order to easily setup environment and also version control your machine learning models.
Related
Environment: Windows 11 + Docker Desktop 4.12.0
I've been digging this the entire morning. There doesn't seem to be a way of installing Laravel in a Docker image. You must curl it in your WSL2 distro. Trying the command curl -s https://laravel.build/example-app | bash in a Docker container command-line immediately returns the dreaded Docker is not running error message
Some suggest that I need to turn on my "WSL2 integration" checkbox in Docker Desktop settings, but that didn't help.
So what if I download the official Ubuntu image from Docker Hub, run it as a container. Can I download (curl) Laravel in that container?
And while we are here, how does the Bitnami Laravel image differ from the standard procedure given in Laravel documentation? I like it because I can download it as a normal Docker image and create as many containers as I want, but I'm unsure how this connects or contrasts with the official Laravel method.
If it helps anyone, the curl -s https://laravel.build/example-app | bash command downloads several Docker images including MariaDB, Redis, mailhog etc. and therefore needs Docker to be running on the host machine (which is not available inside a container, that's why you can't run the curl command there). Once downloaded, it creates a new container containing (no puns intended) one container for each of these images. You can also customize the list of images/containers that your Laravel application needs by passing the list of services in the curl command like this: curl -s https://laravel.build/example-app?with=mysql,redis. Thanks #apokryfos for the helpful comment. Once these containers are running, you can use VSCode (together with GIT) to connect to them and do your development work.
Of course, you can still use the old-school method of Laravel development. Just install XAMPP or one of its cousins on your machine and then use composer create-project command from the terminal to create it on your local file system. Then host your database and website on locally running instances of Apache and MariaDB.
I have yet to check out Bitnami Laravel image and how it works.
I am trying to connect my Windows 10 Home system to be able to run full Linux OS Docker containers. I have installed Docker on both WSL Ubuntu 18.04 as well as a VMware Ubuntu 18.04.
I was trying to follow this guide.
However, I get stuck trying to configure the Daemon as per the instructions.
Can’t use Docker for Windows?
This is only necessary if you are NOT running Docker for Windows!
No problem, just configure your Docker daemon to use -H tcp://0.0.0.0:2375 and --tlsverify=false. Then you can follow along with the rest of this guide exactly.
If you go down this route, I highly recommend rolling your own VM with VMware Player instead of using the Docker Toolbox because VirtualBox has crazy edge case shared folder bugs that will ruin your life at some point. Don’t worry, VMware Player is free. Just Google how to set up Ubuntu 16 server on VMware Player.
When I try to change the Docker Daemon by making a daemon.json file I get errors. I've also tried editing the .profile files and the .bashrc as per other guides (another guide), with no luck.
I am unable to check the DOCKER_HOST variable on the VM Ubuntu.
Don't make things complicated. In your case, why WSL if you just want to connect to a remote daemon? Why not simply use the windows docker client?
Setup you favorite local VM with docker.
Example: I have installed a CentOS distro running on local VMWare Workstation. All Hyper-V of course deinstalled/deactivated.
In this VM, enable tcp access for the daemon.
If you have a systemd linux distro (like mine CentOS), execute this:
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo echo '[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375' >> /etc/systemd/system/docker.service.d/options.conf
sudo systemctl daemon-reload
sudo systemctl restart docker`
Test if the port is open with docker info. You should get an API access warning at the bottom result.
Download the Windows docker cli zip from here: https://download.docker.com/win/static/stable/x86_64/
Move the docker.exe to any folder, for ex. your Documents folder.
Then put this folder path into your Windows PATH variable.
Set the docker host: Open PowerShell, execute setx DOCKER_HOST <VM-IP>:2375 and close it.
Open a new PowerShell and call docker info.
You should see the docker and daemon infos.
Do what ever you like now... :-)
I'm using Windows 7 Pro and have existing shared Docker engine running on a Linux. I would like to use my workstation (with development environment) to access shared Docker engine.
Does someone know how to retrieve Docker client only for Windows 7 ? I have no admin privilege, so I can't install Docker Toolbox.
Older Clients can be found here https://download.docker.com/win/static/stable/x86_64/
And here a some newer Builds by Stefan Scherer (Docker employee) https://github.com/StefanScherer/docker-cli-builder.
Just download docker.exe and add it to your windows path variable.
Then set your DOCKER_HOST variable to define against which Docker daemon you want to speak.
The official Downloads have finally been published:
Mac CLI binaries are available at
https://download.docker.com/mac/static/stable/
Windows CLI (and daemon) binaries at https://download.docker.com/win/
Linux CLI packages are available for each distro as docker-cli (deb and rpm) packages: https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/
From: https://github.com/docker/cli/issues/2281#issuecomment-947699400
Although docker provides a REST-like API, there aren't many clients for it. A quick google turned up one on github, but ymmv. Even if you did find one, you're likely to run into the same problems involved in running docker-ce locally anyway.
There are a handful of gui clients that you could run on that engine and access with a browser, but if you are specifically after a cli you're SOL with this.
If you have an ssh client (git bash, or putty, or something), and you can arrange to run a bastion container on the engine, then you could run a container to ssh into and use that as if it's your local machine. You'd still have to scp resources onto it, but you'll eventually have to solve that problem anyway. Something like:
sudo docker container run --interactive --tty -v /var/run/docker.sock:/var/run/docker.sock image
Where 'image' is a linux image with an ssh server and appropriate keys, git, and docker installed. You could mount a local volume for persistence, or you could just keep everything in git.
It is possible to install Microsoft CNTK on a macbook? I have OS X El Capitan. The official Microsoft documentation at https://github.com/Microsoft/CNTK/wiki/Setup-CNTK-on-your-machine doesn't provide any information for mac users.
Thank you
As of June 2017, you can only run CNTK on OSX using Docker (which will run a Linux container)
Documentation from Microsoft is available here: https://learn.microsoft.com/en-us/cognitive-toolkit/CNTK-Docker-Containers
If you want to run the CPU version of CNTK (as opposed to a GPU enabled) you'll need to pull a particular version of the docker container. See: https://hub.docker.com/r/microsoft/cntk/
I recommend using the following for CPU CNTK:
docker pull microsoft/cntk:2.0-cpu-python3.5
Once you've pulled the container above, you can use Jupyter Notebooks to look at tutorials etc:
First, run the container:
docker run -d -p 8888:8888 --name cntk-jupyter-notebooks -t microsoft/cntk:2.0-cpu-python3.5
Then run this command:
docker exec -it cntk-jupyter-notebooks bash -c "source /cntk/activate-cntk && jupyter-notebook --no-browser --port=8888 --ip=0.0.0.0 --notebook-dir=/cntk/Tutorials --allow-root"
You'll want to access the shell to run CNTK commands. You can attach a bash shell using docker.
Get your container id
docker ps
Then attach a shell
docker exec -it <container_id> bash
While it might not be supported on Mac directly, you can always use a virtual machine to get around.
You can setup docker in your local environment.
https://docs.docker.com/docker-for-mac/
Follow its documentations on how to install on Docker
https://github.com/Microsoft/CNTK/wiki/CNTK-Docker-Containers
We currently support both Linux and Windows. Mac support is on our ToDo or would be interested in community contribution.
I'm currently building CNTK on a linux machine without root access, installing every dependency with linuxbrew (a fork of homebrew). So I think is possible to build on MacOS natively. You can try building it from source with CNTK linux manual to build from source. Let me know if you have any issue.
I have a local jenkins on OSX.
It's working fine, but now I want to perform docker builds inside the jenkins.
So I have to install docker inside my jenkins.
Is this possible, because I thought you can't install docker on osx? (only docker-machine).
I don't know if it's possible to configure jenkins to use docker-machine?
Docker-machine is a Linux VM.
First option: start a Jenkins slave inside this VM, it will have access to the docker daemon to build your image.
Second option: configure Jenkins to use Docker slaves using Docker-machine. You will be able to build docker images from a docker slave.
I would do this for a PoC or an evaluation setup. For a Production setup, I would use real linux hosts to build Docker images.