Install packages in conda terminal jupyter amazon sagemaker - terminal

Question

I have a project in Amazon-Sage-Maker. For this, I have to uninstall specific packages and install others in the terminal. But every time I close or stop the instance I have to go to the terminal and make all the installations again. Why is this happening?
The package with which I am experimenting with this trouble is psycopg2:
import psycopg2
Gives me a warning that suggests that I should uninstall it and install psycopg2-binary.
So I open the terminal and code:
pip uninstall psycopg2
Then in the notebook, I code:
import psycopg2
And have no problem, but if I close and open the instance back, I get the same error and have to go through all the process again.

Answer 1

Thanks for using SageMaker. The packages installed are not persistent when you restart the Notebook Instance. To avoid manually installing it every time, you can create a Lifecycle Config which installs your packages and attach it to you Notebook Instance. Script in Lifecycle Config will be run every time you restart your Notebook Instance.
For more information on how to use Lifecycle Config you can check out:
https://aws.amazon.com/blogs/machine-learning/customize-your-amazon-sagemaker-notebook-instances-with-lifecycle-configurations-and-the-option-to-disable-internet-access/

Answer 2

#anitasp, You have to create a Docker image, by doing the following:
Be sure to set up your SageMaker Execution Role Policy permissions on AWS IAM (besides S3) and also AmazonEC2ContainerServiceFullAccess, AmazonEC2ContainerRegistryFullAccess and AmazonSageMakerFullAccess.
Create and start instance in SageMaker and Open notebook. Clone the directory structure shown here at your instance: https://github.com/RubensZimbres/Repo-2018/tree/master/AWS%20SageMaker/Jupyter-Folder
Inside Jupyter, run:
! sudo service docker start
! sudo usermod -a -G docker ec2-user
! docker info
! chmod +x decision_trees/train
! chmod +x decision_trees/serve
! aws ecr create-repository --repository-name decision-trees
! aws ecr get-login --no-include-email
Copy and paste the login in the command line below
! docker login -u abc -p abc12345 http://abc123
Run
! docker build -t decision-trees .
! docker tag decision-trees your_aws_account_id.dkr.ecr.us-east-1.amazonaws.com/decision-trees:latest
! docker push your_aws_account_id.dkr.ecr.us-east-1.amazonaws.com/decision-trees:latest
! aws ecs register-task-definition --cli-input-json file://decision-trees-task-def.json
And adapt to your needs, according to the algorithm of your choice. You will need the Dockerfile, hyperparameters.json, etc.
The documented project is here: https://github.com/RubensZimbres/Repo-2018/tree/master/AWS%20SageMaker

Answer 3

By default, python packages installed from a Notebook Instance will not be persisted to the next Notebook Instance session. One solution for this problem is to:
1) Create (or clone from a current conda env) a new conda environment into /home/ec2-user/SageMaker, which is persisted between sessions. For example:
conda create --prefix /home/ec2-user/SageMaker/envs/custom-environment --clone tensorflow_p36
2) Next, create a new Lifecycle Configuration for “start notebook” with the following contents:
#!/bin/bash
sudo -u ec2-user -i <<'EOF'
ln -s /home/ec2-user/SageMaker/envs/custom-environment /home/ec2-user/anaconda3/envs/custom-environment
EOF
3) Finally, attach the Lifecycle Configuration to your Notebook Instance
Now, when you restart your Notebook Instance, your custom environment will be detected by conda and Jupyter. Any new packages you install to this environment will be persisted between sessions and then soft-linked back to conda at startup.