EMR failed to bootstrap Airflow - bash

Question

I'm creating a bash to provide the enviroment to execute airflow, but for some reason the script don't work as well.
If I provide first the EMR and after that execute the script, it work's fine. But if I use the script to execute in custom activity don't work. I tried to change the commands to execute like sudo, but still not working.
#!/bin/bash
# check for master node
IS_MASTER=true
if [ -f /mnt/var/lib/info/instance.json ]
then
IS_MASTER=`cat /mnt/var/lib/info/instance.json | tr -d '\n ' | sed -n 's|.*\"isMaster\":\([^,]*\).*|\1|p'`
fi
if [ "$IS_MASTER" = "true}" ];
then
# install mysql jdbc driver on sqoop
wget -qN -O ~/mysql-connector-java-5.1.39.tar.gz "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.39.tar.gz"
tar -zxvf ~/mysql-connector-java-5.1.39.tar.gz && rm ~/mysql-connector-java-5.1.39.tar.gz
sudo mv ~/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar /usr/lib/sqoop/lib
sudo chmod 744 /usr/lib/sqoop/lib/mysql-connector-java-5.1.39-bin.jar
aws s3 cp s3://monet-datapipeline/scripts/emr_boostrap_scripts/airflow_boostrap ~/ --recursive --exclude "*.sh"
#create enviroment for airflow
virtualenv airflowenv -p python3
source ~/airflowenv/bin/activate
pip install --upgrade pip
pip install airflow
pip install boto3
airflow initdb
mv ~/carriola/airflow.cfg ~/airflow
airflow webserver -p 9030
airflow scheduler
fi
This is the code error.
The stderr from master node.
mv: cannot stat ‘/home/hadoop/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar’: No such file or directory
chmod: cannot access ‘/usr/lib/sqoop/lib/mysql-connector-java-5.1.39-bin.jar’: No such file or directory
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 25: /home/hadoop/airflowenv/bin/activate: No such file or directory
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run
root=options.root_path,
File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 687, in install
requirement.uninstall(auto_confirm=True)
File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 730, in uninstall
paths_to_remove.remove(auto_confirm)
File "/usr/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 126, in remove
renames(path, new_path)
File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 292, in renames
shutil.move(old, new)
File "/usr/lib64/python2.7/shutil.py", line 303, in move
os.unlink(src)
OSError: [Errno 13] Permission denied: '/usr/bin/pip'
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Command "/usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/mnt/tmp/pip-build-rmGy3J/sqlalchemy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-J6Ft9n-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /mnt/tmp/pip-build-rmGy3J/sqlalchemy
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run
root=options.root_path,
File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 693, in install
**kwargs
File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 817, in install
self.move_wheel_files(self.source_dir, root=root)
File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 1018, in move_wheel_files
isolated=self.isolated,
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 237, in move_wheel_files
clobber(source, lib_dir, True)
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 208, in clobber
os.makedirs(destdir)
File "/usr/lib64/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/site-packages/s3transfer'
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 31: airflow: command not found
mv: cannot stat ‘/home/hadoop/carriola/airflow.cfg’: No such file or directory
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 35: airflow: command not found
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 37: airflow: command not found

Answer 1

You can use:
cat /mnt/var/lib/info/instance.json | jq .isMaster
instead to find isMaster true or not.