ICP 2.1 Beta 2 can't create Cloudant DB fast enough - ibm-cloud-private

Question

When running the following command from the installation instructions:
sudo docker run --net=host -t -e LICENSE=accept -v $(pwd):/installer/cluster \
ibmcom/icp-inception:2.1.0-beta-2-ee install
The task to create the Cloudant DB times out:
TASK [master : Ensuring that the Cloudant Database is ready] ***********************************************************************************************************************************************
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (20 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (19 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (18 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (17 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (16 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (15 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (14 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (13 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (12 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (11 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (10 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (9 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (8 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (7 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (6 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (5 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (4 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (3 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (2 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (1 retries left).
fatal: [10.20.30.29] => Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>
PLAY RECAP *************************************************************************************************************************************************************************************************
10.20.30.29 : ok=149 changed=57 unreachable=0 failed=1
The container is still using CPU in /usr/bin/python /usr/bin/ansible-playbook -e #cluster/config.yaml playbook/site.yaml so I think it's still installing. How do I increase the number of retries?

Answer 1

Run the command
docker info | grep -i cgroup
and you should see
Cgroup Driver: systemd
...so matched that in ICP by adding the line
kubelet_extra_args: ["--cgroup-driver=systemd"]
to ICP's config.yaml

Answer 2

This usually means cloudant db takes too long to be ready and by that time, 20 times retry limit has been reached. Uninstall ICP 2.1 beta2 and install it usually gets past this issue. In the ICP 2.1 beta3, it will have an increased timeout.
If this does not work, you can check the following:
docker exec -it <cloudant container> bash
now inside the container, to check the database status:
cast cluster status -p $ADMIN_PASSWORD
Also can check:
docker logs <long cloudant container name>
Before installing ICP beta2 again, check whether
docker volume ls
was cleaned up.

Answer 3

In some cases, we find that the size of the ICP datastore, which is based on Cloudant, takes awhile to download. Have you considered pre-pulling the image onto your host?
docker pull ibmcom/icp-datastore:{version}
You would need to do this on each master (but not on the worker nodes).