Permission error when deploying Docker app from CI server to Heroku - heroku

Question

The problem
I'm trying to deploy a Dockerized app from my CI server (Circle) to Heroku.
When I run heroku docker:release --app <app_name>, CI logs show the following:
Error response from daemon: permission denied
! Error in heroku-docker:
! Command failed: docker cp 93a67016632af5cffacab9817b741c4dc69b41f0f6d324c8bae53fe98f4e621d:/tmp/slug.tgz /tmp
However, I can run this command from my local machine just fine. The API key I have provided to Circle is the same as that from my local user. Also, I can run other Heroku commands, such as heroku logs --app <app_name>, from the CI server just fine.
Why are there permission errors?
Extra background info
It appears the only way Heroku supports Docker is using the Docker Toolbelt, with the Docker plugin (heroku docker:release --app <app_name>). At the same time, the recommended way to deploy to Heroku in CircleCI is using a special entry in the proprietary circle.yml file. However, this doesn't support Docker, which is why I'm choosing the heroku docker:release method.

Answer 1

We have seen dockerd report permission denied when the actual error is disk fault or security service preventing access.
Look at /var/log/messages or other system log while doing the docker command that fails.
e.g. look at /var/log/messages while doing docker cp command.
$ docker cp b54433de86ec:/thing ./
Error response from daemon: permission denied
E.g. you might see errors related to vmount
Nov 17 10:57:45 machine dockerd: time="2021-11-17T10:57:45.063942786Z" level=warning msg="b54433de86ecd8426c27e8c8fa7e6a785fa8e513a5133d3a41b163308f0d59a5 unmountVolumes: Failed to do lazy umount fo volume '/logs/docker/overlay/dd77d1edef54cdd4040b8f4c32afca86f7c3870f3b3b353063b406d2c3409220/merged/x/xx/xxx/xxxx': no such file or directory"
Nov 17 10:57:45 machine dockerd: time="2021-11-17T10:57:45.099051149Z" level=error msg="Handler for GET /v1.39/containers/b54433de86ec/archive returned error: permission denied"
possibly disk was corrupt or had a problem
or another example:
Nov 17 10:57:48 machine dbus[719]: [system] Activating service name='org.fedoraproject.Setroubleshootd' (using servicehelper)
Nov 17 10:57:48 machine dbus[719]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Nov 17 10:57:49 machine setroubleshoot: SELinux is preventing /usr/bin/dockerd-ce from mounton access on the directory /logs/docker/overlay/dd77d1edef54cdd4040b8f4c32afca86f7c3870f3b3b353063b406d2c3409220/merged/host/net. For complete SELinux messages run: sealert -l efaad8a5-f04f-4783-9235-7048c915daa3
Nov 17 10:57:49 machine python: SELinux is preventing /usr/bin/dockerd-ce from mounton access on the directory /logs/docker/overlay/dd77d1edef54cdd4040b8f4c32afca86f7c3870f3b3b353063b406d2c3409220/merged/host/net.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that dockerd-ce should be allowed mounton access on the net directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'dockerd' --raw | audit2allow -M my-dockerd#012# semodule -i my-dockerd.pp#012
Selinux is preventing disk access for dockerd process.
So follow the suggested action:
root# ausearch -c 'dockerd' --raw | audit2allow -M my-dockerd
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i my-dockerd.pp
root# semodule -i my-dockerd.pp
Now the docker cp command worked on this system.