AWX fails to retrieve collections from Ansible Galaxy - ansible

Question

I have a project with roles/requirements.yml and collections/requirements.yml. This - I think - per AWX / Tower guidelines.
On project update the roles/requirements.yml is picked up successfully but the collections/requirements.yml fails with a typical Ansible error message unfortunately that doesn't help me understand what is going wrong.
PLAY [Update source tree if necessary] *****************************************
TASK [delete project directory before update] **********************************
changed: [localhost]
TASK [update project using git] ************************************************
changed: [localhost]
TASK [Set the git repository version] ******************************************
ok: [localhost]
TASK [Repository Version] ******************************************************
ok: [localhost] => {
"msg": "Repository Version 812a488b0a09f7fda6a784a8247abaf6892e1f13"
}
PLAY [Install content with ansible-galaxy command if necessary] ****************
TASK [detect roles/requirements.(yml/yaml)] ************************************
ok: [localhost] => (item={'ext': '.yml'})
ok: [localhost] => (item={'ext': '.yaml'})
TASK [fetch galaxy roles from requirements.(yml/yaml)] *************************
changed: [localhost] => (item={'changed': False, 'stat': {'exists': True, 'path': '/var/lib/awx/projects/_6__c2/roles/requirements.yml', 'mode': '0644', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 390, 'inode': 131356, 'dev': 64771, 'nlink': 1, 'atime': 1613038457.4215767, 'mtime': 1613038456.678576, 'ctime': 1613038456.678576, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'root', 'gr_name': 'root', 'checksum': '29ae537db56b12b54f2eecb8668f4185b7b1e91d', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': None, 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/var/lib/awx/projects/_6__c2/roles/requirements.yml', 'follow': False, 'get_md5'…
skipping: [localhost] => (item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/var/lib/awx/projects/_6__c2/roles/requirements.yaml', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': {'ext': '.yaml'}, 'ansible_loop_var': 'item'})
TASK [detect collections/requirements.(yml/yaml)] ******************************
ok: [localhost] => (item={'ext': '.yml'})
ok: [localhost] => (item={'ext': '.yaml'})
TASK [fetch galaxy collections from collections/requirements.(yml/yaml)] *******
failed: [localhost] (item={'changed': False, 'stat': {'exists': True, 'path': '/var/lib/awx/projects/_6__c2/collections/requirements.yml', 'mode': '0644', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 183, 'inode': 131286, 'dev': 64771, 'nlink': 1, 'atime': 1613038457.4195766, 'mtime': 1613038456.673576, 'ctime': 1613038456.673576, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'root', 'gr_name': 'root', 'checksum': '7b8e2a2b64c7d5e64917b548dc40347158f8bb1b', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': None, 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/var/lib/awx/projects/_6__c2/collections/requirements.yml', 'follow': False, '…
skipping: [localhost] => (item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/var/lib/awx/projects/_6__c2/collections/requirements.yaml', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': {'ext': '.yaml'}, 'ansible_loop_var': 'item'})
Of course I tested the requirements locally. It works. Just not in AWX.
With the exception of one task having status failed I really don't see any difference between the error message on failure and the success logging when fetches roles.
What is wrong here?

Answer 1

I just came across this question since I encountered the same situation. I was able to resolve my issue and would like to share my approach. My AWX version is 17.1.0 (Docker image).
How to identify the root cause
enable the -vvv option as sadok-f suggested (see detailed instructions below)
open the job execution logs
click on the logline of the failing task (a small pop-up appears)
click the JSON tab (or Standard Error - Most details are provided in the JSON tab)
read the significantly more descriptive error message
My issue was, that I had a typo in the Ansible Galaxy credentials. Therefore my dependencies within the requirements.yml were not installable.
How to increase the verbosity for project jobs
As mentioned by sadok-f, enable the -vvv option for more verbosity. That option is a bit hidden. You can see the current configuration at Settings > Jobs settings. Now look for "Run Project Updates With Higher Verbosity" (the project update triggers the installation of requirements.yml files).
At least for AWX 17.1.0, editing the settings via the GUI is not implemented. You need to patch the settings via AWX-API to enable the project update-job verbosity. To do so, visit the AWX-API:
<your-awx-fqdn>/api/v2/settings/jobs/
At the bottom of the page, you have a small window that allows you to modify the content. Within the content box look for:
"PROJECT_UPDATE_VVV": false,
and change it to:
"PROJECT_UPDATE_VVV": true,
Click the PATCH button. That enables the -vvv option for project refresh jobs. As written, those refresh jobs install the requirements.
Synch your project and have a look at the significantly more verbose job logs (Projects > Sync Project (the tiny cycle-arrow button)).