Ansible hangs on job submission step Ansible ZOS Core - ansible

Question

I'm using ansible on zlinux. I have a playbook that is using the zos_job_submit command from the zos ansible core modules.
The module is used with a job that generates random data to the jes spool.
//SPOOL1 JOB (UU999999999,1103),'DART JOB',CLASS=0,
// REGION=0M,MSGCLASS=R,TIME=5, LINES=(999999,WARNING),
// NOTIFY=&SYSUID
//* Automatic process will kill the job and cleanup spool.
//* author: xxxxxx , xxxxxxxxx
//STEPNAME EXEC PGM=BPXBATCH
//STDERR DD SYSOUT=*
//STDOUT DD SYSOUT=*
//STDPARM DD *
SH cat /dev/urandom
enter code here
This was working fine until a few days ago when It started to freeze up and error out. It still submits the job but it fails to return its output after the job starts running and then errors out.
Here is the playbook I'm using ( stripped down to only the offending task)
# Author: xxxxxxxxxxxxxx
- name: "DART JES CHAOS EVENT"
hosts: all # WARNING: USE WITH --LIMIT <target> OTHERWISE ALL HOSTS IN INVENTORY WILL BE TARGETED!
vars:
all_jobs:
jobs: [ ]
jobs_file_location: "jobs/{{inventory_hostname}}"
tasks:
- name: "Submit job tasks"
block:
- name: Submit job
ibm.ibm_zos_core.zos_job_submit:
src: "{{uss_jcl_path}}"
location: LOCAL
wait: false
vars:
uss_jcl_path: "{{jcl_lib}}/{{job_jcl}}"
Here is the log using -vvv
ansible-playbook 2.9.27
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/adshome/svc.dart/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /bin/ansible-playbook
python version = 2.7.5 (default, May 27 2022, 07:27:39) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
Using /etc/ansible/ansible.cfg as config file
PLAYBOOK: debug_event.yml **************************************************************************************************************************************************
Positional arguments: /XXXX/dart/ansible/playbooks/debug_event.yml
subset: nwrd
become_method: sudo
inventory: (u'/XXXX/dart/ansible/inventory', u'/XXXX/dart/ansible/auth_inventory')
forks: 5
tags: (u'all',)
extra_vars: (u'jcl_lib=/XXXX/dart/ansible/playbooks/jcl job_jcl=SPOOL',)
verbosity: 4
connection: smart
timeout: 10
1 plays in /XXXX/dart/ansible/playbooks/debug_event.yml
PLAY [DART JES CHAOS EVENT] ************************************************************************************************************************************************
TASK [Submit job] **********************************************************************************************************************************************************
task path: /XXXX/dart/ansible/playbooks/debug_event.yml:12
Using module file /usr/lib/python2.7/site-packages/ansible/modules/files/tempfile.py
Pipelining is enabled.
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'/usr/lpp/izoda/anaconda/bin/python3 && sleep 0'"'"''
<XXXX.XXX.COM> (0, '\n{"changed": true, "path": "/tmp/ansible.nzzb29wz", "uid": 10001120, "gid": 7212, "owner": "XXXXXXX", "group": "GLTCMF", "mode": "0600", "state": "file", "size": 0, "invocation": {"module_args": {"state": "file", "prefix": "ansible.", "suffix": "", "path": null}}}\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'echo ~XXXXXXX && sleep 0'"'"''
<XXXX.XXX.COM> (0, '/u/XXXXXXX\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /u/XXXXXXX/.ansible/tmp `"&& mkdir "` echo /u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409 `" && echo ansible-tmp-1658270106.84-62001-187469557526409="` echo /u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409 `" ) && sleep 0'"'"''
<XXXX.XXX.COM> (0, 'ansible-tmp-1658270106.84-62001-187469557526409=/u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/site-packages/ansible/modules/files/stat.py
Pipelining is enabled.
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'/usr/lpp/izoda/anaconda/bin/python3 && sleep 0'"'"''
<XXXX.XXX.COM> (0, '\n{"changed": false, "stat": {"exists": true, "path": "/tmp/ansible.nzzb29wz", "mode": "0600", "isdir": false, "ischr": false, "isblk": false, "isreg": true, "isfifo": false, "islnk": false, "issock": false, "uid": 10001120, "gid": 7212, "size": 0, "inode": 10691, "dev": 3248, "nlink": 1, "atime": 1658270106, "mtime": 1658270106, "ctime": 1658270106, "wusr": true, "rusr": true, "xusr": false, "wgrp": false, "rgrp": false, "xgrp": false, "woth": false, "roth": false, "xoth": false, "isuid": false, "isgid": false, "blocks": 0, "block_size": 4096, "device_type": 0, "readable": true, "writeable": true, "executable": false, "pw_name": "XXXXXXX", "gr_name": "GLTCMF", "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "mimetype": "unknown", "charset": "unknown", "version": null, "attributes": [], "attr_flags": ""}, "invocation": {"module_args": {"checksum_algorithm": "sha1", "get_checksum": true, "follow": false, "path": "/tmp/ansible.nzzb29wz", "get_md5": false, "get_mime": true, "get_attributes": true}}}\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<XXXX.XXX.COM> PUT /XXXX/dart/ansible/playbooks/jcl/SPOOL TO /u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source
<XXXX.XXX.COM> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 '[XXXX.XXX.COM]'
<XXXX.XXX.COM> (0, 'sftp> put /XXXX/dart/ansible/playbooks/jcl/SPOOL /u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename#openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs#openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs#openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink#openssh.com" revision 1\r\ndebug2: Server supports extension "fsync#openssh.com" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /u/XXXXXXX size 0\r\ndebug3: Looking up /XXXX/dart/ansible/playbooks/jcl/SPOOL\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:395\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 395 bytes at 0\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'chmod u+x /u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/ /u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source && sleep 0'"'"''
<XXXX.XXX.COM> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/site-packages/ansible/modules/files/copy.py
Pipelining is enabled.
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'/usr/lpp/izoda/anaconda/bin/python3 && sleep 0'"'"''
<XXXX.XXX.COM> (0, '\n{"dest": "/tmp/ansible.nzzb29wz", "src": "/u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source", "md5sum": "e18be97081c5a8d8ae1e1c57eb0d2123", "checksum": "b220730079e63acfee97f2b694ae2d31d3074083", "changed": true, "uid": 10001120, "gid": 7212, "owner": "XXXXXXX", "group": "GLTCMF", "mode": "0600", "state": "file", "size": 395, "invocation": {"module_args": {"src": "/u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source", "dest": "/tmp/ansible.nzzb29wz", "_original_basename": "SPOOL", "mode": "0600", "backup": false, "force": true, "follow": false, "unsafe_writes": false, "content": null, "validate": null, "directory_mode": null, "remote_src": null, "local_follow": null, "checksum": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null}}}\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /adshome/svc.dart/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py
Pipelining is enabled.
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'/usr/lpp/izoda/anaconda/bin/python3 && sleep 0'"'"''
<XXXX.XXX.COM> (1, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\nTraceback (most recent call last):\n File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 800, in run_module\n File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/module_utils/job.py", line 58, in job_output\n File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/module_utils/job.py", line 73, in _get_job_output\nRuntimeError: Failed to retrieve job output. RC: -9 Error: \n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 894, in <module>\n File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 890, in main\n File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 805, in run_module\nIndexError: tuple index out of range\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')
<XXXX.XXX.COM> Failed to connect to the host via ssh: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 60: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 4 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 61998
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
Traceback (most recent call last):
File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 800, in run_module
File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/module_utils/job.py", line 58, in job_output
File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/module_utils/job.py", line 73, in _get_job_output
RuntimeError: Failed to retrieve job output. RC: -9 Error:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 102, in <module>
File "<stdin>", line 94, in _ansiballz_main
File "<stdin>", line 40, in invoke_module
File "/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py", line 205, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 894, in <module>
File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 890, in main
File "/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py", line 805, in run_module
IndexError: tuple index out of range
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
<XXXX.XXX.COM> ESTABLISH SSH CONNECTION FOR USER: XXXXXXX
<XXXX.XXX.COM> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'rm -f -r /u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/ > /dev/null 2>&1 && sleep 0'"'"''
<XXXX.XXX.COM> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
fatal: [XXXX.XXX.COM]: FAILED! => {
"changed": true,
"checksum": "b220730079e63acfee97f2b694ae2d31d3074083",
"dest": "/tmp/ansible.nzzb29wz",
"gid": 7212,
"group": "GLTCMF",
"invocation": {
"module_args": {
"_original_basename": "SPOOL",
"attributes": null,
"backup": false,
"checksum": null,
"content": null,
"delimiter": null,
"dest": "/tmp/ansible.nzzb29wz",
"directory_mode": null,
"follow": false,
"force": true,
"group": null,
"local_follow": null,
"mode": "0600",
"owner": null,
"regexp": null,
"remote_src": null,
"selevel": null,
"serole": null,
"setype": null,
"seuser": null,
"src": "/u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source",
"unsafe_writes": false,
"validate": null
}
},
"md5sum": "e18be97081c5a8d8ae1e1c57eb0d2123",
"mode": "0600",
"module_stderr": "OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 60: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 61998\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\nTraceback (most recent call last):\n File \"/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py\", line 800, in run_module\n File \"/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/module_utils/job.py\", line 58, in job_output\n File \"/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/module_utils/job.py\", line 73, in _get_job_output\nRuntimeError: Failed to retrieve job output. RC: -9 Error: \n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lpp/izoda/anaconda/lib/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py\", line 894, in <module>\n File \"/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py\", line 890, in main\n File \"/tmp/ansible_ibm.ibm_zos_core.zos_job_submit_payload_aplq15qp/ansible_ibm.ibm_zos_core.zos_job_submit_payload.zip/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_job_submit.py\", line 805, in run_module\nIndexError: tuple index out of range\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"owner": "XXXXXXX",
"rc": 1,
"size": 395,
"src": "/u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source",
"state": "file",
"uid": 10001120
}
PLAY RECAP *****************************************************************************************************************************************************************
XXXX.XXX.COM : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

Answer 1

I have a few comments and hopefully we can resolve this.
There are a few things that confuse me and make me wonder how it ever worked else I am missing some data points.
The playbook is using option location: LOCAL and the src: appears to be a directory in USS (u/XXXXXXX/.ansible/tmp/ansible-tmp-1658270106.84-62001-187469557526409/source), the option LOCAL is meant for files located on the control node (where Ansible engine is running, in your case zLinux), can you confirm that you do have the file on the control node?
When I look at the verbose environment vars passed to the managed node (z/OS) I see no environment vars for the required dependency ZOAU. The zos_job_submit module uses the ZOAU APIs in this module; I am also questioning how this worked before without the environment vars or has your vars been corrupted (see below)?
SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/XXXX/dart/ansible/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="XXXXXXX"' -o ConnectTimeout=10 -o ControlPath=/adshome/svc.dart/.ansible/cp/7924694a90 XXXX.XXX.COM '/bin/sh -c '"'"'/usr/lpp/izoda/anaconda/bin/python3 && sleep 0'"'"''
I am expecting environment vars similar to this:
environment_vars:
_BPXK_AUTOCVT: "ON"
ZOAU_HOME: "{{ ZOAU }}"
PYTHONPATH: "{{ ZOAU }}/lib"
LIBPATH: "{{ ZOAU }}/lib:{{ PYZ }}/lib:/lib:/usr/lib:."
PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:/bin:/var/bin"
_CEE_RUNOPTS: "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
_TAG_REDIR_ERR: "txt"
_TAG_REDIR_IN: "txt"
_TAG_REDIR_OUT: "txt"
LANG: "C"
Based on the trace it looks like you are using the latest Ansible Core collection version 1.4.0-beta.1 yet you are using Anaconda Python 3.6 from Rocket, that version of Python has long been removed from support.
For your version of the collection you will want to reference these requirements and probably start with a simple example to ensure its functional before doing something more complex , this is a good one to start with.
I am happy to help but supporting what appears to be a mismatch of requirements can be challenging.