pipelining is being ignored by Ansible Tower - ansible

Question

I am trying to enable pipelining but no matter what i'm trying, it doesn't work.
ansible.cfg contents:
[defaults]
transport = ssh
host_key_checking = false
retry_files_enabled = false
nocows=true
remote_user=ansible
# display_skipped_hosts = false
allow_world_readable_tmpfiles = True
warning = False
roles_path = ./roles
ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
callback_whitelist = profile_tasks
[privilege_escalation]
become_method = su
become_exe = 'sx su -'
[ssh_connection]
pipelining = true
ssh_args = -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r
Ansible Tower version: 3.4.1 (by Redhat)
Ansible env version used: 2.9, 2.8 and 2.7
Ansible Tower runs on RHEL 6 and the VM against which i am trying to run the playbook is on RHEL 7.
I do not have access on the Tower VM to check configs.
This is the log of my playbook running:
TASK [pipelining : Check Date] *************************************************
task path: /var/lib/awx/projects/_32459__project/roles/pipelining/tasks/main.yml:1
Friday 06 March 2020 14:46:33 +0000 (0:00:00.210) 0:00:00.210 **********
<myVM> ESTABLISH SSH CONNECTION FOR USER: ansible
<myVM> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 myVM '/bin/sh -c '"'"'echo ~ansible && sleep 0'"'"''
<myVM> (0, '/home/ansible\n', "Warning: Permanently added 'myVM,10.37.30.170' (RSA) to the list of known hosts.")
<myVM> ESTABLISH SSH CONNECTION FOR USER: ansible
<myVM> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 myVM '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658 `" && echo ansible-tmp-1583505993.07-223021018622658="` echo /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658 `" ) && sleep 0'"'"''
<myVM> (0, 'ansible-tmp-1583505993.07-223021018622658=/home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658\n', '')
<VMhost> Attempting python interpreter discovery
<myVM> ESTABLISH SSH CONNECTION FOR USER: ansible
<myVM> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 myVM '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<myVM> (0, 'PLATFORM\nLinux\nFOUND\n/usr/bin/python\n/usr/bin/python2.7\n/usr/libexec/platform-python\n/usr/bin/python\nENDFOUND\n', '')
<myVM> ESTABLISH SSH CONNECTION FOR USER: ansible
<myVM> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 myVM '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<myVM> (0, '{"osrelease_content": "NAME=\\"Red Hat Enterprise Linux Server\\"\\nVERSION=\\"7.7 (Maipo)\\"\\nID=\\"rhel\\"\\nID_LIKE=\\"fedora\\"\\nVARIANT=\\"Server\\"\\nVARIANT_ID=\\"server\\"\\nVERSION_ID=\\"7.7\\"\\nPRETTY_NAME=\\"Red Hat Enterprise Linux Server 7.7 (Maipo)\\"\\nANSI_COLOR=\\"0;31\\"\\nCPE_NAME=\\"cpe:/o:redhat:enterprise_linux:7.7:GA:server\\"\\nHOME_URL=\\"https://www.redhat.com/\\"\\nBUG_REPORT_URL=\\"https://bugzilla.redhat.com/\\"\\n\\nREDHAT_BUGZILLA_PRODUCT=\\"Red Hat Enterprise Linux 7\\"\\nREDHAT_BUGZILLA_PRODUCT_VERSION=7.7\\nREDHAT_SUPPORT_PRODUCT=\\"Red Hat Enterprise Linux\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\"7.7\\"\\n", "platform_dist_result": ["redhat", "7.7", "Maipo"]}\n', '')
Using module file /var/lib/awx/venv/ansible_2_8/lib/python2.7/site-packages/ansible/modules/commands/command.py
<myVM> PUT /var/lib/awx/.ansible/tmp/ansible-local-3omzq9J/tmpquCP2N TO /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658/AnsiballZ_command.py
<myVM> SSH: EXEC sshpass -d10 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 '[myVM]'
<myVM> (0, 'sftp> put /var/lib/awx/.ansible/tmp/ansible-local-3omzq9J/tmpquCP2N /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658/AnsiballZ_command.py\nUploading /var/lib/awx/.ansible/tmp/ansible-local-3omzq9J/tmpquCP2N to /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658/AnsiballZ_command.py\n', '')
<myVM> ESTABLISH SSH CONNECTION FOR USER: ansible
<myVM> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 myVM '/bin/sh -c '"'"'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658/ /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658/AnsiballZ_command.py && sleep 0'"'"''
<myVM> (0, '', '')
<myVM> ESTABLISH SSH CONNECTION FOR USER: ansible
<myVM> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 -tt myVM '/bin/sh -c '"'"'sx su - root -c '"'"'"'"'"'"'"'"'/bin/sh -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-iaimhvmledvpyuy ; /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658/AnsiballZ_command.py'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<myVM> (0, 'Last login: Fri Mar 6 15:12:26 2020 from 10.77.228.130\r\r\n\r\n{"changed": true, "end": "2020-03-06 15:46:35.478978", "stdout": "Fri Mar 6 15:46:35 CET 2020", "cmd": "date", "rc": 0, "start": "2020-03-06 15:46:35.475245", "stderr": "", "delta": "0:00:00.003733", "invocation": {"module_args": {"creates": null, "executable": null, "_uses_shell": true, "strip_empty_ends": true, "_raw_params": "date", "removes": null, "argv": null, "warn": true, "chdir": null, "stdin_add_newline": true, "stdin": null}}}\r\n', 'Shared connection to myVM closed.\r\n')
<myVM> ESTABLISH SSH CONNECTION FOR USER: ansible
<myVM> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=600s -o ControlPath=/tmp/ansible-ssh-%%h-%%p-%%r -o StrictHostKeyChecking=no -o 'User="ansible"' -o ConnectTimeout=10 myVM '/bin/sh -c '"'"'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1583505993.07-223021018622658/ > /dev/null 2>&1 && sleep 0'"'"''
<myVM> (0, '', '')
I was expecting to see only one SSH: EXEC and no PUT.
Am i doing something wrong? I did try mitogen and it works, except the Become part, so i can't use it. I was hoping to be able to enable pipelining instead.
Another thing to mention, the RHEL7 VM, where the playbook will check the date, write a file and delete it, has this line in /etc/sudoers: Defaults !requiretty
I also tried using vars per task and set ansible_ssh_pipelining: true but with the same result.
Could the pipelining be disabled on a higher level in Tower? Am i tired and doing mistakes? I am going crazy with this...

Answer 1

The reason is the usage of "su" as a become_method. It seems that it doesn't support pipelining.
Following comment can be found in the code:
# su does not work with pipelining
https://github.com/ansible/ansible/blob/v2.10.3/lib/ansible/plugins/action/init.py
Ref: https://github.com/ansible/ansible/issues/35698