OpenShift Ansible-based operator hangs inconsistently - ansible

Question

I have an Ansible-based operator running within an OpenShift 4.2 cluster.
Most of times when I apply the relevant CR, the operator runs perfectly.
Occasionally though the operator hangs without reporting any further logs.
The step where this happens is the same however the problem is that this happens inconsistently without any other factors involved and I am not sure how to diagnose it.
Restarting the operator always resolves the issue, but I wonder if there's anything I could do to diagnose it and prevent this from happening altogether?
- name: allow Pods to reference images in myproject project
k8s:
definition:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: "system:image-puller-{{ meta.name }}"
namespace: myproject
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:image-puller
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: "system:serviceaccounts:{{ meta.name }}"
The operator's logs simply hang right after the above step and right before the following step:
- name: fetch some-secret
set_fact:
some_secret: "{{ lookup('k8s', kind='Secret', namespace='myproject', resource_name='some-secret') }}"
oc describe is as follows
oc describe -n openshift-operators pod my-ansible-operator-849b44d6cc-nr5st
Name: my-ansible-operator-849b44d6cc-nr5st
Namespace: openshift-operators
Priority: 0
PriorityClassName: <none>
Node: worker1.openshift.mycompany.com/10.0.8.21
Start Time: Wed, 10 Jun 2020 22:35:45 +0100
Labels: name=my-ansible-operator
pod-template-hash=849b44d6cc
Annotations: k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.254.20.128"
],
"default": true,
"dns": {}
}]
Status: Running
IP: 10.254.20.128
Controlled By: ReplicaSet/my-ansible-operator-849b44d6cc
Containers:
ansible:
Container ID: cri-o://63b86ddef4055be4bcd661a3fcd70d525f9788cb96b7af8dd383ac08ea670047
Image: image-registry.openshift-image-registry.svc:5000/openshift-operators/my-ansible-operator:v0.0.1
Image ID: image-registry.openshift-image-registry.svc:5000/openshift-operators/my-ansible-operator#sha256:fda68898e6fe0c61760fe8c50fd0a55de392e63635c5c8da47fdb081cd126b5a
Port: <none>
Host Port: <none>
Command:
/usr/local/bin/ao-logs
/tmp/ansible-operator/runner
stdout
State: Running
Started: Wed, 10 Jun 2020 22:35:56 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/tmp/ansible-operator/runner from runner (ro)
/var/run/secrets/kubernetes.io/serviceaccount from my-ansible-operator-token-vbwlr (ro)
operator:
Container ID: cri-o://365077a3c1d83b97428d27eebf2f0735c9d670d364b16fad83fff5bb02b479fe
Image: image-registry.openshift-image-registry.svc:5000/openshift-operators/my-ansible-operator:v0.0.1
Image ID: image-registry.openshift-image-registry.svc:5000/openshift-operators/my-ansible-operator#sha256:fda68898e6fe0c61760fe8c50fd0a55de392e63635c5c8da47fdb081cd126b5a
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 10 Jun 2020 22:35:57 +0100
Ready: True
Restart Count: 0
Environment:
WATCH_NAMESPACE: openshift-operators (v1:metadata.namespace)
POD_NAME: my-ansible-operator-849b44d6cc-nr5st (v1:metadata.name)
OPERATOR_NAME: my-ansible-operator
ANSIBLE_GATHERING: explicit
Mounts:
/tmp/ansible-operator/runner from runner (rw)
/var/run/secrets/kubernetes.io/serviceaccount from my-ansible-operator-token-vbwlr (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
runner:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
my-ansible-operator-token-vbwlr:
Type: Secret (a volume populated by a Secret)
SecretName: my-ansible-operator-token-vbwlr
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Is there anything else I could do to diagnose the problem further or prevent the operator from hanging occasionally?

Answer 1

I found a very similar issue in the operator-sdk repository, linking to the root cause in the Ansible k8s module:
Ansible 2.7 stuck on Python 3.7 in docker-ce
From the discussion in the issue it seems that the problem is related to tasks that do not time out and the current workaround seems to be:
For now we just override ansible local connection and normal action plugins, so:
all communicate() calls have 60 second timeout
all raised TimeoutExpired exceptions are retried a few times
Can you check if this resolves your issue? As the issue is still "open", you might need to reach out to the issue as well.