/actuator/health does not detect stopped binders in Spring Cloud Stream - spring-boot

Question

We are using Spring Cloud Streams with multiple bindings based on Kafka Streams binders.
The output of /actuator/health correctly lists all our bindings and their state (RUNNING) - see example below.
Our expectation was, when a binding is stopped using
curl -d '{"state":"STOPPED"}' -H "Content-Type: application/json" -X POST http://<host>:<port>/actuator/bindings/mystep1,
it is still listed, but with threadState = NOT_RUNNING or SHUTDOWN and the overall health status is DOWN.
This is not the case!. After stopping a binder, it is removed from the list and the overall state of /actuator/health is still UP.
Is there a reason for this? We would like to have a alert, on this execution state of our application.
Are there code examples, how we can achieve this by a customized solution based on KafkaStreamsBinderHealthIndicator?
Example output of /actuator/health with Kafka Streams:
{
"status": "UP",
"components": {
"binders": {
"status": "UP",
"components": {
"kstream": {
"status": "UP",
"details": {
"mystep1": {
"threadState": "RUNNING",
...
},
...
},
"mystep2": {
"threadState": "RUNNING",
...
},
...
}
}
}
}
},
"refreshScope": {
"status": "UP"
}
}
}
UPDATE on the exact situation:
We do not stop the binding manually via the bindings endpoint.
We have implemented integrated error queues for runtime errors within all processing steps based on StreamBridge.
The solution has also some kind of circuit breaker feature. This is the one that stops a binding from within the code, when a configurable limit of consecutive runtime errors is reached, because we do not want to flood our internal error queues.
Our application is monitored by Icinga via /actuator/health, therefore we would like to get an alarm, when on of the bindings is stopped.
Switching in Icinga to another endpoint like /actuator/bindings cannot be done easily by our team.

Answer 1

Presently, the Kafka Streams binder health indicator only considers the currently active Kafka Streams for health check. What you are seeing as the output when the binding is stopped is expected. Since you used the bindings endpoint to stop the binding, you can use /actuator/bindings to get the status of the bindings. There you will see the state of all the bindings in the stopped processor as stopped. Does that satisfy your use case? If not, please add a new issue in the repository and we could consider making some changes in the binder so that the health indicator is configurable by the users. At the moment, applications cannot customize the health check implementation. We could also consider adding a property, using which you can force the stopped/inactive kafka streams processors as part of the health check output. This is going to be tricky - for e.g. what will be the overall status of the health if some processors are down?