inconsistent bean validation initialization of ConstraintValidator defined via ServiceLoader - spring

Question

This question asks for some specifics about more general topic regarding modularization of bean validation I asked before.
In question linked above, following this documentation and this post I split annotation and ConstraintValidator definition into 2 java modules, and linked them together using ServiceLoader as shown in documentation here. Works, mostly. But there is one unsolved issue, that it does not work for validation defined via XML, which I did according to documentation again. What does not work: The pairing between annotation and ConstraintValidator is not set, the service loader stuff is not used at all.
To recap: I have working setup using this ServiceLoader approach and it works when validating stuff coming through rest layer. All paired correctly.
BUT! We are getting these DTOs also through kafka. And here we have two different flows. There is some initialization of common ConstraintValidators on startup, and then:
if we first get REST message, ServiceLoader stuff is discovered only at this request time, some next initialization is done seemignly, and after that even kafka messages works, meaning pairing for custom validator is available everywhere. (Great!)
if kafka message arrives first though(typical), no service loader stuff is consulted and somehow it 'destroys' the configuration in way, that even if later rest request comes it won't work either, saying, that there is no ConstraintValidator for given annotation. The initialization is completed somehow defectively.
validation.xml is as easy as:
<validation-config
xmlns="http://xmlns.jcp.org/xml/ns/validation/configuration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/validation/configuration validation-configuration-2.0.xsd"
version="2.0">
<constraint-mapping>/META-INF/validation-constraints.xml</constraint-mapping>
</validation-config>
notes:
2.0 version is because of hibernate-validator 6.2.0 which comes from spring dependency management.
Why not use annotation and dump this xml stuff altogether? Not mine file, unmodifiable.
If there is some trivial newbie mistake, please advise. Maybe there is some way how to kick in service loader functionality into action in validation.xml file, I'm not aware of and cannot find anywhere.
EDITS/suggestions:
A: try to inject validator on startup to make sure it's loaded:
#Autowired
private Validator validator;
#EventListener(ApplicationReadyEvent.class)
public void logReady() {
System.out.println(validator.toString());
}
did print initialized validator, did not help though.