I'm using spring boot.
I want to use different models at both sender and receiver so that they don't depend to the same model (receiver doesn't need to add model of sender to classpath).
1) Should I do that?
2) And how can I do that?
Sender:
AccountEvent accountEvent = new com.dspc.account.domain.dto.AccountEvent(createdAccount.getId(), EventType.CREATED);
jmsMessagingTemplate.convertAndSend(new ActiveMQTopic("VirtualTopic.ACCOUNT-EVENT-TOPIC"), accountEvent);
Receiver:
#JmsListener(destination = "Consumer.AgentGenerator.VirtualTopic.ACCOUNT-EVENT-TOPIC")
public void receive(com.dspc.devicemgmt.domain.dto.AccountEvent accountEvent) {
System.out.println(accountEvent);
}
JMS config of both sender and receiver:
#Bean // Serialize message content to json using TextMessage
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
Get exception when receiving message:
[com.dspc.account.domain.dto.AccountEvent]; nested exception is
java.lang.ClassNotFoundException:
com.dspc.account.domain.dto.AccountEvent
Note that there are 2 different packages:
- com.dspc.account.domain.dto.AccountEvent
- com.dspc.devicemgmt.domain.dto.AccountEvent
I'm thinking about creating a common message model. How do you think?
public class DspcCommonMessage {
private Map<String, String> properties;
private Optional<byte[]> payLoad = Optional.empty();
public Map<String, String> getProperties() {
return properties;
}
public void setProperties(Map<String, String> properties) {
this.properties = properties;
}
public Optional<byte[]> getPayLoad() {
return payLoad;
}
public void setPayLoad(Optional<byte[]> payLoad) {
this.payLoad = payLoad;
}
}
Sender and receiver:
public void publishMessage(com.dspc.account.domain.dto.AccountEvent accountEvent) {
ObjectMapper objectMapper = new ObjectMapper();
String messageAsString = objectMapper.writeValueAsString(accountEvent);
DspcCommonMessage dspcMessage = new DspcCommonMessage();
dspcMessage.setPayLoad(Optional.of(messageAsString.getBytes()));
jmsMessagingTemplate.convertAndSend(new ActiveMQTopic("VirtualTopic.ACCOUNT-EVENT-TOPIC"), dspcMessage);
}
#JmsListener(destination = "Consumer.AgentGenerator.VirtualTopic.ACCOUNT-EVENT-TOPIC")
public void receive(com.dspc.common.domain.DspcCommonMessage dspcCommonMessage) {
String jsonBody = new String(dspcCommonMessage.getPayload());
ObjectMapper objectMapper = new ObjectMapper();
com.dspc.devicemgmt.domain.dto.AccountEvent accountEvent = objectMapper.readValue(jsonBody,
com.dspc.devicemgmt.domain.dto.AccountEvent accountEvent.class);
System.out.println(accountEvent);
}
Related
I have a service which using kafkaTemplate to send the kafka message.
#Service
class KafkaProducer(#Autowired val kafkaTemplate: KafkaTemplate<String, String>) {
final fun sendMessage(msg: String) {
val abc = kafkaTemplate.send(AppConstants.TOPIC_NAME, "abc", msg)
abc.whenComplete {
result, ex ->
if (ex != null) {
print("ex occured")
print(ex.message)
} else {
print("sent successfully")
print(result.producerRecord.value())
}
}
}
}
For unit test, I am trying to mock this KafkaProducer.
Can i use MockProducer (import org.apache.kafka.clients.producer.MockProducer) for this?
#Test
fun verify_test() {
val mockProducer = MockProducer(true,StringSerializer(), StringSerializer())
val kafkaProducer = KafkaProducer(mockProducer)
}
On trying the above code, i am getting error bcoz KafkaProducer takes argument as KafkaTemplate object, and I am providing MockProducer.
Here is an example using a mock ProducerFactory...
#SpringJUnitConfig
class So74993413ApplicationTests {
#Test
void test(#Autowired KafkaTemplate<String, String> template,
#Autowired MockProducer<String, String> producer) throws Exception {
CompletableFuture<SendResult<String, String>> future = template.send("foo", "bar");
SendResult<String, String> sendResult = future.get();
// System.out.println(sendResult);
List<ProducerRecord<String, String>> history = producer.history();
assertThat(history).hasSize(1);
ProducerRecord<String, String> record = history.get(0);
assertThat(record.topic()).isEqualTo("foo");
assertThat(record.value()).isEqualTo("bar");
}
#Configuration
public static class Config {
#Bean
MockProducer<String, String> producer() {
return new MockProducer<>(true, new StringSerializer(), new StringSerializer());
}
#Bean
ProducerFactory<String, String> pf (MockProducer producer) {
return new ProducerFactory<>() {
#Override
public Producer<String, String> createProducer() {
return producer;
}
};
}
#Bean
KafkaTemplate<String, String> template(ProducerFactory<String, String> pf) {
return new KafkaTemplate<>(pf);
}
}
}
KafkaTemplate is not a type of Producer. Native Kafka classes cannot be used in place of Spring's - that'd be a reverse dependency.
You would have to mock the template instead since that's what you're actually using. By doing so, it'll bypass any KafkaProducer instance the template used, mock or not.
I try to test the onFailure case when I send a kafka message with producer but the onFailure method is never fire.
Here is my code where I send a message :
#Component
public class MessageSending {
#Autowired
Map<String, KafkaTemplate<String, String>> producerByCountry;
String topicName = "countryTopic";
public void sendMessage(String data) {
producerByCountry.get("countryName").send(topicName, data).addCallback(
onSuccess -> {},
onFailure -> log.error("failed")
);
}
}
Here is the test class but it's still a success case and I have no idea how I can test the failure case (I want to add some processing inside the onFailure block but I would like to first know how I can trigger onFailure by testing).
#EmbeddedKafka
#SpringBootTest
public class MessageSendingTest {
#MockBean
Map<Country, KafkaTemplate<String, String>> producerByCountry;
#Autowired
EmbeddedKafkaBroker embeddedKafka;
#Autowired
MessageSending messageSending;
#Test
void failTest(CapturedOutput capturedOutput) {
var props = KafkaTestUtils.producerProps(embeddedKafka);
var producerTemplate = new DefaultKafkaProducerFactory<String, String>(props);
var template = new KafkaTemplate<>(producerTemplate);
given(producerByCountry.get("USA"))).willReturn(template);
messageSending.sendMessage("data");
assertThat(capturedOutput).contains("failed");
}
}
I also tried the idea of this topic How to test Kafka OnFailure callback with Junit? by doing
doAnswer(invocationOnMock -> {
ListenableFutureCallback<SendResult<String, String>> listenableFutureCallback = invocationOnMock.getArgument(0);
KafkaProducerException value = new KafkaProducerException(new ProducerRecord<String, String>("myTopic", "myMessage"), "error", ex);
listenableFutureCallback.onFailure(value);
return null;
}).when(mock(ListenableFuture.class)).addCallback(any(ListenableFutureCallback.class));
But I got this mockito exception org.mockito.exceptions.misusing.UnnecessaryStubbingException due by when().addCallback
Can someone can help ?
Thanks.
You can use a mock template; see this answer for an example:
How to mock result from KafkaTemplate
EDIT
You can also mock the underlying Producer object - here is an example that is closer to your use case...
#SpringBootApplication
public class So75074961Application {
public static void main(String[] args) {
SpringApplication.run(So75074961Application.class, args);
}
#Bean
KafkaTemplate<String, String> france(ProducerFactory<String, String> pf) {
return new KafkaTemplate<>(pf, Map.of(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "france:9092"));
}
#Bean
KafkaTemplate<String, String> germany(ProducerFactory<String, String> pf) {
return new KafkaTemplate<>(pf, Map.of(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "germany:9092"));
}
}
#Component
class MessageSending {
private static final Logger log = LoggerFactory.getLogger(MessageSending.class);
#Autowired
Map<String, KafkaTemplate<String, String>> producerByCountry;
String topicName = "countryTopic";
public void sendMessage(String country, String data) {
producerByCountry.get(country).send(topicName, data).addCallback(
onSuccess -> log.info(onSuccess.getRecordMetadata().toString()),
onFailure -> log.error("failed: " + onFailure.getMessage()));
}
}
#SpringBootTest
#ExtendWith(OutputCaptureExtension.class)
class So75074961ApplicationTests {
#Test
void test(#Autowired MessageSending sending, CapturedOutput capture) {
ProducerFactory<String, String> pf = mock(ProducerFactory.class);
Producer<String, String> prod = mock(Producer.class);
given(pf.createProducer()).willReturn(prod);
willAnswer(inv -> {
Callback callback = inv.getArgument(1);
callback.onCompletion(null, new RuntimeException("test"));
return mock(Future.class);
}).given(prod).send(any(), any());
// inject the mock pf into "france" template
Map<?, ?> producers = KafkaTestUtils.getPropertyValue(sending, "producerByCountry", Map.class);
new DirectFieldAccessor(producers.get("france")).setPropertyValue("producerFactory", pf);
sending.sendMessage("france", "foo");
assertThat(capture)
.contains("failed: Failed to send; nested exception is java.lang.RuntimeException: test");
}
}
Use CompletableFuture instead of ListenableFuture for versions 3.0 or later.
public void sendMessage(String country, String data) {
producerByCountry.get(country).send(topicName, data).whenComplete(
(res, ex) -> {
if (ex == null) {
log.info(res.getRecordMetadata().toString());
}
else {
log.error("failed: " + ex.getMessage());
}
});
}
and
assertThat(capture)
.contains("failed: Failed to send");
(the latter because Spring Framework 6.0+ no longer merges nested exception messages; the top level exception is a KafkaProducerException, with the actual exception as its cause).
I have a SOAP service that receives a binary attachment (MTOM). The issue is that I would like to return a non-MTOM response (a "clean" SOAP return). The current code return a multipart response and uses "xop+xml" content-type.
The code is:
#Configuration
public class MyWebServiceConfig {
#Value("${config1}")
private String contextPath;
#Value("${config2}")
private String namespace;
#Value("${config3}")
private String wsdl;
#Bean
ServletRegistrationBean<?> webServicesRegistration(ApplicationContext ctx) {
MessageDispatcherServlet messageDispatcherServlet = new MessageDispatcherServlet();
messageDispatcherServlet.setApplicationContext(ctx);
messageDispatcherServlet.setTransformWsdlLocations(true);
return new ServletRegistrationBean<>(messageDispatcherServlet, contextPath, "*.wsdl");
}
#Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setContextPath(namespace);
marshaller.setMtomEnabled(true);
return marshaller;
}
#Bean
public MarshallingPayloadMethodProcessor methodProcessor(Jaxb2Marshaller marshaller) {
return new MarshallingPayloadMethodProcessor(marshaller);
}
#Bean
DefaultMethodEndpointAdapter endpointAdapter(MarshallingPayloadMethodProcessor methodProcessor) {
DefaultMethodEndpointAdapter adapter = new DefaultMethodEndpointAdapter();
adapter.setMethodArgumentResolvers(Collections.singletonList(methodProcessor));
adapter.setMethodReturnValueHandlers(Collections.singletonList(methodProcessor));
return adapter;
}
#Bean
public SimpleWsdl11Definition contentStore() {
SimpleWsdl11Definition definition = new SimpleWsdl11Definition();
definition.setWsdl(new ClassPathResource(wsdl));
return definition;
}
}
I have tried changing to setMtomEnabled(false) - but then the request is not handled correctly.
Is the anyway of receiving MTOM - but returning a non-MTOM response?
Edit:
Endpoint looks like this:
#Endpoint
public class MyEndpoint {
private ObjectFactory objectFactory;
public MyEndpoint() {
this.objectFactory = new ObjectFactory();
}
#PayloadRoot(localPart = "SendMyRequest", namespace = "somenamespace")
#ResponsePayload
public JAXBElement<MyReceipt> store(#RequestPayload JAXBElement<MyRequest> storeContentRequest) throws IOException {
MyReceipt receipt = new MyReceipt();
receipt.setf1(123);
receipt.setf2(456);
return this.objectFactory.createMyResponse(receipt);
}
}
What I am trying to do? : I am new to Spring Integration and already have read many similar questions regarding error handling but I don't understand how to catch exceptions using error-channel?
What I have done so far:
#EnableIntegration
#IntegrationComponentScan
#Configuration
public class TcpClientConfig implements ApplicationEventPublisherAware {
private ApplicationEventPublisher applicationEventPublisher;
private final ConnectionProperty connectionProperty;
#Override
public void setApplicationEventPublisher(ApplicationEventPublisher applicationEventPublisher) {
this.applicationEventPublisher = applicationEventPublisher;
}
TcpClientConfig(ConnectionProperty connectionProperty) {
this.connectionProperty = connectionProperty;
}
#Bean
public AbstractClientConnectionFactory clientConnectionFactory() {
TcpNioClientConnectionFactory tcpNioClientConnectionFactory =
getTcpNioClientConnectionFactoryOf(
connectionProperty.getPrimaryHSMServerIpAddress(),
connectionProperty.getPrimaryHSMServerPort());
final List<AbstractClientConnectionFactory> fallBackConnections = getFallBackConnections();
fallBackConnections.add(tcpNioClientConnectionFactory);
final FailoverClientConnectionFactory failoverClientConnectionFactory =
new FailoverClientConnectionFactory(fallBackConnections);
return new CachingClientConnectionFactory(
failoverClientConnectionFactory, connectionProperty.getConnectionPoolSize());
}
#Bean
DefaultTcpNioSSLConnectionSupport connectionSupport() {
final DefaultTcpSSLContextSupport defaultTcpSSLContextSupport =
new DefaultTcpSSLContextSupport(
connectionProperty.getKeystorePath(),
connectionProperty.getTrustStorePath(),
connectionProperty.getKeystorePassword(),
connectionProperty.getTruststorePassword());
final String protocol = "TLSv1.2";
defaultTcpSSLContextSupport.setProtocol(protocol);
return new DefaultTcpNioSSLConnectionSupport(defaultTcpSSLContextSupport, false);
}
#Bean
public MessageChannel outboundChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "outboundChannel")
public MessageHandler outboundGateway(AbstractClientConnectionFactory clientConnectionFactory) {
TcpOutboundGateway tcpOutboundGateway = new TcpOutboundGateway();
tcpOutboundGateway.setConnectionFactory(clientConnectionFactory);
return tcpOutboundGateway;
}
#Bean
#ServiceActivator(inputChannel = "error-channel")
public void handleError(ErrorMessage em) {
throw new RuntimeException(String.valueOf(em));
}
private List<AbstractClientConnectionFactory> getFallBackConnections() {
final int size = connectionProperty.getAdditionalHSMServersConfig().size();
List<AbstractClientConnectionFactory> collector = new ArrayList<>(size);
for (final Map.Entry<String, Integer> server :
connectionProperty.getAdditionalHSMServersConfig().entrySet()) {
collector.add(getTcpNioClientConnectionFactoryOf(server.getKey(), server.getValue()));
}
return collector;
}
private TcpNioClientConnectionFactory getTcpNioClientConnectionFactoryOf(
final String ipAddress, final int port) {
TcpNioClientConnectionFactory tcpNioClientConnectionFactory =
new TcpNioClientConnectionFactory(ipAddress, port);
tcpNioClientConnectionFactory.setUsingDirectBuffers(true);
tcpNioClientConnectionFactory.setDeserializer(new CustomDeserializer());
tcpNioClientConnectionFactory.setApplicationEventPublisher(applicationEventPublisher);
tcpNioClientConnectionFactory.setSoKeepAlive(true);
tcpNioClientConnectionFactory.setConnectTimeout(connectionProperty.getConnectionTimeout());
tcpNioClientConnectionFactory.setSoTcpNoDelay(true);
tcpNioClientConnectionFactory.setTcpNioConnectionSupport(connectionSupport());
return tcpNioClientConnectionFactory;
}
}
Gateway
#Component
#MessagingGateway(defaultRequestChannel = "outboundChannel",errorChannel ="error-channel" )
public interface TcpClientGateway {
String send(String message);
}
Also currently, I am facing
required a bean of type org.springframework.messaging.support.ErrorMessage that could not be found
I need some assistance!
Thanking you in advance,
EDIT
#AllArgsConstructor
#Service
public class AsyncNonBlockingClient implements Connector {
TcpClientGateway tcpClientGateway;
#Override
public String send(final String payload) {
return tcpClientGateway.send(payload);
}
}
See documentation about messaging annotation:
Your problem is here: https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#annotations_on_beans
#Bean
#ServiceActivator(inputChannel = "error-channel")
public void handleError(ErrorMessage em) {
This is a plain POJO method, therefore it cannot be marked with a #Bean. You use a #Bean really for beans to expose. Then you decide if that has to be a #ServiceActivator or not. So, just remove #Bean from this method and your error-channel consumer should be OK.
I am trying to consume a JSON message using spring kafka. The message which is consumed by the consumer is like this.
{
"EventHeader": {
"entityName": "Account",
"changedFields": ["Id", "Name"]
},
"NewFields": {
"Id": "001",
"Name": "Test Account",
},
"OldFields": {}
}
So far I have created classes for "EventHeader", "NewFields","OldFields" ,and for "KafkaPayload". And also I have created a custom deserializer to deserialize this JSON payload.Here is my custom deserializer.
public class CustomDeserailizer <T extends Serializable> implements Deserializer<T> {
private ObjectMapper objectMapper = new ObjectMapper();
public static final String VALUE_CLASS_NAME_CONFIG = "value.class.name";
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
Deserializer.super.configure(configs, isKey);
}
#Override
public T deserialize(String topic, byte[] objectData) {
return (objectData == null) ? null : (T) SerializationUtils.deserialize(objectData);
}
#Override
public T deserialize(String topic, Headers headers, byte[] data) {
return Deserializer.super.deserialize(topic, headers, data);
}
#Override
public void close() {
Deserializer.super.close();
}
}
I have set the consumer configurations as below.
public class KafkaConfig {
#Bean
public KafkaConsumer<String, KafkaPayload> consumerFactory(){
Properties config = new Properties();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
config.put(ConsumerConfig.GROUP_ID_CONFIG, "groupId");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, CustomDeserializer.class);
return new KafkaConsumer<>(config);
}
}
Now I need to show the consumed message through a #KafkaListener setting the consumer into ConsumerFactory. But I don't understand how to do that. This is my first time using kafka.So could anyone give me some idea about this?
This is how I am trying to do that.
#Bean
public ConcurrentKafkaListenerContainerFactory<String, KafkaPayload> kafkaListener(){
ConcurrentKafkaListenerContainerFactory factory = new ConcurrentKafkaListenerContainerFactory();
factory.setConsumerFactory(consumerFactory());
return factory;
}
This is my KafkaListener
public class ConsumerService {
#KafkaListener(topics = "Topic", groupId = "sample-group",containerFactory = "kafkaListener")
public void consume(KafkaPayload kafkaPayload){
System.out.println("Consumed Message :"+ kafkaPayload);
}
}
Since you are using Spring Boot, just set the value deserializer class name as a property and Boot will automatically wire it into the container factory for your #KafkaListener. No need to define your own consumer factory or container factory.
spring.kafka.consumer.value-deserializer=com.acme.CustomDeserializer
https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#application-properties.integration.spring.kafka.consumer.value-deserializer