I am using Spring Cloud AWS (1.0.1.RELEASE) with Spring Boot to run a SQS consumer. The application runs fine, but when it looses network connection (for instance if I switch my WIFI off on my laptop when it runs on it), I see errors on the console and the application never recovers. It just hangs there and does not reconnect after the network becomes available. I have to kill it and bring it up. How do I force it to recover by itself?
// Spring Boot entry point:
public static void main(String[] args) {
SpringApplication.run(MyConsumerConfiguration.class, args);
}
// Message Listener (A different class)
#MessageMapping(value = "myLogicalQueueName" )
public void receive(MyPOJO object) {
}
The error I see at console:
Exception in thread "simpleMessageListenerContainer-1" com.amazonaws.AmazonClientException: Unable to execute HTTP request: sqs.us-east-1.amazonaws.com
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:473)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:297)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2422)
at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1130)
at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1678)
at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1676)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745
I just figured out the problem why SQS is not able to reconnect after network connection lost.
Actually seems to be a problem in current Spring AWS implementation of org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.java
private class AsynchronousMessageListener implements Runnable {
private final QueueAttributes queueAttributes;
private final String logicalQueueName;
private AsynchronousMessageListener(String logicalQueueName, QueueAttributes queueAttributes) {
this.logicalQueueName = logicalQueueName;
this.queueAttributes = queueAttributes;
}
#Override
public void run() {
while (isRunning()) {
ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());
CountDownLatch messageBatchLatch = new CountDownLatch(receiveMessageResult.getMessages().size());
for (Message message : receiveMessageResult.getMessages()) {
if (isRunning()) {
MessageExecutor messageExecutor = new MessageExecutor(this.logicalQueueName, message, this.queueAttributes);
getTaskExecutor().execute(new SignalExecutingRunnable(messageBatchLatch, messageExecutor));
} else {
break;
}
}
try {
messageBatchLatch.await();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
Above code spins up a new thread which does the polling to SQS queue to grab messages. Once network connection is dropped getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest()) throws UnknownHostException, which is not handled in the code and causes thread termination.
So when network connection is established later on, there is no thread polling the queue to retrieve the data.
I have already raised a issue with Spring for this. Following is the link: https://github.com/spring-cloud/spring-cloud-aws/issues/82
Hope this explains it all.
Related
I'm testing kafka broker down scenarios in my service. Is there any way to execute piece of code once the service became connected again to the broker ?
For example:-
When the producer fails to send a message or throws an exception the service stores the message in a temp table to resend it once reconnecting to the broker.
#Transactional
#EventListener
public void detailsObjectEventListener(EntityObjectEvent<DetailsEntity> event) {
StreamPayload<Details> payload = new StreamPayload<>(event.getType(), new Details(event.getSource()));
MessageChannel channel = BootstrapApplication.getMessageChannel(DetailsChannel.class);
boolean sent;
try {
sent = channel.send(MessageBuilder.withPayload(payload).build());
} catch (Exception ex) {
sent = false;
}
if (!sent) {
entityManager.persist(new FailedStreamEventEntity(event));
}
}
To handle the problem i made a cron job method to republish failed events again when the broker is UP.
#Scheduled(initialDelay = 10000, cron = "0 * * * * ?")
public void reSendFailedEvents() {
if (kafkaIndicator.health().getStatus().equals(Status.UP)) {
CriteriaBuilder builder = entityManager.getCriteriaBuilder();
Optional.of(FailedStreamEventEntity.class)
.map(builder::createQuery)
.map(query -> query.select(query.from(FailedStreamEventEntity.class)))
.map(entityManager::createQuery)
.map(TypedQuery::getResultList)
.map(List::stream)
.orElse(Stream.empty())
.peek(entityManager::remove)
.map(FailedStreamEventEntity::getEvent)
.forEach(BootstrapApplication.getEventPublisher()::publishEvent);
}
}
But, i believe its the worst solution to handle the problem. Also, i believe that there is a better way that can execute this code once the service reconnected to the broker. (Any help ?)
I have MessageListener.onMessage with a thread sleep. I'm simulating actual processing time the onMessage
method will take by the above mentioned Thread sleep. However what I have noticed is that it is getting called multiple times consecutively for the remaining messages till they get processed by the onMessage method. I see this as an inefficiency.
Actual message count in to queue : 1000
Output of running number for hits
onMessage<<15656
onMessage<<15657
onMessage<<15658
onMessage<<15659
onMessage<<15660
onMessage<<15661
onMessage<<15662
onMessage<<15663
Code block
#Service
class ThreadPooledMessageListener implements MessageListener {
#Autowired
TaskExecutor threadPoolTaskExecutor;
AtomicInteger processedCount = new AtomicInteger();
#Override
public void onMessage(Message message) {
System.out.println("onMessage<<" + processedCount.incrementAndGet());
threadPoolTaskExecutor.execute(new MessageProcessor(message));
}
}
class MessageProcessor implements Runnable {
Message processingMessage;
public MessageProcessor(Message message) {
this.processingMessage = message;
}
#Override
public void run() {
System.out.println("================================"+ Thread.currentThread().getName());
System.out.println(processingMessage);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("================================");
}
}
What are the possible fixes for this.
As #Gary Russell has pointed out; Issue was that I have used non-spring managed container SimpleMessageListenerContainer in my code. Fixed it with spring managed bean and defined concurrency there. Works as expected.
Fixed code segment
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueues(queue);
container.setMessageListener(threadPooledMessageListener);
container.setConcurrentConsumers(4);
container.start();
return container;
}
>I see this as an inefficiency.
It's not clear what you mean. Since you are handing off the processing of a message to another thread, the listener exits immediately and, of course, the next message is delivered.
This will risk message loss in the event of a failure.
If you are trying to achieve concurrency; it's better to set the container concurrentConsumers property and not do your own thread management in the listener. The container will manage the consumers for you.
I am experiencing an odd behavior of my spring boot websocket set-up.
Sometimes it works, sometimes it doesn't, it just feels random.
I have tried the several setups, none proved solid: I moved the last piece of code in a commandlinerunner inside the primary class of the application and the last choice was a different class with #Component annotation.
My setup is the following: I use a jdbc driver (pgjdbc-ng) to use the listen notify function of postgres.I have a function and a trigger that listens to a specific postgres table for inserations. If any occur, notifications are sent through the websocket. The other and is an angular app that uses ng2-stompjs to listen to /topic/notificari for notifications. I am not posting the code because the notifications don't get out of spring, the angular is not the problem.
Kind regards,
This is my WebSocketConfiguration
Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
registry.enableSimpleBroker("/topic", "/queue", "/user", "/notificari");
registry.setApplicationDestinationPrefixes("/app");
registry.setUserDestinationPrefix("/user");
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/socket").setAllowedOrigins("*")
.setHandshakeHandler(new CustomHandshakeHandler());
}
I am using a class ListenNotify and the JDBC driver pgjdbc-ng to connect to the postgresql db and use listen notify functionality
public class ListenNotify {
private BlockingQueue queue = new ArrayBlockingQueue(20);
PGConnection connection;
public ListenNotify() {
PGNotificationListener listener = new PGNotificationListener() {
#Override
public void notification(int processId, String channelName, String payload) {
queue.add(payload);
}
};
try {
PGDataSource dataSource = new PGDataSource();
dataSource.setHost("localhost");
dataSource.setDatabase("db");
dataSource.setPort(5432);
dataSource.setUser("user");
dataSource.setPassword("pass");
connection = (PGConnection) dataSource.getConnection();
connection.addNotificationListener(listener);
Statement statement = connection.createStatement();
statement.execute("LISTEN n_event");
statement.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
public BlockingQueue getQueue() {
return queue;
}
}
And finally this is the code that instantiate the ListenNotify object and listens to postgres for events that might trigger notifications that have to be send using websocket.
#Component
public class InstantaNotificari {
#Autowired
SimpMessagingTemplate template;
#EventListener(ApplicationReadyEvent.class)
public void runn() {
System.out.println("invocare met");
ListenNotify ln = new ListenNotify();
BlockingQueue queue = ln.getQueue();
System.out.println("the que ies "+ queue);
while (true) {
try {
String msg = (String) queue.take();
System.out.println("msg " + msg);
template.convertAndSend("/topic/notificari", msg);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
I didn't use Spring so I can't test your code. Here is my tested version. I think this summarizes the differences -
Change to a try with resources block. This will close the connection on destruction of the class.
Move your while(true) into the try block on the Listener so that the
lines inside the try block doesn't ever get out of execution scope.
The while(true) is blocking, so it needs to be on another thread. ListenNotify extends Thread
I'm sure there are other ways of implementing and welcome corrections to any of my assumptions.
My tested, running code is in this answer JMS Websocket delayed delivery.
I have a CXF client configured in my Spring Boot app like so:
#Bean
public ConsumerSupportService consumerSupportService() {
JaxWsProxyFactoryBean jaxWsProxyFactoryBean = new JaxWsProxyFactoryBean();
jaxWsProxyFactoryBean.setServiceClass(ConsumerSupportService.class);
jaxWsProxyFactoryBean.setAddress("https://www.someservice.com/service?wsdl");
jaxWsProxyFactoryBean.setBindingId(SOAPBinding.SOAP12HTTP_BINDING);
WSAddressingFeature wsAddressingFeature = new WSAddressingFeature();
wsAddressingFeature.setAddressingRequired(true);
jaxWsProxyFactoryBean.getFeatures().add(wsAddressingFeature);
ConsumerSupportService service = (ConsumerSupportService) jaxWsProxyFactoryBean.create();
Client client = ClientProxy.getClient(service);
AddressingProperties addressingProperties = new AddressingProperties();
AttributedURIType to = new AttributedURIType();
to.setValue(applicationProperties.getWex().getServices().getConsumersupport().getTo());
addressingProperties.setTo(to);
AttributedURIType action = new AttributedURIType();
action.setValue("http://serviceaction/SearchConsumer");
addressingProperties.setAction(action);
client.getRequestContext().put("javax.xml.ws.addressing.context", addressingProperties);
setClientTimeout(client);
return service;
}
private void setClientTimeout(Client client) {
HTTPConduit conduit = (HTTPConduit) client.getConduit();
HTTPClientPolicy policy = new HTTPClientPolicy();
policy.setConnectionTimeout(applicationProperties.getWex().getServices().getClient().getConnectionTimeout());
policy.setReceiveTimeout(applicationProperties.getWex().getServices().getClient().getReceiveTimeout());
conduit.setClient(policy);
}
This same service bean is accessed by two different threads in the same application sequence. If I execute this particular sequence 10 times in a row, I will get a connection timeout from the service call at least 3 times. What I'm seeing is:
Caused by: java.io.IOException: Timed out waiting for response to operation {http://theservice.com}SearchConsumer.
at org.apache.cxf.endpoint.ClientImpl.waitResponse(ClientImpl.java:685) ~[cxf-core-3.2.0.jar:3.2.0]
at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:608) ~[cxf-core-3.2.0.jar:3.2.0]
If I change the sequence such that one of the threads does not call this service, then the error goes away. So, it seems like there's some sort of a race condition happening here. If I look at the logs in our proxy manager for this service, I can see that both of the service calls do return a response very quickly, but the second service call seems to get stuck somewhere in the code and never actually lets go of the connection until the timeout value is reached. I've been trying to track down the cause of this for quite a while, but have been unsuccessful.
I've read some mixed opinions as to whether or not CXF client proxies are thread-safe, but I was under the impression that they were. If this actually not the case, and I should be creating a new client proxy for each invocation, or use a pool of proxies?
Turns out that it is an issue with the proxy not being thread-safe. What I wound up doing was leveraging a solution kind of like one posted at the bottom of this post: Is this JAX-WS client call thread safe? - I created a pool for the proxies and I use that to access proxies from multiple threads in a thread-safe manner. This seems to work out pretty well.
public class JaxWSServiceProxyPool<T> extends GenericObjectPool<T> {
JaxWSServiceProxyPool(Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
super(new BasePooledObjectFactory<T>() {
#Override
public T create() throws Exception {
return factory.get();
}
#Override
public PooledObject<T> wrap(T t) {
return new DefaultPooledObject<>(t);
}
}, poolConfig != null ? poolConfig : new GenericObjectPoolConfig());
}
}
I then created a simple "registry" class to keep references to various pools.
#Component
public class JaxWSServiceProxyPoolRegistry {
private static final Map<Class, JaxWSServiceProxyPool> registry = new HashMap<>();
public synchronized <T> void register(Class<T> serviceTypeClass, Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
Assert.notNull(serviceTypeClass);
Assert.notNull(factory);
if (!registry.containsKey(serviceTypeClass)) {
registry.put(serviceTypeClass, new JaxWSServiceProxyPool<>(factory, poolConfig));
}
}
public <T> void register(Class<T> serviceTypeClass, Supplier<T> factory) {
register(serviceTypeClass, factory, null);
}
#SuppressWarnings("unchecked")
public <T> JaxWSServiceProxyPool<T> getServiceProxyPool(Class<T> serviceTypeClass) {
Assert.notNull(serviceTypeClass);
return registry.get(serviceTypeClass);
}
}
To use it, I did:
JaxWSServiceProxyPoolRegistry jaxWSServiceProxyPoolRegistry = new JaxWSServiceProxyPoolRegistry();
jaxWSServiceProxyPoolRegistry.register(ConsumerSupportService.class,
this::buildConsumerSupportServiceClient,
getConsumerSupportServicePoolConfig());
Where buildConsumerSupportServiceClient uses a JaxWsProxyFactoryBean to build up the client.
To retrieve an instance from the pool I inject my registry class and then do:
JaxWSServiceProxyPool<ConsumerSupportService> consumerSupportServiceJaxWSServiceProxyPool = jaxWSServiceProxyPoolRegistry.getServiceProxyPool(ConsumerSupportService.class);
And then borrow/return the object from/to the pool as necessary.
This seems to work well so far. I've executed some fairly heavy load tests against it and it's held up.
I have some web services exposed using spring web services.
I would like to set a maximun timeout on server side, I mean, when a client invokes my web service It could not last more than a fixed time. Is it possible?
I have found lot of information about client timeouts, but not server timeout.
This is set at the level of the server itself and not the application, so it's application server dependent.
The reason for this is that it's the server code that opens the listening socket used by the HTTP connection, so only the server code can set a timeout by passing it to the socket API call that starts listening to a given port.
As an example, this is how to do it in Tomcat in file server.xml:
<Connector connectionTimeout="20000" ... />
You can work around this issue by making the web service server trigger the real work on another thread and countdown the time out it self and return failure if timed out.
Here is an example of how you can do it, it should time out after 10 seconds:
public class Test {
private static final int ONE_SECOND = 1_000;
public String webserviceMethod(String request) {
AtomicInteger counter = new AtomicInteger(0);
final ResponseHolder responseHolder = new ResponseHolder();
// Create another thread
Runnable worker = () -> {
// Do Actual work...
responseHolder.finished = true;
responseHolder.response = "Done"; // Actual response
};
new Thread(worker).start();
while (counter.addAndGet(1) < 10) {
try {
Thread.sleep(ONE_SECOND);
} catch (InterruptedException e) {
e.printStackTrace();
}
if (responseHolder.finished) {
return responseHolder.response;
}
}
return "Operation Timeout"; // Can throw exception here
}
private final class ResponseHolder {
private boolean finished;
private String response; // Can be any type of response needed
}
}