enter code herepublic Mono<Void> handle(#Nonnull WebSocketSession session) {
final WebSocketContext webSocketContext = new WebSocketContext(session);
Mono<Void> output = session.send(Flux.create(webSocketContext::setSink));
Mono<Void> input = session.receive()
.timeout(Duration.ofSeconds(adapterProperties.getSessionTimeout()))
.doOnSubscribe(subscription -> subscription.request(64))
.doOnNext(WebSocketMessage::retain)
.publishOn(Schedulers.boundedElastic())
.concatMap(msg -> {
// ....blocking operation
return Flux.empty();
}).then();
return Mono.zip(input, output).then();
When I use ws client to send information, because the message sending speed is too fast, when 2000 pieces of data are received, the connection is disconnected and there is no exception message,
After I slow down the sending speed of the message on the client side, there is no problem. How can I solve it?
Below is the Flux log information:
2022-07-14 17:19:40.295 adapter-iat [boundedElastic-5] INFO reactor.Flux.PublishOn.5 - | onNext(WebSocket TEXT message (13765 bytes))
2022-07-14 17:19:40.296 adapter-iat [boundedElastic-5] INFO reactor.Flux.PublishOn.5 - | onNext(WebSocket TEXT message (13765 bytes))
2022-07-14 17:19:40.300 adapter-iat [boundedElastic-5] INFO reactor.Flux.PublishOn.5 - | onComplete()????why
Related
This is my code, channel.addConfirmListener() ackCallback Some callbacks will be lost, The message is indeed sent to the rabbitmq server and can be consumed normally , But I sleep for 2ms after sending the message, and all ack callbacks can be received,
I don't know if this is an error in my code or a rabbitmq bug
import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import lombok.extern.log4j.Log4j2;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.TimeoutException;
#Log4j2
public class 异步确认发布{
public static void main(String[] args) throws IOException, TimeoutException, InterruptedException {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("");
connectionFactory.setPort(7005);
connectionFactory.setUsername("");
connectionFactory.setPassword("");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
// 开启确认发布
AMQP.Confirm.SelectOk selectOk = channel.confirmSelect();
channel.queueDeclare("hello", true, false, false, null);
// 异步确认发布消息 回调
channel.addConfirmListener(
(deliveryTag, multiple) -> {
log.info("消息deliveryTag=>{}, send successful", deliveryTag);
},
(deliveryTag, multiple) -> {
log.info("消息deliveryTag=>{}, fail in send", deliveryTag);
}
);
for (int i = 0; i < 5; i++) {
String message = "Hello World!!! " + i;
channel.basicPublish("", "hello", null, message.getBytes(StandardCharsets.UTF_8));
}
}
}
The console shows some callbacks missing
17:04:29.607 [AMQP Connection 27.11.210.232:7005] INFO me.demo.me.rabbitmq.consumer.发布确认.异步确认发布 - ackCallback, deliveryTag=>4, send successful
17:04:29.615 [AMQP Connection 27.11.210.232:7005] INFO me.demo.me.rabbitmq.consumer.发布确认.异步确认发布 - ackCallback, deliveryTag=>5, send successful
But I sleep for 2ms after sending the message, and all callbacks can be received
example code
for (int i = 0; i < 5; i++) {
String message = "Hello World!!! " + i;
channel.basicPublish("", "hello", null, message.getBytes(StandardCharsets.UTF_8));
Thread.sleep(2); // I sleep for 2ms after sending the message, and all ack callbacks can be received
}
console log
17:05:18.037 [AMQP Connection 27.11.210.232:7005] INFO me.demo.me.rabbitmq.consumer.发布确认.异步确认发布 - ackCallback, deliveryTag=>1, send successful
17:05:18.043 [AMQP Connection 27.11.210.232:7005] INFO me.demo.me.rabbitmq.consumer.发布确认.异步确认发布 - ackCallback, deliveryTag=>2, send successful
17:05:18.043 [AMQP Connection 27.11.210.232:7005] INFO me.demo.me.rabbitmq.consumer.发布确认.异步确认发布 - ackCallback, deliveryTag=>3, send successful
17:05:18.043 [AMQP Connection 27.11.210.232:7005] INFO me.demo.me.rabbitmq.consumer.发布确认.异步确认发布 - ackCallback, deliveryTag=>4, send successful
17:05:18.043 [AMQP Connection 27.11.210.232:7005] INFO me.demo.me.rabbitmq.consumer.发布确认.异步确认发布 - ackCallback, deliveryTag=>5, send successful
My RabbitMQ Server Version is 3.9.14 (No configuration has been modified. The default configuration is used), Erlang 24.3.2 ,
Maven Project dependency in
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
<version>2.2.18.RELEASE</version>
</dependency>
I tried to prevent the main thread from shutting down, but it doesn't seem to be the reason for the main thread to shut down, because the main thread won't shut down automatically once the connection is created
I am not sure why you tagged this with spring-rabbit because you are not using the spring-rabbit APIs at all; you are using the amqp-client directly.
This is working as designed; for performance reasons, the confirm callback has the additional argument multiple when true; this means that all tags up to and including this one are confirmed with a single confirmation.
https://www.rabbitmq.com/tutorials/tutorial-seven-java.html
multiple: this is a boolean value. If false, only one message is confirmed/nack-ed, if true, all messages with a lower or equal sequence number are confirmed/nack-ed.
I have the following program through which I can detect the connection failure i.e doBeforeRetry.
Can someone tell me how to detect the successful connection or reconnection. I want to integrate a Health Check program that monitors this connection, but I am unable to capture the event that informs the connections is successfull.
Thanks
requester = RSocketRequester.builder()
.rsocketConnector(connector -> {
connector.reconnect(Retry
.fixedDelay(Integer.MAX_VALUE,Duration.ofSeconds(1))
.doBeforeRetry(e-> System.out.println("doBeforeRetry===>"+e))
.doAfterRetry(e-> System.out.println("doAfterRetry===>"+e))
);
connector.payloadDecoder(PayloadDecoder.ZERO_COPY);
}
).dataMimeType(MediaType.APPLICATION_CBOR)
.rsocketStrategies(strategies)
.tcp("localhost", 7999);
I achieved the detection of successful connection or reconnection with the following approach.
Client Side (Connection initialization)
Mono<RSocketRequester> requester = Mono.just(RSocketRequester.builder()
.rsocketConnector(
// connector configuration goes here
)
.dataMimeType(MediaType.APPLICATION_CBOR)
.setupRoute("client-handshake")
.setupData("caller-name")
.tcp("localhost", 7999)));
One the server side
#ConnectMapping("client-handshake")
public void connect(RSocketRequester requester, #Payload String callerName) {
LOG.info("Client Connection Handshake: [{}]", callerName);
requester
.route("server-handshake")
.data("I am server")
.retrieveMono(Void.class)
.subscribe();
}
On the client side, when I receive the callback on the below method, I detect the connection is successfull.
#MessageMapping("server-handshake")
public Mono<ConsumerPreference> handshake(final String response){
LOG.info("Server Connection Handshake received : Server message [{}]", response.getCallerName());
connectionSuccess.set(true);
return Mono.empty();
}else{
throw new InitializationException("Invalid response message received from Server");
}
}
Additionally, created a application level heartbeat to ensure, the liveliness of the connection.
If you want to know if it's actually healthy, you should probably have a side task that is polling the health of the RSocket, by sending something like a custom ping protocol to your backend. You could time that and confirm that you have a healthy connection, record latencies and success/failures.
I am trying to optimize the performances of Kafka in a scenario where there is high latency (>500ms) and intermittent packet loss. I am working with JAVA and using 'kafka_2.13', version: '2.5.0' API. I have 24
nodes connected to a single broker, each node tries to send a small message to all the other subscribers. I observe that all nodes are able to communicate when there is no packet loss nor latency but they don't seem to be able to communicate soon after I add latency and packet loss. I will do more tests on Monday but I was wondering if anyone had any suggestions on possible configuration improvements.
Following you can see the code that I see to publish and receive messages and then the different configurations that used for consumers and producers.
Publishers:
boolean sendAsyncMessage (byte[] value, String topic) {
ProducerRecord<Long, byte[]> record = new ProducerRecord<> (topic, System.currentTimeMillis (), value);
long msStart = System.currentTimeMillis ();
producer.send (record, (metadata, exception) -> {
long msDelta = System.currentTimeMillis () - msStart;
logger.info ("Message with topic {} sent at {}, was ack after {}", topic, msStart, msDelta);
if (metadata == null) {
logger.info ("An exception was triggered during send:" + exception.toString ());
}
});
producer.flush ();
return true;
}
Subscribers:
while (keepGoing.get ()) {
try {
// java example do it every time!
subscribe ();
ConsumerRecords<Long, byte[]> consumerRecords = consumer.poll (Duration.ofMillis (2000));
manageMessage (consumerRecords);
//Thread processRecords = new Thread (() -> manageMessage (consumerRecords));
//processRecords.start ();
} catch (Exception e) {
logger.error ("Problem in polling: " + e.toString ());
}
}
Producer:
properties.put (ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, KafkaBroker.KEY_SERIALIZER);
properties.put (ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaBroker.VALUE_SERIALIZER);
properties.put (ProducerConfig.ACKS_CONFIG, reliable ? "1" : 0);
// host1:port1,host2:port2
properties.put (ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, server);
// how many bytes to buffer records waiting to be sent to the server
//properties.put (ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432);
properties.put (ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");
properties.put (ProducerConfig.CLIENT_ID_CONFIG, clientID);
//15 MINUTES
properties.put (ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG, 54000000);
// MAX UNCOMPRESSED MESSAGE SIZE
// properties.put (ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 1048576);
// properties.put (ProducerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, 1000);
properties.put (ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG, 300);
properties.put (ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 300);
Consumer
properties.put (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, KafkaBroker.KEY_DESERIALIZER);
properties.put (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaBroker.VALUE_DESERIALIZER);
// host1:port1,host2:port2
properties.put (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, server);
// should be the topic
properties.put (ConsumerConfig.GROUP_ID_CONFIG, groupID);
properties.put (ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "3000");
properties.put (ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
properties.put (ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
Before trying to change all settings, I'd make a few changes in your logic:
Producer
Currently you are calling flush() after sending each message, effectively doing a synchronous send. This is not recommended as it forces the Kafka client to make a request to the cluster for every single message. This is pretty inefficient. In most cases, it's best to let the client decide when to actually send messages and not use flush().
Consumer
In each iteration, you are calling subscribe(), this is not needed. You should only call subscribe() when you want to change the subscription. Also creating a new thread in each poll() loop is not recommended! In addition to being slow, you risk creating hundreds or thousands of threads if the consumer starts fetching large amounts of messages.
Kafka is using a TCP protocol, so packets lost should be automatically retried. By default, Kafka clients are configured to retry most operations and automatically reconnect to brokers if a connection is lost.
When doing your tests, before changing configurations, you should see how the Kafka client is behaving by monitoring its metrics and logs. Are timeouts reached because of the latency? Are messages retried?
In the end, the biggest factor that was preventing my distributed system to correctly communicate was the producer acks option. In the beginning, we had set this to all (the strictest option) and it seems that that, paired with the deteriorated network was preventing Kafka to have performances similar to other TCP based protocols. We now use 0 for unreliable messages and 1 for reliable.
I have this subscriber code:
try {
//subscriber
syncSubscriber.createSubscriber(SdkServiceConfig.s.SUBSCRIPTION_NAME_PARTNER_REQUEST);
final List<ReceivedMessage> messages = syncSubscriber.fetch(10, true);//get all current messages.
List<String> ackIds = new ArrayList<>();
for (ReceivedMessage message : messages) {
requestToCofmanSender.receiveMessage(message.getMessage());
ackIds.add(message.getAckId());
}
//preferred bulk ack, due to network performance
syncSubscriber.sendAck(ackIds);
requestToCofmanSender.getWazePublisher().shutdown();
}
and
public void sendAck(Collection<String> ackIdList) {
if (ackIdList != null && ackIdList.size() != 0) {
String subscriptionName = SubscriptionName.format(this.getProjectId(), this.subscriptionId);
AcknowledgeRequest acknowledgeRequest = AcknowledgeRequest.newBuilder().setSubscription(subscriptionName).addAllAckIds(ackIdList).build();
this.subscriber.acknowledgeCallable().call(acknowledgeRequest);
}
}
I poll the pubsub queue in loop
and even though the code sends ack i still get the same messages.
how should i ack otherwise?
My problem was that i had a break point between receiving the message and sending ack. My pubsub was configured to 10 seconds timeout.
UPDATED w/ additional CODE
Using Netty 4.0.12, we're getting an IllegalStateException from HttpObjectEncoder if we try to send a message immediately after our websocket is connected (see exception at the bottom). If I sleep a 1 - 2 secs, then everything is fine.
I thought this was because we weren't handling the ChannelFuture properly, but I believe I've fixed that by waiting for the future to be completed using the following logic on the ChannelFuture to ensure the connect is completed before we attempt to use it.
Unfortunately, that didn't fix it. If anyone knows what might be causing this, please let me know.
Thanks in advance,
Bob
=====================
WEBSOCKET CREATION
public synchronized Channel createWebSocket(String id, NettyClientConnection connection) throws Exception
{
URI serverUri = connection.getServerUri();
final ClientHandler clientHandler = new ClientHandler(connection);
this.getBootstrap().handler(new ChannelPipelineInitializer(serverUri, clientHandler));
ChannelFuture future = this.getBootstrap().connect(serverUri.getHost(), serverUri.getPort());
// verify connect has completed successfully
NettyUtils.waitForChannelCompletion(future, "connecting websocket");
Channel websocket = future.channel();
return websocket;
}
PIPELINE INIT
public class ChannelPipelineInitializer extends ChannelInitializer<SocketChannel>
{
private URI _serverUri = null;
protected URI getServerUri(){return this._serverUri;}
private ClientHandler _clientHandler = null;
protected ClientHandler getClientHandler(){return this._clientHandler;}
public ChannelPipelineInitializer(URI serverUri, ClientHandler clientHandler)
{
this._serverUri = serverUri;
this._clientHandler = clientHandler;
}
#Override
public void initChannel(SocketChannel channel) throws Exception
{
ChannelPipeline pipeline = channel.pipeline();
boolean handleCloseFrames = false;
WebSocketClientHandshaker handshaker = WebSocketClientHandshakerFactory.newHandshaker(this.getServerUri(), WebSocketVersion.V13, null, false, null);
final WebSocketClientProtocolHandler websocketHandler = new WebSocketClientProtocolHandler(handshaker, handleCloseFrames);
DefaultEventExecutorGroup nettyExecutor = new DefaultEventExecutorGroup(10);
pipeline.addLast(PipelineConstants.HttpClientCodec, new HttpClientCodec());
pipeline.addLast(PipelineConstants.HttpAggregator, new HttpObjectAggregator(65536));
pipeline.addLast(PipelineConstants.WebSocketClientProtocolHandler, websocketHandler);
pipeline.addLast(PipelineConstants.ThingworxMessageCodec, new ThingworxMessageCodec());
// use netty executor to free up initial IO thread
pipeline.addLast(nettyExecutor, this.getClientHandler());
}
}
CHANNEL WAIT LOGIC
public static void waitForChannelCompletion(ChannelFuture future, String operationMessage) throws IOCompletionException
{
future.awaitUninterruptibly();
// Now we are sure the future is completed.
if (future.isDone())
{
if (future.isCancelled())
{
String errorMsg = String.format("IO Operation has been cancelled [operation: %s]", operationMessage);
throw new IOCompletionException(errorMsg);
}
else if (future.isSuccess() == false)
{
String errorMsg = String.format("IO Operation failed [operation: %s]", operationMessage);
throw new IOCompletionException(errorMsg, future.cause());
}
}
else
{
// future should be done, otherwise there's a problem
String errorMsg = String.format("IO Operation never completed [operation: %s]", operationMessage);
throw new IOCompletionException(errorMsg);
}
}
EXCEPTION STACK
2013-11-21 13:32:16.767-0500 [ERROR] [c.t.t.t.SendTask] [T: pool-2-thread-9] Client_9 Attempt to send message failed. java.lang.IllegalStateException: unexpected message type: UnpooledUnsafeDirectByteBuf
at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:80) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:94) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89) ~[netty-all-4.0.12.Final.jar:na] Wrapped by: io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: UnpooledUnsafeDirectByteBuf
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:193) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.invokeWrite(DefaultChannelHandlerContext.java:645) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.write(DefaultChannelHandlerContext.java:699) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.write(DefaultChannelHandlerContext.java:638) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:115) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.invokeWrite(DefaultChannelHandlerContext.java:645) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.write(DefaultChannelHandlerContext.java:699) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.write(DefaultChannelHandlerContext.java:638) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:115) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.invokeWrite(DefaultChannelHandlerContext.java:645) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext.access$2000(DefaultChannelHandlerContext.java:29) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.DefaultChannelHandlerContext$WriteTask.run(DefaultChannelHandlerContext.java:906) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353) ~[netty-all-4.0.12.Final.jar:na]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) ~[netty-all-4.0.12.Final.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
It looks like what's happening is this
1) Client sends upgrade request
2) Server sends upgrade response to client, fires local HANDSHAKE_COMPLETED event
3) Upstream server handlers hear event, start sending websocket frames to client
4) Client receives upgrade response (HTTP 101)
5) Client receives first websocket frame before #4 reaches websocket handshaker
HttpClientCodec throws exception here while processing websocket frame, causing websocket handshaker to close connection
6) Client processes upgrade response and removes HttpClientCodec (oops, too late)
I'm also seeing this with 4.0.14.Final, but with HttpClientCodec at the front of the pipeline.
** UPDATE **
This is fixed, btw: https://github.com/netty/netty/issues/2173