redis timeout increasing set min threads and connection timeout does not resolve - stackexchange.redis

Redis timeout errors occurring in my application when processing thousands of messages
I have an application which takes in contracts, creates a key and value in redis for each contract before it's being sent to Kafka. If I get the same contract, application checks if the key exists and the value is same to the previous contract, the messages is not processed further.
I have tried increasing the connectTimeout and connectRetry in the connection string. Have also tried setting the minthreads in the program main method.
The outer application which calls Redis within is configured by setting min threads and the connection sting with increased connectTimeout and connectRety
public static void Main(string[] args)
{
System.Threading.ThreadPool.SetMinThreads(1000, 300);
CreateWebHostBuilder(args).Build().Run();
}
"Redis": {
"ConnectionString": "localhost:6379",connectTimeout=10000,connectRetry=5
}
public class RedisConnectionFactory
{
public static ConnectionMultiplexer GetConnection(string connectionString)
{
var options = ConfigurationOptions.Parse(connectionString);
return ConnectionMultiplexer.Connect(options);
}
}
public interface IRedisDataAgent
{
string GetStringValue(string key);
void SetStringValue(string key, string value);
void DeleteStringValue(string key);
}
public class RedisDataAgent : IRedisDataAgent
{
private static IDatabase _database;
public RedisDataAgent(IConfiguration configuration)
{
var connection = RedisConnectionFactory.GetConnection(configuration.GetSection("Redis")["ConnectionString"]);
_database = connection.GetDatabase();
}
public string GetStringValue(string key)
{
return _database.StringGet(key);
}
public void SetStringValue(string key, string value)
{
_database.StringSet(key, value);
}
public void DeleteStringValue(string key)
{
_database.KeyDelete(key);
}
}
Application is expected not to throw any timeout errors no matter how many thousands of messages being processed.
I have increased the min threads to 1000, but still throwing errors. No matter how much I increase, it's still throwing errors
An unhandled exception was thrown by the application.
StackExchange.Redis.RedisTimeoutException: Timeout performing SET (5000ms),
IOCP: (Busy=0,Free=1000,Min=300,Max=1000), WORKER: (Busy=1018,Free=31749,Min=1000,Max=32767)

Related

Vert.x aerospike client to use context-bound event loop

Standard java aerospike client's methods have overloads allowing to provide EventLoop as an argument. When running in vert.x that client is not aware of context-bounded event loop and just fallbacks to if (eventLoop == null) { eventLoop = cluster.eventLoops.next(); } which could(and likely does) causes context switching/level of concurrency which in turn affects performance (it's still in theory, but I want to prove it), because there is no guarantee that aerospike requests will run on the same event loop as coming http request according to Vert.x Multi Reactor pattern. Open source aerospike clients like vertx-aerospike-client also have such a disadvantage. Using vert.x there is no way(at least I'm not aware of) to retrieve context-bounded event loop and pass it to aerospike client.
Vert.x has method to retrieve Context Vertx.currentContext() but retrieving EventLoop is not available.
Any ideas?
Finally I've built this:
public class ContextEventLoop {
private final NettyEventLoops eventLoops;
public ContextEventLoop(final NettyEventLoops eventLoops) {
this.eventLoops = Objects.requireNonNull(eventLoops, "eventLoops");
}
public EventLoop resolve() {
final ContextInternal ctx = ContextInternal.current();
final EventLoop eventLoop;
if (ctx != null
&& ctx.isEventLoopContext()
&& (eventLoop = eventLoops.get(ctx.nettyEventLoop())) != null) {
return eventLoop;
}
return eventLoops.next();
}
}
#NotNull
public EventLoops wrap(final EventLoops fallback,
final Supplier<#NotNull EventLoop> next) {
return new EventLoops() {
#Override
public EventLoop[] getArray() {
return fallback.getArray();
}
#Override
public int getSize() {
return fallback.getSize();
}
#Override
public EventLoop get(int index) {
return fallback.get(index);
}
#Override
public EventLoop next() {
return next.get();
}
#Override
public void close() {
fallback.close();
}
};
}

How to accept http requests after shutdown signal in Quarkus?

I tried this:
void onShutdown(#Observes final ShutdownEvent event) throws InterruptedException {
log.infof("ShutdownEvent received, waiting for %s seconds before shutting down", shutdownWaitSeconds);
TimeUnit.SECONDS.sleep(shutdownWaitSeconds);
log.info("Continue shutting down");
}
But after receiving ShutdownEvent Quarkus already responds with 503 to http requests. Looks like this could be done with ShutdownListener in preShutdown method. I have implemented this listener but it does not get called yet. How do I register ShutdownListener?
Use case here is OpenShift sending requests to terminating pod.
Option 1: Create Quarkus extension
Instructions are here. ShutdownController is my own class implementing ShutdownListener where I have a sleep in preShutdown method.
class ShutdownControllerProcessor {
#BuildStep
FeatureBuildItem feature() {
return new FeatureBuildItem("shutdown-controller");
}
#BuildStep
ShutdownListenerBuildItem shutdownListener() {
// Called at build time. Default constructor will be called at runtime.
// Getting MethodNotFoundException when calling default constructor here.
return new ShutdownListenerBuildItem(new ShutdownController(10));
}
}
Option 2: Modify ShutdownRecorder private static final field
New shutdown listener can be added using reflection. This is a bit ugly solution.
registerIfNeeded() need to be called after Quarkus startup, for example with timer 1 second after #PostConstruct.
#ApplicationScoped
public class ListenerRegisterer {
public void registerIfNeeded() {
try {
tryToRegister();
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new IllegalStateException(e);
}
}
private void tryToRegister() throws NoSuchFieldException, IllegalAccessException {
final var field = ShutdownRecorder.class.getDeclaredField("shutdownListeners");
field.setAccessible(true);
final var listeners = (List<ShutdownListener>) field.get(null);
if (listeners != null && !listeners.toString().contains("ShutdownController")) {
listeners.add(new ShutdownController(10));
setFinalStatic(field, listeners);
}
}
private static void setFinalStatic(final Field field, final Object newValue) throws NoSuchFieldException, IllegalAccessException {
field.setAccessible(true);
final var modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);
field.set(null, newValue);
}
}

SpringAMQP - Retry/Resend messages dlx

I'm trying to use a retry mechanism using DLX.
So, basically I want to send an message for 3 times and than stop and keep this message stopped on dlx queue;
What I did:
Created WorkQueue bound to WorkExchange
Created RetryQueue bound to RetryExchange
WorkQueue -> set x-dead-letter-exchange to RetryExchange
RetryQueue -> set x-dead-letter-exchange to WorkExchange AND x-message-ttl to 300000 ms (5 minutes)
So, now when I send any message to WorkQueue and it fail.. this message goes to RetryQueue for 5min and than back to WorkQueue.. but it can keep failing and I would do like to stop it after 3 attemps ...
It is possible? Is possible set to RetryQueue try to 3 times and after stop?
thanks.
There is no way to do this in the broker alone.
You can add code to your listener - examine the x-death header to determine how many times the message has been retried and discard/log it (and/or send it to a third queue) in your listener when you want to give up.
EDIT
#SpringBootApplication
public class So59741067Application {
public static void main(String[] args) {
SpringApplication.run(So59741067Application.class, args);
}
#Bean
public Queue main() {
return QueueBuilder.durable("mainQueue")
.deadLetterExchange("")
.deadLetterRoutingKey("dlQueue")
.build();
}
#Bean
public Queue dlq() {
return QueueBuilder.durable("dlQueue")
.deadLetterExchange("")
.deadLetterRoutingKey("mainQueue")
.ttl(5_000)
.build();
}
#RabbitListener(queues = "mainQueue")
public void listen(String in,
#Header(name = "x-death", required = false) List<Map<String, ?>> xDeath) {
System.out.println(in + xDeath);
if (xDeath != null && (long) xDeath.get(0).get("count") > 2L) {
System.out.println("Given up on this one");
}
else {
throw new AmqpRejectAndDontRequeueException("test");
}
}
}

Netty Latency issue

I am a new user netty. I am using netty 4.0.19-Final version. I am load testing the EchoServer example with 50 clients. Below is my configuration. I am always getting latency of around 300 mcroseconds. I am trying to reduce the latency to around 100 microseconds. Is there anything i can try to achieve desired performance? All my clients are persistent clients and will send messages with a delay of 10 milliseconds.
ServerBootstrap b = new ServerBootstrap();
b.group(workerGroup)
.channel(NioServerSocketChannel.class)
.localAddress(NetUtil.LOCALHOST, Integer.valueOf(8080))
.option(ChannelOption.SO_BACKLOG, 128)
.childOption(ChannelOption.SO_KEEPALIVE, true)
.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
.childOption(ChannelOption.SO_SNDBUF, 1045678)
.childOption(ChannelOption.SO_RCVBUF, 1045678)
.option(ChannelOption.TCP_NODELAY, true)
.childOption(ChannelOption.TCP_NODELAY, true)
// .handler(new LoggingHandler(LogLevel.INFO))
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch)
throws Exception {
ch.pipeline()
//.addLast(new LoggingHandler(LogLevel.INFO))
.addLast(new EchoServerHandler());
}
});
// Start the server.
ChannelFuture f = b.bind(PORT).sync();
// Wait until the server socket is closed.
f.channel().closeFuture().sync();
EchoServerHandler:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
ctx.write(msg);
}
#Override
public void channelReadComplete(ChannelHandlerContext ctx) {
ctx.flush();
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
// Close the connection when an exception is raised.
cause.printStackTrace();
ctx.close();
}
try to call ctx.writeAndFlush(...) Also you may need to adjust buffer sized etc.

MassTransit and event versus command publishing

I'm new to MassTransit, and I miss something in my understanding.
Let's say I have a server farm were all nodes can do the same job. The application framework is CQRS's styled. That means I have two base kind of message to publish :
Commands : must be handled by exactly one of the server, any of them (the first with job slot free)
Events : must be handled by all servers
I've have build an extremely simple MassTransit prototype (a console application that is sending hello every X seconds).
In the API, I can see there is a "publish" method. How can I specify what kind of message it is (one versus all server)?
If I look a the "handler" configuration, I can specify the queue uri. If I specify the same queue for all hosts, all hosts will get the message, but I cannot limit the execution to only one server.
If I listen from a host dedicated queue, only one server will handle the messages, but I don't know how to broadcast the other kind of message.
Please help me to understand what I'm missing.
PS: if it cares, my messaging system is rabbitmq.
In order to test, I have create a common class library with this classes :
public static class ActualProgram
{
private static readonly CancellationTokenSource g_Shutdown = new CancellationTokenSource();
private static readonly Random g_Random = new Random();
public static void ActualMain(int delay, int instanceName)
{
Thread.Sleep(delay);
SetupBus(instanceName);
Task.Factory.StartNew(PublishRandomMessage, g_Shutdown.Token);
Console.WriteLine("Press enter at any time to exit");
Console.ReadLine();
g_Shutdown.Cancel();
Bus.Shutdown();
}
private static void PublishRandomMessage()
{
Bus.Instance.Publish(new Message
{
Id = g_Random.Next(),
Body = "Some message",
Sender = Assembly.GetEntryAssembly().GetName().Name
});
if (!g_Shutdown.IsCancellationRequested)
{
Thread.Sleep(g_Random.Next(500, 10000));
Task.Factory.StartNew(PublishRandomMessage, g_Shutdown.Token);
}
}
private static void SetupBus(int instanceName)
{
Bus.Initialize(sbc =>
{
sbc.UseRabbitMqRouting();
sbc.ReceiveFrom("rabbitmq://localhost/simple" + instanceName);
sbc.Subscribe(subs =>
{
subs.Handler<Message>(MessageHandled);
});
});
}
private static void MessageHandled(Message msg)
{
ConsoleColor color = ConsoleColor.Red;
switch (msg.Sender)
{
case "test_app1":
color = ConsoleColor.Green;
break;
case "test_app2":
color = ConsoleColor.Blue;
break;
case "test_app3":
color = ConsoleColor.Yellow;
break;
}
Console.ForegroundColor = color;
Console.WriteLine(msg.ToString());
Console.ResetColor();
}
private static void MessageConsumed(Message msg)
{
Console.WriteLine(msg.ToString());
}
}
public class Message
{
public long Id { get; set; }
public string Sender { get; set; }
public string Body { get; set; }
public override string ToString()
{
return string.Format("[{0}] {1} : {2}" + Environment.NewLine, Id, Sender, Body);
}
}
I have also 3 console applications that just run the ActualMain method :
internal class Program
{
private static void Main(string[] args)
{
ActualProgram.ActualMain(0, 1);
}
}
What you want is known as Competing Consumers (search SO for that you'll find more info)
Using RabbitMQ makes life easy, all you need to do is specify the same queue name for each consumer you start, the message will be processed by only one of them.
Instead of generating a unique queue each time as you are doing.
private static void SetupBus(int instanceName)
{
Bus.Initialize(sbc =>
{
sbc.UseRabbitMqRouting();
sbc.ReceiveFrom("rabbitmq://localhost/Commands);
sbc.Subscribe(subs =>
{
subs.Handler<Message>(MessageHandled);
});
});
}
AFAIK, you'll need to have a separate process for command handlers as opposed to event handlers. All the command handlers will ReceiveFrom the same queue, all event handlers will ReceiveFrom their own unique queue.
The other piece of the puzzle is how you get messages into the bus. You can still use publish for commands, but if you have configured consumers incorrectly you could get multiple executions as the message will go to all consumers, if you want to guarantee the message ends up on a single queue you can use Send rather than Publish.
Bus.Instance
.GetEndpoint(new Uri("rabbitmq://localhost/Commands"))
.Send(new Message
{
Id = g_Random.Next(),
Body = "Some message",
Sender = Assembly.GetEntryAssembly().GetName().Name
});

Resources