Assuming a large number of log events per second. Generally, it would make sense to bundle several log events, lets say over some temporal window (e.g. 1 min) or by number, before sending them to the Sentry server. It will reduce number of connections to the server.
I am using the Java client but couldn't find any configuration regarding this issue (https://docs.sentry.io/clients/java/config/).
How to configure Sentry appropriately?
Related
I've been considering some multiple consumer problem with my system and unfortunately I'm really stuck. I mean - I see some solutions, which are below, but they are propably not enought efficient. Let me introduce system:
User has their specific Id. User can be logged on diffrent devices - many mobiles, many browsers in this same time. When user is offline every message which user has got should be provide when user is online. When user is online, should be kept informed on a regular basis about messages. Every user, when is online is connected by WebSocket.
So I have been thinking about message brokers from this pool - rabbitmq, kafka, apache pulsar ( All system above will be in Java ). And this is my thoughs about this:
Rabbitmq - Every device gets their own queue associated with client Id. But here i see some troubles. For example - user gonna log in 4 browsers and each will get new queue ( of course I assume that some not used queue will be deleted after time but this solution can be overloaded if somebody just wanna do this ).
One queue with marker for every user which messages were consumed - I tried implement this with apache Pulsar but my every attempt was running out with created new consumer (not continue as the same consumer) - Maybe I can't use this API?
Apache kafka - groups and partitions? Its similiar to point 1.
I will be really gratefull for every single hint - If You see some better solutions with other technology just let me know - I will adjust to this.
( If it matters - Java and SpringBoot are core of this )
I can respond for the apache pulsar part - you need to set SubscriptionName in consumer to be equal to your UserId, this will ensure messages to be consumed starting from the last acknowledged one for that user.
I don't want to use message expiry as it has dependency on sending application and don't want to use pub/sub as well because if the applications don't take the messages it will fill up the filesystem etc. I don't want the messages to be piled up in the queue because application is down.
This setup is required so that there wont be any outage because of this queue and the application consuming it. Any advice?
CAPEXPRY allows the administrator to set message expiry without application changes. See https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.ref.dev.doc/q097495_.htm
We have a microservice which is developed using spring boot. couple of the functionalities it implements is
1) A scheduler that triggers, at a specified time, a file download using webhdfs and process it and once the data is processed, it will send an email to users with the data process summary.
2) Read messages from kafka and once the data is read, send an email to users.
We are now planning to make this application high available either in Active-Active or Active-passive set up. The problem we are facing now is if both the instances of the application are running then both of them will try to download the file/read the data from kafka, process it and send emails. How can this be avoided? I mean to ensure that only one instance triggers the download and process it ?
Please let me know if there is known solution for this kind of scenarios as this seems to be a common scenario in most of the projects? Is master-slave/leader election approach a correct solution?
Thanks
Let the service download that file, extract the information and publish them via kafka.
Check beforehand if the information was already processed by querying kafka or a local DB.
You also could publish an DataProcessed-Event that triggers the EmailService, that sends the corresponding E-Mail.
Most of the articles on the web dealing with WebSockets are about in-memory Chat.
I'm interested in kind of less instant Chat, that is persistent, like a blog's post's comments.
I have a cluster of two servers handling client requests.
I wonder what could be the best strategy to handle pushing of database update to corresponding clients.
As I'm using Heroku to handle this cluster (of 2 web dynos), I obviously read this tutorial aiming to build a Chat Room shared between all clients.
It uses Redis in order to centralize coming messages; each server listening for new messages to propagate to web clients through websocket connections.
My use case differs in that I've got a Neo4j database, persisting into it each message written by any client.
My goal is to notify each client from a specific room that a new message/comment has just been persisted by a client.
With an architecture similar to the tutorial linked above, how could I filter only new messages to propagate to user? Is there an easy and efficient way to tell Redis:
"(WebSocket saying) When my client initiates the websocket connection, I take care to make a query for all persisted messages and sent them to client, however I want you (Redis) to feed me with all NEW messages, that I didn't send to client, so that I will be able to provide them."
How to prevent Redis from publishing the whole conversation each time a websocket connection is made? It would lead to duplications since the database query already provided the existing contents at the moment.
This is actually a pretty common scenario, where you have three components:
A cluster of stateless web servers that maintain open connections with all clients (load balanced across the cluster, obviously)
A persistent main data storage - Neo4j in your case
A messaging/queueing backend for broadcasting messages across channels (thus across the server cluster) - Redis
Your requirement is for new clients to receive an initial feed of the recent messages, and any consequent messages in real-time. All of this is implemented in your connection handlers.
Essentially, this is what your (pseudo-)code should look like:
class ConnectionHandler:
redis = redis.get_connection()
def on_init():
self.send("hello, here are all the recent messages")
recent_msgs = fetch_msgs_from_neo4j()
self.send(recent_msgs)
redis.add_listener(on_msg)
self.send("now listening on new messages")
def on_msg(msg):
self.send("new message: ")
self.send(msg)
The exact implementation really depends on your environment, but this is the general flow of things.
I'm building a simple message delegation application. Messages are being send on both ends via JMS. I'm using a MDB to process incoming messages, transform them and send them to a target queue. Unfortunately the same messages can be send to the incoming queue more than once but it is not allowed to forward duplicates.
So what is the best way to accomplish that?
Since there can be multiple MDBs listening on the incoming queue a need a single cache where I can store the unique message uuids of the incoming messages for at least an hour. How should this cache be accessed? Via a singleton/ static class (I'm running Java EE 5 and thus don't have the singleton annotation)?
In addition I think all operations must be synchronized, right? Does that harm performance too much?
#Ingo: are you OK with database solution. You can full fledged DB server or simple apache derby solution for this..
If so, you can have a simple table where you can store message unique UId and can check against it for uniqueness....this solution will have following benefits:
Simple code
No need of time bound cache(1 hour). You can check for uniqueness of a message forever.
Persistent record of what messages came in.
No need of expensive synchronized, you can rely on DB isolation level to have consistency.
centralized solution for your possibly many deployments of application.