I am streaming a tty's stdout and stderr to RabbitMQ (logs to be exact). These logs can be viewed on a website and while the content is streamed to RabbitMQ they are consumed by the webserver and forwarded to the client using WebSockets. Logs are immediately persisted after sending it to RabbitMQ.
When the user accesses the website the persisted logs are rendered and the consecutive parts are streamed using WebSockets. The problem is that there is a race condition as the persisted logs might be missing chunks of the log that occurred between rendering the site and receiving the first chunk via WebSocket.
My idea was to keep all chunks in the queue and send those via the WebSocket after connecting. Additionally I would add a worker to listen to some kind of a "finished" event which then takes everything in the queue and persists it at once.
The problem is that I don't know if this is possible using RabbitMQ or how. Any ideas or other solutions?
I don't think it really matters but my stack is using Ruby Sinatra and the Bunny RabbitMQ client.
While I agree with your general idea about picking up where you left off, after loading the intial page, what you're trying to do isn't something that should be done from RabbitMQ.
There are a lot of potential problems that this would cause, which I've outlined in a blog post, previously.
Instead of trying to do this w/ RMQ, I would do this from a database layer.
As you push things into the database, you have an ID - hopefully one that is sequential. If not, add a sequence to the entries.
When you load the page for the user, send the current ID that they are at down to the browser.
After the page finishes loading and you're setting up the websocket connection, send the user's current spot in the list of messages via the websocket. then the websocket connection can use that id to say "give me all the messages after this id, and start streaming them"
Again, this is not done via RabbitMQ (see my article on why this is a bad idea), but via your database and sequential IDs.
Related
I have a project that is related to job postings. Consultants or employers register on my website and then start posting jobs. I want to make push notifications for all users. When a consultant or employer posts a job, all online users must get notified that an employer has posted this job without any page refreshes on jquery setInterval or timeout.
I am using Spring framework. I have searched for the solution but found nothing. I want to know whether Spring provided WebSockets in their latest version. Is this possible to do with WebSockets?
I want a proper resource so that I can implement it on my website.
There are two ways to satisfy your need;
First is polling in which you repeatedly send requests from client to the server. On server side you somehow need have a kind of message queue for each client to deliver the incidents on a request. There also is a different type of polling in which you send a request from client and never end the request on the server-side thus you have a kind of pipe between two ends. This is called long polling.
Disadvantage of polling is that you have to send requests to the server forever from the client and in many cases server sends empty messages as there is no events happened.
The real application of pushing messages is recently avaliable with websockets (thanks to html5). However this requires the application server to be capable of websocket functionality. afaik jetty and tomcat has this ability. Spring 4 has websocket here you can find the tutorial; http://syntx.io/using-websockets-in-java-using-spring-4/
You can find a related stackoverflow post here
Most of the articles on the web dealing with WebSockets are about in-memory Chat.
I'm interested in kind of less instant Chat, that is persistent, like a blog's post's comments.
I have a cluster of two servers handling client requests.
I wonder what could be the best strategy to handle pushing of database update to corresponding clients.
As I'm using Heroku to handle this cluster (of 2 web dynos), I obviously read this tutorial aiming to build a Chat Room shared between all clients.
It uses Redis in order to centralize coming messages; each server listening for new messages to propagate to web clients through websocket connections.
My use case differs in that I've got a Neo4j database, persisting into it each message written by any client.
My goal is to notify each client from a specific room that a new message/comment has just been persisted by a client.
With an architecture similar to the tutorial linked above, how could I filter only new messages to propagate to user? Is there an easy and efficient way to tell Redis:
"(WebSocket saying) When my client initiates the websocket connection, I take care to make a query for all persisted messages and sent them to client, however I want you (Redis) to feed me with all NEW messages, that I didn't send to client, so that I will be able to provide them."
How to prevent Redis from publishing the whole conversation each time a websocket connection is made? It would lead to duplications since the database query already provided the existing contents at the moment.
This is actually a pretty common scenario, where you have three components:
A cluster of stateless web servers that maintain open connections with all clients (load balanced across the cluster, obviously)
A persistent main data storage - Neo4j in your case
A messaging/queueing backend for broadcasting messages across channels (thus across the server cluster) - Redis
Your requirement is for new clients to receive an initial feed of the recent messages, and any consequent messages in real-time. All of this is implemented in your connection handlers.
Essentially, this is what your (pseudo-)code should look like:
class ConnectionHandler:
redis = redis.get_connection()
def on_init():
self.send("hello, here are all the recent messages")
recent_msgs = fetch_msgs_from_neo4j()
self.send(recent_msgs)
redis.add_listener(on_msg)
self.send("now listening on new messages")
def on_msg(msg):
self.send("new message: ")
self.send(msg)
The exact implementation really depends on your environment, but this is the general flow of things.
I'm trying to understand whether the HTML5 Web Notifications API can help me out, but I'm falling short in understanding how it works.
I'd like user_a to be able to send user_b a message within my webapp.
I'd like user_b to receive a notification of this.
Can the web notifications API help here? Does it let me specifically target a user (rather than notify everyone the site has been updated_? I can't see how I would create an alert for one person.
Can anyone help me understand a little more?
The notifications API is client side, so it needs to get events from another client-side technology. Here, read THIS: http://nodejs.org/api/. Just kidding. Node.js+socket.io is probably the best way to go here, you can emit events to one or all clients (broadcast). That's a push scenario. Or each user could be pulling their notifications from the server.
HTML5 Web Notifications API gives you ability to display desktop notifications that your application has generated.
What you are trying to achieve is a different thing and web notification is just a part of your scenario.
Depending upon how you are managing your application, for chat and messaging purpose as humbolight mentioned, you should look into node.js. it will provide you the necessary back-end to manage sending and receiving messages between users.
To notify a user that (s)he has received a message, you can opt for ajax polling on client side.
Simply create a javascript that pings the server every x seconds and checks if there is any notification or new message available for this user.
If response is successful, then you can use HTML5 notification API to show a message to user that (s)he has a new message.
The main problem with long polling is server load, and bandwidth usage even when there are no messages, and if number of users are in thousands then you can expect your server always busy responding to poll calls.
An alternate is to use Server Sent Events API, where you send a request to server and then server PUSHES the notifications/messages to the client as soon as they are available.
This reduces the unnecessary client->server polling and seems much better option in your case.
To get started you can check a good tutorial at
HTML5Rocks
What you're looking for is WebSocket. It's the technology that allows a client (browser) to open a persistent connection to the server and receive data from it at the server's whim, rather than having to "poll" the server to see if there's anything new.
Other answers here have already mentioned node.js, but Node is simply one (though arguably the best) option for implementing websockets on your server. You might also be comfortable with Ratchet, which is a websocket server library for PHP, or Tornado which is in Python.
How you handle your real-time communication is up to you. Websockets are merely the underlying technology that you can use to pass data back and forth. The client side of this will be fairly easy, but on the server side, you'll need a mechanism for websocket handlers to get information from each other. Look at tools like ZeroMQ for handling queues, and Memcached or Redis to handle large swaths of data which don't need to be stored permanently.
Goal is, create an HTML page which can send and receive updates from server.
I created HTML page and made connection with Apache ActiveMQ via STOMP over websocket and wrote Java code at server side which will send updates to those queue. This works fine, but ideally HTML page will be used by many users, so I want each to have their separate queue( dynamically generated) and listen and send on their respective queue only. HTML client would be able to register queue but thing which I don't know is how my server side code will get notified, do we have any callback func for this? Tried with DestinationSource class which does tell how many queue's exist but not clear how to get access to that queue
Am I going on wrong path for solving this? Any help/link/info on this will be appreciated
You might want to look into ActiveMQ Advisory Message's which can fire events for various things such as Destination creation and destruction, no consumer, message expiration etc.
The ActiveMQ site has documentation on this.
I am writing an application using jruby on rails. Part of the application initiates a long running process from a web page. The long running process could last for 20 minutes in some cases but will in most cases outlive a web page response in terms of time. I also want the job to continue if the user closes down the browser. The long running process will add records to a database as it is running.
I want to give visual indications of the inserts into the database on the web page and I would prefer to use web sockets rather than polling the database for the inserts.
I am thinking of sending a message to a resque queue with a queue handler that will ensure the job is completed if the user closes down the browser. The queue handler will perform the inserts into the database.
I was thinking of using EM-WebSocket as my websocket server.
The problem I have is:
How can I communicate between the resque process and EM-WebSocket process? I want to somehow pass the details of the new inserts into the database from the resque process to an EM-WebSocket instance that will communicate with the browser?
Anybody solved a problem like this or any ideas how I can do this?
I'm actually working on a gem that makes this pretty simple. Right now it's pretty bare, but it does work. https://github.com/KellyMahan/RealTimeRails
It runs an event-machine server for listening for updates and makes use of em-websockets to send those updates to the browser.
It's meant to watch for active record updates through an after_save call that tells the event-machine server it has an update for it's model with an id. Then it matches the model and id to specific channels to send a message to connections on the web socket server. When the browser receives a notice to update it makes an ajax call to retrieve the latest results.
Still a work in progress but it could help you.