If you were to implement Chat (i.e. a real time text conversation) in Django how would you go about doing it ?
My thoughts on it ( please do have mercy on the noob ) -
Create a table for storing archived messages and one for storing current messages.
And populate the current chat by querying the current messages table every 20 secs using javascript. But this feels so wrong.
There has to be a better way. I've heard something called as the push technology. How do I use it with django ?
P.S. My site is being shared hosted on dreamhost.
Most web chat applications tend to use Comet Programming. Comet is not a real push technology, but more of a simulation of it. First off there are different implementation of comet, but generally pooling and ajax is used to repeatedly check the server to see if there is a message. Comet generally results in too many round trips, and therefore steps have been taken to find more suitable alternatives such as Bosh which simply keep the connection open. This could actually be considered real push as opposed to Comet. The wiki link below list alternatives but in generally I think Comet and Bosh are the most used methods for chat. You could possibly even used both methods together. Here are some links.
Django powered AJAX Chat
Django and Comet
Comet (programming)
If you are mainly concerned with the front end, look into comet.
As for the backend, you have a lot of different options. A more straightforward one would be to store each chat message in the database for the given chat room. Maybe a many to many relation between the chat room table and the chat message.
Other solutions include key value store databases that aren't built in to Django. You could possibly push messages into a redis set, or store active chat room data in memcached.
Since this is on dreamhost, I would look into the database option. The truth is that you have to store some data with each chat message. Who made it, what time it was made, etc. This will have to go into a table at some point in your environment. Try to just make it work and then see if you can make it faster.
Related
I'm doing a pet project with Symfony. In it, I scrape and parse the content of a few websites and APIs (it's all for personal use), and mix everything together. Up until now, I've been separating all the different retrieval processes, and basically it works like this: I have a menu, and each button updates something. When I push it, some website is loaded, the content is parsed and my database is updated. This takes some time, depending on the website loading time, the parsing etc. Basically, when I choose to update something, I lose control and I have no output about the situation until everything is done.
I'm rethinking the whole process, and the way I see this is having a page where I push a button, and a "permanent" connection is established with the server. Then, one thing a time, everything is updated. This could take some time (I would guess even 20 minutes), and therefore the server notifies the client with updates, and possibly even requires the user to make choices (I'm connecting data from different sources, and there are a few edge cases where it just can't automatically guess the right relationships).
I'm thinking about the best way to implement this. At first I thought simple Ajax/jQuery would work, but it seems to me that the relationship between client/server is too permanent and bidirectional to be able to keep everything simple. Then I thought about working with streams and/or websockets, but I don't really know the topics.
What is the best/correct way to do this, especially in a Symfony context?
I don't think this is really tight to Symfony, what you are looking for is called Server Sent Events.
Server-sent events (SSE) is a technology for a browser to get
automatic updates from a server via HTTP connection. The Server-Sent
Events EventSource API is standardized as part of HTML5 by the W3C.
For PHP in general, I generally use the Hoa\EventSource library which makes things easy
For Symfony, you can have a dedicated API endpoint that will use this library.
We are going to develop rest server for our application (and all logic is on client javascript).
So we thought to use Phalcon php, but we also need to create realtime chat system, which is much more easy to do using node.js. This made us think about using node.js instead of phalcon
Unfortunatly, we are not good expirienced in node.js, we love phalcon for its performance and internal beauty.
The quiestion is, did anybody compare phalcon and node.js performance? May be it's better to use node.js only for long polling chat requests, but i dont like when project is connected with so different tools.
You are trying to compare two different things IMO.
node.js has a lot of power and flexibility but so does Phalcon. If you want to create a chat application with Phalcon, then you will need to implement some sort of polling mechanism in your browser that would refresh the chat window every X seconds. Getting/Inserting the data from the database will be Phalcon's job. Javascript will be used to do the polling i.e. refresh the chat page every X seconds.
The problem with this approach is that you might be hitting your web server every X seconds from every client that has the chat application open - and thus refreshing the chat contents, even when there are no messages. This can become very intensive very quickly.
node.js has the ability to send messages to the subscribed clients instantly. Web sockets can do the same thing I believe.
Check this video out, which will give you an idea of how this can be achieved easily:
https://www.youtube.com/watch?v=lW1vsKMUaKg
The idea is to use technologies that will not burden your hardware, rather collaborate with it. Having a "subscription" notification system (such as sockets or node.js) reduces the load on your application since only the subscribed clients receive the new messages and no full refresh is needed from the chat clients.
Phalcon is great for the back end with its speed and it can be used to construct the message which in turn will be passed to the transport layer and sent to the client. Depending on how you want to implement this, there are plenty of options around and you can easily mix and match technologies :)
as #Nikolaos Dimopoulos said, you're trying to compare two different things.
But here is my advice, while you're experienced with PhalconPHP framework, and you want to benefit from Phalcon speed and performance, you can implement the web app in Phalcon FW, and the chatting system in Node.JS as a service.
If your web application "The Phalcon app" needs to push messages from the backend, you can use http://elephant.io/ library for that, I have done this before with Yii framework and Node, and it's working perfectly.
My advice is to use what you already know, experimenting with nodejs just for the chat application.
Mainly because you said you do not have experience with it, so, because the chat app is something a lot of people made you'll find plenty of examples.
By doing so you will learn a lot from node and might even think about migrating from Phalcon if it suits your needs, using the features offered by expressjs for example.
I would not choose one over the other based on performance.
I am looking for a realtime hosted push/socket service (paid is fine) which will handle many connections/channels from many clients (JS) and server api which can subscribe/publish to those channels from a PHP script.
Here is an example:
The client UI has a fleet of 100 trucks rendered, when a truck is modified its data is pushed to channel (eg. /updates/truck/34) to server (PHP subscriber), DB is updated and receipt/data is sent back to the single truck channel.
We have a prototype working in Firebase.io but we don't need the firebase database, we just need the realtime transmission. One of the great features of firebase.io is that its light and we can subscribe to many small channels at once. This helps reduce payload as only that object data that has changed is transmitted.
Correct me if I am wrong but I think pusher and pubnub will allow me to create 100 truck pub/subs (in this example) for each client that opens the site?
Can anyone offer a recommendation?
I can confirm that you can use Pusher to achieve this - I work for Pusher.
PubNub previously counted each channel as a connection, but they now seem to have introduced multiplexing. This FAQ states you can support 100 channels over the multiplexed connection.
So, both of these services will be able to achieve what you are looking for. There will also be more options available via this Realtime Web Tech guide which I maintain.
[I work for Firebase]
Firebase should continue to work well for you even if you don't need the persistence features. We're not aware of any case where our persistence actually makes things harder, and in many cases it actually makes your life a lot easier. For example, you probably want to be able to ask "what is the current position of a truck" without needing to wait for the next time an update is sent.
If you've encountered a situation where persistence is actually a negative for you, we'd love to hear about it. That certainly isn't our intention.
Also - we're not Firebase.io -- we're just Firebase (though we do own the firebase.io domain name).
Maybe the title could be rephrased slightly better but basically I'm wondering how the likes of facebook have implemented a 'live' interface with regards to new notifications/messages etc. I know that the complexity behind such a social network is too much to discuss in this one small SO thread but if anyone has any idea as to the technologies used in order to notify it's users almost immediately of new records (I'm assuming DB records) then I'd be curious to hear it.
If I were to guess, I'd say that there were timers on client-side code that would periodically check the database via AJAX and then react accordingly. Is this right?
It is via Comet and long polling via node.js or similar non-thread based web servers.
If I were to guess, I'd say that there were timers on client-side code that would periodically check the database via AJAX and then react accordingly. Is this right?
You are partially right. The client opens a connection and the server only responds once it has something to return to the client.
I think they are trying to use HTML5 WebSockets but as a fallback they using Comet, AJAX, Long pooling with a good backend.
Of course I am aware of Ajax, but the problem with Ajax is that the browser should poll the server frequently to find whether there is new data. This increases server load.
Is there any better method (even using Ajax) other than polling the server frequently?
Yes, what you're looking for is COMET http://en.wikipedia.org/wiki/Comet_(programming). Other good Google terms to search for are AJAX-push and reverse-ajax.
Yes, it's called Reverse Ajax or Comet. Comet is basically an umbrella term for different ways of opening long-lived HTTP requests in order to push data in real-time to a web browser. I'd recommend StreamHub Push Server, they have some cool demos and it's much easier to get started with than any of the other servers. Check out the Getting Started with Comet and StreamHub Tutorial for a quick intro. You can use the Community Edition which is available to download for free but is limited to 20 concurrent users. The commercial version is well worth it for the support alone plus you get SSL and Desktop .NET & Java client adapters. Help is available via the Google Group, there's a good bunch of tutorials on the net and there's a GWT Comet adapter too.
Nowadays you should use WebSockets.
This is 2011 standard that allows to initiate connections with HTTP and then upgrade them to two-directional client-server message-based communication.
You can easily initiate the connection from javascript:
var ws = new WebSocket("ws://your.domain.com/somePathIfYouNeed?args=any");
ws.onmessage = function (evt)
{
var message = evt.data;
//decode message (with JSON or something) and do the needed
};
The sever-side handling depend on your tenchnology stack.
Look into Comet (a spoof on the fact that Ajax is a cleaning agent and so is Comet) which is basically "reverse Ajax." Be aware that this requires a long-lived server connection for each user to receive notifications so be aware of the performance implications when writing your app.
http://en.wikipedia.org/wiki/Comet_(programming)
Comet is definitely what you want. Depending on your language/framework requirements, there are different server libraries available. For example, WebSync is an IIS-integrated comet server for ASP.NET/C#/IIS developers, and there are a bunch of other standalone servers as well if you need tighter integration with other languages.
I would strongly suggest to invest some time on Comet, but I dont know an actual implementation or library you could use.
For an sort of "callcenter control panel" of a web app that involved updating agent and call-queue status for a live Callcenter we developed an in-house solution that works, but is far away from a library you could use.
What we did was to implement a small service on the server that talks to the phone-system, waits for new events and maintains a photograph of the situation. This service provides a small webserver.
Our web-clients connects over HTTP to this webserver and ask for the last photo (coded in XML), displays it and then goes again, asking for the new photo. The webserver at this point can:
Return the new photo, if there is one
Block the client for some seconds (30 in our setup) waiting for some event to ocurr and change the photograph. If no event was generated at that point, it returns the same photo, only to allow the connection to stay alive and not timeout the client.
This way, when clients polls, it get a response in 0 to 30 seconds max. If a new event was already generated it gets it immediately), otherwise it blocks until new event is generated.
It's basically polling, but it somewhat smart polling to not overheat the webserver. If Comet is not your answer, I'm sure this could be implemented using the same idea but using more extensively AJAX or coding in JSON for better results. This was designed pre-AJAX era, so there are lots of room for improvement.
If someone can provide a actual lightweight implementation of this, great!
An interesting alternative to Comet is to use sockets in Flash.
Yet another, standard, way is SSE (Server-Sent Events, also known as EventSource, after the JavaScript object).
Comet was actually coined by Alex Russell from Dojo Toolkit ( http://www.dojotoolkit.org ). Here is a link to more infomration http://cometdproject.dojotoolkit.org/
There are other methods. Not sure if they are "better" in your situation. You could have a Java applet that connects to the server on page load and waits for stuff to be sent by the server. It would be a quite a bit slower on start-up, but would allow the browser to receive data from the server on an infrequent basis, without polling.
You can use a Flash/Flex application on the client with BlazeDS or LiveCycle on the server side. Data can be pushed to the client using an RTMP connection. Be aware that RTMP uses a non standard port. But you can easily fall back to polling if the port is blocked.
It's possible to achive what you're aiming at through the use of persistent http connections.
Check out the Comet article over at wikipedia, that's a good place to start.
You're not providing much info but if you're looking at building some kind of event-driven site (a'la digg spy) or something along the lines of that you'll probably be looking at implementing a hidden IFRAME that connects to a url where the connection never closes and then you'll push script-tags from the server to the client in order to perform the updates.
Might be worth checking out Meteor Server which is a web server designed for COMET. Nice demo and it also is used by twitterfall.
Once a connection is opened to the server it can be kept open and the server can Push content a long while ago I did with using multipart/x-mixed-replace but this didn't work in IE.
I think you can do clever stuff with polling that makes it work more like push by not sending content unchanged headers but leaving the connection open but I've never done this.
You could try out our Comet Component - though it's extremely experimental...!
please check this library https://github.com/SignalR/SignalR to know how to push data to clients dynamically as it becomes available
You can also look into Java Pushlets if you are using jsp pages.
Might want to look at ReverseHTTP also.