Related
Recently, I found myself wanting to develop a web application using real-time technologies. As far as I see, I have two ideal solutions.
Use WebSockets. (in my case Socket.io)
use a real-time database like RethinkDB (I guess stuff like Pusher and PubNub also fall into this category too)
I am lost, however. What are the technical differences between these two solutions? When should I prefer one over the other?
They actually work really well together. Using RethinkDB changefeeds to watch for changes in the database and then using socket.io to pass those changes to the client(s) is a good way to use them in tandem. I'd recommend taking a look at Jorge Silva's answer to this similar question.
socket.io vs RethinkDB changefeed
I was only familiar with this because of a tutorial I did with an interesting tech stack of React, Redux, RethinkDB, Express, & Socket.io. I'm not sure what you're using for your front-end/view layer but this might be helpful for learning how to integrate RethinkDB+Socket.io
http://webapplog.com/reactive-web-stack/
Good Luck!
I have run into trouble with Socket.io regarding memory leaks and scaling issues lately. My decision to use Socket.io was made over a year ago when it was undoubtedly the best library to use.
Now that Socket.io causes much trouble, I spent time looking for alternatives that became available in the meantime and think that both Engine.io and SockJS are generally well suited for me. However, in my opinion both have some disadvantages and I am not sure which one to choose.
Engine.io is basically the perfect lightweight version of Socket.io that does not contain all the features I do not require anyway. I have already written my own reconnection and heartbeat logic for Socket.io, because I was not satisfied with the default logics and I never intended to use rooms or other features that Socket.io offers.
But - in my opinion - the major disadvantage of Engine.io is the way connections are established. Clients start with slower jsonp-polling and are upgraded if they support better transports. The fact that the clients which support websockets natively (number increasing steadily) have a disadvantage in the form of a longer and unstable connection procedure over those clients which use outdated browsers, contradicts my sense of how it should be handled.
SockJS on the other hand handles the connections exactly as I would like to. From what I have read it seems to be pretty stable while Engine.io has some issues at this time.
My app is running behind an Nginx router on a single domain, therefore I do not need the cross-domain functionality SockJS offers. Because of providing this functionality, however, SockJS does not expose the cookie data of the client at all. So far I had a 2-factor authorization with Socket.io via cookie AND query string token and this would not be possible with SockJS (with Engine.io it would).
I have read pretty much all what is avilable about and pros and cons of both, but it seems there is not much being discussed or published so far, espacially about Engine.io (there are only 8 questions tagged with engine.io here).
Which of the 2 libraries do you prefer and for which reason? Do you use them in production?
Which one will likely be maintained more actively and could have a major advantage over the other in the future?
Have you looked at Primus? It offers the cookie requirements you mention, it supports all of the major 'real-time'/websocket libraries available and is a pretty active project. To me it also sounds like vendor lock-in could be a concern for you and Primus would address that.
The fact that it uses a plugin system should also a) make it easier for you to extend if needed and b) may actually have a community plugin that already does what you need.
Which of the 2 libraries do you prefer and for which reason? Do you use them in production?
I have only used SockJS via the Vert.x API and it was for an internal project that I would consider 'production', but not a production facing consumer app. That said, it performed very well.
Which one will likely be maintained more actively and could have a major advantage over the other in the future?
Just looking over the commit history of Engine.io and SockJS, and the fact that Auttomatic is supporting Engine.io makes me inclined to think that it will be more stable, for a longer period of time, but of course that's debatable. Looking at the issues for Engine.io and SockJS is another good place to evaluate, but since they're both split over multiple repos it should be taken with a grain of salt. I'm not sure where/how Automattic is using Engine/Socket.io, but if it's in WordPress.com or one of their plugins, it has substantial production-at-scale battle testing.
edit: change answer to reflect cookie support confirmed by Primus author in comments below
I'd like to redirect you to this (quite detailed) discussion thread about SockJS and Engine.io
https://groups.google.com/forum/#!topic/sockjs/WSIdcY14ciI
Basically,
SockJS detects working transports before marking the connection
as open. Engine.io will immediately open the connection and upgrade
it later.
flash, one of the Engine.io fallbacks
(and not present in SockJS) loads slowly and in environments
behind proxies takes 3 seconds to timeout.
SockJS doesn't use flash and therefore doesn't need to work around
this issue.
SockJS does the upgrade on start. After that you have
a consistent experience. You send what you send, you receive
what you receive.
Also, as far as I can tell, engine.io-client (the client-side) library for engine.io, does not support requirejs builds, so that's another negative point. (SockJS does build perfectly).
You may also consider node-walve. Complete WebSocket basic. Extremely performant as fully stream based.
Example of how to use:
walve.createServer(function(wsocket) {
wsocket.on('incoming', function(incoming) {
incoming.pipe(process.stdout, { end: false });
});
}).listen(server);
It may not be the best choice if you feel not secure in the nodejs environment (e.g. extending prototypes for API sugar), contributing to the project (though the code is more readable as socket.io).
Hy,
I'm implementing an IMAP client as a Mac OSX application using MacRuby.
For the sake of offline availability, I wanted to allow fulltext indexing and attribute based indexing of all messages. Attributes include common E-Mail stuff like from:, to:, etc...
This would allow for advanced results sprinkled with faceting, analytic calculations and such.
Now I'm unsure about the choices and good practices when it comes to integrating such a search feature. I have a strong web development background, therefore my intuitive action would be to setup a Solr server and start feeding it with data. This might just work theoretically, as I could write an Agent that manages the solr instance for my application in the background. But to me, this approach seems like an infrastructure hassle.
On the other side, I've read about people using the FTS3 functionality from SQLite. This approach is easily accessible by CoreData. I haven't used SQLite's FTS3 but I don't think it is as powerful as Solr can be.
What is your weapon of choice for a use case like mine?
I'm mainly interested in solutions that are actually in use by Objective-C/Cocoa/MacRuby developers.
In you're going to develop the app with Ruby give a try to picky. It is very simple to use.
There is an Objective-C Lucene port
http://svn.gna.org/viewcvs/etoile/trunk/Etoile/Frameworks/LuceneKit/
I have not used it, but in your situation, I'd at least check it out. In my experience, SQL based full text search can't compete with Lucene, but haven't tried SQLite for this.
EDIT: just noticed the ruby tag -- this started out as port of Lucene
https://github.com/dbalmain/ferret
EDIT: I forgot to include the prime candidate for web applications: JSON over HTTP/REST + Comet. It combines the best features of the others (below)
Persevere basically bundles everything I need in a server
The focus for Java and such is definitely on Comet servers, but it can't be too hard to use/write a client.
I'm embarking on an application with a server holding data, and clients executing operations which would affect this data, and thus require some sort of notification across all interested/subscribed clients.
The first client will probably be written in WPF, but we'll probably need to add clients written in other languages, e.g. a Java (Swing?) client, and possibly, a web client.
The actual question(s): What protocol should I use to implement this? How easy would it be to integrate with JS, Java and .NET (precisely, C#) clients?
I could use several interfaces/protocols, but it'd be easier overall to use one that is interoperable. Given that interoperability is important, I have researched a few options:
JSON-RPC
lightweight
supports notifications
The only .NET lib I could find, Jayrock doesn't support notifications
works well with JS
also true of XML-based stuff (and possibly, even binary protocols) BUT this would probably be more efficient, thanks to native support
Protobuf/Thrift
IDL makes it easy to spit out model classes in each language
doesn't seem to support notifications
Thrift comes with RPC out of the box, but protobufs don't
not sure about JS
XML-RPC
simple enough, but doesn't support notifications
SOAP: I'm not even sure about this one; I haven't grokked this yet.
seems rather complex
Message Queues/PubSub approach: Not strictly a protocol, but might be fitting
I hardly know anything about them, and got lost amongst the buzzwords`-- JMS? **MQ?
Perhaps combined with some RPC mechanism above, although that might not be strictly necessary, and possibly, overkill.
Other options are, of course, welcome.
I am partial to the pub/sub design you've suggested. I'd take a look at ZeroMQ. It has bindings to C#, Java, and many other platforms.
Bindings list: http://www.zeromq.org/bindings:clr
I also found this conversation on the ZeroMQ dev listing that may answer some questions you have about multiple clients and ZeroMQ: http://lists.zeromq.org/pipermail/zeromq-dev/2010-February/002146.html
As XMPP was mentioned, SIP has a similar functionality. This might be more accessible for you.
We use Servoy for this. It does automatic data broadcasting to web-clients and java-clients. I'm not sure if broadcasts can be sent to other platforms, you might be able to find an answer to that on their forum.
If you want to easily publish events to clients across networks, you may wish to look at a the XMPP standard. (Used by, amongst other things, Jabber and Google Talk.)
See the extension for publish-subscribe functionality.
There are a number of libraries in different languages including C#, Java and Javascript.
You can use SOAP over HTTP to modify the data on the server and SOAP over SMTP to notify the subscribed clients.
OR
The server doesn't know anything about the subscription and the clients call the server by timeout to track updates they are interested in, using XML-RPC, SOAP (generated using WSDL), or simply HTTP GET if there is no need to pass back complex data on tracking.
For a .NET component that will be used in both web applications and rich client applications, there seem to be two obvious options for caching: System.Web.Caching or the Ent. Lib. Caching Block.
What do you use?
Why?
System.Web.Caching
Is this safe to use outside of web apps? I've seen mixed information, but I think the answer is maybe-kind-of-not-really.
a KB article warning against 1.0 and 1.1 non web app use
The 2.0 page has a comment that indicates it's OK: http://msdn.microsoft.com/en-us/library/system.web.caching.cache(VS.80).aspx
Scott Hanselman is creeped out by the notion
The 3.5 page includes a warning against such use
Rob Howard encouraged use outside of web apps
I don't expect to use one of its highlights, SqlCacheDependency, but the addition of CacheItemUpdateCallback in .NET 3.5 seems like a Really Good Thing.
Enterprise Library Caching Application Block
other blocks are already in use so the dependency already exists
cache persistence isn't necessary; regenerating the cache on restart is OK
Some cache items should always be available, but be refreshed periodically. For these items, getting a callback after an item has been removed is not very convenient. It looks like a client will have to just sleep and poll until the cache item is repopulated.
Memcached for Win32 + .NET client
What are the pros and cons when you don't need a distributed cache?
These are the items that I consider for the topic of Caching:
MemCached Win32
Velocity
.net Cache
Enterprise Library Caching Application Block
MemCached Win32: Up until recently I have used MemCached Win32. This is a akin to a web farm (many servers serving the same content for high availability) but it is a cache farm. This means that you can install it locally on your web server initially if you don't have the resources to go bigger. Then as you go down the road you can scale horizontally (more servers) or vertically (more hardware). This is a product that was ported from the original MemCached to work on Windows. This product has been used extensively in very high traffic sites. http://lineofthought.com/tools/memcached
Velocity: This is Microsofts answer to products such as MemCached. MemCached has been out for quite some time, Velocity is in CTP mode. I must say that from what I have read so far this product will certainly turn my head once it is out. But I can't bring myself to run big production projects on a CTP product with zero track record. I have started playing with it though as once it gains momentum MemCached won't even compare for those locked in the windows world! http://blogs.msdn.com/velocity/
.NET Cache: There is no reason to discount the standard .NET Cache. It is built in and ready to use for free and with no (major) set up required. It offers flexibility by way of offering mechanisms for storing items in local memory, a SINGLE state server, or a centralized database. Where Velocity steps in is when you need more than a single state server (cache in memory) and don't want to use a slow database for holding your cache.
Enterprise Application Block: I stay away from all of the Enterprise Application Blocks. They are heavy frameworks that give more than I generally require! As long as you remember to wrap everything that touches code that is not your own and follow simple rules for coding, stick to any of the other methods over this one! (just my opinion of course - MySpace leverages as much as they can out of Enterprise Application Blocks!)
You don't have to choose up front! I generally create a cache wrapper that I communicate with in my code for methods such as Get, Set, Exists, Remove, ListKeys, etc. This then points to an underlying level of cache abstraction that can point to MemCached, Velocity, or .NET cache. I use StructureMap (or choose another IoC container) to inject which form of cache I want to use for a given environment. In my local dev box I might use .NET cache in the session. In production I generally use MemCached Win 32. But regardless of how it is set up you can easily swap things around to try each system out to see what works best for you. You just need to make sure that you application knows as little as possible about how things are cached! Once this layer of abstraction is in place you can then do things such as run a compression algorithm (gzip) for all the data that is going in and out of cache which would allow you to store 10 times the amount of data in cache. - transparently.
I cover .NET Cache, MemCached Win32, StructureMap, and the appropriate abstractions in my book if you are interested!
ASP.NET 3.5 Social Networking (http://www.amazon.com/ASP-NET-3-5-Social-Networking-Enterprise-ready/dp/1847194788/ref=sr_1_1?ie=UTF8&s=books&qid=1225408005&sr=8-1 )
Andrew Siemer www.andrewsiemer.com blog.andrewsiemer.com www.socialnetworkingin.net
Update
Changed the link that lists sites using memcached. Thank you David for noticing that it was broken!
Bear in mind that the EntLib documentation specifically steers you towards the ASP.NET cache for ASP.NET applications. That's probably the strongest recommendation towards using it here. Plus the EntLib cache doesn't have dependencies, which for me is a big reason not to use it.
I don't think there's a technical limitation as such on shipping System.Web as part of your app, though it's slightly odd that they've put that notice in on the .NET 3.5 page. Hanselman actually says he started out being creeped out by this notion, but became convinced. Also if you read the comments, he says that the block has too many moving parts and the ASP.NET Cache is much more lightweght.
I think this is exactly the kind of problem that Velocity is going to solve, but that's only a preview for now :-(
I'd say use Web.Caching and see how you get on. If you put some kind of abstraction layer over the top of it, you've always got the option to swap it out for the EntLib block later on if you find problems.
Take a look at memcached. It is a really cool, fast and lightweight distributed caching system. There are APIs for several of the most popular languages, including C#. It may not serve well on the client side (unless of course the client is obtaining the cached data from a server of some kind), but if you abstract your usage of memcached to a specific interface, you could then implement the interface with another caching system.
#Davide Vosti
"If they put it in the web namespace, I think's it's for a good reason."
Does that same logic apply to the Concurrency and Coordination Runtime (CCR) in the robotic studio? no? didn't think so.