I am going to user rabbitmq cluster for my web app. As I am implementing auto scaling using AutoScaling service of AWS, I am planning to use mirrored queue. I have read this article of rabbitmq mirrored queue. Taking performance as a consideration I would like to know that will it be recommended to implement mirrored queue in rabbtimq cluster? i.e. will the performance decrease compared with non-persistent queue?
I haven't found any good articles of benchmark with mirrored queue. Can anybody share their suggestion?
Related
all new to Masstransit and are currently evaluating it for a larger project and
wondering if anyone could help get a better understanding of the following challenges:
"Single consumer of events in a loadbalanced environment"
In production our services will be runinng multiple instances for scalability and failvover and be
part of a larger ecosystem of microservices. The overall architecture is based on Microsofts eshoponcontainers
reference implementation where different microservices are communicating with each other via "integrationevents".
When publishing a IntegrationEvent to other services which I assume should be done as described in Masstransit Producers / Publish
, https://masstransit-project.com/usage/producers.html#publish, how can I assure that only ONE instance in a specific microservice are processing
the event but of course that the event reaches ALL microsystem that depends on the event? When we have done similar solutions based on Azure Functions
this requirement has been solved by using the "Singelton" attribute (https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-how-to#singleton-attribute).
Azure Service Bus
Reading the documentation my impression is that Masstransit is very RabbitMQ centric.
Since we will be on the Azure Service bus when moving to production is there any limitations or features
not available on that "transport"?
Regards Niclas
For the first question, it's a normal publish-subscribe with competing consumers. It works like this out of the box, there's nothing that needs to be done to achieve this.
When running multiple instances of the same service
Use the same queue name for each instance
Messages from the queue will be load balanced across all instances (the competing consumer pattern)
It's from the RMQ Guidances, but it's like this for all transports.
Concerning the Azure Service Bus transport, it works as expected and has a lot of production users. It's properly documented as well.
I'd say for both of your questions the answer is "it just works".
I am a backend developer and I would like to know what are the common technologies for building real-time servers. I know I could use a service like Firebase, but I really want to create it. I have some experience using Websockets on Java, but I would like to know more ways to achieve a real-time server. When I say real-time, I mean something like Facebook. I also would like to know how to scale real-time servers.
Thank you all!
I've asked the same in multiple forums. Common answer to this question is strangely enough still:
WebSocket
Socket.io
Server-Sent Events (SSE)
But those are mainly ways of transporting or streaming events to the clients. Something needs to be built on top of it. And there are multiple other things to consider, such as:
Considerations for real-time API's
What events to send to the client
How to send each client only the events they need
How to handle authorization for events
Where to keep state on the event subscriptions (for stateless services)
How to recover from missed events due to lost connections and service crashes
Producing events for search-, or pagination queries
How to scale
Publish/Subscribe solutions
There are multiple pub/sub solutions out there, such as:
Pusher
PubNub
SocketCluster
etc.
But because of the limitation of a topic based pub/sub architecture, some of the above questions are still left unanswered and has to be dealt with by yourself. Examples are lost connections, where Pusher has no fallback, neither does SocketCluster, and PubNub has a limited queue.
Resgate - Realtime API Gateway
An alternative to the traditional topic based pub/sub pattern is using a resource-aware realtime API Gateway, such as Resgate.
Instead of the client subscribing to topics, the gateway keeps track on which resources (objects or arrays) that the client has fetched, keeping the client data up to date until it unsubscribes.
As a developer of Resgate, I can really recommend checking it out as it solves all above question, is language agnostic, simple and light-weight, and blazingly fast.
Read more at NATS blog.
Scaling
Let's say you want to scale both in the number of concurrent clients and the number of events that is produced. You will eventually need to ensure each client only gets the data they are interested in through either traditional topic based publish/subscribe, or through resource subscriptions. All above solutions handles that.
I also assume all the above mentioned solutions scales concurrent clients by allowing you to add more nodes/servers that handles the persistent WebSocket connections.
With Resgate, first level of scaling is done by simply running multiple instances (it is a simple executable), and adding a load balancer that distributes the connection evenly between them:
Handling 100M concurrent clients
Let's say a single Resgate instance handles 10000 persistent WebSocket connections, and you can add 10000 Resgates (distributed to multiple data centers) to a single NATS Server. This would allow a total of 100M connections. Of course, depending on your data, you might have other scaling issues as well, such as network traffic ;) .
A second layer of scaling (and adding redundancy) would be to replicate the whole setup to different data centers, and have the services synchronize their data between the data centers using other tools like Kafka, CockroachDB, etc.
Scaling data retrieval
With the traditional publish/subscribe solution that only deals with events, you will also have to handle scaling for the HTTP (REST) requests.
With Resgate, this is not required, as resource data is also fetched over the WebSocket connection. This allows Resgate not only to ensure that resource data and events are synchronized (another issue with separate pub/sub solutions), but also that the data can be cached. If multiple clients requests the same data, Resgate will only need to fetch it from the service once, effectively improving scalability.
Butterfly Server .NET is a real-time server written in C# allowing you to create real-time apps. You can see the source at https://github.com/firesharkstudios/butterfly-server-dotnet.
I dont have a tonne of experience with heroku, and even less with phoenix, so this may be a stupid question... but want to make sure I am making a good choice on hosting :)
From what I understand, the way you scale phoenix is add another server, launch another node, and connect them, then let BEAM / OTP work its magic to handle work load balancing. On heroku, dynos can't really talk together over a local network, which from what I understand is something that BEAM requires to cluster. So adding dynos will result in a more "traditional" scaling model, where you have an external load balancer balancing connections between unconnected nodes, with the db being shared state.
My question here is how big of an impact will this have? Is it more only an issue when you are hitting serious levels of load / scale, or will it mean spending a lot more money on infrastructure then is necessary?
You'll get the best performance on a host that supports clustering, but Phoenix has a PubSub adapter system exactly for deployments like heroku:
https://github.com/phoenixframework/phoenix_pubsub
One line config change and mix.exs deps entry and you'll have multinode channels on heroku via our Redis adapter.
This is very open question, so I am sure my answer won't be comprehensive.
In your situation the most important question is: will I Phoenix use channels?
If you use plain old HTTP, it can be mostly stateless. There are lots of methods to simulate stateful connection like storing sessions in cookies. At the end of the day, it doesn't matter if your backend servers are connected with each other, because each of them is doing independent computations. Your load balancer can randomly select any server and it will always work. This cool feature of http enables this protocol to scale so well. You can definitely use Heroku in that scenario and it will work great.
If you use Phoenix channels, things get complicated. You still want to be able to connect to any of the servers, but you will probably send messages to other users real time and they can be connected to other servers. Phoenix solves this problem for you by clustering using BEAM and this will be hard on Heroku. Or even impossible.
To sum up: it is not a question of small scale/big scale. It is a question of features. Scaling channels will require clustering, scaling plain old HTTP will not.
I'm about to build a new system and I want maximum availability! I'll have to use Windows!
I will have clients talking to my system using webservices. I'll also get data from surrounding systems. This data is delivered using messaging, MQ-series and MSMQ.
The system will produce some data that is sent back to the surrounding systems using queues.
After new data has come to the system different processes will use this data to do diffrent tasks, like printing, writing to databases etc.
To achieve high availablity I'm planning to have two versions of the system running in parallel on two different machines. The clients will try to use the first server thats responds correctly.
I think an ideal soultion would be that the incomming data from anyone of the two servers is placed in a COMMON queue(on a third machine?). Data in the queue can be picked up by processes on both servers(think producer-consumer pattern).
I think that maybe NServiceBus will suits my needs. I have a few questions according to the above.
Can a queue be shared between two servers? I dont want data to be stuck on a server if its gets down. I that case I want the other server to keep processing.
Can two(or more) "consumers"/processes on different machines pick data from a common queue?
Any advice is welcome!
The purpose of NSB distributor is not to address availability issues but to address scale issues, distributors help scaling out systems at a low cost.
By looking at the description, your system consist of WebService endpoits, multiple databases and queuing infrastructure. If you want to achieve complete high-availability you will have to make sure there are no single points of failures. In order to do that you will need,
A load balanced web farm for web service endpoints (2 or more servers)
Application cluster for queues and applications that relies on those queues.
Highly available database server, again clustered.
On top of everything a good SAN.
But if you are referring to being available to consumers, you just have to make sure target queues and webservice endpoints are available. And making sure the overall architecture promotes deferred execution.
Two or more applications can read a MSMQ queue remotely but thats something you don't want to do since it's based on DTC. And that's a real performance killer.
Some references
[http://blogs.msdn.com/b/clustering/archive/2012/05/01/10299698.aspx][1]
[http://msdn.microsoft.com/en-us/library/ms190202.aspx][2]
In short you will want to use the distributor... http://support.nservicebus.com/customer/portal/articles/859556-load-balancing-with-the-distributor
The key thing here is that the distributor node is a single point of failure so you want to run it on a cluster.
So I have seen a number of references and links from a year +/- ago asking about support for NServiceBus on Amazon EC2. Wondering if anyone out there has attempted to do anything with this recently?
I have seen the following articles/posts but fear the information and related links are dated.
A Less Than Positive Experience w/NServiceBus on Amazon EC2
The right idea, any movement on this?
Azure Love, but no Amazon?
I see a lot of chatter on the NServiceBus forums about the "next version" having a big focus on support for the cloud (at the time the current version was 2.5). I have a scenario where I would like to run NServiceBus w/MSMQ or RabbitMQ on a cluster of Amazon EC2 instances but it concerns me that there is not more discussion around people actually using NServiceBus on Amazon.
Anyone doing it successfully or have reasons to avoid considering it?
[EDIT] - Does anyone know if using reserved instances gets around the issue with EC2 restarts described in the article above?
There are different ways to successfully run NServiceBus on EC2. Picking which option to go with requires weighing the balance of cost, scalability, & operational overhead.
MSMQ
NServiceBus runs fine on EC2 with MSMQ, but there are a few obstacles that need attention. The main issue is that the computer names / DNS names on EC2 instances change during each restart. This is an issue because the computer name is used when sending messages to endpoint as well when subscribing to messages. One simple option to overcome this overhead is to attach an elastic IP to the instance & use its DNS name. The benefit is that it's pretty easy to do this. The downside is that you're only given 5 Elastic IPs by default. You can ask for more & Amazon is usually pretty liberal with handing out extra Elastic IPs. You will also be limited in how you scale. For instance, you won't be able to simply plug into the elastic scaling features of AWS. You also have to deal with backups. I would put the queues on a separate EBS volume & take snapshots on an interval.
I'd pick this option if you want to use messaging, but you don't have really crazy SLA's, you don't need to scale up and down machines quickly, & you don't need to deal with high message volumes. This is the case with most projects.
Amazon SQS
You could write a custom transport for SQS. The benefit of using NSB with SQS remote queues is that you get highly available queues, you don't have to manage them on your EC2 instances & you don't have to worry about backups. It's also easier to leverage elastic scaling with this approach. The downside is that each read costs $$$, so it may not be economically feasible to read at the same speeds as MSMQ or RabbitMQ - although this problem is mitigated by support for long polling and the ability to download many messages in a single call. Another downside is that it doesn't support distributed transactions with DTC. If you're using NServiceBus 5 or later, you could implement the Outbox pattern in your transport as described here, to ensure your messages are still processed only once. Otherwise it's up to you to ensure that your endpoints and handlers have idempotency solutions in place. You can play around with speed vs cost by adjusting the polling intervals of each of your endpoints & perhaps even have a back-off strategy where you decrease your polling intervals if you haven't received messages in a while. You will also have to worry about the size of your messages, as SQS has a small size limitation (256 K). You don't hit this in most messages.
I'd pick this option if read / write speeds aren't an issue, but you don't want to worry about operationally supporting your queuing infrastructure.
RabbitMQ
I haven't personally played with RabbitMQ on EC2, but a quick search came up with a few articles on how to get it up and running on an EC2 instance. There is a mature RabbitMQ transport available and it supports guaranteed once-only processing of messages as of NServiceBus version 5, as described in the link above. This would be cheaper to operate than SQS & I've heard that it's easier to cluster than MSMQ. Finally, like MSMQ, you would have to come up with a backup strategy (probably using snapshots).
Mixed
Nobody says that you have to pick one queuing system. You could use SQS for endpoints that need high availability & you don't mind paying the $$$, then use MSMQ / RabbitMQ for the rest of your system.