I'm using MassTransit with Azure Service Bus as a transport. Some endpoints will live outside of our network, so I'd like to restrict the connection strings to those endpoint queues/topics while allowing the endpoints that are on our network to send to all of the other endpoints.
Is this possible? If I try to set a connectionstring like that, errors indicating the lack of permissions to a topic that I don't think I need it to access.
MassTransit requires Manage since it creates any topics/queues at startup.
If you are only sending messages to a specific queue, I have heard of some having success by ensuring the queue already exists and has the appropriate access for the credentials, but I don't know the details. In the one case I know of, they were using queue:name with GetSendEndpoint on IBus, and then calling Send.
Related
I got this error when I'm using streaming subscription with impersonation.
After the connection opened and receive notification successfully for minutes, it just pops up a bunch of this for almost all subscriptions.
How can I avoid this error?
One or more subscriptions in the request reside on another Client Access server. GetStreamingEvents won't proxy in the event of a batch request., The Availability Web Service instance doesn't have sufficient permissions to perform the request
I need to keep the connection stable and avoid this error.
Sounds like you haven't use affinity https://learn.microsoft.com/en-us/exchange/client-developer/exchange-web-services/how-to-maintain-affinity-between-group-of-subscriptions-and-mailbox-server
Also if its a multi threaded application ExchangeService isn't thread safe and shouldn't be used across multiple threads.
I have exposed a websocket enabled service endpoint through Azure Application Gateway and the service is hosted on azure service fabric. Client initiates a websocket connection with my endpoint and is able to exchange data. During certain message flows, my Web Socket enabled service calls other services hosted on the service fabric using azure service bus. These are handled in a completely async manner. Once the other services finish processing, they post a message to the service bus which my WebSocket service reads back.
The problem I am having is to route the messages back to the right service fabric node so that it can be pushed back to the client at the other end of the WebSocket connection
In the picture below, you can imagine each node containing multiple services including the web socket enabled service. Once the Websocket service posts a message to the service bus, the downstream services start processing and finally they post a message back to the service bus which the websocket service reads back. Here a random node will pick up the message and it might not have the relevent websocket connection to push the processed data back
Sample Design
I have looked at redis pubsub model and it looks like I have to maintain last message processed on the nodes. It also means, every node on the cluster will need to read the message and discard it if they don't have the websocket connection with the client. I am looking for any suggested design models for this kind of problem
I ran into a similar scenario and didn't like the idea of using a new external service (Redis/SQL Server) as a backplane that would simply duplicate each message/event across all nodes.
The solution I settled on was to lean on a property of actor proxies, using actor events to call-back to a specific instance of a stateless service. Creating an actor service to act as a pub/sub backplane.
The solution is summarised in this blog post and this GitHub repo. It's worth pointing out that the documentation states actor events are best effort. This hasn't really been an issue when the application is running as normal, I presume that during a deployment or failover, some events may get lost, however this could be mitigated with additional work.
It's also worth noting that your load balancing rules should maintain sticky connections between clients and back-end instances. You could create separate rules for websockets if you only wanted this to apply to them and not your regular HTTP traffic.
I want to use AWS SQS for communication between my microservices (and later possibly SNS). Each microservice can have multiple instances up.
Currently I'm trying to implement the Request/Response pattern of message queues.
As I understand it, the normal way is to have one request queue, and pass a unique response queue per service instance.
The consuming service will process the message and send the response to the given response queue. Thus, the response will always be returned to the correct instance of the requesting service.
My problem now comes with Cloudfoundry.
How it should work:
Service A needs to request data from Service B.
There is one queue named A-request-B.
Service A starts with 6 instances.
Every instance creates its own queue: B-response-A-instance[x]
Every request from an instance of A sends their response queue name in the request so response is routed to the correct queue.
This is the only way I know to guarantee that the response from B gets to the correct instance of A.
This doesn't work as Cloudfoundry doesn't allow the "create-queue" call from SQS, even if I can connect to the SQS instance to send and receive messages.
The only way to create a queue is via the command line.
So I would have to create these 6 response-queues manually beforehand.
And if I start a 7th instance of A, it will fail as it doesn't have its own response queue.
I also tried using the SQS temporary queues, but they also work by creating queues dynamically which is not possible in Cloudfoundry.
I'm currently stuck with SQS, so switching to kafka/rabbitmq or something else is not possible.
Is there any other way to pass a response to the matching service instance? Or is there another way to create queues in cloud foundry?
Summary from comments above...
This doesn't work as Cloudfoundry doesn't allow the "create-queue" call from SQS
Cloud Foundry doesn't really care what messaging system you're using, unless you're using a Marketplace service to create it. In that case, Cloud Foundry will work on your behalf to create a service instance. It does this by talking to a service broker, which does the actual creation of the service instance and user credentials.
In your case, Cloud foundry handles creating the credentials to the AWS SQS through the AWS Service Broker. Unfortunately, the credentials the broker gives you don't have the permission to create queues. The creds are only allowed to send and receive messages for the specific queue that was created by the broker.
There's not a lot you can do about this, but there's a couple options:
Don't use the Marketplace service. Instead, just go to AWS directly, create an IAM user, create your SQS resources, and give the IAM user permissions to them.
Then create a user provided service with the credentials and information for the resources you created. You can bind the user provided service to your apps just like a service created by the AWS Service broker. You'd lose the convenience of using the broker, but you won't have to jump through the hoops you listed when scaling up/down your app instances.
You could create a service instance through the broker, then create a service key. The service key is a long-lived set of credentials so you could then go into AWS, look up the IAM user associated with that service key and adjust the permissions so that you can create queues.
You would then need to create a user provided service, like the first option, insert the credentials and information for your service key and bind the user provided service to any apps that you'd like to use that service.
Don't delete the service key, or your modified user will be removed and your user provided service will stop working.
Hope that helps!
Currently I'm building an application in a micro service architecture.
The first application is an API that does the user authentication, receive requests to initiate/keep a realtime connection with the user (via Socket.io or SockJS) and the system store the socket id into the User object.
The second application is a WORKER doing some stuff and sometime he has to send realtime data to the user.
The question is: How should the second application (the WORKER) send realtime data to the user?
Should the WORKER send a message to the API then the API forward this message to the user?
Or the WORKER can directly send the message to the user?
Thank you
In a perfect world example, the service that are responsible to send "publish" a real time push notifications should be separated from other services. Since the micro service is a set of narrowly related methods, and there is no relation between the authentication "user" service, and the realtime push notification service. And to a deep break down, the authentication actually is a separate service, this only FYI, There might be a reason you did this way.
How the service would communicate? There is actually many ways how to implement the internal communication between the services, MQ solution, which could add more technology to your stack, like Rabbit MQ, Beanstalk, Gearman, etc...
And also you can do the communication on top of HTTP protocal, but you need to consider that the HTTP call will add more cost.
The perfect solution is that each service will have to interfaces to execute on their behalf, HTTP interface and an MQ interface (console)
I'm designing a system where one server must send messages to lots of independent clients. The clients doesn't know about each other and should not be able to consume, peek or in any other way acquire knowledge about each others messages.
I therefore wonder if JMS / ActiveMq have the ability to control which clients get which messages?
I want all the clients to connect to the same JSM provider (the 'destination') and consume only messages meant for them. This would be a simple setup from the servers point of view.
An alternative would be to acquire webservice endpoints from all the clients and perform ws-calls every time the server have a message for a client. I think this alternative sound 'wrong' as I think ws calls are bloated. There is a great overhead for each ws call, and this server would have to make 1000's of call each day. In my opinion this would be suboptimal for the server...
Short answer: Use Message selector.
Detail answer:
The question doesn't mention about how conversation is initiated. So here my answers for both scenarios.
a) If client initiates the conversation (i.e. Client sends a message to server and waiting for a reply).
This is a request/reply scenario. Messaging/JMS is a decoupled communication system. But request/reply is a common pattern in JMS. It can be implemented using correlation pattern.
A unique identifier(correlation id) is sent part of the request message.
Server receives the message and sets the correlation id in the reply message.
Client uses Message selector to receive the message with the correct correlation id.
b) If server initiates the conversation (i.e. Server sends messages to the clients without client request).
In this case, similar approach can be used.
A fixed client id is assigned to each client.
Server maintains all client ids and sets client id of the recipient as correlation id of the message.
Client uses Message selector to receive the message which has correlation id equals to its client id.
Update about confidentiality.
Following info extracted from this link useful for you to understand JMS security.
JMS does not specify a security
contract or an API for controlling
message confidentiality and integrity.
Security is considered to be a
JMS-provider-specific feature. It is
controlled by a System Administrator
rather than implemented
programmatically or by the J2EE server
runtime.
Two major features of JMS security are Authentication and Authorization. According to my knowledge, JMS security for client access is focusing on protecting the JMS destinations (not the individual messages). As long as a client has access to a destination, the security role assigned to the client is applicable for all the messages belongs to the destination.
Based on this,
Solution 1: If the client code is controlled by a trusted party.
Follow my solutions in my original answer.
This will make sure the message is delivered to the right person. But will not protect anything if the client code is purposely modified to receive all messages.
Solution 2: Assign private destination and user account to each client and configure security such that user account of a client can access only its destination.
Note: Found a link about "Restrictions for message selectors to provide message level authorization". But I think it is a vendor specific custom feature.
Hope this will be helpful.