I am implementing an API with the following basic structure:
Run serverA
Client connects to serverA and sends data
ServerA processes the data and sends it on to serverB
ServerB replies to serverA
ServerA responds to client's request based on the input received from serverB
So far I have tried two solutions:
1) Create a standard non-twisted TCP connection using httplib to process the request from serverA to serverB. This however effectively blocks the server for the duration of the httplib call.
2) Create a second class inheriting from protocol.Protocol and use
factory = protocol.ClientFactory()
factory.protocol = Authenticate
reactor.connectSSL("localhost",31337,factory, ssl.ClientContextFactory())
to create the connection between serverA and serverB. However when doing this, I don't seem to be able to access the original client-to-serverA connection from within the callbacks of the request class.
What would be the correct way to handle such a setting in Twisted?
The "client-to-serverA" connection is represented by a protocol instance associated with a transport. These are both regular old Python objects, so you can do things like pass them as arguments to functions or class initializers, or set them as attributes on other objects.
For example, if you have ClientToServerAProtocol with a fetchServerBData method which is invoked in response to some bytes being received from the client, you might write it something like this:
class ClientToServerAProtocol(Protocol):
...
def fetchServerBData(self, anArg):
factory = protocol.ClientFactory()
factory.protocol = Authenticate
factory.clientToServerAProtocol = self
reactor.connectSSL("localhost",31337, factory, ssl.ClientContextFactory())
Since ClientFactory sets itself as the factory attribute on any protocol it creates, the Authenticate instance which will result from this will be able to say `self.factory.clientToServerAProtocol and get a reference to that "client-to-serverA" connection.
There are lots of variations on this approach. Here's another one, using the more recently introduced endpoint API:
from twisted.internet.endpoints import SSL4ClientEndpoint
class ClientToServerAProtocol(Protocol):
...
def fetchServerBData(self, anArg):
e = SSL4ClientEndpoint(reactor, "localhost", 31337, ssl.ClientContextFactory())
factory = protocol.Factory()
factory.protocol = Authenticate
connectDeferred = e.connect(factory)
def connected(authProto):
authProto.doSomethingWith(self)
connectDeferred.addCallback(connected)
Same basic idea here - use self to give a reference to the "client-to-serverA" connection you're interested in to the Authenticate protocol. Here, I used a nested function to ''close over'' self. That's just another of the many options you have for getting references into the right part of your program.
Related
Disclaimer: This is my first time working with WS and MQTT, so structure may be wrong. Please point this out.
I am using autoban with asyncio to receive and send messages to a HA (HomeAssistant) instance through websockets.
Once my python code receives messages, I want to forward them using MQTT to AWS IoT service. This communication needs to work both ways.
I have made this work as a script where everything is floating within a file.
I am trying to make this work in a class structure, which is how my final work will be done.
In order to do that, I need my WebSocketClientProtocol to have access to AWSIoTClient .publish and .subscribe. Although WebSocketClientProtocol initialization is done through a factory, as a result I am not sure how to pass any arguments to it. For instance:
if __name__ == "__main__":
aws_iot_client = AWSIoTClient(...)
factory = WebSocketServerFactory('ws://localhost:8123/api/websocket')
factory.protocol = HomeAssistantProtocol
How can I pass aws_iot_client to HomeAssistantProtocol?
I have found examples of Autobahn - Twisted that do this using self.factory on the WebSocketClientProtocol subclass, but this is not available for asyncio.
I found that calling run_until_complete on returns transport, protocol instances, so I can then I can pass the AWS client to it.
loop = asyncio.get_event_loop()
coro = loop.create_connection(factory, '127.0.0.1', 9000)
transport, protocol = loop.run_until_complete(coro)
I need horizontally scalable WebSocket connection server for chat like system, where browser clients connected to different WebSocket servers coould exchange messages within separate chat rooms.
Clients HaProxy WebSocket server1 WebSocket server2 Redis/ZeroMQ
| | | |
client A ----=------------>o<----------------|------------------>|
| | | |
client B ----=-------------|---------------->o<----------------->|
| | | |
Here client A and client B are connected through HaProxy to two different WebSocket servers, which exchange messages through Redis/ZeroMQ backend, like in that and that questions.
Thinking of building that architecture I wonder if already there is an opensource analog. What such a project would you suggest to look at?
Look into the Plezi Ruby framework. I'm the author and it has automatic Redis scalability built in.
(you just setup the ENV['PL_REDIS_URL'] with the Redis URL)
As for the architecture to achieve this, it's fairly simple... I think.
Each server instance "subscribes" to two channels: a global channel for "broadcasting" (messages sent to all users or a large "family" of users) and a unique channel for "unicasting" (messages intended for a specific user connected to the server).
Each server manages it's internal broadcasting system, so that messages are either routed to a specific user, to a family of connections or all users, as par their target audience.
You can find the source code here. The Redis integration is handled using this code together with the websocket object code.
Web socket broadcasts are handled using both the websocket object on_broadcast callback. The Iodine server handles the inner broadcasting within each server instance using the websocket implementation.
I already posted the inner process architecture details as an answer to this question
I think socket.io has cross server support as well.
Edit (some code)
Due to the comment, I thought I'd put in some code... if you edit your question and add more specifications about the feature you're looking for, I can edit the code here.
I'm using the term "room" since this is what you referred to, although I didn't envision Plezi as just a "chat" framework, it is a very simple use case to demonstrate it's real-time abilities.
If you're using Ruby, you can run the following in the irb terminal (make sure to install Plezi first):
require 'plezi'
class MultiRoom
def on_open
return close unless params[:room] && params[:name]
#name = params[:name]
puts "connected to room #{params[:room]}"
# # if you use JSON to get room data,
# # you can use room arrays like so:
# params[:room] = params[:room].split(',') unless params[:room].is_a?(Array)
end
def on_message data
to_room = params[:room]
# # if you use JSON you can try:
# to_room = JSON.parse(data)['room'] rescue nil
# # we can use class `broadcast`, to broadcast also to self
MultiRoom.broadcast :got_msg, to_room, data, #name if to_room
end
protected
def got_msg room, data, from
write ::ERB::Util.html_escape("#{from}: #{data}") if params[:room] == room
# # OR, on JSON, with room arrays, try something like:
# write data if params[:room].include?(room)
end
end
class EchoConnection
def on_message data
write data
MultiRoom.broadcast "myroom", "Echo?", "system" if data == /^test/i
end
end
route '/echo', EchoConnection
route '/:name/(:room)', MultiRoom
# # add Redis auto-scaling with:
# ENV['PL_REDIS_URL'] = "redis://:password#my.host:6389/0"
exit # if running in terminal, using irb
You can test it out by connecting to: ws://localhost:3000/nickname/myroom
to connect to multiple "rooms" (you need to re-write the code for JSON and multi-room), try: ws://localhost:3000/nickname/myroom,your_room
test the echo by connecting to ws://localhost:3000/echo
Notice that the echo acts differently and allows you to have different websockets for different concerns - i.e., having one connection for updates and messages using JSON and another for multiple file uploading using raw binary data over websockets.
Using shelf_auth I can extract current user information from request this way:
getAuthenticatedContext(request)
.map((ac) => ac.principal.name)
.getOrElse(() => 'guest')
but, obviously, I need a request for that to work :)
On the other hand, using shelf_web_socket, establishing of websocket connection executes handler like that:
handleWS(CompatibleWebSocket ws){
// Here I should get user from getAuthenticatedContext()
// but I cannot due to absence of Request here.
ws.messages.listen(...);
};
rootRouter.get('/ws', webSocketHandler(handleWS), middleWare: authMiddleware);
But I have no idea how to forward original socket connection request to my handleWS, to be able to know which user just connected to server.
Another question is what is best practice to store these open sockets, to being able to send broadcast messages to all connected clients, and delete corresponding socket when it's closed.
First what is coming to my mind is to store sockets in a map like Map<int,CompatibleWebSocket>, where int is CompatibleWebSocket.hash().
Then I can iterate over Map.values() to do broadcast, and delete by key when connection is closed.
I can't get if that technique overcomplicated for such task and maybe exist more convenient way doing that, like storing them in a list? Or can I join Streams somehow for broadcasting?
I have a Cocoa client and server application that communicate using distributed objects implemented via NSSocketPorts and NSConnections in the standard way. The server vends a single object to the client applications, of which there may be several. Each client application can gain access to the same distributed object getting its own proxy.
The vended object supports a particular protocol, which includes a method similar to the following:
#protocol VendedObjectProtocol
- (void) acquireServerResource:(ServerResourceID)rsc;
#end
When a client application invokes this method, the server is supposed to allocate the requested resource to that client application. But there could be several clients that request the same resource, and the server needs to track which clients have requested it.
What I'd like to be able to do on the server-side is determine the NSConnection used by the client's method invocation. How can I do that?
One way I have thought about is this (server-side):
- (void) acquireServerResource:(ServerResourceID)rsc withDummyObject:(id)dummy {
NSConnection* conn = [dummy connectionForProxy];
// Map the connection to a particular client
// ...
}
However, I don't really want the client to have to pass through a dummy object for no real purpose (from the client's perspective). I could make the ServerResourceID be a class so that it gets passed through as a proxy, but I don't want to really do that either.
It seems to me that if I were doing the communications with raw sockets, I would be able to figure out which socket the message came in on and therefore be able to work out which client sent it without the client needing to send anything special as part of the message. I am needing a way to do this with a distributed objects method invocation.
Can anyone suggest a mechanism for doing this?
What you are looking for are NSConnection's delegate methods. For example:
- (BOOL)connection:(NSConnection *)parentConnection shouldMakeNewConnection:(NSConnection *)newConnnection {
// setup and map newConnnection to a particular client
id<VendedObjectProtocol> obj = //...
obj.connection = newConnection;
return YES;
}
You could design an object for each individual connection (like VendedObjectProtocol) and get the connection with self.connection.
- (void) acquireServerResource:(ServerResourceID)rsc {
NSConnection* conn = self.connection;
// Map the connection to a particular client
// ...
}
Or
You can make use of conversation tokens using +createConversationForConnection: and +currentConversation
I have been dabling in some Twisted and I've come across a problem. I'm implementing a couple servers that can be connected to using telnet localhost x
My code handles it like this:
reactor.listenTCP(12001, server1, ....)
reactor.listenTCP(12002, server2, ....)
etc.
These then use my factory to build a protocol
I'm wondering how I can get these servers to interact with each other. For example, let's say a client sends a request to update a value common across each server and I want to relay this value to the other servers to update their current value. I looked into reactor.connectTCP but this doesn't seem to do what I want. Is there a way to connect to my servers without using the telnet command?
Sure. Use normal Python techniques for this. Let's take your example of a value being updated. I'll make it a simple counter that might be incremented by connections to one server and the value sent out to connections to the other server:
from twisted.internet import reactor
from twisted.internet.protocol import ServerFactory, Protocol
class Counter(object):
def __init__(self):
self._value = 0
def increment(self):
self._value += 1
def get(self):
return self._value
class Incrementor(Protocol):
def connectionMade(self):
self.factory.value.increment()
self.transport.loseConnection()
class Reporter(Protocol):
def connectionMade(self):
self.transport.write("Value is %d\n" % (self.factory.value.get(),))
self.transport.loseConnection()
server1 = ServerFactory()
server1.protocol = Incrementor
server2 = ServerFactory()
server2.protocol = Reporter
server1.value = server2.value = Value()
reactor.listenTCP(12001, server1)
reactor.listenTCP(12002, server2)
reactor.run()
The two factories are now sharing a piece of mutable state. The protocols can access their factories because the default ServerFactory helpfully sets itself as an attribute on protocols it instantiates, so the protocols can also reach the shared mutable state on the factories. One increments the counter, the other retrieves the value.
You can construct many scenarios like this with shared state and method calls onto non-protocol, non-factory objects.