I have this code
context = zmq.Context()
app_worker = context.socket(zmq.PUSH)
app_worker.bind("tcp://127.0.0.1:9005")
app_sub = context.socket(zmq.SUB)
app_sub.connect("tcp://127.0.0.1:9004")
app_sub.setsockopt(zmq.SUBSCRIBE,'sometopic')
while True:
msg = app_sub.recv()
msg_data = msg.split(' ',1)
app_worker.send_json(msg_data[1])
print msg_data[1]
but when i run this, it is unable to receive any message from the publisher but when i comment this lines
app_worker = context.socket(zmq.PUSH)
app_worker.bind("tcp://127.0.0.1:9005")
it suddenly works. it is stated in the zeromq guide chapter 5 black box pattern that this is possible. if so, what am i doing wrong here?
You didn't supply enough data to solve this question with 100% assurances.
But based on what you did post the most obvious problem is that the port 9005 was already binded by someone else.
Its very likely your app_worker.send_json(msg_data[1]) is blocking (the entire thread) if there are no downstream nodes to PULL the messages.
Set the send_json to non blocking mode and check the error/exception returned
app_worker.send_json(msg_data[1], zmq.NOBLOCK)
The reason it "works" when you comment out the bind is because the send is just failing and not blocking.
Related
I'm working on a project where I want to have a Julia server process periodically publishing messages that involve some binary data. The initial client will be written in javascript, but we'd ultimately like the capability of having multiple clients, implemented in various ways. For that reason, I'd like to define the message using protocol buffers. I've turned up 3 Julia websocket implementations: WebSockets.jl, SimpleSockets.jl and HTTP.jl's WebSockets. I've done some naive experimenting with WebSockets.jl and got an error ("WebSockets does not support byte I/O"). My inclination is to shift my focus to the HTTP.jl implementation; I get the impression that it is under more active development than WebSockets.jl.
Update: I've continued with my experimentation. I was guided to a previous SO question, unable to write binary data in websocket, that was instructive. I modeled my server implementation after that in the link, yielding:
include("testmessage_pb.jl")
text = "A man spekith"
msg = TestMessage(someText=text)
function server(port)
#async HTTP.listen(Sockets.localhost, port) do http::HTTP.Stream
if HTTP.WebSockets.is_upgrade(http.message)
HTTP.WebSockets.upgrade(http, binary=true) do ws
while !eof(ws)
data = readavailable(ws)
IOExtras.startwrite(ws)
writeproto(ws, msg)
IOExtras.closewrite(ws)
end
end
end
end
end
Running it, I got an error very much like the previous one I reported:
HTTP.WebSockets.WebSocket{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}} does not support byte I/O
I'm wondering if anyone reading this has any experience with this particular cluster of technologies (julia, protobuf, websockets) and suggestions on how to proceed ("don't try" would count as useful feedback).
HTTP.jl websockets are working fine. Havn't tried it with protobuf, but hardly there should be any problem with that. It's basically
HTTP.WebSockets.open(data_url) do ws
x = readavailable(ws)
# Do protobuf related things, generate response
write(ws, response)
end
readavailable returns you UInt8[] so you can do whatever you want with it.
I was able to tweak the server implementation that I showed in the question, and it now works. It's a little clunky, and I'll hold off from accepting my own answer in hopes that a nicer approach comes along. In any event, here's the new version:
function server(port)
#async HTTP.listen(Sockets.localhost, port) do http::HTTP.Stream
if HTTP.WebSockets.is_upgrade(http.message)
HTTP.WebSockets.upgrade(http, binary=true) do ws
while !eof(ws)
data = readavailable(ws)
iob = PipeBuffer()
writeproto(iob, msg)
write(ws, take!(iob))
end
end
end
end
end
I'm learning ZeroMQ and trying to build a simple message queue in Python.
I noticed basically all code samples contain some kind of sleep() operation.
Even the hello world example on the ZeroMQ guide does, with the comment "Do some work".
I find this a little unclear, is the motivation to simulate the act of processing the message? Why is this necessary?
import time
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
# Wait for next request from client
message = socket.recv()
print("Received request: %s" % message)
# Do some 'work'
time.sleep(1)
# Send reply back to client
socket.send(b"World")
is the motivation to simulate the act of processing the message ?
Sort of yes. Launching while True: "without" any handbrake would be soon pretty ugly on screen with a literally endlessly running river of print()-s, wouldn't it?
Why is this necessary ?
Just a cheap SLOC / convenience-trick. Except for cases, where some latency needs to get injected, there is no technical reason for sleep()-s
I am creating an rabbitmq rpc in ruby 2.3 using bunny 2.7.0
I've made it with one reply queue per client. But I am expected to have quite a large amount of clients and it is not efficient to do it in this way. I want to use a direct reply feature of rabbitmq
connection = Bunny.new(rabbitmq_url, :automatically_recover => true)
connection.start
channel = connection.create_channel
reply_queue = channel.queue('amq.rabbitmq.reply-to', no_ack: true)
on the last line of code I receive error
Bunny::AccessRefused: ACCESS_REFUSED - queue name 'amq.rabbitmq.reply-to' contains reserved prefix 'amq.*'
in theory that is expected due to http://rubybunny.info/articles/queues.html
but on other hand - there is an article https://www.rabbitmq.com/direct-reply-to.html that describes existance an usability of this queue.
i want to declare a queue because i need to subscribe to it to receive respond
consumer = reply_queue.subscribe do |_, properties, payload|
# action
end
I dont understand what am I doing wrong with it (
there are similar topics with examples of such approach but created on other languages and tools like nodejs and that seems to work fine. What am I doing wrong with bunny ?
Update
found the problem - I used odler version of rabbitmq server. That one that id not support direct reply queue yet
I think it's trying to create it which you're not allowed to do.
https://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-September/030095.html
My ruby is a tad rusty but give this a try:
channel = connection.create_channel
channel.queue_declare('amq.rabbitmq.reply-to', :passive => true)
Fairly new to zeromq and trying to get a basic pub/sub to work. When I run the following (sub starting before pub) the publisher finishes but the subscriber hangs having not received all the messages - why ?
I think the socket is being closed but the messages have been sent ? Is there a way of ensuring all messages are received ?
Publisher:
import zmq
import random
import time
import tnetstring
context=zmq.Context()
socket=context.socket(zmq.PUB)
socket.bind("tcp://*:5556")
y=0
for x in xrange(5000):
st = random.randrange(1,10)
data = []
data.append(random.randrange(1,100000))
data.append(int(time.time()))
data.append(random.uniform(1.0,10.0))
s = tnetstring.dumps(data)
print 'Sending ...%d %s' % (st,s)
socket.send("%d %s" % (st,s))
print "Messages sent: %d" % x
y+=1
print '*** SERVER FINISHED. # MESSAGES SENT = ' + str(y)
Subscriber :-
import sys
import zmq
import tnetstring
# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5556")
filter = "" # get all messages
socket.setsockopt(zmq.SUBSCRIBE, filter)
x=0
while True:
topic,data = socket.recv().split()
print "Topic: %s, Data = %s. Total # Messages = %d" % (topic,data,x)
x+=1
In ZeroMQ, clients and servers always try to reconnect; they won't go down if the other side disconnects (because in many cases you'd want them to resume talking if the other side comes up again). So in your test code, the client will just wait until the server starts sending messages again, unless you stop recv()ing messages at some point.
In your specific instance, you may want to investigate using the socket.close() and context.term(). It will block until all the messages have been sent. You also have the problem of a slow joiner. You can add a sleep after the bind, but before you start publishing. This works in a test case, but you will want to really understand what is the solution vs a band-aid.
You need to think of the PUB/SUB pattern like a radio. The sender and receiver are both asynchronous. The Publisher will continue to send even if no one is listening. The subscriber will only receive data if it is listening. If the network goes down in the middle, the data will be lost.
You need to understand this in order to design your messages. For example, if you design your messages to be "idempotent", it doesn't matter if you lose data. An example of this would be a status type message. It doesn't matter if you have any of the previous statuses. The latest one is correct and message loss doesn't matter. The benefits to this approach is that you end up with a more robust and performant system. The downsides are when you can't design your messages this way.
Your example includes a type of message that requires no loss. Another type of message would be transactional. For example, if you just sent the deltas of what changed in your system, you would not be able to lose the messages. Database replication is often managed this way which is why db replication is often so fragile. To try to provide guarantees, you need to do a couple things. One thing is to add a persistent cache. Each message sent needs to be logged in the persistent cache. Each message needs to be assigned a unique id (preferably a sequence) so that the clients can determine if they are missing a message. A second socket (ROUTER/REQ) needs to be added for the client to request the missing messages individually. Alternatively, you could just use the secondary socket to request resending over the PUB/SUB. The clients would then all receive the messages again (which works for the multicast version). The clients would ignore the messages they had already seen. NOTE: this follows the MAJORDOMO pattern found in the ZeroMQ guide.
An alternative approach is to create your own broker using the ROUTER/DEALER sockets. When the ROUTER socket saw each DEALER connect, it would store its ID. When the ROUTER needed to send data, it would iterate over all client IDs and publish the message. Each message should contain a sequence so that the client can know what missing messages to request. NOTE: this is a sort of reimplementation of Kafka from linkedin.
I have to send a lot of data to I client connected to my server in small blocks.
So, I have something like:
for(;;) {
messageEvent.getChannel().write("Hello World");
}
The problem is that, for some reason, client is receiving dirty data, like Netty buffer is not clear at each iteration, so we got something like "Hello WorldHello".
If I make a little change in my code putting a thread sleep everything works fine:
for(;;) {
messageEvent.getChannel().write("Hello World");
Thread.sleep(1000);
}
As MRAB said, if the server is sending multiple messages on a channel without indicating the end of each message, then client can not always read the messages correctly. By adding sleep time after writing a message, will not solve the root cause of the problem either.
To fix this problem, have to mark the end of each message in a way that other party can identify, if client and server both are using Netty, you can add LengthFieldPrepender and LengthFieldBasedFrameDecoder before your json handlers.
String encodedMsg = new Gson().toJson(
sendToClient,newTypeToken<ArrayList<CoordinateVO>>() {}.getType());
By default, Gson uses html escaping for content, sometime this will lead to wired encoding, you can disable this if required by using a Gson factory
final static GsonBuilder gsonBuilder = new GsonBuilder().disableHtmlEscaping();
....
String encodedMsg = gsonBuilder.create().toJson(object);
In neither case are you sending anything to indicate where one item ends and the next begins, or how long each item is.
In the second case the sleep is getting the channel time out and flush, so the client sees a 'break', which it interprets as the end of the item.
The client should never see this "dirty data". If thats really the case then its a bug. But to be hornest I can't think of anything that could lead to this in netty. As every Channel.write(..) event will be added to a queue which then get written to the client when possible. So every data that is passed in the write(..) method will just get written. There is no "concat" of the data.
Do you maybe have some custom Encoder in the pipeline that buffers the data before sending it to the client ?
It would also help if you could show the complete code that gives this behavoir so we see what handlers are in the pipeline etc.