I'm working on a project where I want to have a Julia server process periodically publishing messages that involve some binary data. The initial client will be written in javascript, but we'd ultimately like the capability of having multiple clients, implemented in various ways. For that reason, I'd like to define the message using protocol buffers. I've turned up 3 Julia websocket implementations: WebSockets.jl, SimpleSockets.jl and HTTP.jl's WebSockets. I've done some naive experimenting with WebSockets.jl and got an error ("WebSockets does not support byte I/O"). My inclination is to shift my focus to the HTTP.jl implementation; I get the impression that it is under more active development than WebSockets.jl.
Update: I've continued with my experimentation. I was guided to a previous SO question, unable to write binary data in websocket, that was instructive. I modeled my server implementation after that in the link, yielding:
include("testmessage_pb.jl")
text = "A man spekith"
msg = TestMessage(someText=text)
function server(port)
#async HTTP.listen(Sockets.localhost, port) do http::HTTP.Stream
if HTTP.WebSockets.is_upgrade(http.message)
HTTP.WebSockets.upgrade(http, binary=true) do ws
while !eof(ws)
data = readavailable(ws)
IOExtras.startwrite(ws)
writeproto(ws, msg)
IOExtras.closewrite(ws)
end
end
end
end
end
Running it, I got an error very much like the previous one I reported:
HTTP.WebSockets.WebSocket{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}} does not support byte I/O
I'm wondering if anyone reading this has any experience with this particular cluster of technologies (julia, protobuf, websockets) and suggestions on how to proceed ("don't try" would count as useful feedback).
HTTP.jl websockets are working fine. Havn't tried it with protobuf, but hardly there should be any problem with that. It's basically
HTTP.WebSockets.open(data_url) do ws
x = readavailable(ws)
# Do protobuf related things, generate response
write(ws, response)
end
readavailable returns you UInt8[] so you can do whatever you want with it.
I was able to tweak the server implementation that I showed in the question, and it now works. It's a little clunky, and I'll hold off from accepting my own answer in hopes that a nicer approach comes along. In any event, here's the new version:
function server(port)
#async HTTP.listen(Sockets.localhost, port) do http::HTTP.Stream
if HTTP.WebSockets.is_upgrade(http.message)
HTTP.WebSockets.upgrade(http, binary=true) do ws
while !eof(ws)
data = readavailable(ws)
iob = PipeBuffer()
writeproto(iob, msg)
write(ws, take!(iob))
end
end
end
end
end
Related
I see the basic example for Elixir's WebSockex library here but it doesn't really explain how I can expose my own websocket to the internet. This question has answers which explain how to chat to an existing websocket externally, but I want to expose my own websocket. I'm actually using websockex as part of a Phoenix application, so perhaps bits of Phoenix might help here?
I obviously know the ip:port combo of my phoenix application so given these, how do I expose a websockex websocket on that ip:port? In other words, what should I pass as the URL? in this basic example code:
defmodule WebSocketExample do
use WebSockex
def start_link(url, state) do
WebSockex.start_link(url, __MODULE__, state)
end
def handle_frame({type, msg}, state) do
IO.puts "Received Message - Type: #{inspect type} -- Message: #{inspect msg}"
{:ok, state}
end
def handle_cast({:send, {type, msg} = frame}, state) do
IO.puts "Sending #{type} frame with payload: #{msg}"
{:reply, frame, state}
end
end
Please note that I need to expose a raw websocket, not a Phoenix channel, as the consumer doesn't understand Phoenix channels. If Phoenix can expose a raw websocket then I'll consider that a solution too.
If neither Phoenix nor WebSockex can help, what are my options?
Websockex is a client library, I don't think it has any code for exposing a websocket. Since you're already using phoenix, you probably can do what you need with phoenix channels.
If you're on cowboy (and you probably are, since it's the default), then you can also use it to expose a raw websocket. However, it requires some fiddling with routing. You will need to replace YourAppWeb.Endpoint with a manual configuration of cowboy:
{
Plug.Cowboy,
scheme: :http,
plug: YourAppWeb.Endpoint,
options: endpoint_options(),
dispatch: [
_: [
# Dispatch paths beginning with /ws to a websocket handler
{"/ws/[...]", YourApp.WebsocketHandler, []},
# Dispatch other paths to the phoenix endpoint
{:_, Plug.Cowboy.Handler, {YourAppWeb.Endpoint, endpoint_options()}}
]
]
}
I have honestly only done this with raw plug, so you might need to convert the endpoint to be a Plug instead of a Phoenix.Endpoint. Then, you need to implement YourApp.WebsocketHandler to conform to cowboy's API and perform a websocket upgrade (and handle sending/receiving messages), as described in cowboy docs. You can also see this gist for a more fleshed-out example.
WebSockex implements many callbacks, including, but not limited to WebSockex.handle_connect/2. It holds WebSockex.Conn in a state and passes it to all callbacks.
WebSockex.Conn is a plain old good struct, having socket field.
So from any callback (I’d do it from WebSockex.handle_connect/2) you might share this socket with the process which needs it and use it then from there.
Also, you can borrow some internals and check how the connection is being created.
You’ll see it uses WebSockex.Conn.new/2 that returns an initialized connection, that, in turn, holds a socket. In that case, you’ll be obliged to supervise the process that holds the socket manually.
The power of OSS is all answers are one mouse click far from questions.
I'm learning ZeroMQ and trying to build a simple message queue in Python.
I noticed basically all code samples contain some kind of sleep() operation.
Even the hello world example on the ZeroMQ guide does, with the comment "Do some work".
I find this a little unclear, is the motivation to simulate the act of processing the message? Why is this necessary?
import time
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
# Wait for next request from client
message = socket.recv()
print("Received request: %s" % message)
# Do some 'work'
time.sleep(1)
# Send reply back to client
socket.send(b"World")
is the motivation to simulate the act of processing the message ?
Sort of yes. Launching while True: "without" any handbrake would be soon pretty ugly on screen with a literally endlessly running river of print()-s, wouldn't it?
Why is this necessary ?
Just a cheap SLOC / convenience-trick. Except for cases, where some latency needs to get injected, there is no technical reason for sleep()-s
I have to send a lot of data to I client connected to my server in small blocks.
So, I have something like:
for(;;) {
messageEvent.getChannel().write("Hello World");
}
The problem is that, for some reason, client is receiving dirty data, like Netty buffer is not clear at each iteration, so we got something like "Hello WorldHello".
If I make a little change in my code putting a thread sleep everything works fine:
for(;;) {
messageEvent.getChannel().write("Hello World");
Thread.sleep(1000);
}
As MRAB said, if the server is sending multiple messages on a channel without indicating the end of each message, then client can not always read the messages correctly. By adding sleep time after writing a message, will not solve the root cause of the problem either.
To fix this problem, have to mark the end of each message in a way that other party can identify, if client and server both are using Netty, you can add LengthFieldPrepender and LengthFieldBasedFrameDecoder before your json handlers.
String encodedMsg = new Gson().toJson(
sendToClient,newTypeToken<ArrayList<CoordinateVO>>() {}.getType());
By default, Gson uses html escaping for content, sometime this will lead to wired encoding, you can disable this if required by using a Gson factory
final static GsonBuilder gsonBuilder = new GsonBuilder().disableHtmlEscaping();
....
String encodedMsg = gsonBuilder.create().toJson(object);
In neither case are you sending anything to indicate where one item ends and the next begins, or how long each item is.
In the second case the sleep is getting the channel time out and flush, so the client sees a 'break', which it interprets as the end of the item.
The client should never see this "dirty data". If thats really the case then its a bug. But to be hornest I can't think of anything that could lead to this in netty. As every Channel.write(..) event will be added to a queue which then get written to the client when possible. So every data that is passed in the write(..) method will just get written. There is no "concat" of the data.
Do you maybe have some custom Encoder in the pipeline that buffers the data before sending it to the client ?
It would also help if you could show the complete code that gives this behavoir so we see what handlers are in the pipeline etc.
There is a great example of a chat app using Server-Sent Events by Konstantin Haase. I am trying to run it and have a problem with callbacks (I use Sinatra 1.3.2 and browse with Chrome 16). They simply do not run (e.g. after page reload), and therefore the number of connections is growing.
Also, connection is closed in 30-60 sec unless one sets a periodic timer to send empty data, as suggested by Konstantin elsewhere.
Can you replicate it? If yes, is it possible to fix these issues somehow? WebSockets work seamlessly in this respect...
# ruby
get '/stream', provides: 'text/event-stream' do
stream :keep_open do |out|
EventMachine::PeriodicTimer.new(20) { out << "data: \n\n" } # added
settings.connections << out
puts settings.connections.count # added
out.callback { puts 'closed'; settings.connections.delete(out) } # modified
end
end
# javascript
var es = new EventSource('/stream');
es.onmessage = function(e) { if (e.data != '') $('#chat').append(e.data + "\n") }; // modified
This was a bug in Sinatra https://github.com/sinatra/sinatra/issues/446
Neat bit of code. But you're right, WebSockets would address these problems. I think there are two problems here:
1) Your browser, the Web server, or a proxy in-between may shut down your connection after a period of time, idle or not. Your suggestion of a periodic timer sending empty data will help, but is no guarantee.
2) As far as I know, there's no built-in way to tell if/when one of these connections is still working. To keep your list of connections from growing, you're going to have to keep track of when each connection was last "used" (maybe the client should ping occasionally, and you would store this datetime.) Then add a periodic timer to check for and kill "stale" connections.
An easier, though perhaps uglier option, is to store the creation time of each connection, and kill it off after n minutes. The client should be smart enough to reconnect.
I know that takes some of the simplicity out of the code. As neat as the example is, I think it's a better candidate for WebSockets.
I'm creating a ruby server which is connecting to a TCP client. My server is using a TCPServer and I'm attempting to use TCPServer::recv(), but it doesn't wait for data, so just continues in a tight loop until data is received.
What is the most efficient way to process intermittant data? I'm unable to change the data being sent in since I'm attempting to emulate another server. Which read like statement from TCPServer/TCPSocket would wait for data being sent?
require "socket"
dts = TCPServer.new('localhost', 20000)
s = dts.accept
print(s, " is accepted\n")
loopCount = 0;
loop do
Thread.start(s) do
loopCount = loopCount + 1
lineRcvd = s.recv(1024)
if ( !lineRcvd.empty? )
puts("#{loopCount} Received: #{lineRcvd}")
s.write(Time.now)
end
end
end
s.close
print(s, " is gone\n")
Thanks for your time.
are you sure recv isn't returning "" -- meaning the socket is closed?
If not then perhaps your sockets are set to non blocking somehow?
EventMachine is indeed far faster than using threads for socket programming :)
GL.
-r
Based on the questions you've been asking, I think you should try a framework like EventMachine and write a server that implements what you want instead of trying to fuss around with writing a server wrapper.
That being said, the most efficient way to read from a socket is to use a proper select call and poll all the open connections. While this is a fairly academic thing for anyone who's developed client-server applications before, it is a nuisance because there are a lot of things you can easily get wrong. For example. handling multiple connections can lead to all kinds of troublesome situations if you're not especially careful to avoid blocking calls.
The EventMachine framework makes it easy to develop query/response-type servers because you can always start with a template and work from there, for example, the built-in EventMachine::Protocols::LineAndTextProtocol one works as a great basis for most.