I am implementing a very simple websocket server using Netty (v4).
I am basing my code on the websocket example source code that comes with Netty.
Do I need to do anything extra to handle framing/fragmentation/packet boundaries?
Or can I assume that the example code reassembles the data correctly?
The example uses calls like:
String request = ((TextWebSocketFrame) frame).text();
or:
ctx.channel().write(new TextWebSocketFrame(request.toUpperCase()));
So it doesn't seem to be dealing with fragmentation (at that layer at least).
Many Thanks.
The frames are handled as complete frames. If you need to aggregate also ContinumWebSocketFrames you can put WebSocketFrameAggregator in the ChannelPipeline.
Related
Best way to send & receive an array over websockets
On the client side I'm using javascript that sends the data.
I'm sending to a microcontroller (ESP8266) that is programmed in c++ using the websocket library with the arduino IDE
At the moment I'm sending the variable which I build up on the client side.
It is then sent to the microcontroller and received by the payload buffer.
I am sending this from the client
#,tank,pond,1537272000,1537272000,Normal,4789,12
I received here in the code:
case WStype_TEXT: Serial.printf("[%u] get Text: %s\n", num, payload);
this is the result of what I receive
[0] here it is: #,tank,pond,1537272000,1537272000,Normal,4789,12
I am using a hash(#) to mark the start of the data.
I have been googling and searching forums for days but can't fathom which is the best way to do this.
What is the fastest most elegant code to split this up into different variables so that they can be compared?
Briefly, I prefer JSON. You can use the library below for ESP8266:
https://github.com/bblanchon/ArduinoJson
I think your websocket server is written in a high level language so neither JSON parsing nor composing will not be an issue for you.
Good luck.
We're using Kafka as a broker which takes notifications from different message sources and then routes them to one or more target apps like Slack or E-Mail. Having such an approach it is necessary to convert the Kafka message into different output formats like JSON or E-Mail before they are sent to the apps.
I thought of having Microservices with SpringBoot at the target ends which takes the message from Kafka, converts it using one of the common template languages like Velocity or Freemarker into the target format and then forwards the converted result to the given target app.
Would you agree with such an approach or are there better ways, some caveats or even no-gos to do it this way? What about performance? Any experience in this?
Thanks for your honest assessment.
Why not have a single serialization format and let each service deserialize the payload for their use case? Templating with something like Velocity or Freemarker seems like a specific concern independent of the data used to populate the template. Maybe focus on broadcasting the raw data.
I have created a OSB Proxy Service(Messaging Service) which loading the data with a MFL file.
The format of data is:
1/1/2007;00:11:00;2.500;0.000;242.880;10.200;0.000;0.000;0.000;
1/1/2007;00:12:00;2.494;0.000;242.570;10.200;0.000;0.000;0.000;
All the data records are : 2075259
The total size of file(.txt or .data) is : 130MB.
Which is best way to handling all these data in order to inserted to an OSB Proxy and transformed all the data in a simple xml file?
I have tested with a small size of records(5000) and it works as expected but how i should insert all this data in the proxy?
The MFL transformation is a valid idea or i should create a FileAdapter Proxy which will received the data from a dbtable?
Please for your suggestion
Thank you in advance.
ESBs are efficient at handling messages in the order of KBs, not MBs, although this is very subjective and depends a lot on the number of concurrent requests, transactions per second, sizing of hardware etcetera. As Trent points out in a comment, you could implement a claim check pattern and delegate the file transformation to an external utility, such as perl or similar.
Using Talend ESB 5.2.0, I want to create a mediation route that will call a processing job on the payload of an inbound request to a CXF messaging endpoint, however my current implementation is suffering some performance issues with large payloads.
I’ve investigated the issue and found that the bottleneck is in marshalling my inbound XML payload from the tRouteInput component to the internal row structure for processing, using a tXMLMap.
Is it possible, using a built-in type converter in the route, to marshal the internal row structure from the route and stream through POJOs or transport objects that are cheaper to process in the job? Or is there a better way to marshal XML to Talend’s internal row structure from a route using a less expensive transform?
Any thoughts would be welcome.
Cheers,
mids
It turns out that the issue was caused by the format of the inbound XML payload - having more than one loop element mapping to separate output flows from the tXMLMap generates relative links for each item for each output flow, enabling more advanced processing involving the loops if required.
This caused the large memory overhead that led to poor throughput.
Not requiring any more advanced processing in the XML to Talend row conversion, we overcame this issue by splitting the payload to its distinct loop elements using tReplicate and tExtractXMLField components before mapping out of the XML in separate tXMLMaps to avoid the auto-generation of those links.- mids
I'm creating a client-server relationship whereby a single client will be connected to an arbitrary number of servers using persistent TCP connections. The actual number of servers is as-of-yet undetermined, but the design goal is to shoot for 1000.
I found an example using direct Java NIO that nearly completely matches my mental model of how this could work:
http://drdobbs.com/jvm/184406242
In general, it opens up all of the channels and adds them to a single thread monitoring java.nio.channels.Selector. The use of the Selector, in particular, is what allows this to scale far better than using the standard thread-per-channel.
I would rather use a (slightly) higher level socket framework like Netty, than direct Java NIO. Unfortunately, I have not been able to determine how Netty would handle a case like this. That is, the examples and discussions I've found all tend to center around the server side, with accepting scores of concurrent connections.
But what about doing this from the client side? If I create a large number of channels and just wait on their events, how is Netty going to handle this at the back-end?
This isn't a direct answer to your question but I hope it is helpful nonetheless. Below, I describe a way for you to determine the answer that you are looking for. This is something that I recently did myself for an upcoming project.
Compared to OIO (Old IO) the asynchronous nature of the Netty framework and NIO will indeed provide much better memory and CPU usage characteristics for your application. The way buffers are handled in Netty will also be of benefit as it will help you to avoid copying byte buffers. The point is that all of the thread pool and NIO details will be handled for you allowing you to focus on your business logic. You mentioned the NIO Selector and you will benefit from that; the nice thing about Netty is that you get the benefits without having to worry about that implementation yourself because it is already done for you.
My understanding of the client side is that it is very similar to the server side and should provide you with commensurate performance gains (as long as your business logic doesn't introduce any performance issues).
My advice would be to throw together a prototype that more or less does what you want. Leave out any time consuming details and just add in the basic Netty handlers that you need to make something that works.
Then I would use jmeter to invoke your client to apply load to the server and client. Using something like jconsole or jvisualvm will show you the performance characteristics of the client and server under load. You could also try jprobe. You can add a listener in jmeter that will indicate the throughput. I would advise to use jmeter in server mode, the client on another machine and the server on yet another. This is a bit of up front work but if you decide to move forward you will have these tools ready to go for further testing as your proceed.
I suspect a decent Netty implementation that doesn't introduce any extraneous poorly performing components will give you the performance characteristics you are looking for, but, the only way to know for sure is to measure the system under the expected load.
You need to define what the expected load looks like and the desired performance characteristics under such load. Given these inputs you can measure your system to find out if it will meet your expectations. I personally don't think anyone can tell you if it will behave in the desired manner. You have to measure it. It's the only reliable way to know if the system can meet your needs.
I would rather use a (slightly) higher level socket framework like Netty, than direct Java NIO.
This is the correct approach. You can try implementing your own NIO server and client but why do that when you have the benefit of a highly refined framework at your fingertips already?
Netty will use up to x worker threads that handle the work for you. Each worker thread will have one Selector that is used to register Channels to it. The number of used workers is configurable and by default 2 * cpu-count.
As you can see in the example from Netty's doc [http://netty.io/docs/stable/guide/html/#start.9][1] you can control exactly the number of worker threads (meaning the number of underlying selectors) on the Client side.
Netty solves a numbers of issues that are very hard to handle in a simple way such as NIO vs SSL, and have a lot of default encoder/decoder for Zip... etc.
I started using Netty a few week ago and it was quite fast to came into. (I recommend dowloading the project with all the example code inside, there is a lot of documentation in it that can not be found on the url above.
ChannelFactory factory = new NioClientSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool());
ClientBootstrap bootstrap = new ClientBootstrap(factory);
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() {
return Channels.pipeline(new TimeClientHandler());
}
});
bootstrap.setOption("tcpNoDelay", true);
bootstrap.setOption("keepAlive", true);
bootstrap.connect(new InetSocketAddress(host, port));
Good luck,
Renaud