For the past few days I've been struggling to analyze the traffic that an IoT Edge device generates when transmitting data to IoT Hub.
The metrics in the hub fall in line perfectly with my expectation of around 120KB per hour, which is roughly the size of the messages I`m sending with the module client.
But when I monitor the network traffic of the device the result is 20MB sent and around 10 MB received, for a total of over 30MB per hour for AMQPS which is a huge difference.
Has anyone encountered this and is there some way to find out the reason for the discrepancy in the data.
IoT Hub provides several metrics to give you an overview of the health of your hub and the total number of connected devices. In your mentions, the metric only included d2c messages. The communication between the client and service includes not only d2c messeage protocol but also other protocols.
Azure IoT Edge bridges the traffic to IoT Hub over AMQP 1.0. It plugs in components for specialized processing such as custom authentication, message transformations, compression/decompression, or encryption/decryption of traffic between the devices and IoT Hub.
The Azure IoT protocol gateway and MQTT/AMQP implementation are provided in an open-source software project.You can refer to Microsoft.Azure.Devices.Edge.Hub.Amqp.
This ended up being a bug in IoT Edge runtime and was resolved with version 1.0.2 more info can be found on GitHub
Related
I am working on a project where I want to use MQTT, some of my project requirements are around 25k clients connected and a message rate around 4000 messages/sec, after looking some open-source broker option I have been making some test whit mosquitto.
I am using a software called JMTER(it can simulate by threads the clients and messages that I need).
The machine where I am doing the test has 2 Intel® Xeon® Processors E5-2620 v3, it has 6 cores and 12 threads each one, and 9 GB ram,15M Cache, 2.40 GHz, my OS is windows server 2012 R2, so I have a good machine to host a broker.
To monitor my testing attempts I use MQTT explorer which is a plugging specially design to mosquitto.
I have been making some test trying 2k clients (1k publishing 1000 messages/second during 15 seconds, the message was “Hola1” and also 1k subscribers) these numbers where the highest ones that I could get, every time I tried to pass this number of clients mosquitto just died and I lost the connections.
I have been looking in web sites and some people say that mosquitto can handle up to 100k connections, some people say that you can configure your broker to support more connections but I haven’t figured out the way to configure my broker, is there any guide or documentation available to do this?
Anybody here with some experience in websockets and webRTC using TURN/STUN servers?
Requirement:
Send real-time video feed from local IP to browser in an external network and I need some help implementing via raspberry pi 3b+. My camera source is android device, and using 3rd party apps I am able to generate the video feed over local network. Using the same app I can stream via Youtube Live,but getting a latency of about 2 secs in ultra low latency mode and dvr enabled. And I am trying to reduce the latency of the stream.
Q1. Do the semi-public TURN server provide a one to one peer. Or anyone can just jump into the URL and view and override what I am streaming?Please provide few list of service providers.
Just for information there would be 1-2 users browser connected at max.
Q2. Do I need Janus gateway to send webRTC/websockets data into the TURN/STUN server? Since my raspberry is connected to a different network and I cannot port forward due to carrier constraints.
Q3. Do I need both STUN/TURN servers or do I even need webRTC instead of websockets to send my video stream over the internet. Is websockets not sufficient?
Q4. Since we are not implementing over local network do we need to install coTURN too on raspberry pi?
Q5. Is there any android app that can publish the data from camera to websocket/werRTC server with a public ws URL?
Any help would be really helpful.
Q1. TURN servers relay media. They do this by allocating for every connecting peer a relay port between 49152–65535. This relay port will then be used to transmit the media to the second peer. The peers will know which relay ports to use automatically since this is part of the ice gathering process. To get back to your question: Other Peers cannot write to that relay port, it is 1 to 1 with handshakes, there is no chance of someone else overwriting it.
Q2. You definitely do not need a Janus Gateway to use TURN. TURN and STUN will probably work fine for NAT-Traversal without port forwarding.
Q3. You need at least a TURN server (but you ideally want to use 1 STUN server and 1 TURN server). STUN will work in most cases, but will fail if there are firewalls or complicated NATs, which block inbound udp connections. TURN is just the fallback for those cases.
Needing WebRTC? For just streaming videos, it depends on the use case. A sequence of images can be transmitted over websockets, they can handle Blobs fine. But you won't have a very fluent, high fps AND high resolution video stream this way. And of course, I know of no usable way to transmit audio over websocket.
Q4. The raspberry pi is a Peer that transmits media? Peers do not need a local TURN server installation, you will only need 1 TURN server (which should not be behind a NAT, probably running on some web server). The TURN server is a separate instance.
EDIT
For your private testing and development purposes, you may use https://numb.viagenie.ca/ . I don't know much about commercial turn server hosters, except that some exist. For someone who owns a v-server or root server, installing coTURN may be an option, this Tutorial might be helpful. To check if the server is working, I also found this snippet to be very useful.
END EDIT
Q5. There is no android app that publishes webRTC streams to a ws URL since websocket
messages are used by webrtc only for signalling (that is, telling peers their host candidates, those are the IP adresses and ports learned by the ice gathering process, this includes the TURN and STUN ip and port combinations).
Couldn't find a clear answer to either:
WebSockets: There is support for WebSockets (http://www.pubnub.com/websockets/) and socket.io, however do the other SDKs use web sockets?
XMPP: Does PubNub use it as a communication protocol?
PubNub WebSockets and/or XMPP
Update 2019 🌟 PubNub is planning to add additional protocols. MQTT is supported today mqtt.pubnub.com, additionally we will be adding WebSockets and SEE and connectionless push with UDP.
At PubNub we use many protocols in our Client SDKs starting with an always-on forever lived TCP Socket. Our TTL policy on TCP Sockets is unlimited. We provide the best protocol and we roll in updates under the covers so developers don't have to sweat the details of how messages are delivered.
The PubNub Data Stream Network believes in a protocol independent open mobile web; meaning that we will use the best protocol to get connectivity through any environment. Protocols, like WebSockets, can get tripped up by cell tower switching, double NAT environments, and even some anti-virus software or proxy boarder authorities.
PubNub provides client libraries specifically so we can auto-switch the protocol and remove socket-level complexities making it easy for developers to build apps that can communicate in realtime.
PubNub has deployed a variety of protocols over time, like WebSockets, MQTT, COMET, BOSH, long polling and others. We are exploring currently prototyping future designs using SPDY, HTTP 2.0, and others. The bottom line is that PubNub will work in every network environment, and has very low network bandwidth overhead, as well as low battery drain on mobile devices compared to connection based push implementations.
Can one say an architecture using websocket technology is based on client-server model?
By definition The client–server model is a distributed application structure that partitions tasks or workloads between the providers of a resource or service, called servers, and service requesters, called clients.
However using the websocket technology, two endpoints can both act as providers of a resource or service and also service requesters.
Say for example in a situation where the two endpoints are: a user device with a gps sensor and a computer machine, both connected in the network using websocket. And the computer machine is sending requests to obtain the current position of the user device (here the user device is acting as a resource provider and the computer machine as a requester). Later on the user device uses the websocket connection to request all its positions on the last 5 days to the computer machine (now the user device is acting as the requester and the computer machine as the resource provider).
If both devices can act as resource provider and requester, are they complying with the client-server model definition or not?
No it's not breaking anything. End Points are not devices they are connections between devices.
ie if we were asking each other questions and answering them
There would two connections between two 'devices' giving four endpoints. You to me and me to you. No conflict.
TCP is full duplex capable, and particularly WebSockets are full duplex. As #Tony Hopkinson pointed out, there is no conflict at all. This means, you can write and read at the same time.
WebSockets are push technology, more suited for events; while usual request-response models are pull technology.
You can have both client-server or peer to peer architectures with push approach, but pull is the normal choice for pull architectures.
Peer-to-peer Architecture: A peer-to-peer network is designed
around the notion of equal peer nodes simultaneously functioning as
both "clients" and "servers" to the other nodes on the network. This
model of network arrangement differs from the client–server model
where communication is usually to and from a central server. A typical
example of a file transfer that uses the client-server model is the
File Transfer Protocol (FTP) service in which the client and server
programs are distinct: the clients initiate the transfer, and the
servers satisfy these requests.
You can also provide a mix of peer-to-peer and client-server. For example, you can do requests via WebSocket, and at the same time, the server could send updates on its own initiative. I don't understand what you mean with "breaking the model". WebSocket is just a communication channel. In your app both models can coexists and use the same communication channel.
we using Tibco RVRD for both Unix and windows as the messaging system. Just wonder, other than buy HAWK from Tibco, is there anyway to measure the network usage, before and after RVRD compression?
There is a really great tool for this called Rai Insight
Basically what it can do is to sit on a box and silently listen all the multicast data and represent statistics even in real time. We used it to monitor traffic flow spikes with just few seconds delay.
It can give you traffic statistics braked down by multicast group, service number or even sending machine. Traffic flow peak/average, retransmission rate peak/average. All you can think of.
I haven't really used it for such, but the rvrd web gui (default http://server:7580) provides some statistics on inbound/outbound messages and bytes.