I'm working on IoT project where I receive datastream in CoAP protocol.
I want to process the data in Heron by doing some transformations on top of it.
Is it possible to integrate CoAP protocol objects to heron?
I think it is not very important from the CoAP endpoint's view where do you put the received data.
Use this link as a initial point:
http://coap.technology/impls.html
There you can find brief descriptions for implementations for several languages/platforms.
Unfortunately, I am not familiar with Twitter Heron and don't know which language is best for implementing a Heron data provider.
If such language is Java or Heron is language-agnostic (say, has a REST API as a primary interface) - I'd consider the https://eclipse.org/californium/ as a very mature implementation. That way (sure in 5000ft view as I don't know the details) you could write an app which uses Californium and CoapHandlers might push data to Heron.
Related
I'm developing an application that must both handle events coming from other systems and provide a REST API. I want to split the applications into micro services and I'm trying to figure out which approach I should use. I drew attention to the Spring Cloud Netflix and the Spring Cloud Data Flow toolkit, but it's not clear to me whether they can be integrated and how.
As an example, suppose we have the following functionality in the system:
1. information about users
card of orders
product catalog
sending various notifications
obtaining information about the orders from third-party systems
processing, filtering, and transformation of order events
processing of various rules based on orders and sending notifications
sending information about user orders from third-party systems to other users using websockets (with pre-filtering)
Point 1-4 - there I see the classical micro service architecture. Framework - Spring Netflix Stack.
Point 5-9 - it's best to use an event-driven approach. Toolkit - Spring Data Flow.
The question is how to build communication between these platforms.
In particular - POPULATE ORDER DETAILS SERVICE must transform the incoming orders and save additional information (in case it needed) in the database. ORDER RULE EXECUTOR SERVICE should obtain information about the current saved rules, execute them and send notifications. WEB SOCKET SERVICE should send orders information only if a particular user has set the filters, and ORDER SAVER SERVICE should store the information about the transformed orders in the database.
1.
Communication between the micro-services within the two platforms could be using the API GATEWAY, but in this case, I have the following questions:
Does the Spring Cloud platform allow to work with micro services that way?
Performance - the number of events is very huge, which can significantly slow down the processing of events. Is it possible to use other approaches, for example, communication not through the API Gateway but through in-memory cache?
2.
Since some functionality intersects between these services, I have a question about what is "microservice" in the understanding of the Spring Cloud Stream framework. In particular, does it make sense to have separate services? Can the microservice in the Spring Cloud Stream have a REST API, work with the database and simultaneously process the events? Does such a diagram make sense and is it possible to build such a stack at the moment?
The question is which of these approaches is more correct? What did Spring Data Streams mean by "microservice"?
Given the limited information in the post, it is hard to convince on all the matters pertaining to this type of architecture, but I'll attempt to share some specifics, and point to samples. Also for the same reasons, it is hard to solve for your needs end-to-end. From the surface, it appears you're attempting to build event-driven applications and wondering whether Spring Cloud Stream (SCSt) and Spring Cloud Data Flow (SCDF) could help.
They can, yes.
The Order, User, and Catalog seem like domain objects and it would all come together to solve for a use-case. For instance, querying for a number of orders for a particular product, and group by the user. There are a few samples that articulate the data flow between the entities to solve similar problems. Here's a live code-walkthrough of event-driven systems in action. There's another example of social-graph application, too.
Though these event-driven applications can run standalone as individual services with the help of of message broker (eg: Kafka or RabbitMQ), you could of course also register them in SCDF and use them in the SCDF DSL to build a coherent data pipeline. We are expanding on more direct capabilities in SCDF for these types of use-cases, but there are ways to orchestrate them today with current abilities, too. Follow spring-cloud/spring-cloud-#2331#issuecomment-406444350 for more details.
I hope this gives an idea. Try to build something small using SCSt/SCDF, prove it out, and expand to more complex use-cases.
I have been toying with the idea of using gRPC for 'IoT' type devices; not very constrained things like sensors; more like single board computer inbuilt devices like robots, drones and the like. What is needed a interface between device and centralized controller as the devices are developed separately and possible by other companies. So a versioned interface language is a must; and it should not be just in a word document; something programmable like a header file or WSDL or IDL or ProtocolBuffer. Also between device and controller the behavior should be specified for common use case like registration, re-registration etc.This can be in word file or in some non technical document.
The (rpc) interface specification in Protocol Buffer (ver 3) along with the efficient implementation via gRPC is leading me to choose this over CoAP/ LWM2M (Leshan Java and C++ implementation).
Having used both LWM2M and grPC, I would say that gRPC is more developer friendly; interface definition and implementation is fast,compared to OMA LWM2M process.Of course there is no Observer-Notify in gRPC, but for that MQTT should suffice.
Strictly one cannot compare LWM2M to gRPC. LWM2M is not just the interface but defines behavior in lot of IoT cases like BootStrap, Registration, KeepAlive , SW Upgrade etc and its universal HTTP like GET, PUT on an URL type addressable resource makes it very complete. However most of these behaviors can be custom defined with some effort.
Some of the IoT things which we plan to orchestrate are far from little brained devices like bulbs, and more like robots. Has anyone used gRPC for similar purposes. Any success of failure stories to share
The thing I would miss in gRPC for IoT are the MQTT MQ capabilities like queueing of messages, broker bridging QoS Parameter. Or for CoAP the Out of Band messages over SMS Transport. In this context gRPC is "just" an RPC framework which assumes to be always connected over TCP.
We were thinking in the same solution as Joe's, which includes CoAP + Protobuf:
Protobuf & its IDL are a really helpful way to define and govern our APIs in a centralized way. Google's api-linter as well as AIPs are high quality resources for keeping everything sane.
CoAP for transport: Lightweight protocol for IoT devices. It's available in practically every RTOS/embedded/platform and it's adapted to the low bandwitdh we usually face in this world. Being the protocol chosen by Thread and the support Project Connected Home Over IP is giving to it does also help.
Use Protobuf for the messages no matter they are Request/Response or even event messages pushed by the IoT device. An already solved problem by nanopb which is a protobuf wire compatible C code generator for embedded systems.
Generate our own stubs based on the IDL to create our own wrapper over CoAP and nanopb code. Unary calls are supported and also server streaming by leveraging this to the CoAP observable mechanisms.
I have taken the plunge and used it in a project with connected 'Devices'; These are small computers like Raspberry-pi. Overall it has been good experience; and languages used are C++ and Java mainly and also JavaScript in Node.js . We have used this as Dockerized microservices; Load Balancing is what we have not done; and I read that HTTP/2 based load balancing is tricky; Will update that part; planning to use Kubernetes for that. Overall Container technology with versioned interfaces - GRPC seems like a good fit for (micro) services
I use an esp32 and R_Pi with CoAP and protobufs. As far as I know gRPC is not supported for esp32/8266. I'm pretty happy with it, but didn't do any concrete testing against lwm2m. Implementation is here
Any ideas how to start with MapServer with NativeScript. I don't have necessary skills yet to make own plugin but I must try if nothing else is found.
Does anyone know any resources to get started implementing mapserver maps for nativescript. Plugin, code anything.
I assume you have an application using NativeScript framework and you want to include some map services from MapServer.
If that is the case, you may not need to write your own plugin but you still do some level of coding to achieve the goal.
My recommendation would be :
1) At your front end, choose a GIS map library such as Google maps API, Leaflet or OpenLayers API. The GIS library would handle a lot of critical tasks in the process such as handling communication with GIS server, user interface function etc. So that you don't have to reinvent the wheels again.
https://developers.google.com/maps/
2) Based on your system requirement, you may or may not need a GIS server such as MapServer. Data could come from directly from clients for example. But since you mentioned it i would think you may need a GIS server to publish some data for the application to consume the services.
MapServer services via OGC standard protocols are easy to integred with most of the popular GIS libraries mentioned earlier. So once you publish Map service (wms) for example, it would be as easy as add one layer in such GIS library point to the service endpoint you published via Http request and by passing some parameters you would be able to get the GIS contents.
This is the link about how to consume MapServer service on the client side:
http://mapserver.org/ogc/wms_client.html
How does telepat-io differ from socket-io and other socket based real time sytems ? what is the underlying technology - is it a wrapper on top of socket-io ?
Reading through their website, you can see references to socket.io...
Reading through their code, for example, their client code, you can also find references to socket.io...
It seems to me that the word wrapper doesn't fit, as they focus on creating an optimized design for meshing different technologies to create a real-time application backend... I would go with the word framework if I had to put a name on it. If you like their approach, you'll probably enjoy simplified scaling as this is one of their main concerns.
As Myst pointed out, Telepat is more of a framework, a full stack software. This framework uses socket.io for the notification part of the system: clients manipulate application resources -> API -> workers -> subscribed clients get notified of the changes through various means (Apple Push Notifications, Google Cloud Messaging and web sockets for any other client).
So in short: Telepat uses socket.io for client notifications.
I am making a turn-based card game that will have clients, a lobby server and a game server. What methologies are there that are both cross-language and bi-directional (e.g. client request -> server server response-> client, as well as server request-> client client response -> server)?
I have looked into JMS but believe it is too heavyweight for my needs (this program will just be small scale, and I don't think the complexities make this solution suitable). I have briefly looked into REST but I believe that wouldn't fit the bi-directional requirement. Of course, there is RMI but I would like to be able to develop clients in C++ and other languages as another learning exercise.
If I'm honest, I'm at a bit of loss because I don't want to use JMS as I think it is too complex for this, but I don't think just using TCP sockets and say using a basic XML based protocol for the messages will provide a good structure of communication for the program.
The research lab that I do some work with develops a system called "Object Oriented Distributed Semantic Services."
We leverage some work that we do with cross-language serialization to allow you to write clients/servers in different languages, and the underlying messages to be a format that be serialized and deserialized by clients/servers regardless of their implementation language.
Right now we mostly support Java/ObjectiveC. You can take a look at the chat room tutorial, which should give you a basic idea of how requests / responses work.
http://ecologylab.net/research/simplGuide/oodss/index.html
OODSS is designed to work well for game scenarios... the system was originally written to support a game one of the researchers in our lab was working on. The original paper on OODSS discusses the development of a game from the ground up. That may work out well for you: http://ecologylab.net/technicalReports/oodss_TR_10_01.pdf
You could apply a similar idea to allow for multiple clients in languages that aren't supported yet. (you may have to write some serialization/deserialization code on your own, to start.)
Good luck! Hope that helps!