Substrate uses libp2p to do peer discovery and transport.
Is there functionality to advertise additional information from peers using this layer? Or alternatively, use substrate to advertise information without needing to sink that information to the chain?
For instance, the location of additional RPC endpoints attached to the light clients.
Re: shawntabrizi
one big issue about light clients and not storing that data to chain is that light client inherently depend on merkle proofs and this merkle trie is where all the overhead of runtime storage comes from practically speaking, so if you want some light-client verifiable content using only the runtime state, then you are basically stuck with this but you could store only a hash of some file
and then use any other p2p protocol to share that file
and let the light client verify based on the hash
but this would be a third party tool or possibly an offchain worker on top of a substrate client|
You should be able to make modifications to networking to enable different kinds of gossiping for your needs. AFAIK, Polkadot does this: https://github.com/paritytech/polkadot/tree/master/network
a few parties use IPFS, including ourselves at parity, that have investigated IPFS integration. I am not to certain on the latest progress from 3rd party teams, but we had a very old branch that adds an IPFS node along side the Substrate node. Because Substrate and IPFS use LibP2P, this was relatively painless from what I understand.
Related
The reason I write this post is I would like to avoid re-inventing the wheel if possible, as I think this is definitely a solved problem, but I am struggling to find a SaaS provider which offers an existing solution that fits my use-case.
The problem goes as follows:
I am running a fleet of data hungry IoT devices.
These devices need to send data to multiple 3rd party endpoints (all HTTPS).
Currently as each new endpoint is added, the mobile data usage (3G/4G/5G) scales in a linear manner, as it is sending to each 3rd-party from the IoT device itself.
Proposed solution:
The IoT devices transmit their data to a HTTPS "hub", this then forwards the data to a list of specified endpoints. Like how a HTTPS load-balancer would work, but operating in a send-to-all mode.
This would keep the IoT device data usage constant, while increasing the cloud data usage (which is orders of magnitude cheaper). Resulting in a cost saving.
Now I had imagined that this is a fairly common problem with IoT devices, but am struggling to find any provider offering this type of service already. I may be lacking knowledge of what terminology to search for if this already exists. If anybody knows the name of a service which offers something like this then this is the type of answer I am looking for.
Ideal features:
HTTPS retry. The service should cache a request if it fails to forward to one or more of the destinations, it should then attempt to re-transmit to the failed destinations after some amount of time.
A SLA regarding uptime, as downtime of this service would expectedly cause a larger outage than duplicating the requests in the original method.
Reverse proxy to preserve the original IP address if possible.
A web GUI (a nice-to-have but not essential).
If this doesn't exist I will likely write my own solution. Thanks in advance.
gRPC is a modern open source high performance RPC framework that can
run in any environment. It can efficiently connect services in and
across data centers with pluggable support for load balancing,
tracing, health checking and authentication. It is also applicable in
last mile of distributed computing to connect devices, mobile
applications and browsers to backend services.
I'm finding GRPC is becoming increasingly more pertinent in backend infrastructure, and would've liked to have it in my favorite language/tsdb kdb+/q.
I was surprised to find that kdb+ does not have a grpc implementation. Obviously, the (https://code.kx.com/q/interfaces/protobuf/)
package doesn't support the parsing of rpc's, is there anything quantitatively preventing there being a KDB+ implementation of the rpc requests/services etc. found in grpc?
Why would one not want to implement rpc's (grpc) in kdb+ and would it be a good idea to wrap a c++/c implemetation therin inorder to achieve this functionality.
Thanks for your advice.
Interesting post:
https://zimarev.com/blog/event-sourcing/myth-busting/2020-07-09-overselling-event-sourcing/
outlines event sourcing, which I think might be a better fit for kdb?
What is the main issue with services using RPC calls to exchange information? Well, it’s the high degree of coupling introduced by RPC by its nature. The whole group of services or even the whole system can go down if only one of the services stops working. This approach diminishes the whole idea of independent components.
In my practice I hardly encounter any need to use RPC for inter-service communication. Partially because I often use Event Sourcing, more about it later. But we always use asynchronous communication and exchange information between services using events, even without Event Sourcing.
For example, an order microservice in an e-commerce system needs customer data from the customer microservice. These dependencies between microservices are not ideal. Other microservices can go down and synchronous RESTful requests over https do not scale well due to their blocking nature. If there was a way to completely eliminate dependencies between microservices completely the result would be a more robust architecture with less bottlenecks.
You don’t need Event Sourcing to fix this issue. Event-driven systems are perfectly capable of doing that. Event Sourcing can eliminate some of the associated issues like two-phase commits, but again, not a requirement to remove the temporal coupling from your system.
If I want to write a node for a P2P application (like Bitcoin, Bitorrent, etc.) there are a lot of parts that are the same:
I need to bootstrap to the network (discover other peers)
I need to manage a list of peers, and monitor their states
I need to retrieve lists of more peers from my neighbour peers
Etc, etc.
Since I don't want to re-invent the wheel, is their a framework that I could as a sort of base library to build on?
You mention both bitcoin and bittorrent, which are quite different, so I'm assuming you don't want to be bound to any specific protocol or even serialization format.
And yet you mention peer-discovery and stats-management which are high-level concerns, be built on top of some network protocol.
But the protocol dictates how such a mechanism would work.
It sortof sounds like you're asking if there are pre-built roofs that would fit on skyscrapers just as well as on a wood cabin.
So if you actually want to design your own protocol you probably should look more at the foundation first.
which language do you want to use
what IO / event processing libraries are available
what protocol parsers and serializers are available
do you aim for throughput? low memory footprint? low latency? minimal amount of programmer-hours spent?
what kind of security is needed? heavy crypto use at the protocol level will need a trustworthy crypto library (don't roll your own!)
what kind of auxiliary things do you need (where does the data go? filesystem? databases? do you need a UI?)
Alternatively, depending how one interprets your question, if you want to write a client for a specific network then you should simply look for a library implementing the core concepts of that specific network while freeing you up to implement the rest of the application.
In bittorrent's case such an example would be libtorrent
I'm probably lacking in some fundamental understanding of how BitTorrent, DHT, and the "swarm" work, as I'm not even sure if DHT and the "swarm" are one and the same.
However, I'm trying to find peers, the number of peers, and some statistics about a torrent from its magnet link (and hash).
I've looked for some libraries to accomplish this, but they seem either outdated or unrelated, or just bencode things.
How do I go about connecting and requesting information? A brief explanation would be delightful.
official specs: http://www.bittorrent.org/beps/bep_0000.html
non-official spec: http://wiki.theory.org/BitTorrentSpecification
BEncoding is a data serialization format used in bittorrent for several purposes
The DHT is a global, decentralized, iterative, UDP-based lookup system which can be used to locate clients participating in a particular swarm based on an infohash, which can be either obtained directly from a magnet link or calculated from a .torrent metadata file.
If you have the announce URL for a tracker (an optional part of the torrent file or the magnet link) you can obtain client addresses directly from a tracker.
Once you have obtained client addresses for a particular swarm you can connect to them - or they will connect to you if you have announced yourself to the DHT/a responsible tracker - using the bittorrent wire protocol, which is basically an asynchronous, binary messaging protocol.
To get a fully-functioning, state-of-the-art bittorrent client you have to implement the following:
a DHT node (bencoding over UDP)
tracker announce and scrape protocol (uses bencoding over HTTP and a custom binary prtocol over UDP)
the bittorrent wire protocol (custom binary protocol over TCP with an optional extension layer. some of the messages are bencoded. A congestion-avoiding UDP-based transport protocol called µTP also exists as alternative to TCP)
a torrent file parser (bencoding, obviously)
general stuff like queueing active torrents, file management, highly concurrent network IO
This is a lot of work which to my knowledge has not been done in ruby. So either you have a lot ahead of yourself or you might want to use a bittorrent library written in a different language (e.g. libtorrent) or interface with a client providing a web service (e.g. transmission).
What are methods for undocumented client/server communication to be captured and analyzed for information you want and then have your program looking for this information in real time? For example, programs that look at online game client/server communication and get information and use it to do things like show location on a 3rd party map, etc.
Wireshark will allow you to inspect communication between the client-server (assuming you're running one of them on your machine). As you want to perform this snooping in your own application, look at WinPcap. Being able to reverse engineer the protocol is a whole other kettle of fish, mind.
In general, wireshark is an excellent recommendation for traffic/protocol analysis- however, you seem to be looking for something else:
For example, programs that look at online game client/server communication and get information and use it to do things like show location on a 3rd party map, etc.
I assume you are referring to multiplayer games and game servers?
If so, these programs are usually using a dedicated service connection to query the corresponding server for positional updates and other meta information on a different port, they don't actually intercept or inspect client/server communciations at realtime, and they don't really interfere with these updates, either.
So, you'll find that most game servers provide support for a very simply passive connection (i.e. output only), that's merely there for getting certain runtime state, which in turn is often simply polled by a corresponding external script/webpage.
Similarly, there's often also a dedicated administration interface provided on a different port, as well as another one that publishes server statistics, so that these can be easily queried for embedding neat stats in webpages.
Depending on the type of game server, these may offer public/anonymous use, or they may require certain credentials to access such a data port.
More complex systems will also allow you to subscribe only to specific state and updates, so that you can dynamically configure what data you are interested in.
So, even if you had complete documentation on the underlying protocol, you wouldn't really be able to directly inspect client/server communications without being in between these communications. This can however not be easily achieved. In theory, this would basically require a SOCKS proxy server to be set up and used by all clients, so that you can actually inspect the communications going on.
Programs like wireshark will generally only provide useful information for communications going on on your own machine/network, and will not provide any information about communications going on in between machines that you do not have access to.
In other words, even if you used wireshark to a) reverse engineer the protocol, b) come up with a way to inspect the traffic, c) create a positional map - all this would only work for those communications that you have access to, i.e. those that are taking place on your own machine/network. So, a corresponding online map would only show your own position.
Of course, there's an alternative: you can emulate a client, so that you are being provided with server-side updates from other clients, this will mostly have to be in spectator mode.
This in turn would mean that you are a passive client that's just consuming server-side state, but not providing any.
So that you can in turn use all these updates to populate an online map or use it for whatever else is on your mind.
This will however require your spectator/client to be connected to the server all the time, possibly taking up precious game server slots.
Some game servers provide dedicated spectator modes, so that you can observe the whole game play using a live feed. Most game servers will however automatically kick spectators after a certain idle timeout.