Related
Substrate uses libp2p to do peer discovery and transport.
Is there functionality to advertise additional information from peers using this layer? Or alternatively, use substrate to advertise information without needing to sink that information to the chain?
For instance, the location of additional RPC endpoints attached to the light clients.
Re: shawntabrizi
one big issue about light clients and not storing that data to chain is that light client inherently depend on merkle proofs and this merkle trie is where all the overhead of runtime storage comes from practically speaking, so if you want some light-client verifiable content using only the runtime state, then you are basically stuck with this but you could store only a hash of some file
and then use any other p2p protocol to share that file
and let the light client verify based on the hash
but this would be a third party tool or possibly an offchain worker on top of a substrate client|
You should be able to make modifications to networking to enable different kinds of gossiping for your needs. AFAIK, Polkadot does this: https://github.com/paritytech/polkadot/tree/master/network
a few parties use IPFS, including ourselves at parity, that have investigated IPFS integration. I am not to certain on the latest progress from 3rd party teams, but we had a very old branch that adds an IPFS node along side the Substrate node. Because Substrate and IPFS use LibP2P, this was relatively painless from what I understand.
I'm working on a project that heavily uses Apache NiFi v1.10.0. I'm getting tired of clicking through hundreds of process groups to apply small fixes that are essentially the same.
I've recently discovered Remote Process Groups and I was wondering if there is a way to connect NiFi instance to itself and implement DRY this way? I was thinking of implementing repeating components inside the root component and accessing them with remote access inside other process groups. Is this possible?
Right now I'm getting only SSLHandshakeException / PKIX path building failed
If there are other ways to implement DRY - please tell me.
#Alex. I feel your pain, in a previous role, they had a process group of 100s of flows, and would copy and paste the entire main group, turning into 1000s of flows. All copies with small modifications in random places.
Although I am advocate of programming this way to get a POC operational, I am a huge advocate of evaluating how to make the flows dynamic from the highest level. The process I use to do this is to go through DFDLC and versioning flow until, for example, I have 1 process group that can replace 2 by cleaning up the flow design differences between each other. We consider this part of optimizing the flow to reduce the total number of active processors too.
I highly recommend you do NOT use remote process groups within the same cluster. I also recommend you make common flows on the main canvas, and connect them with input/output ports when you need to move from a deeper process group back up to the main canvas. You will end up with a flow like this:
You can definitely do site-to-site to self, however it will be less performant because now you are taking local flow files and transferring them over a network connection to all the nodes in the cluster, even though some of them will go back to the same node they are on.
You could use NiFi Registry and created versioned flows that contain reusable functionality. Then you make the change once, commit it back to NiFi Registry, and then update the other instances of that versioned flow.
Bryan and Steven both offered good solutions. I will address the PKIX path building error you are encountering -- this indicates that NiFi is attempting to make an HTTPS connection to another service (likely in this case itself), and doesn't know how to verify the presented public certificate. The solution is to reference an SSLContextService configured with a truststore that contains the certificate. The Apache NiFi walkthroughs provide step-by-step instructions for performing these tasks.
I want to simulate a WBAN routing protocol in Castalia simulator.
First of all, I want nodes to send broadcast messages to all of their neighbors. In Castalia, as a default manner nodes send Broadcast messages not only to neighbors but also to all nodes in the network. as a result, we don't have any relay node to forward packets.
So, I think that Castalia uses single-hop routing and therefore
multi-hop routing is not defined in it.
How can I implement multi-hop routing in Castalia?
In Castalia, as a default manner nodes send Broadcast messages not only to neighbors but also to all nodes in the network. as a result, we don't have any relay node to forward packets.
[...] So, I think that Castalia uses single-hop routing and therefore multi-hop routing is not defined in it.
I am note sure where you are getting this information, but this is wrong. Castalia is capable of multihop routing, there is even a Routing module called MultipathRingsRouting in the default distribution.
Moreover, your statements seem to be contradictory with each other. How can Castalia send messages to all nodes in the network (not just single-hop neighbours) and yet not support any multihop routing? Can you see how these are contradictory statements? In the same spirit how can Castalia send messages to all nodes in the network and "as a result, we don't have any relay node to forward packets". Can you see how this does not make much sense.
Perhaps you are confused by the BAN simulation examples in Castalia which define networks where all the nodes are a single hop from the BAN hub. This does not mean that Castalia in general cannot support multihop routing. It does.
You mention a WBAN Routing protocol in your question. You should note that the current implementation of the BaselineBANMAC in Castalia does not support multi-hop communication. Just to be clear, I am talking about this specific MAC and not Castalia as a whole. This MAC does not support any routing on top of it because it only allows communication between the hub and the slave nodes. There can only be one hub per network. You can have multiple networks (each having a hub) in your simulation, but these will be logically seperate and can only interfere with each other, not communicate. Perhaps, you wanted to use this MAC in your simulations. You will not be able to explore routing with this MAC.
You can however use other MACs (such as TunableMAC, or TMAC) that allow routing to be used on top of them. To be clear, the MAC does not do any routing, this is done by the routing layer above it (a separate module), but a MAC design can allow a routing layer to operate above it, or not. Make sure you read the manual carefully and have a solid understanding of the basics.
i would like to scale kubernetes nodes according to unscheduled pods. if i have pods that can't be scheduled because of their resource requirements, i want to add a new node to the cluster.
looking at autoscaling feature of managed groups in GCE, this doesn't seem to be possible, as their model requires a metric per node in the cluster, while my metric is global.
can anyone confirm that this can't be achieved with current GCE solution?
anyone know of any existing tool/blogpost whatever that could help implementing a solution?
assuming i'm going to roll my own, i'm having problems finding an api that controls GCE managed groups (allows to add a node, remove a node)
thanks,
Nathan
If you are fine with the standard per-node metrics, read the "Horizontal auto-scaling of nodes (GCE)" section of the kubernetes cluster management guide to enable to autoscaler.
If you want custom metrics, you can check out the GCE document.
There is also a similar question on stackoverflow, and the author of one of the answers said that after writing their own custom metrics, the standard per-node metrics was found to be just as good, if not better, for their use case.
I have been working in Flume to fetch logs from a server machine to HDFS. I was able to achieve this if the server and client machines are connected in same network. But how can i achieve the same if the server and client are in different networks.
Do i need to write a custom source for this? [Just checked with twitter example from cloudera in which they're using their own custom source to fetch twitter tweets.]
Any help would be greatly appreciated.
Thanks,
Kalai
If you have a multihomed host joining two non-talking networks you'd like to ship across, you can have a flume agent running there to bridge logs incoming from one network and deliver it to the other one. So your multihomed host will act as a sort of proxy. I don't know if this is necessarily a good idea, as your proxy is probably already busy doing other things if it's the only link between the networks. But if you can set this up, you won't need custom sinks or sources.
If you have two disjoint networks that can both see the internet, you can have one agent post to a web server over HTTP (or TCP for that matter, but it's more work), and another fetch it from the same website. You'll need to write two custom agents (source & sink) for that to work in a performant, reliable and secure fashion, not to mention the web service itself.
Finally, if you have two networks that are completely disconnected (with an air gap), then you may consider writing a custom sink that will, for example, autodetect an inserted tape and copy logs to the tape. Then you take the tape, walk over to another network, plug it in, and have another agent there autodetect it as well and ingest the data :)
Flume agents need to be able to connect to transport events. This means they need to be on the same network.
I'm not sure I understand your question. Why would you expect it to work at all?