Related
I am planning to use nifi to ingest data from more than 10,000 sensors. There are 50-100 types of sensors which will send a specific metric to nifi.
I am pondering over whether I should assign 1 port number to listen to all the sensors or I should assign 1 port for each type of sensor to facilitate my data pipeline. which is the better option?
Is there a upper limit of the no of ports which I can "listen" using nifi?
#ilovetolearn
NiFi is such a powerful tool. You can do either of your ideas, but I would recommend to do what is easier for you. If you have data source sensors that need different data flows, use different ports. However, if you can fire everything at a single port, I would do this. This makes it easier to implement, consistent, easier to support later, and easier to scale.
In large scale highly available NiFi, you may want a Load Balancer to handle the inbound data. This would push the sensor data toward a single host:port on the LB appliance, that then directs to NiFi with 3-5-10+ nodes.
I agree with the other answer that once scaling comes into play, an external load balancer in front of NiFi would be helpful.
In regards to the flow design, I would suggest using a single exposed port to ingest all the data, and then use RouteOnAttribute or RouteOnContent processors to direct specific sensor inputs into different flow segments.
One of the strengths of NiFi is the generic nature of flows given sufficient parameterization, so taking advantage of flowfile attributes to handle different data types dynamically scales and performs better than duplicating a lot of flow segments to statically handle slightly differing data.
The performance overhead to run multiple ingestion ports vs. a single port and routed flowfiles is substantial, so this will give you a large performance improvement. You can also organize your flow segments into hierarchical nested groups using the Process Group features, to keep different flow segments cleanly organized and enforce access controls as well.
2020-06-02 Edit to answer questions in comments
Yes, you would have a lot of relationships coming out of the initial RouteOnAttribute processor at the ingestion port. However, you can segment these (route all flowfiles with X attribute in "family" X here, Y here, etc.) and send each to a different process group which encapsulates more specific logic.
Think of it like a physical network: at a large organization, you don't buy 1000 external network connections and hook each individual user's machine directly to the internet. Instead, you obtain one (plus redundancy/backup) large connection to the internet and use a router internally to direct the traffic to the appropriate endpoint. This has management benefits as well as cost, scalability, etc.
The overhead of multiple ingestion ports is that you have additional network requirements (S2S is very efficient when communicating, but there is overhead on a connection basis), multiple ports to be opened and monitored, and CPU to schedule & run each port's ingestion logic.
I've observed this pattern in practice at scale in multinational commercial and government organizations, and the performance improvement was significant when switching to a "single port; route flowfiles" pattern vs. "input port per flow" design. It is possible to accomplish what you want with either design, but I think this will be much more performant and easier to build & maintain.
In our application we have an elasticsearch cluster that is shared for all of our clients.
The types of requests made against this cluster are computationally intense and may take minutes to complete. Because of the type of data in the cluster, the types of requests we get, and the irregularity at which they are used, we can't predict when these requests will be made or do any caching before hand. If multiple clients make requests at the same time, they will experience slower response speeds than normal.
For most clients this isn't an issue. Their data isn't large enough to make a noticeable difference (3s -> 10s occasionally isn't a big deal). But for larger clients, the time difference may be minutes and is very noticeable.
What we'd like, above all else, is consistency- even if these operations were slower on average. To do that, we'd like to give these special clients dedicated nodes, while all other clients use the shared nodes. At a first glance, it seems the only way to do that is to create a dedicated cluster.
But this adds overhead in any application that interfaces with elasticsearch to first do a lookup for the cluster to route to. Ideally, we'd be able to create dedicated nodes within the cluster. This way, the application doesn't need to be aware of cluster routing, and the index configuration can be shared between these "virtual clusters".
Each document has a client id, which we can use to distribute across nodes using _routing. But this has its own problems. Firstly, this doesn't allow us to create a generic default cluster. Secondly, this may mean the dedicated nodes share client data with other clients- the goal is to get consistent speeds by removing node resource contention. And finally, this doesn't allow us to allocate multiple nodes for a given route.
Is there a way to create a routing rule that nodes can be added to explicitly. e.g. I want to add 3 nodes to routing key 582123.
Is there a way to create a default routing rule for nodes that don't match existing routes? If not we could always have an explicit default route. We'll still need to do a route lookup application side but it would still reduce complexity in a multi-cluster scenario.
Depending on the version of elasticsearch you are using and the setup of your indices, you could use per-index allocation. Basically you give a node an attribute and then you specify on index level settings where that index (more accurately, its shards) should end up. As you will read in the documentation, you need to make sure that other constraints are not being violated e.g.
Shards are only relocated if it is possible to do so without breaking another routing constraint, such as never allocating a primary and replica shard on the same node.
This means that you need to have different indices though.
Is there any way in ODL to split the same namespace into a few shards? The current performance of our application for topology discovery does not satisfy us. It would be good if we could use a few controllers to do it. For this purpose, we would divide our controllers into two sets: frontend and backend controllers. Each frontend controller would only collect and store the data from some subnet of the network to data store.. Each of the frontend controllers should use the same namespace but a different shard because we want it to be a shard leader for the high performance. Backend controllers would use the collected data. They should receive all data from all frontend controllers. Currently we don’t care about HA.
Is it possible with the current ODL?
If not, are there any plans to have horizontal scalability in the next future?
Currently ODL clustering is centered around HA. There have been some building blocks put in place to do sub-sharding for scale out but none of the stakeholders that currently contribute to ODL are currently investing resources in horizontal scalability. If you have a strong interest and use case, the ODL community would welcome your contributions.
I have a shared hosting plan and am designing a single page site which will include a slideshow. The browser typically limits the number of simultaneous requests to a single domain. I don't expect a lot of traffic, but I would like the traffic I do receive to have fast load times. I may be able to add unlimited subdomains, but does that really affect the speed for the customer considering they are probably the only one polling my server and all subdomains point to the same processor? I have already created two versions of every image, one for the slideshow, and one for larger format via AJAX request, but the lag times are still a little long for my taste. Any suggestions?
Before you contrive a bunch of subdomains to maximize parallel connections, you should profile your page load behavior so you know where most of the time is being spent. There might be easier and more rewarding optimizations to make first.
There are several tools that can help with this, use all of them:
https://developers.google.com/speed/pagespeed/
http://developer.yahoo.com/yslow/
http://www.webpagetest.org/
Some important factors to look at are cache optimization and image compression.
If you've done all those things, and you are sure that you want to use multiple (sub)domains, then I would recommend using a content delivery network (CDN) instead of hosting the static files (images) on the same shared server. You might consider Amazon's CloudFront service. It's super easy to set up, and reasonably priced.
Lastly, don't get carried away with too many (sub)domains, because each host name will require a separate DNS lookup; find a balance.
I'm working on creating a version of Pastry natively in Go. From the design [PDF]:
It is assumed that the application
provides a function that allows each Pastry node to determine the “distance” of a node
with a given IP address to itself. A node with a lower distance value is assumed to be
more desirable. An application is expected to implements this function depending on its
choice of a proximity metric, using network services like traceroute or Internet subnet
maps, and appropriate caching and approximation techniques to minimize overhead.
I'm trying to figure out what the best way to determine the "proximity" (i.e., network latency) between two EC2 instances programmatically from Go. Unfortunately, I'm not familiar enough with low-level networking to be able to differentiate between the different types of requests I could use. Googling did not turn up any suggestions for measuring latency from Go, and general latency techniques always seem to be Linux binaries, which I'm hoping to avoid in the name of fewer dependencies. Any help?
Also, I note that the latency should be on the scale of 1ms between two EC2 instances. While I plan to use the implementation on EC2, it could hypothetically be used anywhere. Is latency generally so bad that I should expend the effort to ensure the network proximity of two nodes? Keep in mind that most Pastry requests can be served in log base 16 of the number of servers in the cluster (so for 10,000 servers, it would take approximately 3 requests, on average, to find the key being searched for). Is the latency from, for example, EC2's Asia-Pacific region to EC2's US-East region enough to justify the increased complexity and the overhead introduced by the latency checks when adding nodes?
A common distance metric in networking is to count the number of hops (node-hops in-between) a packet needs to reach its destination. This metric was also mentioned in the text you quoted. This could give you adequate distance values even for the low-latency environment you mentioned (EC2 “local”).
For the go logic itself, one would think the net package is what you are looking for. And indeed, for latency tests (ICMP ping) you could use it to create an IP connection
conn, err := net.Dial("ip4", "127.0.0.1")
create your ICMP package structure and data, and send it. (See Wikipedia page on ICMP; IPv6 needs a different format.) Unfortunately you can’t create an ICMP connection directly, like you can with TCP and UDP, thus you will have to handle the package structure yourself.
As conn of type Conn is a Writer, you can then pass it your data, the ICMP data you defined.
In the ICMP Type field you can specify the message type. Values 8, 1 and 30 are the ones you are looking for. 8 for your echo request, the reply will be of type 1. And maybe 30 gives you some more information.
Unfortunately, for counting the network hops, you will need the IP packet header fields. This means, you will have to construct your own IP packets, which net does not seem to allow.
Checking the source of Dial(), it uses internetSocket, which is not exported/public. I’m not really sure if I’m missing something, but it seems there is no simple way to construct your own IP packets to send, with customizable header values. You’d have to further check how DialIP sends packages with internetSocket and duplicate and adapt that code/concept. Alternatively, you could use cgo and a system library to construct your own packages (this would add yet more complexity though).
If you are planning on using IPv6, you will (also) have to look into ICMPv6. Both packages have a different structure over their v4 versions.
So, I’d suggest using simple latency (timed ping) as a simple(r) implementation and then add node-hops at a later time/afterwards, if you need it. If you have both in place, maybe you also want to combine those 2 (less hops does not automatically mean better; think long overseas-cables etc).