I'm wondering if I'm missing a concept here somewhere, and wondering if someone can explain how this might work.
My understanding of consistent hashing makes perfect sense where I want to map a particular key to one server. I can just map the key to a single server or virtual node clockwise or counterclockwise from the key.
How does consistent hashing work if I want to specify that each key should be mapped to some quorum of servers I define? For example, what if I have 5 servers and want each key mapped on at least two servers? Would I just choose the two unique servers clockwise on the ring, or is there some other strategy you need? Could you equivalently choose one server clockwise and one counterclockwise? What if you want a key mapped to an arbitrary number of servers?
My issue may be also that I just don't know the right terminology to search for. My particular use case is that I want to have a cluster of Prometheus collectors, say 7, and say I have a pool of 150 exporters. I want each exporter to be collected by at least 3 of the collectors. What's a good way to think about this problem? Appreciate thoughts, thanks!
It turns out that consistent hashing is actually a special case of rendezvous hashing, which is what I was looking for.
Related
I have a set of nodes, each identified by a unique integer (UID), which are arranged into one or more networks. Each network has one or more root nodes. I need to know which nodes are not connected to any root nodes.
At present, from a previous product iteration, my connectivity check starts at each root node and follows all connections. For every found node, a bit in a bitmap is set so that you can quickly check if a node has already been found / processed. Once all paths for all root nodes have been followed, the complete set of nodes is compared against the 'found' bitmap to show all the unconnected nodes.
Previously, UIDs were sequential and I could consolidate them to remove gaps. So using the first ID as an offset, I just made my found array quite large and indexed found nodes into the bitmap directly using the UID (i.e., if I found node 1000, I'd set the 1000th bit in the bitmap). However, with the current version of the product, I have less control over the node UIDs. I can't consolidate them, and third party interaction is unpredictably making very large gaps (e.g., UIDs might jump from being in the thousands to being in the tens of millions). I have come across instances where my bitmap array is too small to accommodate the gaps and the connectivity check fails.
Now, I could just go on increasing the size of my bitmap array, but that always runs the risk of still being too small and is not very resource efficient. Thus, I'm looking to replace it with something else. Obviously, I'd like it to be as fast and as resource efficient as possible - I think some sort of hashed map is what I need. Unfortunately, I have to make this work in Fortran, so I don't have access to <map> etc.
What is the best way to hash my UIDs into a lookup structure, such that I can easily check if I already found that node?
If you can modify the node types themselves, you could add a found field, and use that?
If that is not possible, then yes, a hash table sounds like the obvious solution. Depending on how the UID's are distributed, you might get away with something as simple as mod(UID, bitmap_size). But, you still need to handle collisions somehow. There's lot of literature on that topic, so I won't go into that here, except to note that robinhood hashing is pretty cool (except maybe a bit complicated for a one-off use).
Assume we have a distributed system and there are K machines in the cluster. Each machine stores several integers. I would like to remove all the duplicate values from the system. So if integer 123 appears in machine1 and machine2, we should only keep one 123 in the system. How should I handle this?
My idea is to first let each machine do a removeDuplicate operation using something like bucket-sorting (all nubmers are integer), and then let one machine be the master-node to do a reduce. Is there any better idea?
The easy answer would be to not end up with unmanaged duplicate values on different machines in the first place, by using a distributed hash ring or similar technology to make sure a certain value ends up on a certain node.
If that's not good enough, I'd look into heuristic optimizations. Since you already have multiple copies on different machines, I'm assuming that you want to deduplicate these values for a little bit of extra performance, rather than application correctness.
If this is the case, let each node slowly pass through its keyspace (foreach integer on node) and ask all other nodes if they have a copy of that same value. If they do, we deduplicate it. If someone doesn't respond (fast enough); ignore them and continue. This allows for a decentralized deduplication algorithm that handles node failures and that can be run in any speed, allowing more important traffic to be prioritized when needed.
I'm guessing that the keys are accessed according to a power law distribution, so sweaping through the most commonly updated keys more often could be more efficient, but there's no guarantee for it.
Not sure what type of system you are interested in but if a shared memory is an option you can keep a counter array. Since all your numbers are integers you can flag each integer that appears in this shared array. Also, if this integer is already flagged then drop it. This will result in o(k) operations for each integer received and no duplicates.
In a distributed systems, all agents are expected to use the same key to encrypt information. The system is completely distributed, no agent is privileged, agents can come and leave.
Each agent is holding a key, and I want any agent to be able replacing the key at once. That is, from some time t, all agents will use a new key K' instead of previous key K.
Connection is reliable (tcp), but more versatile solution should also be considered.
Is there a standard algorithm to do that?
I'm missing a textbook of similar consensus algorithm.
My idea:
All agents will connect in a circle.
Each agent will send the new key to his neighbours, once he got one, with the time of the key's creation.
When an agent is receiving a new key, it'll compare the creation date of the two keys and will discard the older.
When an agent throws his previous key - he'll send his new key to his neighbours.
If I get it right, you want to have an agreement protocol. Depending on you assumptions (e.g., synchronization, possible faults, etc.) there are lots of such protocols.
E.g., im many cases are simpe group communication protocols (e.g., ISIS) sufficient, for more sophisticated fault modes you need Byzantine Agreement.
You can see this for few generic approaches, and even this for few famous ones.
I've been reading Introduction to Algorithms and started to get a few ideas and questions popping up in my head. The one that's baffled me most is how you would approach designing an algorithm to schedule items/messages in a queue that is distributed.
My thoughts have lead me to browsing Wikipedia on topics such as Sorting,Message queues,Sheduling, Distributed hashtables, to name a few.
The scenario:
Say you wanted to have a system that queued messages (strings or some serialized object for example). A key feature of this system is to avoid any single point of failure. The system had to be distributed across multiple nodes within some cluster and had to consistently (or as best as possible) even the work load of each node within the cluster to avoid hotspots.
You want to avoid the use of a master/slave design for replication and scaling (no single point of failure). The system totally avoids writing to disc and maintains in memory data structures.
Since this is meant to be a queue of some sort the system should be able to use varying scheduling algorithms (FIFO,Earliest deadline,round robin etc...) to determine which message should be returned on the next request regardless of which node in the cluster the request is made to.
My initial thoughts
I can imagine how this would work on a single machine but when I start thinking about how you'd distribute something like this questions like:
How would I hash each message?
How would I know which node a message was sent to?
How would I schedule each item so that I can determine which message and from which node should be returned next?
I started reading about distributed hash tables and how projects like Apache Cassandra use some sort of consistent hashing to distribute data but then I thought, since the query won't supply a key I need to know where the next item is and just supply it...
This lead into reading about peer to peer protocols and how they approach the synchronization problem across nodes.
So my question is, how would you approach a problem like the one described above, or is this too far fetched and is simply a stupid idea...?
Just an overview, pointers,different approaches, pitfalls and benefits of each. The technologies/concepts/design/theory that may be appropriate. Basically anything that could be of use in understanding how something like this may work.
And if you're wondering, no I'm not intending to implement anything like this, its just popped into my head while reading (It happens, I get distracted by wild ideas when I read a good book).
UPDATE
Another interesting point that would become an issue is distributed deletes.I know systems like Cassandra have tackled this by implementing HintedHandoff,Read Repair and AntiEntropy and it seems to work work well but are there any other (viable and efficient) means of tackling this?
Overview, as you wanted
There are some popular techniques for distributed algorithms, e.g. using clocks, waves or general purpose routing algorithms.
You can find these in the great distributed algorithm books Introduction to distributed algorithms by Tel and Distributed Algorithms by Lynch.
Reductions
are particularly useful since general distributed algorithms can become quite complex. You might be able to use a reduction to a simpler, more specific case.
If, for instance, you want to avoid having a single point of failure, but a symmetric distributed algorithm is too complex, you can use the standard distributed algorithm of (leader) election and afterwards use a simpler asymmetric algorithm, i.e. one which can make use of a master.
Similarly, you can use synchronizers to transform a synchronous network model to an asynchronous one.
You can use snapshots to be able to analyze offline instead of having to deal with varying online process states.
Does anyone know of the logic behind a server-less chat client that would be able to interconnect not LAN-wide but internet-wide? It doesn't need to be able to detect other users world wide, it just needs to obtain some kind of unique identification to be able to connect to a user, like an IP or a unique ID of some sort.
To start, you need some information from somewhere. You can't just turn it on and start chatting without knowing where everyone is. You might have one node that is online all the time and knows a few other nodes. The other nodes would know other nodes and those would know more, etc. It is debatable whether or not you would call that static node a "server" or not. It could just be your friend's node, or a publicly available IP. Once you are up and running, you wont need the start node anymore.
In this type of system, you would need to query your neighbors if you want some sort of identification besides IP address. An IP address has its own drawbacks as well, because you might have two people behind a router at a home DSL connection. Unique ID's would require a recursive query across the whole mesh to find out if your ID is unique.
In this type of system, you would only need to know a limited subset of people in order to chat with anyone, as you can query everyone around you (and the query happens recursively) for the location of that person. An artificial limit on the number of people stored on the local node might be implemented with a Least Recently Used algorithm, kind of like a CPU cache.