etcd allows clients to safely wait for changes of individual k/v nodes, by supplying a last known index of a node to the wait command. etcd also allows to wait ("recursively") for any changes to child nodes under a certain parent node.
Now, the problem is: is it possible to recursively wait on a parent node in such a way, as to guarantee that no child node changes are ever missed by the client? Parent node index is of no use in this case, as it would not change on child node modification.
If you're just starting up, presumably you have just retrieved the subtree you're watching. The reply has an etcd_index field. Use that as the starting point.
Otherwise, your wait contains the modification index of the change. Use that as a starting point for the next call.
You may have to increase one or two of these values to ensure that you don't get duplicate replies. I don't remember which of these I need to increment on purpose; the code needs tests which ensure that I get every change exactly once, so I adjust the values based on that.
Related
I have been working on a position-based protocol using veins-inet and I want to get the position of the destination node.
In my code, I got the IP Address of the destination from the datagram.
const L3Address& destAddr = datagram->getDestinationAddress();
and I want to get the current position of this node.
I already checked the following question
How to get RSU coordinate from TraCIDem11p.cc?
But it seems that it refers to the node by using the node ID.
Is there a way to get the position of the node by referring to its IP Address?
I am using instant veins-4.7.1
A very simple solution would be to have each node publish its current L3Address and Coord to a lookup table whenever it moves. This lookup table could be located in a shared module or every node could have its own lookup table. Remember, you are writing C++ code, so even a simple singleton class with methods for getting/setting information is enough to coordinate this.
If, however, the process of "a node figures out where another node is" is something you would like to model (e.g., this should be a process that takes some time, can fail, causes load on the wireless channel, ...) you would first need to decide how this information would be transferred in real life, then model this using messages exchanged between nodes.
I've an zset on my redis filled with several elements.
By other hand, I have several service instances need to "consume" those elements.
When I say consume, I mean, each instance:
Get the first element from zset
I process it
If everything has been FINE, remove it from zset
Problems here:
two instances could proces the same element twice due to race conditions.
By other hand, I could pop first element:
Pop me the first element from zset
I process it
If something has been WRONG, push it again
Problems here:
If process stops at step 2., the element is lost forever since it's not added at zset again.
Any ideas?
you have several options
do it yourself, then you need to manage the state of the jobs in the zset
states: active / failed / completed ...
on fail => emit error event to push it back to queue
on completed => emit complete event so that we can safely remove it from the queue
retry ? retry delay ?
lock duration?
etc...
OR if you want to stick with redis? find a queue lib built on top of redis. those kind of libs can do all the above for you and better tested.
OR if you're using a recent version of redis, you may want to take a look into disque (https://github.com/antirez/disque-module). This module is from redis author himself.
OR use other proper queue solution
Can I call cleanup manually after I am done with some task in my bolt? Lets say clear a hash which I am using in the code after I am finished processing a set of input?
Also is there a way to trigger a run of whole topology on getting an event? For example if I am reading from a message queue and just got a new message, how do I enforce another run for the topology? Will I need to create a new topology now?
Before the answer, a warning. It isn't necessarily safe to assume only one starting tuple is going through your topology at a time, search for documentation for the MaxSpoutPending parameter.
In order to clean up your data you have a few options, the first is to use Coordination, this will trigger a callback on each bolt when it has ack'ed all of the incoming tuples for a given starting tuple.
If all you are doing is clearing a cache, we use a Guava Cache with an expiry timeout equal to, or a bit greater than, the messageTimeoutSecs setting. Basically the cache clears itself after enough time has passed that the original tuple has either finished processing or timed out
I'm working on a project where I use Riak with Ripple, and I've stumbled on a problem.
For some reason I get duplicates when link-walking a structure of links. When I link walk using curl I don't get the duplicates as far as I can see.
The difference between my curl based link-walk
curl -v http://127.0.0.1:8098/riak/users/2306403e5177b4716da9df93b67300824aa2fd0e/_,projects,0/_,tasks,1
and my ruby ripple/riak-client based link walk
result = Riak::MapReduce.new(self.robject.bucket.client).
add(self.robject.bucket,self.key).
link(Riak::WalkSpec.new({:key => 'projects'})).
link(Riak::WalkSpec.new({:key => 'tasks', :bucket=>'tasks'})).
map("function(v){ if(!JSON.parse(v.values[0].data).completed) {return [v];} else { return [];} }", {:keep => true}).run
is as far as I can tell the map at the end.
However the result of the map/reduce contains several duplicates. I can't wrap my head around why. Now I've settled for removing the duplicates based on the key, but I wish that the riak result wouldn't contain duplicates, since it seems like waste to remove duplicates at the end.
I've tried the following:
Making sure there are no duplicates in the links sets of my ripple objects
Loading the data without the map reduce, but the link walk contains duplicate keys.
Any help is appreciated.
What you're running into here is an interesting side-effect/challenge of Map/Reduce queries.
M/R queries don't have any notion of read quorum values, and they necessarily have to hit every object (within the limitations of input filtering, of course) on every node.
Which means, when N > 1, the queries have to hit every copy of every object.
For example, let's say N=3, as per default. That means, for each written object, there are 3 copies, one each on 3 different nodes.
When you issue a read for an object (let's say with the default quorum value of R=2), the coordinating node (which received the read request from your client) contacts all 3 nodes (and potentially receives 3 different values, 3 different copies of the object).
It then checks to make sure that at least 2 of those copies have the same values (to satisfy the R=2 requirement), returns that agreed-upon value to the requesting client, and discards the other copies.
So, in regular operations (reads/writes, but also link walking), the coordinating node filters out the duplicates for you.
Map/Reduce queries don't have that luxury. They don't really have quorum values associated with them -- they are made to iterate over every (relevant) key and object on all the nodes. And because the M/R code runs on each individual node (close to the data) instead of just on the coordinating node, they can't really filter out any duplicates intrinsically. One of the things they're designed for, for example, is to update (or delete) all of the copies of the objects on all the nodes. So, each Map phase (in your case above) runs on every node, returns the matched 'completed' values for each copy, and ships the results back to the coordinating node to return to the client. And since it's very likely that your N>1, there's going to be duplicates in the result set.
Now, you can probably filter out duplicates explicitly, by writing code in the Reduce phase, to check if there's already a key present and reject duplicates if it is, etc.
But honestly, if I was in your situation, I would just filter out the duplicates in ruby on the client side, rather than mess with the reduce code.
Anyways, I hope that sheds some light on this mystery.
I'm wondering what kind(s) of data structures / algorithms might help facilitate handling the following situation; I'm not sure if I need a single FIFO, or a priority queue, or multiple FIFOs.
I have N objects that must proceed through a predefined workflow. Each object must complete step 1, then step 2, then step 3, then step 4, etc. Each step is either done quickly or involves a "wait" that depends on something external to finish (like the completion of a file operation or whatever). Each object maintains its own state. If I had to define an interface for these objects, it would be something like this (written below in pseudo-Java, but this question is language-agnostic):
public interface TaskObject
{
public enum State { READY, WAITING, DONE };
// READY = ready to execute next step
// WAITING = awaiting some external condition
// DONE = finished all steps
public int getCurrentStep();
// returns # of current step
public int getEndStep();
// returns # of step which is the DONE case.
public State getState();
// checks state and returns it.
// multiple calls will always be identical,
// except WAITING which can transition to READY or DONE.
public State executeStep();
// if READY, executes next step and returns getState().
// otherwise, returns getState().
}
I need to write a single-threaded scheduler that calls executeStep() on the "next" object. My problem is, I'm not sure exactly what technique I should use to determine what the "next" object is. I want it to be fair (first-come, first-serve for objects not in the WAITING state).
My gut call is to have 3 FIFOs, READY, WAITING and DONE. In the beginning all objects are placed in the READY queue, and the scheduler repeats a loop where it takes the first object off the READY queue, calls executeStep(), and places it onto the queue that's appropriate the the result of executeStep(). Except that items in the WAITING queue need to be put into the READY or DONE queue when their state changes.... argh!
Any advice?
If this has to be single threaded you can use a single FIFO queue for the ready and waiting objects and use your thread to process each object as it comes out. If it's state changes to WAITING then simply stick it back into the queue and it will be reprocessed.
Something like (psuedocode):
var item = queue.getNextItem();
var state = item.executeStep ();
if (state == WAITING)
queue.AddItem (item);
else if (state == DONE)
// add to collection of done objects
Depending on the time executeStep takes to run you may need to introduce a delay (Sleep not for) to prevent a tight polling loop. Ideally you would have the objects publish state change events and do-away with the polling altogether.
This is the kind of timeslicing approach that was commonplace in hardware and comms software before multithreading was widespread.
You don't have any way for the task object to notify you when it changes from WAITING to READY except polling it, so the WAITING and READY queues could really just be one. You can just loop around it calling executeStep() on each one in turn. If as a return value from executeStep() you receive DONE, then you remove it from that queue and stick it on the DONE queue and forget about it.
If you wanted to give "more priority" towards READY objects and attempt to run through all possible READY objects before wasting any resources polling WAITING you can maintain 3 queues like you said and only process the WAITING queue when you have nothing in the READY queue.
I personally would spend some effort to eliminate the polling of the state, and instead define an interface that the object could use to notify your scheduler when a state changes.
You might want to study the design of an operating system scheduler. Check out the Linux and *BSD for example.
Some pointers for the Linux scheduler: Inside the Linux scheduler and Understanding the Linux Kernel
NOTE - this does not address your question of how to schedule, but I would use a separate state class that defines the states and transitions. The objects should not know what states they should go through. They can be informed of what "Step" they are at, etc.
there are some patterns for that as well.
You should read up a little on operating systems - specifically the scheduler. Your example is a scaled down set of that problem and if you copy the relevant parts it should work great for you.
You can then add priority, etc.
The simplest technique that satisfies the requirements in your question is to repeatedly iterate over all TaskObjects calling executeStep() on each one.
This requires only one construct to hold the TaskObjects, and it can be any iterable structure, e.g. an array.
Since a TaskObject can transition from WAITING to READY asynchronously, you have to poll every TaskObject that you don't know is DONE.
The performance gained from not polling the DONE TaskObjects may be negligible. It depends on the processing load of calling executeStep() on a DONE TaskObject, which should be small.
A simple round-robin polling assures that once a READY TaskObject has executed a step, it will not execute another step until all other TaskObjects have had a chance to execute.
One obvious additional requirement is detecting when all TaskObjects are in the DONE state so you can stop processing.
To avoid polling DONE TaskObjects you will need to either maintain a flag for each one, or chain the TaskObjects in two queues: READY/WAITING and DONE.
If you store the TaskObjects in an array, make it an array of records, with members DoneFlag and TaskObject.
If for some reason you are storing the TaskObjects in a queue, with available enqueue() and dequeue() methods, then the overhead of two queues instead of one may be small.
-Al.
Take a look a this link.
Boost state machines vs uml
Boost has state machines. Why reinvent?