I have a channel that emits events. I'd like to set up an "oscillometer" on top of this channel, i.e. some sort of a "frequency counter" that gets constantly updated (with some interval, of course, e.g. one second).
My first thought is to create a counter that will be incremented every time an event arrives on the channel and then have a ticker that every second prints out this counter and resets it to 0. It looks like the most straightforward approach, but I'm wondering if, maybe, there are some built-in tools for this in Go? Or maybe some libraries that can do it for me (I've tried finding one, but haven't succeeded)?
Related
I'm new to programming. I want to create a flowchart for my algorithm. In my algorithm there is a coordinator which waits for values from several devices. Each time it receives a value, it does some operation on it and after finishing the operation, waits for the next value to come.
So my question is that should I put an oval at the end of the the flowchart to show the end of each operation on a value or I should just connect the last rectangle (showing the last operation done on the received value) to the start part of the flowchart?
The attached photo is the flowchart I mean. Is it wrong to omit the end oval? If I put an end symbol (oval), won't it be wrong since the coordinator should wait for the next values?
I would put a wait for values node before the start, which would go with a directed vertice into start and operation2 would go into the wait.
Your chart, as it is suggests that the next chunk of work is started right away when the current is finished, yet, you need to illustrate the fact that there might be a wait time. Take a look at this chart:
You see the "Wait for data processing result" node. This is the wait symbol, which you need before start, as your very first node.
I am looking for a queue algorithm that fulfills the following properties:
Processes communicate using only a shared dictionary (key-value-store)
Does not use any atomic operations other than load and store (no CAS, for example)
Supports multiple producers
Supports a single consumer
Producers can die at any time and queue must remain operational
The consumer can also die at any time and be restarted later, but there will never be more than one consumer-process running at a time
This is meant as a general question about a suitable algorithm, since I'd like to use it in a couple of different scenarios. But to help visualize the requirements, here is an example use-case:
I have a website with two pages: producer.html and consumer.html
producer.html can be opened in multiple tabs simultaneously
Each producer.html adds events to the queue
One copy of consumer.html is open and consumes these events (to aggregate and stream them to a webserver, for example)
If the multiple producer-tabs are opened by the user rather than the page, these tabs do not have references to each other available, so the usual communication methods (postMessage or calling directly into the other tab's JS code) are out. One of the ways they can still communicate with each other is via LocalStorage as suggested here: Javascript; communication between tabs/windows with same origin. But LocalStorage is not "thread-safe" as detailed here.
Note: There may be other ways to implement cross-tab communication in the browser (Flash, ...), but these are NOT the aim of this question as they won't translate to my other use-cases. This is really just an example use-case for the general queue algorithm that I am trying to find.
A couple more parameters:
The number of producers will never be very large (10s or 100s maybe), so the scaling of the number of reads and writes needed with respect to the number of producers is not really a concern.
I don't know before hand how many producers I might have and there is no immediately obvious way to assign a number or index to them. (Many mutex algorithms (Lamport's Bakery, Eisenberg&McGuire, Szymański's, ...) maintain an array of state for each process, which wouldn't necessarily be a natural approach here, although I do not want to exclude these approaches ex ante, if they can be implemented using the shared dictionary in some way...)
The algorithm should be 100% reliable. So, I'd like to avoid things like the delay in Lamport's first Fast Mutex algorithm (page 2 in the PDF) since I don't have any kind of real-time guarantees.
It would be very helpful if the queue was FIFO, but it's not strictly required.
The algorithm should not be encumbered by any patents, etc.
Update:
The Two-Lock Concurrent Queue Algorithm by Michael and Scott looks like it could work, but I would need two things to implement it:
A locking mechanism using the shared dictionary that can survive the crash of a lock-holder
A reliable way to allocate a new node (if I move the allocation into the locked section, I could just generate new random keys until I find one that's not in use yet, but there might be a better way?)
Update 2:
It seems, I wasn't being specific enough about the dictionary:
It's really nothing more than a trivial key-value-store. It provides the functions get(key) to read the value of a key, put(key, value) to change the value of a key, and delete(key) to remove a key. In some of my use-cases, I can also iterate over keys, but if possible, I'd like to avoid it for generality. Keys are arbitrary and the producers and consumers can create or calculate them as needed. The dictionary does not provide any facilities for automatically generating unique keys.
Examples are HTML LocalStorage, Google AppEngine's Datastore, a Java Map, a Python dictionary, or even a file-system with only a single directory (where the keys would be the file-names and the values the content of the files).
After quite a bit of further reading and sleeping on things for a night, I came up with one way that should be able to accomplish what I need, but it might not be the most elegant:
The paper Wait-Free Algorithms for Fast, Long-Lived Renaming by Moir and Anderson generalizes Lamport's Fast Mutex Algorithm #2 (page 6 here) into the following building block (Figure 2):
When n processes enter this section of code, at most one of them will stop, at most n-1 will move right and at most n-1 will move down.
In Lamport's algorithm, stopping means the process acquired the lock, whereas moving right or left will simply send the process back to the beginning of this section of code. To release the lock, a process simply sets Y back to false. (Not quite correct, actually... See "Update" below...)
The big problem with this is that if any of the processes ever die while holding the lock (i.e. before releasing it), the block will simply stay locked forever.
Another problem is that every process needs to be assigned a unique process ID p.
The locked-forever problem can be fixed by borrowing an idea from Moir and Anderson, namely to send processes that end up moving right or down into a different building block rather than back to this one, leading to a structure like this (Figure 3 in the paper):
Except that in this case, I won't be using this grid to assign process IDs as M&A did (although I could probably solve the problem of the unique values for p with this). Instead, every box in the grid will correspond to a very simple queue. If a process stops on a box, it acquired the tail-lock for the corresponding queue (e.g. as per the algorithm by Michael and Scott) and proceeds to enqueue a new element to that queue. Upon completion, it sets the Y value of the box back to false to allow other processes to use this queue. This way, if there is high contention or if processes die before releasing locks, new queues will be created dynamically as needed.
The consumer-process doesn't need to worry about locking the heads of the queues when dequeuing elements, since it's the only process to ever do so. So, it simply traverses the tree of boxes to find all queues and trivially helps itself to their contained elements. One thing to note is that while each individual queue will be FIFO, there is no synchronization between the queues, so the combined queue will not necessarily be FIFO.
If we now change the boolean Y to a time-stamp (or null/0 to indicate false), the consumer can also expire locks after some safe timeout to re-activate dead queues.
A note about implementation using the dictionary:
The shared variables X and Y can be entries in the dictionaries with key-names X_123 and Y_123, where 123 is the number of the box.
p can simply be any unique random string and will be stored as the value of key X_123.
The boolean or time-stamp is also simply stored as the value of key Y_123. The producer-processes interpret a missing entry for Y_123 as false or null/0.
The box-numbers 123 need to be calculated from the move-pattern. One way to do this would be to start with 1 in the top-left corner. If the process stops in that box, we're done. If not, the current number (starting with 1) is shifted left by 1 (i.e. multiplied by 2) and, if the process moved down, also incremented by 1. Smaller (and fewer) numbers can be calculated with a different numbering scheme (I still need to work it out), but this one should work.
The queues then consist of one entry with key H_123 that holds the index of the current head of the queue in its value and one entry with key T_123 that holds the index of the tail. Both default to 0 if they don't exist.
To enqueue an item into queue 123, the tail index is read from T_123 (let's say it yields 48) and an entry with key Q_123_48 is put into the dictionary with its value containing the enqueued item. After, T_123 is incremented by 1.
After the item is enqueued, the Y_123 entry is set back to false or null/0 (not deleted!)
To dequeue an item, the head index is read from H_123 (let's say it yields 39) and compared to the tail index T_123. If it is smaller, an item is available at Q_123_39, which is then read and deleted from the dictionary. After, H_123 is incremented by 1.
To traverse the box-tree, the consumer starts with the box in the top left corner. For each box (e.g. 123), if a key Y_123 exists in the dictionary (even if it contains values null/0 or false), the consumer dequeues items from the corresponding queue, and then recursively moves right and down to the adjacent boxes. If no key Y_123 exists, this box hasn't been used by any processes yet and doesn't need to be considered (and neither do the boxes below or to its right).
I haven't actually implemented this yet, but I'll do that next. I just wanted to post this already to see if it could inspire other approaches or if anyone can see anything wrong with this idea.
Update:
I just noticed one complication: It is possible that if two processes are trying to acquire the lock for a queue simultaneously, both will fail and move on to the next block. This will leave that queue locked forever as no-one will be left to set Y back to false or null/0.
This is the reason why the "Long-Lived Renaming" algorithm by M&A as well as Lamport's algorithm #2 use an array of Y-values in which every process has its own entry that it resets also if it moves on to another block. Y is then only considered false if all entries are false.
Since I don't know before-hand how many processes I will have, I could implement this only if the dictionary had some way of enumerating keys (the keys would then be Y_123_456 where 456 is the value of p for each process).
But, with rare contention and the above described timeout-mechanism for reactivating dead queues, the issue might lead to only a little bit of memory inefficiency, rather than a major problem.
Update 2:
A better way to label the boxes would be this pattern:
If we call the total number of moves n (counting the move into the top left box also, i.e. n ≥ 1) and the number of moves to the right r, then the box-number can be calculated using
box = (n × (n - 1))/2 + r
Just use a RDBMS. It's pretty simple in MS SQL, for PostgresSQL you'd have to use the RETURNING keyword and for MySQL you'd probably have to use triggers.
CREATE TABLE Q ([Key] BIGINT IDENTITY(1,1) PRIMARY KEY, [Message] NVARCHAR(4000))
INSERT INTO Q OUTPUT inserted.* VALUE(#message)
DELETE TOP(1) Q WITH (READPAST) OUTPUT deleted.*
If you were really hoping for an algorithmic solution, just use a ring buffer.
const int MAX_Q_SIZE = 20000000;
static string[] Q = new string[MAX_Q_SIZE];
static long ProducerID = 0;
static long ConsumerID = 0;
public static long Produce(string message) {
long key = Interlocked.Increment(ref ProducerID);
int idx = (int)(key % MAX_Q_SIZE);
Q[idx] = message;
return key;
}
public static string Consume() {
long key = Interlocked.Increment(ref ConsumerID);
int idx = (int)(key % MAX_Q_SIZE);
string message = Q[idx];
return message;
}
I am new to parallelization and MPI. I am learning and experimenting with mpi4py. Currently I am trying to optimize the performance of a method which randomly samples for a desired point(satisfying a condition) in an interval.
To give you a detailed idea, i created a sample program which is similar to my program. The aim of this program is to output 20 numbers between 9.9999 and 10.0. This is done by randomly sampling from [0.0,1.0] and multiplying it by 10(varies by iteration by a infinitesimally small amount).
The following is the function and comments are provided.
import numpy as np
import time
def fit(iterations, value):
array = []
sample = None
# This runs for a fixed number of iterations. For one iteration one sample needs to go to the accumulator array (in this case i.e array)
for iteration in range(iterations):
while True:
arbit = np.random.uniform(0,1)
# The main condition for sampling. In my original program this is bound to change with each
# iteration. so I made it depend on the iteration in this test program.
if((10-0.000001*(iteration))*arbit > value):
sample = 10*arbit
break
# The accumulation of accepted sample. If there are n iterations, n samples are expected.
array.append(sample)
print "Result: "+ str(array)
if __name__ == '__main__':
startTime = time.time()
fit(20, 9.9999)
elapsedTime = time.time() - startTime
print "\n"+"time taken: "+str(elapsedTime)+"\n"
As you can see all the action happens in the while loop in the fit() method. What I want to do is to take advantage of parallelization and mpi4py to make this method faster. For example, I want to start n processes and when the while loop comes the processes are fired parallely and which ever one finds the desired value first I want to take that value add it to accumulator array and abort all other processes. I want to continue this routine again in the next iteration and so on until the method finishes. Is something like this possible ? If not this way, What other way can I use parallelization in the above function ?
Thanks
The general ideas behind parallelization are heavily application-dependent. The more independent you can make your processes, the better. Inter-process communication adds hassle and overhead. This is especially true if your processes reside in different computers.
With your sample code above the simple way to make it parallel would be to split it by iterations. You would have a list of iterations and a number of worker processes which would churn through one iteration cycle at a time. Even if you needed to have the results in order, you can sort them afterwards. So, it does not really matter, if you go through iterations 0, 1, 2, 3... or e.g. 17, 3, 14, 1, 5...
But what you seem to suggest is that you split each iteration cycle into parallel loops looking for a suitable number. That is possible (but make sure you use different random seeds in different processes, otherwise they are replicating the same sequence), and the communication needed is very simple:
worker processes need to be able to send "I found it!"
worker processes need to stop when another process sends "I found it!"
worker processes need to be able to fetch a new starting value after they are done
There are several ways to accomplish this. The description above assumes the workers are active, but it is often easier to make stupid workers which only indicate when they are done and start doing things when they are told to. In that case you only need point-to-point communication between the master and the slaves.
In any case the workers have to poll the communication regularly when they are doing their work, and from the performance point of view the polling interval is important. If you poll very frequently, you lose time polling. If your poll interval is very long, then you lose time when something happens. The situation is easier with the master which can use blocking communication and just sit and wait until the workers say something.
So, you may use broadcasts between workers, or you may use master-slave communication, or you may use a combination of these. There are pros and cons in each approach, and the optimal solution depends on your application and requirements. (I usually pick the solution which is simplest to write and optimize only if there is a clear need.)
I am only superficially familiar with MPI, but the answer to your question is "yes", it can be done with MPI.
I'm not sure if the title is right but...
I want to animate (with html + canvas + javascript) a section of a road with a given density/flow/speed configuration. For that, I need to have a "source" of vehicles in one end, and a "sink" in the other end. Then, a certain parameter would determine how many vehicles per time unit are created, and their (constant) speed. Then, I guess I should have a "clock" loop, to increment the position of the vehicles at a given frame-rate. Preferrably, a user could change some values in a form, and the running animation would update accordingly.
The end result should be a (much more sophisticated, hopefully) variation of this (sorry for the blinking):
Actually this is a very common problem, there are thousands of screen-savers that use this effect, most notably the "star field", which has parameters for star generation and star movement. So, I believe there must be some "design pattern", or more widespread form (maybe even a name) for this algoritm. What would solve my problem would be some example or tutorial on how to achieve this with common control flows (loops, counters, ifs).
Any idea is much appreciated!
I'm not sure of your question, this doesn't seem an algorithm question, more like programming advice. I have a game which needs exactly this (for monsters not cars), this is what I did. It is in a sort of .Net psuedocode but similar stuff exists in other environments.
If you are running an animation by hand, you essentially need a "game-loop".
while (noinput):
timenow = getsystemtime();
timedelta = timenow - timeprevious;
update_object_positions(timedelta);
draw_stuff_to_screen();
timeprevious = timenow;
noinput = check_for_input()
The update_object_positions(timedelta) moves everything along timedelta, which is how long since this loop last executed. It will run flat-out redrawing every timedelta. If you want it to run at a constant speed, say once every 20 mS, you can stick in a thread.sleep(20-timedelta) to pad out the time to 20mS.
Returning to your question. I had a car class that included its speed, lane, type etc as well as the time it appears. I had a finite number of "cars" so these were pre-generated. I held these in a list which I sorted by the time they appeared. Then in the update_object_position(time) routine, I saw if the next car had a start time before the current time, and if so I popped cars off the list until the first (next) car had a start time in the future.
You want (I guess) an infinite number of cars. This requires only a slight variation. Generate the first car for each lane, record its start time. When you call update_object_position(), if you start a car, find the next car for that lane and its time and make that the next car. If you have patterns that you want to repeat, generate the whole pattern in one go into a list, and then generate a new pattern when that list is emptied. This would also work well in terms of letting users specify variable pattern flows.
Finally, have you looked at what happens in real traffic flows as the volume mounts? Random small braking activities cause cars behind to slightly over-react, and as the slight over-reactions accumulate it turns into cars completely stopping a kilometre back up the road. Its quite strange, and so might be a great effect in your wallpaper/screensaver whatever as well as being a proper simulation.
Situation: There are several entities in a simulated environment, which has an artificial notion of time called "ticks", which has no link to real time. Each entity takes it in turns to move, but some are faster than others. This is expressed by a delay, in ticks. So entity A might have a delay of 10, and B 25. In this case the turn order would go:
A A B A A
I'm wondering what data structure to use. At first I automatically thought "priority queue" but the delays are relative to "current time" which complicates matters. Also, there will be entities with larger delays and it's not unforseeable that the program will run through millions of ticks. It seems silly for an internal counter to be building higher and higher when the delays themselves stay relatively small and don't increase.
So how would you solve this?
You store the entities in a Heap and group them by their time left to wait. The group of entities that are next to move would be at the top of the Heap. You only have to update these entities. When their time remaining to wait drops to 0, you remove them from the heap. Put the next group of entities in line at the top of the Heap while decrementing their time to wait by the amount of time that just passed before the previous move.
For example:
Your Heap has 3 nodes (A,B, and C), the top is node A with two entities both having 5 ticks remaining. The childern have 10 and 12 ticks remaining respectively.
At time t=5 you move all the entities that are bucketed in node A
Remove A from the Heap
B moves to the top of the Heap with 10-5 = 5 ticks remaining then
repeat.
It seems to me by your description that the concept of "what's next?" is more important than "how long is it until the next action?". This being the case, sort your queue by "next-to-go" or lowest number of ticks remaining to highest. Inserts, of course, get entered in their appropriate order, and altered entries ("Speed up" spells) get removed trom the queue, altered, and then re-entered appropriately.
Then, you just pop the next job off the queue. Whatever number of ticks it had remaining must be the "time-elapsed". Makes a pass over the queue, decrementing the ticks remaining field of each entry by the amount of ticks you just discovered.
This has the advantage of keeping track of the concept of time remaining, but also of not having to fire events or execute any other code for ticks that go by when there is no action to take. You can afford this, since there is no relation to real time at all. There is only "what's next?", and "How long did it take to get there?".
If we assume your entities are observing or watching the simulation time, they could each implement an interface which makes them track the ticks left and provides a method to get how many ticks are left for a particular entity. At each tick, the entity reduces its ticks left by 1.
You could then keep a sorted set queue (set because each entity will be in the queue only once) of these entities, sorted based on get ticks left, so that the 0th entity is the one to move next, and the Nth entity is the "slowest".
When the entity's get ticks left method is 0, it is removed from the sorted set, the ticks left timer is reset, and it is re-inserted into the sorted set.
Option #1: Polling
I would probably build a controller that can discover the delay for all the different entities and maintain a ticks-remaining for each entity. The controller would cycle through ticks and on each tick it would reduce the ticks-remaining for all entities in the game.
Once an entities ticks-remaining value reaches zero you know it's their turn, either controlled by the heartbeat method that handles the ticks or a method that you call.
Option #2 Events
Think like the UI paradigm, the interface doesn't constantly poll the button to see when it is clicked. Rather it let's the button notify the UI when it has been clicked via events. Have your Entities (or an EntityBattleContext) fire an event when it is ready. You will have to handle you game time in some manner, since it isn't based off real world time at all you may need to have all the Entities listen for a GameTick event and when they receiver that event update their interal TicksRemaining variable.
Before following the event driven route make sure the polling route won't work. Remember the cardinal rule always optimize later because more often then not you don't need the optimization.
Look at how Java's DelayQueue is implemented.