Related
I'm currently playing with some ideas wrt CRF-ish work and I have an idea that I need help with.
Minimal Problem
I've got a bunch of function objects (think something expensive like neural nets). They are applied onto a linear buffer (think an array of floats or bytes) but at varying intervals. So they look like that (think of Start and End as "apply Object to buf[Start:End]":
| Object | Start | End |
|--------|-------|-----|
| A | 0 | 4 |
| B | 4 | 10 |
| C | 13 | 15 |
Interval Characteristics
There may be some skips (for example, see the start of C vs the end of B)
There will definitely be changes to the intervals, both positive or negative (for example, B may change from [4:10] to [4:12].
When this happens, the object(s) associated with the intervals may have to be reapplied.
If the interval changes overlaps with another interval, both objects will have to be reapplied. For example, if B changes from [4:10] to [3:12], A would have to be applied to the range [0:3] and B would have to be applied to the range [3:12]
Depending on operation, downstream intervals will have to be updated as well, but the objects will not necessarily have to be reapplied. For example, if it were an insertion that changed the interval range for B, then the interval ranges for C will also increment by 2, but will not trigger a reapplication of C.
Program Characteristics
The intervals change a lot (it's a machine learning training loop).
Supported forms of interval updates are: insert, delete, shiftleft, shiftright. The latter two are the same as insert/delete but applied at the ends of the intervals.
Changes to the interval typically comes as a tuple (index, and size) or as a single index.
Application of function is fairly expensive operation and is CPU bound.
However, being that I am using Go, a couple of mutexes + goroutine solves a majority of the problem (there are some finer points but by large swarths it can be ignored).
One epoch can have anywhere from 5-60ish interval-object pairs.
Buffer is linear, but not necessarily contiguous.
Task
The tasks can be summarized as follows:
Query by index: returns the interval and the object associated with the interval
Update interval: must also update downstream if necessary (which is the majority case)
Insertion of new intervals: must also update downstream
What I've Tried
Map with intervals as a key. This was a bad idea because I had to know if a given index that changed was within a interval or not
Linear structure to keep track of Starts. Discovered a bug immediately when I realized there may be skips.
Linear structures with "holes" to keep track of Starts. This turns out to be similar to a rope.
Ropes and Skip lists. Ended up refactoring what I had into the skiprope package that works for strings. More yak shaving. Yay.
Interval/Segment trees. Implementation is a bitch. I also tried a concrete variant of gods/augmentedtree but couldn't actually get the call-backing to work properly to evaluate it.
The Question
Is there any good data structure that I'm missing out on that would make these tasks easier?
Am I missing out on something blindingly obvious?
A friend suggested I look up incremental compilation methods because it's similar. An analogy used would be that Roslyn would parse/reparse fragments of text in a ranged fashion. That would be quite similar to my problem - just replace linear buffer of floats with linear buffer of tokens.
The problem is I couldn't find any solid useful information about how Roslyn does it.
This solution isn't particularly memory-efficient, but if I understand you correctly, it should allow for a relatively simple implementation of the functionality you want.
Keep an array or slice funcs of all your function objects, so that they each have a canonical integer index, and can be looked up by that index.
Keep a slice of ints s that is always the same size as your buffer of floats; it maps a particular index in your buffer to a "function index" in the slice of functions. You can use -1 to represent a number that is not part of any interval.
Keep a slice of (int, int) pairs intervals such that intervals[i] contains the start-end indices for the function stored at funcs[i].
I believe this enables you to implement your desired functionality without too much hassle. For example, to query by index i, look up s[i], then return funcs[s[i]] and intervals[s[i]]. When changes occur to the buffer, change s as well, cross-referencing between s and the intervals slice to figure out if neighboring intervals are affected. I'm happy to explain this part in more detail, but I don't totally understand the requirements for interval updates. (When you do an interval insert, does it correspond to an insert in the underlying buffer? Or are you just changing which buffer elements are associated with which functions? In which case, does an insert cause a deletion at the beginning of the next interval? Most schemes should work, but it changes the procedure.)
I would like a simple way to represent the order of a list of objects. When an object changes position in that list I would like to update just one record. I don't know if this can be done but I'm interested to ask the SO hive...
Wish-list constraints
the algorithm (or data structure) should allow for items to be repositioned in the list by updating the properties of a single item
the algorithm (or data structure) should require no housekeeping to maintain the integrity of the list
the algorithm (or data structure) should allow for the insertion of new items or the removal of existing items
Why I care about only updating one item at a time...
[UPDATED to clarify question]
The use-case for this algorithm is a web application with a CRUDy, resourceful server setup and a clean (Angular) client.
It's good practice to keep to the pure CRUD actions where possible and makes for cleaner code all round. If I can do this operation in a single resource#update request then I don't need any additional serverside code to handle the re-ordering and it can all be done using CRUD with no alterations.
If more than one item in the list needs to be updated for each move then I need a new action on my controller to handle it. It's not a showstopper but it starts spilling over into Angular and everything becomes less clean than it ideally should be.
Example
Let's say we have a magazine and the magazine has a number of pages :
Original magazine
- double page advert for Ford (page=1)
- article about Jeremy Clarkson (page=2)
- double page advert for Audi (page=3)
- article by James May (page=4)
- article by Richard Hammond (page=5)
- advert for Volkswagen (page=6)
Option 1: Store integer page numbers
... in which we update up to N records per move
If I want to pull Richard Hammond's page up from page 5 to page 2 I can do so by altering its page number. However I also have to alter all the pages which it then displaces:
Updated magazine
- double page advert for Ford (page=1)
- article by Richard Hammond (page=2)(old_value=5)*
- article about Jeremy Clarkson (page=3)(old_value=2)*
- double page advert for Audi (page=4)(old_value=3)*
- article by James May (page=5)(old_value=4)*
- advert for Volkswagen (page=6)
* properties updated
However I don't want to update lots of records
- it doesn't fit my architecture
Let's say this is being done using javascript drag-n-drop re-ordering via Angular.js. I would ideally like to just update a value on the page which has been moved and leave the other pages alone. I want to send an http request to the CRUD resource for Richard Hammond's page saying that it's now been moved to the second page.
- and it doesn't scale
It's not a problem for me yet but at some point I may have 10,000 pages. I'd rather not update 9,999 of them when I move a new page to the front page.
Option 2: a linked list
... in which we update 3 records per move
If instead of storing the page's position, I instead store the page that comes before it then I reduce the number of actions from a maximum of N to 3.
Original magazine
- double page advert for Ford (id = ford, page_before = nil)
- article about Jeremy Clarkson (id = clarkson, page_before = ford)
- article by James May (id = captain_slow, page_before = clarkson)
- double page advert for Audi (id = audi, page_before = captain_slow)
- article by Richard Hammond (id = hamster, page_before = audi)
- advert for Volkswagen (id = vw, page_before = hamster)
again we move the cheeky hamster up...
Updated magazine
- double page advert for Ford (id = ford, page_before = nil)
- article by Richard Hammond (id = hamster, page_before = ford)*
- article about Jeremy Clarkson (id = clarkson, page_before = hamster)*
- article by James May (id = captain_slow, page_before = clarkson)
- double page advert for Audi (id = audi, page_before = captain_slow)
- advert for volkswagen (id = vw, page_before = audi)*
* properties updated
This requires updating three rows in the database: the page we moved, the page just below its old position and the page just below its new position.
It's better but it still involves updating three records and doesn't give me the resourceful CRUD behaviour I'm looking for.
Option 3: Non-integer positioning
...in which we update only 1 record per move (but need to housekeep)
Remember though, I still want to update only one record for each repositioning. In my quest to do this I take a different approach. Instead of storing the page position as an integer I store it as a float. This allows me to move an item by slipping it between two others:
Original magazine
- double page advert for Ford (page=1.0)
- article about Jeremy Clarkson (page=2.0)
- double page advert for Audi (page=3.0)
- article by James May (page=4.0)
- article by Richard Hammond (page=5.0)
- advert for Volkswagen (page=6.0)
and then we move Hamster again:
Updated magazine
- double page advert for Ford (page=1.0)
- article by Richard Hammond (page=1.5)*
- article about Jeremy Clarkson (page=2.0)
- double page advert for Audi (page=3.0)
- article by James May (page=4.0)
- advert for Volkswagen (page=6.0)
* properties updated
Each time we move an item, we chose a value somewhere between the item above and below it (say by taking the average of the two items we're slipping between).
Eventually though you need to reset...
Whatever algorithm you use for inserting the pages into each other will eventually run out of decimal places since you have to keep using smaller numbers. As you move items more and more times you gradually move down the floating point chain and eventually need a new position which is smaller than anything available.
Every now and then you therefore have to do a reset to re-index the list and bring it all back within range. This is ok but I'm interested to see whether there is a way to encode the ordering which doesn't require this housekeeping.
Is there an algorithm which requires only 1 update and no housekeeping?
Does an algorithm (or perhaps more accurately, a data encoding) exist for this problem which requires only one update and no housekeeping? If so can you explain it in plain english how it works (i.g. no reference to directed graphs or vertices...)? Muchos gracias.
UPDATE (post points-awarding)
I've awarded the bounty on this to the question I feel had the most interesting answer. Nobody was able to offer a solution (since from the looks of things there isn't one) so I've not marked any particular question as correct.
Adjusting the no-housekeeping criterion
After having spent even more time thinking about this problem, it occurs to me that the housekeeping criterion should actually be adjusted. The real danger with housekeeping is not that it's a hassle to do but that it should ideally be robust to a client who has an outstanding copy of a pre-housekept set.
Let's say that Joe loads up a page containing a list (using Angular) and then goes off to make a cup of tea. Just after he downloads it the housekeeping happens and re-indexes all items (1000, 2000, 3000 etc).. After he comes back from his cup of tea, he moves an item from 1010 1011. There is a risk at this point that the re-indexing will place his item into a position it wasn't intended to go.
As a note for the future - any housekeeping algorithm should ideally be robust to items submitted across different housekept versions of the list too. Alternatively you should version the housekeeping and create an error if someone tries to update across versions.
Issues with the linked list
While the linked list requires only a few updates it's got some drawbacks too:
it's not trivial to deal with deletions from the list (and you may have to adjust your #destroy method accordingly
it's not easy to order the list for retrieval
The method I would choose
I think that having seen all the discussion, I think I would choose the non-integer (or string) positioning:
it's robust to inserts and deletions
it works of a single update
It does however need housekeeping and as mentioned above, if you're going to be complete you will also need to version each housekeeping and raise an error if someone tries to update based on a previous list version.
You should add one more sensible constraint to your wish-list:
max O(log N) space for each item (N being total number of items)
For example, the linked-list solution holds to this - you need at least N possible values for pointer, so the pointer takes up log N space. If you don't have this limit, trivial solution (growing strings) already mentioned by Lasse Karlsen and tmyklebu are solution to your problem, but the memory grows one character up (in the worst case) for each operation). You need some limit and this is a sensible one.
Then, hear the answer:
No, there is no such algorithm.
Well, this is a strong statement, and not easy to hear, so I guess proof is required :) I tried to figure out general proof, posted a question on Computer Science Theory, but the general proof is really hard to do. Say we make it easier and we will explicitly assume there are two classes of solutions:
absolute addressing - address of each item is specified by some absolute reference (integer, float, string)
relative addressing - address of each item is specified relatively to other items (e.g. the linked list, tree, etc.)
To disprove the existence of absolute addressing algorithm is easy. Just take 3 items, A, B, C, and keep moving the last one between the first two. You will soon run out of the possible combinations for the address of the moved element and will need more bits. You will break the constraint of the limited space.
Disproving the existence of relative addressing is also easy. For non-trivial arrangement, certainly some two different positions exist to which some other items are referring to. Then if you move some item between these two positions, at least two items have to be changed - the one which referred to the old position and the one which will refer to the new position. This violates the constraint of only one item changed.
Q.E.D.
Don't be fascinated by complexity - it doesn't work
Now that we (and you) can admit your desired solution does not exist, why would you complicate your life with complex solution that do not work? They can't work, as we proved above. I think we got lost here. Guys here spent immense effort just to end up with overly complicated solutions that are even worse than the simplest solution proposed:
Gene's rational numbers - they grow 4-6 bits in his example, instead of just 1 bit which is required by the most trivial algorithm (described below). 9/14 has 4 + 4 = 8 bits, 19/21 has 5 + 5 = 10 bits, and the resultant number 65/84 has 7 + 7 = 14 bits!! And if we just look at those numbers, we see that 10/14 or 2/3 are much better solutions. It can be easily proven that the growing string solution is unbeatable, see below.
mhelvens' solution - in the worst case he will add a new correcting item after each operation. This will for sure occupy much more than one bit more.
These guys are very clever but obviously cannot bring something sensible. Someone has to tell them - STOP, there's no solution, and what you do simply can't be better than the most trivial solution you are afraid to offer :-)
Go back to square one, go simple
Now, go back to the list of your restrictions. One of them must be broken, you know that. Go through the list and ask, which one of these is least painful?
1) Violate memory constraint
This is hard to violate infinitely, because you have limited space... so be prepared to also violate the housekeeping constraint from time to time.
The solution to this is the solution already proposed by tmyklebu and mentioned by Lasse Karlsen - growing strings. Just consider binary strings of 0 and 1. You have items A, B and C and moving C between A and B. If there is no space between A and B, i.e. they look
A xxx0
B xxx1
Then just add one more bit for C:
A xxx0
C xxx01
B xxx1
In worst case, you need 1 bit after every operation. You can also work on bytes, not bits. Then in the worst case, you will have to add one byte for every 8 operations. It's all the same. And, it can be easily seen that this solution cannot be beaten. You must add at least one bit, and you cannot add less. In other words, no matter how the solution is complex, it can't be better than this.
Pros:
you have one update per item
can compare any two elements, but slow
Cons:
comparing or sorting will get very very slow as the strings grow
there will be a housekeeping
2) Violate one item modified constraint
This leads to the original linked-list solution. Also, there are plenty of balanced tree data structures, which are even better if you need to look up or compare items (which you didn't mention).
These can go with 3 items modified, balanced trees sometimes need more (when balance operations are needed), but as it is amortized O(1), in a long row of operations the number of modifications per operation is constant. In your case, I would use tree solution only if you need to look up or compare items. Otherwise, the linked-list solution rocks. Throwing it out just because they need 3 operations instead of 1? C'mon :)
Pros:
optimal memory use
fast generation of ordered list (one linear pass), no need to sort
fast operations
no housekeeping
Cons:
cannot easily compare two items. Can easily generate the order of all the items, but given two items randomly, comparing them will take O(N) for list and O(log N) for balanced trees.
3 modified items instead of 1 (... letting up to you how much of a "con" this is)
3) Violate "no housekeeping" constraint
These are the solution with integers and floats, best described by Lasse Karlsen here. Also, the solutions from point 1) will fall here :). The key question was already mentioned by Lasse:
How often will housekeeping have to take place?
If you will use k-bit integers, then from the optimal state, when items are spread evenly in the integer space, the housekeeping will have to take place every k - log N operations, in the worst-case. You might then use more ore less sophisticated algorithms to restrict the number of items you "housekeep".
Pros:
optimal memory use
fast operation
can compare any two elements
one item modified per operation
Cons:
housekeeping
Conclusion - hope never dies
I think the best way, and the answers here prove that, is to decide which one of those constraints is least pain and just take one of those simple solutions formerly frowned upon.
But, hope never dies. When writing this, I realized that there would be your desired solution, if we just were able to ask the server!! Depends on the type of the server of course, but the classical SQL server already has the trees/linked-list implemented - for indices. The server is already doing the operations like "move this item before this one in the tree"!! But the server is doing based on the data, not based on our request. If we were able somehow to ask server to do this without the need to create perverse, endlessly growing data, that would be your desired solution! As I said, the server already does it - the solution is sooo close, but so far. If you can write your own server, you can do it :-)
#tmyklebu has the answer, but he never quite got to the punch line: The answer to your question is "no" unless you are willing to accept a worst case key length of n-1 bits to store n items.
This means that total key storage for n items is O(n^2).
There is an "adversary" information-theoretic argument that says no matter what scheme for assigning keys you choose for a database of n items, I can always come up with a series of n item re-positionings ("Move item k to position p.") that will force you to use a key with n-1 bits. Or by extension, if we start with an empty database, and you give me items to insert, I can choose a sequence of insertion positions that will require you to use at least zero bits for the first, one for the second, etc. indefinitely.
Edit
I earlier had an idea here about using rational numbers for keys. But it was more expensive than just adding one bit of length to split the gap between pairs of keys that differ by one. So I've removed it.
You can also interpret option 3 as storing positions as an unbounded-length string. That way you don't "run out of decimal places" or anything of that nature. Give the first item, say 'foo', position 1. Recursively partition your universe into "the stuff that's less than foo", which get a 0 prefix, and "the stuff that's bigger than foo", which get a 1 prefix.
This sucks in a lot of ways, notably that the position of an object can need as many bits to represent as you've done object moves.
I was fascinated by this question, so I started working on an idea. Unfortunately, it's complicated (you probably knew it would be) and I don't have time to work it all out. I just thought I'd share my progress.
It's based on a doubly-linked list, but with extra bookkeeping information in every moved item. With some clever tricks, I suspect that each of the n items in the set will require less than O(n) extra space, even in the worst case, but I have no proof of this. It will also take extra time to figure out the view order.
For example, take the following initial configuration:
A (-,B|0)
B (A,C|0)
C (B,D|0)
D (C,E|0)
E (D,-|0)
The top-to-bottom ordering is derived purely from the meta-data, which consists of a sequence of states (predecessor,successor|timestamp) for each item.
When moving D between A and B, you push a new state (A,B|1) to the front of its sequence with a fresh timestamp, which you get by incrementing a shared counter:
A (-,B|0)
D (A,B|1) (C,E|0)
B (A,C|0)
C (B,D|0)
E (D,-|0)
As you see, we keep the old information around in order to connect C to E.
Here is roughly how you derive the proper order from the meta-data:
You keep a pointer to A.
A agrees it has no predecessor. So insert A. It leads you to B.
B agrees it wants to be successor to A. So insert B after A. It leads you to C.
C agrees it wants to be successor to B. So insert C after B. It leads you to D.
D disagrees. It wants to be successor to A. Start recursion to insert it and find the real successor:
D wins from B because it has a more recent timestamp. Insert D after A. It leads you to B.
B is already D's successor. Look back in D's history, which leads you to E.
E agrees it wants to be successor to D with timestamp 0. So return E.
So the successor is E. Insert E after C. It tells you it has no successor. You are finished.
This is not exactly an algorithm yet, because it doesn't cover all cases. For example, when you move an item forwards instead of backwards. When moving B between D and E:
A (-,B|0)
C (B,D|0)
D (C,E|0)
B (D,E|1)(A,C|0)
E (D,-|0)
The 'move' operation is the same. But the algorithm to derive the proper order is a bit different. From A it will run into B, able to get the real successor C from it, but with no place to insert B itself yet. You can keep it in reserve as a candidate for insertion after D, where it will eventually match timestamps against E for the privilege of that position.
I wrote some Angular.js code on Plunker that can be used as a starting-point to implement and test this algorithm. The relevant function is called findNext. It doesn't do anything clever yet.
There are optimizations to reduce the amount of metadata. For example, when moving an item away from where it was recently placed, and its neighbors are still linked of their own accord, you won't have to preserve its newest state but can just replace it. And there are probably situations where you can discard all of an item's sufficiently old states (when you move it).
It's a shame I don't have time to fully work this out. It's an interesting problem.
Good luck!
Edit: I felt I needed to clarify the above-mentioned optimization ideas. First, there is no need to push a new history configuration if the original links still hold. For example, it is fine to go from here (moved D between A and B):
A (-,B|0)
D (A,B|1) (C,E|0)
B (A,C|0)
C (B,D|0)
E (D,-|0)
to here (then moved D between B and C):
A (-,B|0)
B (A,C|0)
D (B,C|2) (C,E|0)
C (B,D|0)
E (D,-|0)
We are able to discard the (A,B|1) configuration because A and B were still connected by themselves. Any number of 'unrelated' movements can come inbetween without changing that.
Secondly, imagine that eventually C and E are moved away from each other, so the (C,E|0) configuration can be dropped the next time D is moved. This is trickier to prove, though.
All of this considered, I believe there is a good chance that the list requires less than O(n+k) space (n being the number of items in the list, k being the number of operations) in the worst case; especially in the average case.
The way to prove any of this is to come up with a simpler model for this data-structure, most likely based on graph theory. Again, I regret that I don't have time to work on this.
Your best option is "Option 3", although "non-integer" doesn't necessarily have to be involved.
"Non-integer" can mean anything that have some kind of accuracy definition, which means:
Integers (you just don't use 1, 2, 3, etc.)
Strings (you just tuck on more characters to ensure the proper "sort order")
Floating point values (adding more decimal points, somewhat the same as strings)
In each case you're going to have accuracy problems. For floating point types, there might be a hard limit in the database engine, but for strings, the limit will be the amount of space you allow for this. Please note that your question can be understood to mean "with no limits", meaning that for such a solution to work, you really need infinite accuracy/space for the keys.
However, I think that you don't need that.
Let's assume that you initially allocate every 1000th index to each row, meaning you will have:
1000 A
2000 B
3000 C
4000 D
... and so on
Then you move as follows:
D up between A and B (gets index 1500)
C up between A and D (gets index 1250)
B up between A and C (gets index 1125)
D up between A and B (gets index 1062)
C up between A and D (gets index 1031)
B up between A and C (gets index 1015)
D up between A and B (gets index 1007)
C up between A and D (gets index 1004)
B up between A and C (gets index 1002)
D up between A and B (gets index 1001)
At this point, the list looks like this:
1000 A
1001 D
1002 B
1004 C
Now, then you want to move C up between A and D.
This is currently not possible, so you're going to have to renumber some items.
You can get by by updating B to have number 1003, trying to update the minimum number of rows, and thus you get:
1000 A
1001 C
1002 D
1003 B
but now, if you want to move B up between A and C, you're going to renumber everything except A.
The question is this: How likely is it that you have this pathological sequence of events?
If the answer is very likely then you will have problems, regardless of what you do.
If the answer is likely seldom, then you might decide that the "problems" with the above approach are manageable. Note that renumbering and ordering more than one row will likely be the exceptions here, and you would get something like "amortized 1 row updated per move". Amortized means that you spread the cost of those occasions where you have to update more than one row out over all the other occasions where you don't.
What if you store the original order and don't change it after saving it once and then store the number of increments up the list or down the list?
Then by moving something up 3 levels you would store this action only.
in the database you can then order by a mathematically counted column.
First time insert:
ord1 | ord2 | value
-----+------+--------
1 | 0 | A
2 | 0 | B
3 | 0 | C
4 | 0 | D
5 | 0 | E
6 | 0 | F
Update order, move D up 2 levels
ord1 | ord2 | value | ord1 + ord2
-----+------+-------+-------------
1 | 0 | A | 1
2 | 0 | B | 2
3 | 0 | C | 3
4 | -2 | D | 2
5 | 0 | E | 5
6 | 0 | F | 6
Order by ord1 + ord2
ord1 | ord2 | value | ord1 + ord2
-----+------+-------+-------------
1 | 0 | A | 1
2 | 0 | B | 2
4 | -2 | D | 2
3 | 0 | C | 3
5 | 0 | E | 5
6 | 0 | F | 6
Order by ord1 + ord2 ASC, ord2 ASC
ord1 | ord2 | value | ord1 + ord2
-----+------+-------+-------------
1 | 0 | A | 1
4 | -2 | D | 2
2 | 0 | B | 2
3 | 0 | C | 3
5 | 0 | E | 5
6 | 0 | F | 6
Move E up 4 levels
ord1 | ord2 | value | ord1 + ord2
-----+------+-------+-------------
5 | -4 | E | 1
1 | 0 | A | 1
4 | -2 | D | 2
2 | 0 | B | 2
3 | 0 | C | 3
6 | 0 | F | 6
Something like relative ordering, where ord1 is the absolute order while ord2 is the relative order.
Along with the same idea of just storing the history of movements and sorting based on that.
Not tested, not tried, just wrote down what I thought at this moment, maybe it can point you in some direction :)
I am unsure if you will call this cheating, but why not create a separate page list resource that references the page resources?
If you change the order of the pages you need not update any of the pages, just the list that stores the order if the IDs.
Original page list
[ford, clarkson, captain_slow, audi, hamster, vw]
Update to
[ford, hamster, clarkson, captain_slow, audi, vw]
Leave the page resources untouched.
You could always store the ordering permutation separately as a ln(num_records!)/ln(2) bit bitstring and figure out how to transform/CRUD that yourself so that you'd only need to update a single bit for simple operations, if updating 2/3 records is not good enough for you.
What about the following very simple algorithm:
(let's take the analogy with page numbers in a book)
If you move a page to become the "new" page 3, you now have "at least" one page 3, possibly two, or even more. So, which one is the "right" page 3?
Solution: the "newest". So, we make use of the fact that a record also has an "updated date/time", to determine who the real page 3 is.
If you need to represent the entire list in its right order, you have to sort with two keys, one for the page number, and one for the "updated date/time" field.
Some time ago I learned that rsync deletes files much faster that many other tools.
A few days ago I came across this wonderful answer on Serverfault which explains why rsync is so good at deleting files.
Quotation from that answer:
I revisited this today, because most filesystems store their directory
structures in a btree format, the order of which you delete files is
also important. One needs to avoid rebalancing the btree when you
perform the unlink. As such I added a sort before deletes occur.
Could you explain how does removing files in-order prevents or reduces the number of btree rebalancings?
I expect the answer to show how deleting in order increase deletion speed, with details of what happens at btree level. People, who wrote rsync and another programs (see links in the question) used this knowledge to create better programs. I think it's important for other programmers to have this understanding to be able to write better soft.
It is not important, nor b-tree issue. It is just a coincidence.
First of all, this is very much implementation dependent and very much ext3 specific. That's why I said it's not important (for general use). Otherwise, put the ext3 tag or edit the summary line.
Second of all, ext3 does not use b-tree for the directory entry index. It uses Htree. The Htree is similar to b-tree but different and does not require balancing. Search "htree" in fs/ext3/dir.c.
Because of the htree based index, a) ext3 has a faster lookup compare to ext2, but b) readdir() returns entries in hash value order. The hash value order is random relative to file creation time or physical layout of data. As we all know random access is much slower than sequential access on a rotating media.
A paper on ext3 published for OLS 2005 by Mingming Cao, et al. suggests (emphasis mine):
to sort the directory entries returned by readdir() by inode number.
Now, onto rsync. Rsync sorts files by file name. See flist.c::fsort(), flist.c::file_compare(), and flist.c::f_name_cmp().
I did not test the following hypothesis because I do not have the data sets from which #MIfe got 43 seconds. but I assume that sorted-by-name was much closer to the optimal order compare to the random order returned by readdir(). That was why you saw much faster result with rsync on ext3. What if you generate 1000000 files with random file names then delete them with rsync? Do you see the same result?
Let's assume that the answer you posted is correct, and that the given file system does indeed store things in a balanced tree. Balancing a tree is a very expensive operation. Keeping a tree "partially" balanced is pretty simple, in that when you allow for a tree to be imbalanced slightly, you only worry about moving things around the point of insertion/deletion. However, when talking about completely balanced trees, when you remove a given node, you may find that suddenly, the children of this node could belong on the complete opposite side of the tree, or a child node on the opposite side has become the root node, and all of it's children need to be rotated up the tree. This requires you to do either a long series of rotations, or to place all the items into an array and re-create the tree.
5
3 7
2 4 6 8
now remove the 7, easy right?
5
3 8
2 4 6
Now remove the 6, still easy, yes...?
5
3 8
2 4
Now remove the 8, uh oh
5
3
2 4
Getting this tree to be the proper balanced form like:
4
3 5
2
Is quite expensive, compared at least to the other removals we have done, and gets exponentially worse as the depth of our tree increases. We could make this go much(exponentially) faster by removing the 2 and the 4, before removing the 8. Particularly if our tree was more than 3 levels deep.
Without sorting removal is on average a O(K * log_I(N)^2). N representing the number of elements total, and K the number to be removed, I the number of children a given node is permitted, log_I(N) then being the depth, and for each level of depth we increase the number of operations quadratically.
Removal with some ordering help is on average O(K * log_I(N)), though sometimes ordering cannot help you and you are stuck removing something that will require a re-balance. Still, minimizing this is optimal.
EDIT:
Another possible tree ordering scheme:
8
6 7
1 2 3 4
Accomplishing optimal removal under this circumstance would be easier, because we can take advantage of our knowledge of how things are sorted. Under either situation it is possible, and in fact both are identical, under this one the logic is just a little simpler to understand, because the ordering is more human friendly for the given scenario. In either case in-order is defined as "remove the farthest leaf first", in this case it just so happens to be that the farthest leaves are also the smallest numbers, a fact that we could take advantage of to make it even a little more optimal, but this fact is not necessarily true for the file system example presented(though it may be).
I am not convinced that the number of B-tree rebalancing changes significantly if you delete the files in-order. However I do believe that the number of different seeks to external storage will be significantly smaller if you do this. At any time, the only nodes in the B-tree that need be visited will then be the far right boundary of the tree, whereas, with a random order, each leaf block in the B tree is visited with equal probability for each file.
Rebalancing for B-Trees are cheaper than B-Tree+ implementations that's why most filesystems and database indexes implementations use them.
There are many approaches when deletion, depending on the approach it can be more efficient in terms of time and the need to rebalance the tree. You'll also have to consider the size of the node, since the number of keys the node can store will affect the need for rebalancing the tree. A large node size will just reorder keys inside the node, but a small one probably will make the tree rebalance many times.
A great resource for understanding this is the famous CLR (Thomas Cormen) book "Introduction to Algorithms".
On storage systems where hosting huge directories, the buffer cache will be under stress and buffers may get recycled. So, if you have deletes spaced apart by time, then the number of disk reads to get the btree back in core into the buffer cache, between deletes, may be high.
If you sort the files to be deleted, effectively you are delaying the deletes and bunching them. This may have the side effect of more deletes per block of btree paged in. If there are stats to say what the buffer cache hits are between the two experiments, it may tell if this hypo is wrong or not.
But, if there is no stress on the buffer cache during the deletes, then the btree blocks could stay in core and then my hypothesis is not a valid one.
We have two offline systems that normally can not communicate with each other. Both systems maintain the same ordered list of items. Only rarely will they be able to communicate with each other to synchronize the list.
Items are marked with a modification timestamp to detect edits. Items are identified by UUIDs to avoid conflicts when inserting new items (as opposed to using auto-incrementing integers). When synchronizing new UUIDs are detected and copied to the other system. Likewise for deletions.
The above data structure is fine for an unordered list, but how can we handle ordering? If we added an integer "rank", that would need renumbering when inserting a new item (thus requiring synchronizing all successor items due to only 1 insertion). Alternatively, we could use fractional ranks (use the average of the ranks of the predecessor and successor item), but that doesn't seem like a robust solution as it will quickly run into accuracy problems when many new items are inserted.
We also considered implementing this as a doubly linked-list with each item holding the UUID of its predecessor and successor item. However, that would still require synchronizing 3 items when 1 new items was inserted (or synchronizing the 2 remaining items when 1 item was deleted).
Preferably, we would like to use a data structure or algorithm where only the newly inserted item needs to be synchronized. Does such a data structure exist?
Edit: we need to be able to handle moving an existing item to a different position too!
There is really no problem with the interpolated rank approach. Just define your own numbering system based on variable length bit vectors representing binary fractions between 0 and 1 with no trailing zeros. The binary point is to the left of the first digit.
The only inconvenience of this system is that the minimum possible key is 0 given by the empty bit vector. Therefore you use this only if you're positive the associated item will forever be the first list element. Normally, just give the first item the key 1. That's equivalent to 1/2, so random insertions in the range (0..1) will tend to minimize bit usage. To interpolate an item before and after,
01 < newly interpolated = 1/4
1
11 < newly interpolated = 3/4
To interpolate again:
001 < newly interpolated = 1/8
01
011 < newly interpolated = 3/8
1
101 < newly interpolated = 5/8
11
111 < newly interpolated = 7/8
Note that if you wish you can omit storing the final 1! All keys (except 0 which you won't normally use) end in 1, so storing it is supefluous.
Comparison of binary fractions is a lot like lexical comparison: 0<1 and the first bit difference in a left-to-right scan tells you which is less. If no differences occur, i.e. one vector is a strict prefix of the other, then the shorter one is smaller.
With these rules it's pretty simple to come up with an algorithm that accepts two bit vectors and computes a result that's roughly (or exactly in some cases) between them. Just add the bit strings, and shift right 1, dropping unnecessary trailing bits, i.e. take the average of the two to split the range between.
In the example above, if deletions had left us with:
01
111
and we need to interpolate these, add 01(0) and and 111 to obtain 1.001, then shift to get 1001. This works fine as an interpolant. But note the final 1 unnecessarily makes it longer than either of the operands. An easy optimization is to drop the final 1 bit along with trailing zeros to get simply 1. Sure enough, 1 is about half way between as we'd hope.
Of course if you do many inserts in the same location (think e.g. of successive inserts at the start of the list), the bit vectors will get long. This is exactly the same phenomenon as inserting at the same point in a binary tree. It grows long and stringy. To fix this, you must "rebalance" during a synchronization by renumbering with the shortest possible bit vectors, e.g. for 14 you'd use the sequence above.
Addition
Though I haven't tried it, the Postgres bit string type seems to suffice for the keys I've described. The thing I'd need to verify is that the collation order is correct.
Also, the same reasoning works just fine with base-k digits for any k>=2. The first item gets key k/2. There is also a simple optimization that prevents the very common cases of appending and prepending elements at the end and front respectively from causing keys of length O(n). It maintains O(log n) for those cases (though inserting at the same place internally can still produce O(p) keys after p insertions). I'll let you work that out. With k=256, you can use indefinite length byte strings. In SQL, I believe you'd want varbinary(max). SQL provides the correct lexicographic sort order. Implementation of the interpolation ops is easy if you have a BigInteger package similar to Java's. If you like human-readable data, you can convert the byte strings to e.g. hex strings (0-9a-f) and store those. Then normal UTF8 string sort order is correct.
You can add two fields to each item - 'creation timestamp' and 'inserted after' (containing the id of the item after which the new item was inserted). Once you synchronize a list, send all the new items. That information is enough for you to be able to construct the list on the other side.
With the list of newly added items received, do this (on the receiving end): sort by creation timestamp, then go one by one, and use the 'inserted after' field to add the new item in the appropriate place.
You may face trouble if an item A is added, then B is added after A, then A is removed. If this can happen, you will need to sync A as well (basically syncing the operations that took place on the list since the last sync, and not just the content of the current list). It's basically a form of log-shipping.
You could have a look at "lenses", which is bidirectional programming concept.
For instance, your problem seems to be solved my "matching lenses", described in this paper.
I think the datastructure that is appropriate here is order statistic tree. In order statistic tree you also need to maintain sizes of subtrees along with other data, the size field helps easy to find element by rank as you need it to be. All operations like rank,delete,change position,insert are O(logn).
I think you can try kind of transactional approach here. For example you do not delete items physically but mark them for deletion and commit changes only during synchronization. I'm not absolutely sure which data type you should choose, it depends on which operations you want to be more productive (insertions, deletions, search or iteration).
Let we have the following initial state on both systems:
|1| |2|
--- ---
|A| |A|
|B| |B|
|C| |C|
|D| |D|
After that the first system marks element A for deletion and the second system inserts element BC between B and C:
|1 | |2 |
------------ --------------
|A | |A |
|B[deleted]| |B |
|C | |BC[inserted]|
|D | |C |
|D |
Both systems continue processing taking into account local changes, System 1 ignores element B and System 2 treats element BC as normal element.
When synchronization occurs:
As I understand, each system receives the list snapshot from other system and both systems freeze processing until synchronization will be finished.
So each system iterates sequentially through received snapshot and local list and writes changes to local list (resolving possible conflicts according to modified timestamp) after that 'transaction is commited', all local changes are finally applied and information about them erases.
For example for system one:
|1 pre-sync| |2-SNAPSHOT | |1 result|
------------ -------------- ----------
|A | <the same> |A | |A |
|B[deleted]| <delete B> |B |
<insert BC> |BC[inserted]| |BC |
|C | <same> |C | |C |
|D | <same> |D | |D |
Systems wake up and continue processing.
Items are sorted by insertion order, moving can be implemented as simultaneous deletion and insertion. Also I think that it will be possible not to transfer the whole list shapshot but only list of items that were actually modified.
I think, broadly, Operational Transformation could be related to the problem you are describing here. For instance, consider the problem of Real-Time Collaborative text editing.
We essentially have a sorted list of items( words) which needs to be kept synchronized, and which could be added/modified/deleted at random within the list. The only major difference I see is in the periodicity of modifications to the list.( You say it does not happen often)
Operational Transformation does happen to be well studied field. I could find this blog article giving pointers and introduction. Plus, for all the problems Google Wave had, they actually made significant advancements to the domain of Operational Transform. Check this out. . There is quite a bit of literature available on this subject. Look at this stackoverflow thread, and about Differential Synchronisation
Another parallel that struck me was the data structure used in Text Editors - Ropes.
So if you have a log of operations,lets say, "Index 5 deleted", "Index 6 modified to ABC","Index 8 inserted",what you might now have to do is to transmit a log of the changes from System A to System B, and then reconstruct the operations sequentially on the other side.
The other "pragmatic Engineer" 's choice would be to simply reconstruct the entire list on System B when System A changes. Depending on actual frequency and size of changes, this might not be as bad as it sounds.
I have tentatively solved a similar problem by including a PrecedingItemID (which can be null if the item is the top/root of the ordered list) on each item, and then having a sort of local cache that keeps a list of all items in sorted order (this is purely for efficiency—so you don't have to recursively query for or build the list based on PrecedingItemIDs every time there is a re-ordering on the local client). Then when it comes time to sync I do the slightly more expensive operation of looking for cases where two items are requesting the same PrecedingItemID. In those cases, I simply order by creation time (or however you want to reconcile which one wins and comes first), put the second (or others) behind it, and move on ordering the list. I then store this new ordering in the local ordering cache and go on using that until the next sync (just making sure to keep the PrecedingItemID updated as you go).
I haven't unit tested this approach yet—so I'm not 100% sure I'm not missing some problematic conflict scenario—but it appears at least conceptually to handle my needs, which sound similar to those of the OP.
Please explain what is the advantage of linked list over an array. And also is there any advantage of using array compared to linked list.
Regards,
Shoaib
Both store a sequence of elements, but using different techniques.
An array stores elements in successive order in memory, i.e. it looks like follows:
--------------------------------------------------------------------------------------
| item 1 | item 2 | item 3 | ... ... | item x | //here comes other stuff
--------------------------------------------------------------------------------------
This means, elements are one after another consecutive in memory.
A ((doubly) linked) list, on the other hand, stores the items the following way: It creates an own "list item" for each element; this "list item" holds the actual element and a pointer/reference/hint/etc to the next "list item". If it is doubly linked, it also contains a pointer/reference/hint/etc to the previous "list item":
------------
------------ ---------- | item 4 |
| item 1 | | item 2 | | next ---+----...
| next ---+------->| next | ------------
------------ ---+------ ^
| |
| |
v |
---------- |
| item 3 | |
| next --+---+
----------
This means, the elements can be spread all over the memory and are not stored at specific memory locations.
Now that we know this, we can compare some usual operations on sequences of elements:
Accessing an element at a specific index: Using an array, we simply compute the offset for this index and have the element at the index.
This is very cheap. With a list on the other hand, we do not know a priori where the element at index is stored (since all elements can be anywhere in memory), so we have to walk the list item by item until we find the element at the index. This is an expensive operation.
Adding a new element at the end of the sequence: Using an array this can lead to the following problem: Arrays are usually of fixed size, so if we have the situation that our array is already completely filled (see //here comes other stuff), we probably cant put the new element there because there might already be something else. So, maybe we have to copy the whole array. However, if the array is not filled, we can simply put the element there.
Using a list, we have to generate a new "list item", put the element into it and set the next pointer of the currently last element to this new list item. Usually, we store a reference to the currently last element so that we don't have to search through the whole list to find it. Thus, inserting a new element is no real problem with lists.
Adding a new element somewhere in the middle: Let's first consider arrays: here, we might get into the following situation: We have an array with elements 1 to 1000:
1 | 2 | 3 | 4 | 5 | 6 | ... | 999 | 1000 | free | free
Now, we want to insert 4.5 between 4 and 5: This means, we have to move all elements from 5 to 1000 one position right in order to make space for the 4.5:
1 | 2 | 3 | 4 | free | 5 | 6 | ... | 999 | 1000 | free
||
vv
1 | 2 | 3 | 4 | 4.5 | 5 | 6 | ... | 999 | 1000 | free
Moving all these elements, is a very expensive operation. So better don't do this too often.
Now we consider, how a list can perform this task: Let's say we have currently the following list:
1 -> 2 -> 3 -> 4 -> 5 -> ... -> 999 -> 1000
Again, we want to insert 4.5 between 4 and 5. This can be done very easily: We generate a new list item and update the pointers/references:
1 -> 2 -> 3 -> 4 5 -> ... -> 999 -> 1000
| ^
+-> 4.5 -+
We have simply created a new list element and generated sort of "bypass" to insert it - very cheap (as long as we have a pointer/reference to the list item the new element will be inserted after).
So, let's sum up: Linked lists are really nice when it comes to inserting at random positions (as long as you have a pointer to the adequate list item). If your operation involves adding lots of elements dynamically and traversing all elements anyway, a list might be a good choice.
An array is very good when it comes to index accesses. If your application needs to access elements at specific positions very often, you should rather use an array.
Notable things:
Solving the fixed-size problem for arrays: As mentioned before, (raw) arrays are usually of fixed size. However, this problem is nowadays no real point anymore, since almost all programming languages provide mechanisms to generate arrays that grow (and possibly shrink) dynamically - just as needed. This growing and shrinking can be implemented such that we have amortized runtime of O(1) for inserting and removing elements (at the end of the array) and such that the programmer doesn't have to call grow and shrink explicitly.
Since lists provide such nice properties for insertions, they can be used as underlying data structures for search trees, etc. I.e. you construct a search tree, whose lowest level consists of the linked list.
Arrays have a fixed size but are faster to access: they are allocated in one place and the location of each element is known (you can jump to the right element).
Lists are not limited in size but for the amount of available memory. They are slower to access since to find an element you have to traverse the list.
This is a very short explanation: I would suggest you to get a book on data structures and read it. These are basic concepts that you will need to fully understand.
Since you tagged the question with "data structures" I will answer this in that context.
An array is fixed in size when you declare/create it meaning you can't add more elements to it. Thus, if I have an array of, say, 5 elements, you can do whatever you want with it but you can't add more elements to it.
A linked-list is basically a way to represent a list where you can have as many "items" as you'd like. It consists of a head (the first element), a tail (the last element), and elements ( called nodes ) in-between the list.
There are many types of linked-lists that you will probably encounter in any data structures class.
The key thing you will learn with linked-lists is proficiency when learning how to create fields in your classes to point to other objects, which is the case for linked-lists where you need to construct your list such that each node points to the next node.
Obviously, this is a very generalized answer. It should give you an idea for your class.
Advantages of Array over Linked List
The array has a specific address for each element stored in it and thus we can access any memory directly.
As we know the position of the middle element and other elements are easily accessible too, we can easily perform BINARY SEARCH in the array.
Disadvantages of Array over Linked List
Total number of elements need to be mentioned or the memory allocation needs to be done at the time of array creation
The size of the array, once mentioned, cannot be increased in the program. If the number of elements entered exceed the size of the array ARRAY OVERFLOW EXCEPTION occurs.
Advantages of Linked List over Array
Size of the list doesn't need to be mentioned at the beginning of the program.
As the linked list doesn't have a size limit, we can go on adding new nodes (elements) and increasing the size of the list to any extent.
Disadvantages of Linked List over Array
Nodes do not have their own address. Only the address of the first node is stored and in order to reach any node, we need to traverse the whole list from beginning to the desired node.
As all Nodes don't have their particular address, BINARY SEARCH cannot be performed.
If you don't know the amount of objects you need to store beforehand, a list is probably what you want, since it's very easy to dynamically shrink or grow the list as needed. With this also comes the advantage of being able to easily insert elements mid-list without any need for reallocation.
The disadvantage of a list vis-à-vis an array, on the other hand, is that it's slower to select individual elements, since you need to iterate. With an array, you won't have this problem. Arrays, on the other hand, are troublesome to use if you need to resize it, as this operation is more costly than adding or subtracting elements from a linked list.
Lists should be used more commonly, since the ease of use is often more beneficial than the small performance gain from using a static size array.
While it has been mentioned that arrays have better performance than linked lists, I am surprised to see no mention of the word "cache" anywhere. The problem with linked lists is mainly that they pretty much guarantee that every jump from node to node will be a cache miss, which is incredibly, brutally expensive, performance-wise.
To put it crudely, if performance matters even the slightest bit in your program, you should never use a linked list. Ever. There is no excuse for it. They are far beyond "slow", they are catastrophically slow, and nothing about them can make up for that fact. Using them is like an intentional sabotage to your performance, like inserting a "wait()" function into your code for absolutely no reason.