Is there a nearest-key map datastructure? - algorithm

I have a situation where I need to find the value with the key closest to the one I request. It's kind of like a nearest map that defines distance between keys.
For example, if I have the keys {A, C, M, Z} in the map, a request for D would return C's value.
Any idea?

Most tree data structures use some sort of sorting algorithm to store and find keys. Many implementations of such can locate a close key to the key you probe with (usually it either the closest below or the closest above). For example Java's TreeMap implements such a data structure and you can tell it to get you the closest key below your lookup key, or the closest key above your lookup key (higherKey and lowerKey).
If you can calculate distances (its not always easy - Java's interface only require you to know if any given key is "below" or "above" any other given key) then you can ask for both closest above and closest below and then calculate for yourself which one is closer.

What's the dimensionality of your data? If it's just one dimensional, a sorted array will do it - a binary search will locate the exact match and/or reveal betweeen which two keys your search key lies - and a simple test will tell you which is closer.
If you need to locate not just the nearest key, but an associated value, maintain an identically sorted array of values - the index of the retrieved key in the key array is then the index of the value in the value array.
Of course, there are many alternative approaches - which one to use depends on many other factors, such as memory consumption, whether you need to insert values, if you control the order of insertion, deletions, threading issues, etc...

BK-trees do precisely what you want. Here's a good article on implementing them.
And here is a Scala implementation:
class BKTree[T](computeDistance: (T, T) => Int, node: T) {
val subnodes = scala.collection.mutable.HashMap.empty[Int,BKTree[T]]
def query(what: T, distance: Int): List[T] = {
val currentDistance = computeDistance(node, what)
val minDistance = currentDistance - distance
val maxDistance = currentDistance + distance
val elegibleNodes = (
subnodes.keys.toList
filter (key => minDistance to maxDistance contains key)
map subnodes
)
val partialResult = elegibleNodes flatMap (_.query(what, distance))
if (currentDistance <= distance) node :: partialResult else partialResult
}
def insert(what: T): Boolean = if (node == what) false else (
subnodes.get(computeDistance(node, what))
map (_.insert(what))
getOrElse {
subnodes(computeDistance(node, what)) = new BKTree(computeDistance, what)
true
}
)
override def toString = node.toString+"("+subnodes.toString+")"
}
object Test {
def main(args: Array[String]) {
val root = new BKTree(distance, 'A')
root.insert('C')
root.insert('M')
root.insert('Z')
println(findClosest(root, 'D'))
}
def charDistance(a: Char, b: Char) = a - b abs
def findClosest[T](root: BKTree[T], what: T): List[T] = {
var distance = 0
var closest = root.query(what, distance)
while(closest.isEmpty) {
distance += 1
closest = root.query(what, distance)
}
closest
}
}
I'll admit to a certain dirt&uglyness about it, and of being way too clever with the insertion algorithm. Also, it will only work fine for small distance, otherwise you'll search repeatedly the tree. Here's an alternate implementation that does a better job of it:
class BKTree[T](computeDistance: (T, T) => Int, node: T) {
val subnodes = scala.collection.mutable.HashMap.empty[Int,BKTree[T]]
def query(what: T, distance: Int): List[T] = {
val currentDistance = computeDistance(node, what)
val minDistance = currentDistance - distance
val maxDistance = currentDistance + distance
val elegibleNodes = (
subnodes.keys.toList
filter (key => minDistance to maxDistance contains key)
map subnodes
)
val partialResult = elegibleNodes flatMap (_.query(what, distance))
if (currentDistance <= distance) node :: partialResult else partialResult
}
private def find(what: T, bestDistance: Int): (Int,List[T]) = {
val currentDistance = computeDistance(node, what)
val presentSolution = if (currentDistance <= bestDistance) List(node) else Nil
val best = currentDistance min bestDistance
subnodes.keys.foldLeft((best, presentSolution))(
(acc, key) => {
val (currentBest, currentSolution) = acc
val (possibleBest, possibleSolution) =
if (key <= currentDistance + currentBest)
subnodes(key).find(what, currentBest)
else
(0, Nil)
(possibleBest, possibleSolution) match {
case (_, Nil) => acc
case (better, solution) if better < currentBest => (better, solution)
case (_, solution) => (currentBest, currentSolution ::: solution)
}
}
)
}
def findClosest(what: T): List[T] = find(what, computeDistance(node, what))._2
def insert(what: T): Boolean = if (node == what) false else (
subnodes.get(computeDistance(node, what))
map (_.insert(what))
getOrElse {
subnodes(computeDistance(node, what)) = new BKTree(computeDistance, what)
true
}
)
override def toString = node.toString+"("+subnodes.toString+")"
}
object Test {
def main(args: Array[String]) {
val root = new BKTree(distance, 'A')
root.insert('C')
root.insert('E')
root.insert('M')
root.insert('Z')
println(root.findClosest('D'))
}
def charDistance(a: Char, b: Char) = a - b abs
}

With C++ and STL containers (std::map) you can use the following template function:
#include <iostream>
#include <map>
//!This function returns nearest by metric specified in "operator -" of type T
//!If two items in map are equidistant from item_to_find, the earlier occured by key will be returned
template <class T,class U> typename std::map<T,U>::iterator find_nearest(std::map<T,U> map_for_search,const T& item_to_find)
{
typename std::map<T,U>::iterator itlow,itprev;
itlow=map_for_search.lower_bound(item_to_find);
itprev=itlow;
itprev--;
//for cases when we have "item_to_find" element in our map
//or "item_to_find" occures before the first element of map
if ((itlow->first==item_to_find) || (itprev==map_for_search.begin()))
return itlow;
//if "item"to_find" is besides the last element of map
if (itlow==map_for_search.end())
return itprev;
return (itlow->first-item_to_find < item_to_find-itprev->first)?itlow:itprev; // C will be returned
//note that "operator -" is used here as a function for distance metric
}
int main ()
{
std::map<char,int> mymap;
std::map<char,int>::iterator nearest;
//fill map with some information
mymap['B']=20;
mymap['C']=40;
mymap['M']=60;
mymap['Z']=80;
char ch='D'; //C should be returned
nearest=find_nearest<char,int>(mymap,ch);
std::cout << nearest->first << " => " << nearest->second << '\n';
ch='Z'; //Z should be returned
nearest=find_nearest<char,int>(mymap,ch);
std::cout << nearest->first << " => " << nearest->second << '\n';
ch='A'; //B should be returned
nearest=find_nearest<char,int>(mymap,ch);
std::cout << nearest->first << " => " << nearest->second << '\n';
ch='H'; // equidistant to C and M -> C is returned
nearest=find_nearest<char,int>(mymap,ch);
std::cout << nearest->first << " => " << nearest->second << '\n';
return 0;
}
Output:
C => 40
Z => 80
B => 20
C => 40
It is assumed that an operator - is used as a function to evaluate distance. You should implement that operator if class T is your own class, objects of which serve as keys in a map.
You could also change the code to use special class T static member function (say, distance), not operator -, instead:
return (T::distance(itlow->first,item_to_find) < T::distance(item_to_find,itprev->first))?itlow:itprev;
where distance should be smth. like
static distance_type some_type::distance()(const some_type& first, const some_type& second){//...}
and distance_type should support comparison by operator <

You can implement something like this as a tree. A simple approach is to assign each node in the tree a bitstring. Each level of the tree is stored as a bit. All parent information is encoded in the node's bitstring. You can then easily locate arbitrary nodes, and find parents and children. This is how Morton ordering works, for example. It has the extra advantage that you can calculate distances between nodes by simple binary subtraction.
If you have multiple links between data values, then your data structure is a graph rather than a tree. In that case, you need a slightly more sophisticated indexing system. Distributed hash tables do this sort of thing. They typically have a way of calculating the distance between any two nodes in the index space. For example, the Kademlia algorithm (used by Bittorrent) uses XOR distances applied to bitstring ids. This allows Bittorrent clients to lookup ids in a chain, converging on the unknown target location. You can use a similar approach to find the node(s) closest to your target node.

If your keys are strings and your similarity function is Levenshtein distance, then you can use finite-state machines:
Your map is a trie built as a finite-state machine (by unionizing all key/value pairs and determinizing). Then, compose your input query with a simple finite-state transducer that encodes the Levenshtein distance, and compose that with your trie. Then, use the Viterbi algorithm to extract the shortest path.
You can implement all this with only a few function calls using a finite-state toolkit.

in scala this is a technique I use to find the closest Int <= to the key you are looking for
val sMap = SortedMap(1 -> "A", 2 -> "B", 3 -> "C")
sMap.to(4).lastOption.get // Returns 3
sMap.to(-1) // Returns an empty Map

Related

Given: aabcdddeabb => Expected: [(a,2),(b,1),(c,1),(d,3),(e,1),(a,1),(b,1)] in Scala

I'm really interested in how this algorithm can be implemented. If possible, it would be great to see an implementation with and without recursion. I am new to the language so I would be very grateful for help. All I could come up with was this code and it goes no further:
print(counterOccur("aabcdddeabb"))
def counterOccur(string: String) =
string.toCharArray.toList.map(char => {
if (!char.charValue().equals(char.charValue() + 1)) (char, counter)
else (char, counter + 1)
})
I realize that it's not even close to the truth, I just don't even have a clue what else could be used.
First solution with using recursion. I take Char by Char from string and check if last element in the Vector is the same as current. If elements the same I update last element by increasing count(It is first case). If last element does not the same I just add new element to the Vector(second case). When I took all Chars from the string I just return result.
def counterOccur(string: String): Vector[(Char, Int)] = {
#tailrec
def loop(str: List[Char], result: Vector[(Char, Int)]): Vector[(Char, Int)] = {
str match {
case x :: xs if result.lastOption.exists(_._1.equals(x)) =>
val count = result(result.size - 1)._2
loop(xs, result.updated(result.size - 1, (x, count + 1)))
case x :: xs =>
loop(xs, result :+ (x, 1))
case Nil => result
}
}
loop(string.toList, Vector.empty[(Char, Int)])
}
println(counterOccur("aabcdddeabb"))
Second solution that does not use recursion. It works the same, but instead of the recursion it is using foldLeft.
def counterOccur2(string: String): Vector[(Char, Int)] = {
string.foldLeft(Vector.empty[(Char, Int)])((r, v) => {
val lastElementIndex = r.size - 1
if (r.lastOption.exists(lv => lv._1.equals(v))) {
r.updated(lastElementIndex, (v, r(lastElementIndex)._2 + 1))
} else {
r :+ (v, 1)
}
})
}
println(counterOccur2("aabcdddeabb"))
You can use a very simple foldLeft to accumulate. You also don't need toCharArray and toList because strings are implicitly convertible to Seq[Char]:
"aabcdddeabb".foldLeft(collection.mutable.ListBuffer[(Char,Int)]()){ (acc, elm) =>
acc.lastOption match {
case Some((c, i)) if c == elm =>
acc.dropRightInPlace(1).addOne((elm, i+1))
case _ =>
acc.addOne((elm, 1))
}
}
Here is a solution using foldLeft and a custom State case class:
def countConsecutives[A](data: List[A]): List[(A, Int)] = {
final case class State(currentElem: A, currentCount: Int, acc: List[(A, Int)]) {
def result: List[(A, Int)] =
((currentElem -> currentCount) :: acc).reverse
def nextState(newElem: A): State =
if (newElem == currentElem)
this.copy(currentCount = this.currentCount + 1)
else
State(
currentElem = newElem,
currentCount = 1,
acc = (this.currentElem -> this.currentCount) :: this.acc
)
}
object State {
def initial(a: A): State =
State(
currentElem = a,
currentCount = 1,
acc = List.empty
)
}
data match {
case a :: tail =>
tail.foldLeft(State.initial(a)) {
case (state, newElem) =>
state.nextState(newElem)
}.result
case Nil =>
List.empty
}
}
You can see the code running here.
One possibility is to use the unfold method. This method is defined for several collection types, here I'm using it to produce an Iterator (documented here for version 2.13.8):
def spans[A](as: Seq[A]): Iterator[Seq[A]] =
Iterator.unfold(as) {
case head +: tail =>
val (span, rest) = tail.span(_ == head)
Some((head +: span, rest))
case _ =>
None
}
unfold starts from a state and applies a function that returns, either:
None if we want to signal that the collection ended
Some of a pair that contains the next item of the collection we want to produce and the "remaining" state that will be fed to the next iteration.
In this example in particular, we start from a sequence of A called as (which can be a sequence of characters) and at each iteration:
if there's at least one item
we split head and tail
we further split the tail into the longest prefix that contains items equal to the head and the rest
we return the head and the prefix we got above as the next item
we return the rest of the collection as the state for the following iteration
otherwise, we return None as there's nothing more to be done
The result is a fairly flexible function that can be used to group together spans of equal items. You can then define the function you wanted initially in terms of this:
def spanLengths[A](as: Seq[A]): Iterator[(A, Int)] =
spans(as).map(a => a.head -> a.length)
This can be probably made more generic and its performance improved, but I hope this can be an helpful example about another possible approach. While folding a collection is a recursive approach, unfolding is referred to as a corecursive one (Wikipedia article).
You can play around with this code here on Scastie.
For
str = "aabcdddeabb"
you could extract matches of the regular expression
rgx = /(.)\1*/
to obtain the array
["aa", "b", "c", "ddd", "e", "a", "bb"]
and then map each element of the array to the desired string.1
def counterOccur(str: String): List[(Char, Int)] = {
"""(.)\1*""".r
.findAllIn(str)
.map(m => (m.charAt(0), m.length)).toList
}
counterOccur("aabcdddeabb")
#=> res0: List[(Char, Int)] = List((a,2), (b,1), (c,1), (d,3), (e,1), (a,1), (b,2))
The regular expression reads, "match any character and save it to capture group 1 ((.)), then match the content of capture group 1 zero or more times (\1*).
1. Scala code kindly provided by #Thefourthbird.

How to best build a persistent binary tree from a sorted stream

For a side project I wanted a simple way to generate a persistent binary search tree from a sorted stream. After some cursory searching I was only able to find descriptions of techniques that involved storing a sorted array where you can access any element by index. I ended up writing something that works but I figured this is well trodden territory and a canonical example is probably documented somewhere (and probably has a name).
The make shift code I made is included just for clarity. (It's also short)
object TreeFromStream {
sealed trait ImmutableTree[T] {
def height: Int
}
case class ImmutableTreeNode[T](
value: T,
left: ImmutableTree[T],
right: ImmutableTree[T]
) extends ImmutableTree[T] {
lazy val height = left.height + 1
}
case class NilTree[T]() extends ImmutableTree[T] {
def height = 0
}
#tailrec
def treeFromStream[T](
stream: Stream[T],
tree: ImmutableTree[T] = NilTree[T](),
ancestors: List[ImmutableTreeNode[T]] = Nil
): ImmutableTree[T] = {
(stream, ancestors) match {
case (Stream.Empty, _) =>
ancestors.foldLeft(tree) { case(right, root) => root.copy(right=right) }
case (_, ancestor :: nextAncestors) if ancestor.left.height == tree.height =>
treeFromStream(stream, ancestor.copy(right=tree), nextAncestors)
case (next #:: rest, _) =>
treeFromStream(
rest, NilTree(),
ImmutableTreeNode(next, tree, NilTree()) :: ancestors
)
}
}
}
To create a balanced tree, which I will guess you want to do, you will need to visit each node at least once. First, collect all the nodes into a buffer, and then recursively convert the buffer into a tree:
def tfs[T](stream: Stream[T]): ImmutableTree[T] = {
val ss = scala.collection.mutable.ArrayBuffer.empty[T]
def treeFromSubsequence(start: Int, end: Int): ImmutableTree[T] =
if (end == start) NilTree()
else if (end - start == 1) ImmutableTreeNode(ss(start), NilTree(), NilTree())
else {
val mid = (end - start) / 2
ImmutableTreeNode(ss(mid), treeFromSubsequence(start, mid), treeFromSubsequence(mid + 1, end))
}
stream.foreach { x => ss += x }
treeFromSubsequence(0, ss.length)
}
It will visit each value exactly twice, once to collect it and once to put it into the value field of a tree.

Maximum element in a tree

I have the following ADT implementation in Scala.
How to find the maximum element in the tree? Can I introduce some helper function, and if yes, then how?
abstract class MySet {
def max: Int
def contains(tweet: Tweet): Boolean = false
}
class Empty extends MySet {
def max: throw new NoSuchElementExeption("max called on empty tree")
def contains(x: Int): Boolean =
if (x < elem) left.contains(x)
else if (elem < x) right.contains(x)
else true
}
class Node(elem: Int, left: MySet, right: MySet) extends Set {
def max: { ... }
def contains(x: Int): Boolean =
if (x < elem) left.contains(x)
else if (elem < x) right.contains(x)
else true
}
I found a solution in Haskell which feels quite intuitive can I convert it to Scala somehow?
data Tree a = Nil | Node a (Tree a) (Tree a)
maxElement Nil = error "maxElement called on empty tree"
maxElement (Node x Nil Nil) = x
maxElement (Node x Nil r) = max x (maxElement r)
maxElement (Node x l Nil) = max x (maxElement l)
maxElement (Node x l r) = maximum [x, maxElement l, maxElement r]
Update
I am not interested in copying the Haskell code in Scala instead I think Haskell version is more intuitive but because of this keyword and other stuff in Object oriented language. How can I write the equivalent code in object oriented style without pattern matching?
Your tree is heterogeneous, which means that each node can be either a full node with a value, or an empty leaf. Hence you need to distinguish which is which, otherwise you can call max on an empty node. There are many ways:
Classic OOP:
abstract class MySet {
def isEmpty: Boolean
...
}
class Empty extends MySet {
def isEmpty = true
...
}
class Node(...) extends MySet {
def isEmpty = false
...
}
So you do something like this:
var maxElem = elem
if(!left.isEmpty)
maxElem = maxElem.max(left.max)
end
if(!right.isEmpty)
maxElem = maxElem.max(right.max)
end
Since JVM has class information at runtime you can skip the definition of isEmpty:
var maxElem = elem
if(left.isInstanceOf[Node])
maxElem = maxElem.max(left.asInstanceOf[Node].max)
end
if(left.isInstanceOf[Node])
maxElem = maxElem.max(right.asInstanceOf[Node].max)
end
(asInstanceOf is not required if you defined max in MySet, but this pattern covers the case when you didn't)
Well, Scala has a syntactic sugar for the latter, and not surprisingly it's the pattern matching:
var maxElem = elem
left match {
case node: Node =>
maxElem = maxElem.max(node.max)
case _ =>
}
right match {
case node: Node =>
maxElem = maxElem.max(node.max)
case _ =>
}
maxElem
You can take it slightly further and write something like this:
def max = (left, right) match {
case (_: Empty, _: Empty) => elem
case (_: Empty, node: Node) => elem.max(node.max)
case (node: Node, _: Empty) => elem.max(node.max)
case (leftNode: Node, rightNode: Node) =>
elem.max(leftNode.max).max(rightNode.max)
}
If you don't want to use pattern matching, you will need to implement an isEmpty operation or its equivalent, to avoid calling max on an empty set.
The important thing is how the tree is organized. Based on the implementation of contains, it looks like you have an ordered tree (a "binary search tree") where every element in the left part is less than or equal to every element in the right part. If that's the case, your problem is fairly simple. Either the right sub tree is empty and the current element is the max, or the max element of the tree is the max of the right sub tree. That should be a simple recursive implementation with nothing fancy required.
Full disclosure, still learning Scala myself, but here is two versions I came up with (which the pattern match looks like a fair translation of the Haskell code)
sealed trait Tree {
def max: Int
def maxMatch: Int
}
case object EmptyTree extends Tree {
def max = 0
def maxMatch = 0
}
case class Node(data:Int,
left:Tree = EmptyTree,
right:Tree = EmptyTree) extends Tree {
def max:Int = {
data
.max(left.max)
.max(right.max)
}
def maxMatch: Int = {
this match {
case Node(x,EmptyTree,EmptyTree) => x
case Node(x,l:Node,EmptyTree) => x max l.maxMatch
case Node(x,EmptyTree,r:Node) => x max r.maxMatch
case Node(x,l:Node,r:Node) => x max (l.maxMatch max r.maxMatch)
}
}
}
Tests (all passing)
val simpleNode = Node(3)
assert(simpleNode.max == 3)
assert(simpleNode.maxMatch == 3)
val leftLeaf = Node(1, Node(5))
assert(leftLeaf.max == 5)
assert(leftLeaf.maxMatch == 5)
val leftLeafMaxRoot = Node(5,
EmptyTree, Node(2))
assert(leftLeafMaxRoot.max == 5)
assert(leftLeafMaxRoot.maxMatch == 5)
val nestedRightTree = Node(1,
EmptyTree,
Node(2,
EmptyTree, Node(3)))
assert(nestedRightTree.max == 3)
assert(nestedRightTree.maxMatch == 3)
val partialFullTree = Node(1,
Node(2,
Node(4)),
Node(3,
Node(6, Node(7))))
assert(partialFullTree.max == 7)
assert(partialFullTree.maxMatch == 7)

Trial division for primes with immutable collections in Scala

I am trying to learn Scala and functional programming ideology by rewriting basic exercises. Currently I have trouble with naive approach for generating primes "trial division".
The trouble described below is that I could not rewrite well-known algorithm in functional style preserving efficiency, because I have no suitable immutable data structure, like a List but with fast operations not only on head, but also on the very end.
I started with writing java code which for every odd number tests its divisibility by already found primes (limited by square root of value being tested) - and adds it to the end of the list if no divisor was found.
http://ideone.com/QE8U0I
List<Integer> primes = new ArrayList<>();
primes.add(2);
int cur = 3;
while (primes.size() < 100000) {
for (Integer x : primes) {
if (x * x > cur) {
primes.add(cur);
break;
}
if (cur % x == 0) {
break;
}
}
cur += 2;
}
Now I tried to rewrite it in "functional way" - there was no problem with using recursion instead of loops, but I stuck with immutable collections. Core idea is as following:
http://ideone.com/4DQ6mi
def primes(n: Int) = {
#tailrec
def divisibleByAny(x: Int, list: List[Int]): Boolean = {
if (list.isEmpty) false else {
val h = list.head
h * h <= x && (x % h == 0 || divisibleByAny(x, list.tail))
}
}
#tailrec
def morePrimes(from: Int, prev: List[Int]): List[Int] = {
if (prev.size == n) prev else
morePrimes(from + 2, if (divisibleByAny(from, prev)) prev else prev :+ from)
}
morePrimes(3, List(2))
}
But it is slow - if I understand correctly because operation of adding to the end of immutable list requires creation of new copy of the whole stuff.
I searched over documentation to find more suitable data structure and tried to substitute list with immutable Queue, for it is said:
Adding items to the queue always has cost O(1) ... Removing an item is on average O(1).
But it is still even slower:
http://ideone.com/v8BsuQ
def primes(n: Int) = {
#tailrec
def divisibleByAny(x: Int, list: Queue[Int]): Boolean = {
if (list.isEmpty) false else {
val (h, t) = list.dequeue
h * h <= x && (x % h == 0 || divisibleByAny(x, t))
}
}
#tailrec
def morePrimes(from: Int, prev: Queue[Int]): Queue[Int] = {
if (prev.size == n) prev else
morePrimes(from + 2, if (divisibleByAny(from, prev)) prev else prev.enqueue(from))
}
morePrimes(3, Queue(2))
}
What is going wrong or am I missing something?
P.S. I believe there are other algorithms for generating primes which are more suitable for functional style. I think I've seen some paper. But now I'm interested in this one, or more precisely in existence of suitable data structure.
According to http://docs.scala-lang.org/overviews/collections/performance-characteristics.html Vectors have an amortised constant cost for appending, prepending and seeking. Indeed, using vectors instead of lists in your solution is much faster
def primes(n: Int) = {
#tailrec
def divisibleByAny(x: Int, list: Vector[Int]): Boolean = {
if (list.isEmpty) false else {
val (h +: t) = list
h * h <= x && (x % h == 0 || divisibleByAny(x, t))
}
}
#tailrec
def morePrimes(from: Int, prev: Vector[Int]): Vector[Int] = {
if (prev.length == n) prev else
morePrimes(from + 2, if (divisibleByAny(from, prev)) prev else prev :+ from)
}
morePrimes(3, Vector(2))
}
http://ideone.com/x3k4A3
I think you have 2 main options
Use a Vector - which is better than a list for appending. It is a Bitmapped Trie data structure (http://en.wikipedia.org/wiki/Trie). It’s “effectively” O(1) for appending to (i.e. O(1) on average)
Or...possibly the answer you're not looking for
Use a mutable data structure like ListBuffer - immutability it great to try achieve, and should be your go to collections - but sometimes for efficiency reasons, you may use mutable structures . What is key it to make sure it does not “leak out” of your classes. If you look at the List.scala implementation, you’ll see ListBuffer used a lot internally. However, its coverted back to a List just before it leaves the class. If its good enough for the core Scala libraries, its probably ok for you to use under exceptional cases that warrant it.
Except using Vector, also consider using higher-order functions instead of recursion. That's also a completely valid functional style. On my machine the following implementation of divisibleByAny is about 8x faster, than #Pyetras tailrec implementation when running primes(1000000):
def divisibleByAny(x: Int, list: Vector[Int]): Boolean =
list.view.takeWhile(el => el * el <= x).exists(x % _ == 0)

How to design an algorithm to calculate countdown style maths number puzzle

I have always wanted to do this but every time I start thinking about the problem it blows my mind because of its exponential nature.
The problem solver I want to be able to understand and code is for the countdown maths problem:
Given set of number X1 to X5 calculate how they can be combined using mathematical operations to make Y.
You can apply multiplication, division, addition and subtraction.
So how does 1,3,7,6,8,3 make 348?
Answer: (((8 * 7) + 3) -1) *6 = 348.
How to write an algorithm that can solve this problem? Where do you begin when trying to solve a problem like this? What important considerations do you have to think about when designing such an algorithm?
Very quick and dirty solution in Java:
public class JavaApplication1
{
public static void main(String[] args)
{
List<Integer> list = Arrays.asList(1, 3, 7, 6, 8, 3);
for (Integer integer : list) {
List<Integer> runList = new ArrayList<>(list);
runList.remove(integer);
Result result = getOperations(runList, integer, 348);
if (result.success) {
System.out.println(integer + result.output);
return;
}
}
}
public static class Result
{
public String output;
public boolean success;
}
public static Result getOperations(List<Integer> numbers, int midNumber, int target)
{
Result midResult = new Result();
if (midNumber == target) {
midResult.success = true;
midResult.output = "";
return midResult;
}
for (Integer number : numbers) {
List<Integer> newList = new ArrayList<Integer>(numbers);
newList.remove(number);
if (newList.isEmpty()) {
if (midNumber - number == target) {
midResult.success = true;
midResult.output = "-" + number;
return midResult;
}
if (midNumber + number == target) {
midResult.success = true;
midResult.output = "+" + number;
return midResult;
}
if (midNumber * number == target) {
midResult.success = true;
midResult.output = "*" + number;
return midResult;
}
if (midNumber / number == target) {
midResult.success = true;
midResult.output = "/" + number;
return midResult;
}
midResult.success = false;
midResult.output = "f" + number;
return midResult;
} else {
midResult = getOperations(newList, midNumber - number, target);
if (midResult.success) {
midResult.output = "-" + number + midResult.output;
return midResult;
}
midResult = getOperations(newList, midNumber + number, target);
if (midResult.success) {
midResult.output = "+" + number + midResult.output;
return midResult;
}
midResult = getOperations(newList, midNumber * number, target);
if (midResult.success) {
midResult.output = "*" + number + midResult.output;
return midResult;
}
midResult = getOperations(newList, midNumber / number, target);
if (midResult.success) {
midResult.output = "/" + number + midResult.output;
return midResult
}
}
}
return midResult;
}
}
UPDATE
It's basically just simple brute force algorithm with exponential complexity.
However you can gain some improvemens by leveraging some heuristic function which will help you to order sequence of numbers or(and) operations you will process in each level of getOperatiosn() function recursion.
Example of such heuristic function is for example difference between mid result and total target result.
This way however only best-case and average-case complexities get improved. Worst case complexity remains untouched.
Worst case complexity can be improved by some kind of branch cutting. I'm not sure if it's possible in this case.
Sure it's exponential but it's tiny so a good (enough) naive implementation would be a good start. I suggest you drop the usual infix notation with bracketing, and use postfix, it's easier to program. You can always prettify the outputs as a separate stage.
Start by listing and evaluating all the (valid) sequences of numbers and operators. For example (in postfix):
1 3 7 6 8 3 + + + + + -> 28
1 3 7 6 8 3 + + + + - -> 26
My Java is laughable, I don't come here to be laughed at so I'll leave coding this up to you.
To all the smart people reading this: yes, I know that for even a small problem like this there are smarter approaches which are likely to be faster, I'm just pointing OP towards an initial working solution. Someone else can write the answer with the smarter solution(s).
So, to answer your questions:
I begin with an algorithm that I think will lead me quickly to a working solution. In this case the obvious (to me) choice is exhaustive enumeration and testing of all possible calculations.
If the obvious algorithm looks unappealing for performance reasons I'll start thinking more deeply about it, recalling other algorithms that I know about which are likely to deliver better performance. I may start coding one of those first instead.
If I stick with the exhaustive algorithm and find that the run-time is, in practice, too long, then I might go back to the previous step and code again. But it has to be worth my while, there's a cost/benefit assessment to be made -- as long as my code can outperform Rachel Riley I'd be satisfied.
Important considerations include my time vs computer time, mine costs a helluva lot more.
A working solution in c++11 below.
The basic idea is to use a stack-based evaluation (see RPN) and convert the viable solutions to infix notation for display purposes only.
If we have N input digits, we'll use (N-1) operators, as each operator is binary.
First we create valid permutations of operands and operators (the selector_ array). A valid permutation is one that can be evaluated without stack underflow and which ends with exactly one value (the result) on the stack. Thus 1 1 + is valid, but 1 + 1 is not.
We test each such operand-operator permutation with every permutation of operands (the values_ array) and every combination of operators (the ops_ array). Matching results are pretty-printed.
Arguments are taken from command line as [-s] <target> <digit>[ <digit>...]. The -s switch prevents exhaustive search, only the first matching result is printed.
(use ./mathpuzzle 348 1 3 7 6 8 3 to get the answer for the original question)
This solution doesn't allow concatenating the input digits to form numbers. That could be added as an additional outer loop.
The working code can be downloaded from here. (Note: I updated that code with support for concatenating input digits to form a solution)
See code comments for additional explanation.
#include <iostream>
#include <vector>
#include <algorithm>
#include <stack>
#include <iterator>
#include <string>
namespace {
enum class Op {
Add,
Sub,
Mul,
Div,
};
const std::size_t NumOps = static_cast<std::size_t>(Op::Div) + 1;
const Op FirstOp = Op::Add;
using Number = int;
class Evaluator {
std::vector<Number> values_; // stores our digits/number we can use
std::vector<Op> ops_; // stores the operators
std::vector<char> selector_; // used to select digit (0) or operator (1) when evaluating. should be std::vector<bool>, but that's broken
template <typename T>
using Stack = std::stack<T, std::vector<T>>;
// checks if a given number/operator order can be evaluated or not
bool isSelectorValid() const {
int numValues = 0;
for (auto s : selector_) {
if (s) {
if (--numValues <= 0) {
return false;
}
}
else {
++numValues;
}
}
return (numValues == 1);
}
// evaluates the current values_ and ops_ based on selector_
Number eval(Stack<Number> &stack) const {
auto vi = values_.cbegin();
auto oi = ops_.cbegin();
for (auto s : selector_) {
if (!s) {
stack.push(*(vi++));
continue;
}
Number top = stack.top();
stack.pop();
switch (*(oi++)) {
case Op::Add:
stack.top() += top;
break;
case Op::Sub:
stack.top() -= top;
break;
case Op::Mul:
stack.top() *= top;
break;
case Op::Div:
if (top == 0) {
return std::numeric_limits<Number>::max();
}
Number res = stack.top() / top;
if (res * top != stack.top()) {
return std::numeric_limits<Number>::max();
}
stack.top() = res;
break;
}
}
Number res = stack.top();
stack.pop();
return res;
}
bool nextValuesPermutation() {
return std::next_permutation(values_.begin(), values_.end());
}
bool nextOps() {
for (auto i = ops_.rbegin(), end = ops_.rend(); i != end; ++i) {
std::size_t next = static_cast<std::size_t>(*i) + 1;
if (next < NumOps) {
*i = static_cast<Op>(next);
return true;
}
*i = FirstOp;
}
return false;
}
bool nextSelectorPermutation() {
// the start permutation is always valid
do {
if (!std::next_permutation(selector_.begin(), selector_.end())) {
return false;
}
} while (!isSelectorValid());
return true;
}
static std::string buildExpr(const std::string& left, char op, const std::string &right) {
return std::string("(") + left + ' ' + op + ' ' + right + ')';
}
std::string toString() const {
Stack<std::string> stack;
auto vi = values_.cbegin();
auto oi = ops_.cbegin();
for (auto s : selector_) {
if (!s) {
stack.push(std::to_string(*(vi++)));
continue;
}
std::string top = stack.top();
stack.pop();
switch (*(oi++)) {
case Op::Add:
stack.top() = buildExpr(stack.top(), '+', top);
break;
case Op::Sub:
stack.top() = buildExpr(stack.top(), '-', top);
break;
case Op::Mul:
stack.top() = buildExpr(stack.top(), '*', top);
break;
case Op::Div:
stack.top() = buildExpr(stack.top(), '/', top);
break;
}
}
return stack.top();
}
public:
Evaluator(const std::vector<Number>& values) :
values_(values),
ops_(values.size() - 1, FirstOp),
selector_(2 * values.size() - 1, 0) {
std::fill(selector_.begin() + values_.size(), selector_.end(), 1);
std::sort(values_.begin(), values_.end());
}
// check for solutions
// 1) we create valid permutations of our selector_ array (eg: "1 1 + 1 +",
// "1 1 1 + +", but skip "1 + 1 1 +" as that cannot be evaluated
// 2) for each evaluation order, we permutate our values
// 3) for each value permutation we check with each combination of
// operators
//
// In the first version I used a local stack in eval() (see toString()) but
// it turned out to be a performance bottleneck, so now I use a cached
// stack. Reusing the stack gives an order of magnitude speed-up (from
// 4.3sec to 0.7sec) due to avoiding repeated allocations. Using
// std::vector as a backing store also gives a slight performance boost
// over the default std::deque.
std::size_t check(Number target, bool singleResult = false) {
Stack<Number> stack;
std::size_t res = 0;
do {
do {
do {
Number value = eval(stack);
if (value == target) {
++res;
std::cout << target << " = " << toString() << "\n";
if (singleResult) {
return res;
}
}
} while (nextOps());
} while (nextValuesPermutation());
} while (nextSelectorPermutation());
return res;
}
};
} // namespace
int main(int argc, const char **argv) {
int i = 1;
bool singleResult = false;
if (argc > 1 && std::string("-s") == argv[1]) {
singleResult = true;
++i;
}
if (argc < i + 2) {
std::cerr << argv[0] << " [-s] <target> <digit>[ <digit>]...\n";
std::exit(1);
}
Number target = std::stoi(argv[i]);
std::vector<Number> values;
while (++i < argc) {
values.push_back(std::stoi(argv[i]));
}
Evaluator evaluator{values};
std::size_t res = evaluator.check(target, singleResult);
if (!singleResult) {
std::cout << "Number of solutions: " << res << "\n";
}
return 0;
}
Input is obviously a set of digits and operators: D={1,3,3,6,7,8,3} and Op={+,-,*,/}. The most straight forward algorithm would be a brute force solver, which enumerates all possible combinations of these sets. Where the elements of set Op can be used as often as wanted, but elements from set D are used exactly once. Pseudo code:
D={1,3,3,6,7,8,3}
Op={+,-,*,/}
Solution=348
for each permutation D_ of D:
for each binary tree T with D_ as its leafs:
for each sequence of operators Op_ from Op with length |D_|-1:
label each inner tree node with operators from Op_
result = compute T using infix traversal
if result==Solution
return T
return nil
Other than that: read jedrus07's and HPM's answers.
By far the easiest approach is to intelligently brute force it. There is only a finite amount of expressions you can build out of 6 numbers and 4 operators, simply go through all of them.
How many? Since you don't have to use all numbers and may use the same operator multiple times, This problem is equivalent to "how many labeled strictly binary trees (aka full binary trees) can you make with at most 6 leaves, and four possible labels for each non-leaf node?".
The amount of full binary trees with n leaves is equal to catalan(n-1). You can see this as follows:
Every full binary tree with n leaves has n-1 internal nodes and corresponds to a non-full binary tree with n-1 nodes in a unique way (just delete all the leaves from the full one to get it). There happen to be catalan(n) possible binary trees with n nodes, so we can say that a strictly binary tree with n leaves has catalan(n-1) possible different structures.
There are 4 possible operators for each non-leaf node: 4^(n-1) possibilities
The leaves can be numbered in n! * (6 choose (n-1)) different ways. (Divide this by k! for each number that occurs k times, or just make sure all numbers are different)
So for 6 different numbers and 4 possible operators you get Sum(n=1...6) [ Catalan(n-1) * 6!/(6-n)! * 4^(n-1) ] possible expressions for a total of 33,665,406. Not a lot.
How do you enumerate these trees?
Given a collection of all trees with n-1 or less nodes, you can create all trees with n nodes by systematically pairing all of the n-1 trees with the empty tree, all n-2 trees with the 1 node tree, all n-3 trees with all 2 node tree etc. and using them as the left and right sub trees of a newly formed tree.
So starting with an empty set you first generate the tree that has just a root node, then from a new root you can use that either as a left or right sub tree which yields the two trees that look like this: / and . And so on.
You can turn them into a set of expressions on the fly (just loop over the operators and numbers) and evaluate them as you go until one yields the target number.
I've written my own countdown solver, in Python.
Here's the code; it is also available on GitHub:
#!/usr/bin/env python3
import sys
from itertools import combinations, product, zip_longest
from functools import lru_cache
assert sys.version_info >= (3, 6)
class Solutions:
def __init__(self, numbers):
self.all_numbers = numbers
self.size = len(numbers)
self.all_groups = self.unique_groups()
def unique_groups(self):
all_groups = {}
all_numbers, size = self.all_numbers, self.size
for m in range(1, size+1):
for numbers in combinations(all_numbers, m):
if numbers in all_groups:
continue
all_groups[numbers] = Group(numbers, all_groups)
return all_groups
def walk(self):
for group in self.all_groups.values():
yield from group.calculations
class Group:
def __init__(self, numbers, all_groups):
self.numbers = numbers
self.size = len(numbers)
self.partitions = list(self.partition_into_unique_pairs(all_groups))
self.calculations = list(self.perform_calculations())
def __repr__(self):
return str(self.numbers)
def partition_into_unique_pairs(self, all_groups):
# The pairs are unordered: a pair (a, b) is equivalent to (b, a).
# Therefore, for pairs of equal length only half of all combinations
# need to be generated to obtain all pairs; this is set by the limit.
if self.size == 1:
return
numbers, size = self.numbers, self.size
limits = (self.halfbinom(size, size//2), )
unique_numbers = set()
for m, limit in zip_longest(range((size+1)//2, size), limits):
for numbers1, numbers2 in self.paired_combinations(numbers, m, limit):
if numbers1 in unique_numbers:
continue
unique_numbers.add(numbers1)
group1, group2 = all_groups[numbers1], all_groups[numbers2]
yield (group1, group2)
def perform_calculations(self):
if self.size == 1:
yield Calculation.singleton(self.numbers[0])
return
for group1, group2 in self.partitions:
for calc1, calc2 in product(group1.calculations, group2.calculations):
yield from Calculation.generate(calc1, calc2)
#classmethod
def paired_combinations(cls, numbers, m, limit):
for cnt, numbers1 in enumerate(combinations(numbers, m), 1):
numbers2 = tuple(cls.filtering(numbers, numbers1))
yield (numbers1, numbers2)
if cnt == limit:
return
#staticmethod
def filtering(iterable, elements):
# filter elements out of an iterable, return the remaining elements
elems = iter(elements)
k = next(elems, None)
for n in iterable:
if n == k:
k = next(elems, None)
else:
yield n
#staticmethod
#lru_cache()
def halfbinom(n, k):
if n % 2 == 1:
return None
prod = 1
for m, l in zip(reversed(range(n+1-k, n+1)), range(1, k+1)):
prod = (prod*m)//l
return prod//2
class Calculation:
def __init__(self, expression, result, is_singleton=False):
self.expr = expression
self.result = result
self.is_singleton = is_singleton
def __repr__(self):
return self.expr
#classmethod
def singleton(cls, n):
return cls(f"{n}", n, is_singleton=True)
#classmethod
def generate(cls, calca, calcb):
if calca.result < calcb.result:
calca, calcb = calcb, calca
for result, op in cls.operations(calca.result, calcb.result):
expr1 = f"{calca.expr}" if calca.is_singleton else f"({calca.expr})"
expr2 = f"{calcb.expr}" if calcb.is_singleton else f"({calcb.expr})"
yield cls(f"{expr1} {op} {expr2}", result)
#staticmethod
def operations(x, y):
yield (x + y, '+')
if x > y: # exclude non-positive results
yield (x - y, '-')
if y > 1 and x > 1: # exclude trivial results
yield (x * y, 'x')
if y > 1 and x % y == 0: # exclude trivial and non-integer results
yield (x // y, '/')
def countdown_solver():
# input: target and numbers. If you want to play with more or less than
# 6 numbers, use the second version of 'unsorted_numbers'.
try:
target = int(sys.argv[1])
unsorted_numbers = (int(sys.argv[n+2]) for n in range(6)) # for 6 numbers
# unsorted_numbers = (int(n) for n in sys.argv[2:]) # for any numbers
numbers = tuple(sorted(unsorted_numbers, reverse=True))
except (IndexError, ValueError):
print("You must provide a target and numbers!")
return
solutions = Solutions(numbers)
smallest_difference = target
bestresults = []
for calculation in solutions.walk():
diff = abs(calculation.result - target)
if diff <= smallest_difference:
if diff < smallest_difference:
bestresults = [calculation]
smallest_difference = diff
else:
bestresults.append(calculation)
output(target, smallest_difference, bestresults)
def output(target, diff, results):
print(f"\nThe closest results differ from {target} by {diff}. They are:\n")
for calculation in results:
print(f"{calculation.result} = {calculation.expr}")
if __name__ == "__main__":
countdown_solver()
The algorithm works as follows:
The numbers are put into a tuple of length 6 in descending order. Then, all unique subgroups of lengths 1 to 6 are created, the smallest groups first.
Example: (75, 50, 5, 9, 1, 1) -> {(75), (50), (9), (5), (1), (75, 50), (75, 9), (75, 5), ..., (75, 50, 9, 5, 1, 1)}.
Next, the groups are organised into a hierarchical tree: every group is partitioned into all unique unordered pairs of its non-empty subgroups.
Example: (9, 5, 1, 1) -> [(9, 5, 1) + (1), (9, 1, 1) + (5), (5, 1, 1) + (9), (9, 5) + (1, 1), (9, 1) + (5, 1)].
Within each group of numbers, the calculations are performed and the results are stored. For groups of length 1, the result is simply the number itself. For larger groups, the calculations are carried out on every pair of subgroups: in each pair, all results of the first subgroup are combined with all results of the second subgroup using +, -, x and /, and the valid outcomes are stored.
Example: (75, 5) consists of the pair ((75), (5)). The result of (75) is 75; the result of (5) is 5; the results of (75, 5) are [75+5=80, 75-5=70, 75*5=375, 75/5=15].
In this manner, all results are generated, from the smallest groups to the largest. Finally, the algorithm iterates through all results and selects the ones that are the closest match to the target number.
For a group of m numbers, the maximum number of arithmetic computations is
comps[m] = 4*sum(binom(m, k)*comps[k]*comps[m-k]//(1 + (2*k)//m) for k in range(1, m//2+1))
For all groups of length 1 to 6, the maximum total number of computations is then
total = sum(binom(n, m)*comps[m] for m in range(1, n+1))
which is 1144386. In practice, it will be much less, because the algorithm reuses the results of duplicate groups, ignores trivial operations (adding 0, multiplying by 1, etc), and because the rules of the game dictate that intermediate results must be positive integers (which limits the use of the division operator).
I think, you need to strictly define the problem first. What you are allowed to do and what you are not. You can start by making it simple and only allowing multiplication, division, substraction and addition.
Now you know your problem space- set of inputs, set of available operations and desired input. If you have only 4 operations and x inputs, the number of combinations is less than:
The number of order in which you can carry out operations (x!) times the possible choices of operations on every step: 4^x. As you can see for 6 numbers it gives reasonable 2949120 operations. This means that this may be your limit for brute force algorithm.
Once you have brute force and you know it works, you can start improving your algorithm with some sort of A* algorithm which would require you to define heuristic functions.
In my opinion the best way to think about it is as the search problem. The main difficulty will be finding good heuristics, or ways to reduce your problem space (if you have numbers that do not add up to the answer, you will need at least one multiplication etc.). Start small, build on that and ask follow up questions once you have some code.
I wrote a terminal application to do this:
https://github.com/pg328/CountdownNumbersGame/tree/main
Inside, I've included an illustration of the calculation of the size of the solution space (it's n*((n-1)!^2)*(2^n-1), so: n=6 -> 2,764,800. I know, gross), and more importantly why that is. My implementation is there if you care to check it out, but in case you don't I'll explain here.
Essentially, at worst it is brute force because as far as I know it's impossible to determine whether any specific branch will result in a valid answer without explicitly checking. Having said that, the average case is some fraction of that; it's {that number} divided by the number of valid solutions (I tend to see around 1000 on my program, where 10 or so are unique and the rest are permutations fo those 10). If I handwaved a number, I'd say roughly 2,765 branches to check which takes like no time. (Yes, even in Python.)
TL;DR: Even though the solution space is huge and it takes a couple million operations to find all solutions, only one answer is needed. Best route is brute force til you find one and spit it out.
I wrote a slightly simpler version:
for every combination of 2 (distinct) elements from the list and combine them using +,-,*,/ (note that since a>b then only a-b is needed and only a/b if a%b=0)
if combination is target then record solution
recursively call on the reduced lists
import sys
def driver():
try:
target = int(sys.argv[1])
nums = list((int(sys.argv[i+2]) for i in range(6)))
except (IndexError, ValueError):
print("Provide a list of 7 numbers")
return
solutions = list()
solve(target, nums, list(), solutions)
unique = set()
final = list()
for s in solutions:
a = '-'.join(sorted(s))
if not a in unique:
unique.add(a)
final.append(s)
for s in final: #print them out
print(s)
def solve(target, nums, path, solutions):
if len(nums) == 1:
return
distinct = sorted(list(set(nums)), reverse = True)
rem1 = list(distinct)
for n1 in distinct: #reduce list by combining a pair
rem1.remove(n1)
for n2 in rem1:
rem2 = list(nums) # in case of duplicates we need to start with full list and take out the n1,n2 pair of elements
rem2.remove(n1)
rem2.remove(n2)
combine(target, solutions, path, rem2, n1, n2, '+')
combine(target, solutions, path, rem2, n1, n2, '-')
if n2 > 1:
combine(target, solutions, path, rem2, n1, n2, '*')
if not n1 % n2:
combine(target, solutions, path, rem2, n1, n2, '//')
def combine(target, solutions, path, rem2, n1, n2, symb):
lst = list(rem2)
ans = eval("{0}{2}{1}".format(n1, n2, symb))
newpath = path + ["{0}{3}{1}={2}".format(n1, n2, ans, symb[0])]
if ans == target:
solutions += [newpath]
else:
lst.append(ans)
solve(target, lst, newpath, solutions)
if __name__ == "__main__":
driver()

Resources