Algorithm to find optimal groups in 2D array - algorithm

I have a deck of 24 cards - 8 red, 8 blue and 8 yellow cards.
red |1|2|3|4|5|6|7|8|
yellow |1|2|3|4|5|6|7|8|
blue |1|2|3|4|5|6|7|8|
I can take 3 of cards (same numbers, straight, straigh flush), whereas each of the type is scored differently.
My question is, how to calculate maximal possible score (find optimal groups) for a game in progress, where some cards are already missing.
for example:
red |1|2|3|4|5|6|7|8|
yellow |1|2|3| |5| |7|8|
blue |1|2| |4|5|6| |8|
The score for a three-of-a-kind is:
1-1-1 20
2-2-2 30
3-3-3 40
4-4-4 50
5-5-5 60
6-6-6 70
7-7-7 80
8-8-8 90
The score for a straight is:
1-2-3 10
2-3-4 20
3-4-5 30
4-5-6 40
5-6-7 50
6-7-8 60
The score for a straight flush is:
1-2-3 50
2-3-4 60
3-4-5 70
4-5-6 80
5-6-7 90
6-7-8 100

A solution which recursively tries every combination would go like this:
Start looking at combinations that have a red 8 as the highest card: three-of-a-kind r8-y8-b8, straight flush r6-r7-r8, and every possible straight *6-*7-r8. For each of these, remove the cards from the set, and recurse to check combinations with the yellow 8, then blue 8, then red 7, yellow 7, blue 7, red 6 ... until you've checked everything except the 2's and 1's; then add three-of-a-kind 2-2-2 and 1-1-1 if available. At each step, check which recursion returns the maximum score, and return this maximum.
Let's look at what happens in each of these steps. Say we're looking at combinations with red 8; we have available cards like:
red ...|6|7|8|
yellow ...|6| |8|
blue ...| |7|8|
First, use three-of-a-kind r8-y8-b8, if possible. Create a copy of the available cards, remove the 8's, and recurse straight to the 7's:
score = 90 + max_score(cards_copy, next = red 7)
(Trying the three-of-a-kind should only be done when the current card is red, to avoid duplicate solutions.)
Then, use straight flush r6-r7-r8, if possible. Create a copy of the available cards, remove r6, r7 and r8, and recurse to yellow 8:
score = 100 + max_score(cards_copy, next = yellow 8)
Then, use every possible non-flush straight containing red 8; in the example, those are r6-b7-r8, y6-r7-r8 and y6-b7-r8 (there could be up to nine). For each of these, create a copy of the available cards, remove the three cards and recurse to yellow 8:
score = 60 + max_score(cards_copy, next = yellow 8)
Then, finally, recurse without using red 8: create a copy of the available cards, remove red 8 and recurse to yellow 8:
score = max_score(cards_copy, next = yellow 8)
You then calculate which of these options has the greatest score (with the score returned by its recursion added), and return that maximum score.
A quick test in JavaScript shows that for a full set of 24 cards, the algorithm goes through 30 million recursions to find the maximum score 560, and becomes quite slow. However, as soon as 3 higher-value cards have been removed, the number of recursions falls below one million and it takes around 1 second, and with 6 higher-value cards removed, it falls below 20,000 and returns almost instantly.
For almost-complete sets, you could pre-compute the maximum scores, and only calculate the score once a certain number of cards have been removed. A lot of sets will be duplicates anyway; removing r6-r7-r8 will result in the same maximum score as removing y6-y7-y8; removing r6-y7-b8 is a duplicate of removing b6-y7-r8... So first you change the input to a canonical version, and then you look up the pre-computed score. E.g. using pre-computed scores for all sets with 3 or 6 cards removed would require storing 45,340 scores.
As a code example, here's the JavaScript code I tested the algorithm with:
function clone(array) { // copy 2-dimensional array
var copy = [];
array.forEach(function(item) {copy.push(item.slice())});
return copy;
}
function max_score(cards, suit, rank) {
suit = suit || 0; rank = rank || 7; // start at red 8
var max = 0;
if (rank < 2) { // try 3-of-a-kind for rank 1 and 2
if (cards[0][0] && cards[1][0] && cards[2][0]) max += 20;
if (cards[0][1] && cards[1][1] && cards[2][1]) max += 30;
return max;
}
var next_rank = suit == 2 ? rank - 1: rank;
var next_suit = (suit + 1) % 3;
max = max_score(clone(cards), next_suit, next_rank); // try skipping this card
if (! cards[suit][rank]) return max;
if (suit == 0 && cards[1][rank] && cards[2][rank]) { // try 3-of-a-kind
var score = rank * 10 + 20 + max_score(clone(cards), 0, rank - 1);
if (score > max) max = score;
}
for (var i = 0; i < 3; i++) { // try all possible straights
if (! cards[i][rank - 2]) continue;
for (var j = 0; j < 3; j++) {
if (! cards[j][rank - 1]) continue;
var copy = clone(cards);
copy[j][rank - 1] = 0; copy[i][rank - 2] = 0;
var score = rank * 10 - 10 + max_score(copy, next_suit, next_rank);
if (i == suit && j == suit) score += 40; // straight is straight flush
if (score > max) max = score;
}
}
return max;
}
document.write(max_score([[1,1,1,1,1,0,1,1], [1,1,1,1,1,1,1,0], [1,1,1,0,1,1,1,1]]));
An obvious way to speed up the algorithm is to use a 24-bit pattern instead of a 3x8 bit array to represent the cards; that way the array cloning is no longer necessary, and most of the code is turned into bit manipulation. In JavaScript, it's about 8 times faster:
function max_score(cards, suit, rank) {
suit = suit || 0; rank = rank || 7; // start at red 8
var max = 0;
if (rank < 2) { // try 3-of-a-kind for rank 1 and 2
if ((cards & 65793) == 65793) max += 20; // 65793 = rank 1 of all suits
if ((cards & 131586) == 131586) max += 30; // 131586 = rank 2 of all suits
return max;
}
var next_rank = suit == 2 ? rank - 1: rank;
var next_suit = (suit + 1) % 3;
var this_card = 1 << rank << suit * 8;
max = max_score(cards, next_suit, next_rank); // try skipping this card
if (! (cards & this_card)) return max;
if (suit == 0 && cards & this_card << 8 && cards & this_card << 16) { // try 3oaK
var score = rank * 10 + 20 + max_score(cards, 0, rank - 1);
if (score > max) max = score;
}
for (var i = 0; i < 3; i++) { // try all possible straights
var mid_card = 1 << rank - 1 << i * 8;
if (! (cards & mid_card)) continue;
for (var j = 0; j < 3; j++) {
var low_card = 1 << rank - 2 << j * 8;
if (! (cards & low_card)) continue;
var cards_copy = cards - mid_card - low_card;
var score = rank * 10 - 10 + max_score(cards_copy, next_suit, next_rank);
if (i == suit && j == suit) score += 40; // straight is straight flush
if (score > max) max = score;
}
}
return max;
}
document.write(max_score(parseInt("111101110111111111011111", 2)));
// B Y R
// 876543218765432187654321
The speed for almost-complete sets can be further improved by using the observation that if a straight flush for all three suits can be be made for the current rank, then this is always the best option. This reduces the number of recursions drastically, because nine cards can be skipped at once. This check should be added immediately after trying 3-of-a-kind for rank 1 and 2:
if (suit == 0) { // try straight flush for all suits
var flush3 = 460551 << rank - 2; // 460551 = rank 1, 2 and 3 of all suits
if ((cards & flush3) == flush3) {
max = rank * 30 + 90;
if (rank > 2) max += max_score(cards - flush3, 0, rank - 3);
return max;
}
}

Related

Algorithm for equiprobable random square binary matrices with two non-adjacent non-zeros in each row and column

It would be great if someone could point me towards an algorithm that would allow me to :
create a random square matrix, with entries 0 and 1, such that
every row and every column contain exactly two non-zero entries,
two non-zero entries cannot be adjacent,
all possible matrices are equiprobable.
Right now I manage to achieve points 1 and 2 doing the following : such a matrix can be transformed, using suitable permutations of rows and columns, into a diagonal block matrix with blocks of the form
1 1 0 0 ... 0
0 1 1 0 ... 0
0 0 1 1 ... 0
.............
1 0 0 0 ... 1
So I start from such a matrix using a partition of [0, ..., n-1] and scramble it by permuting rows and columns randomly. Unfortunately, I can't find a way to integrate the adjacency condition, and I am quite sure that my algorithm won't treat all the matrices equally.
Update
I have managed to achieve point 3. The answer was actually straight under my nose : the block matrix I am creating contains all the information needed to take into account the adjacency condition. First some properties and definitions:
a suitable matrix defines permutations of [1, ..., n] that can be build like so: select a 1 in row 1. The column containing this entry contains exactly one other entry equal to 1 on a row a different from 1. Again, row a contains another entry 1 in a column which contains a second entry 1 on a row b, and so on. This starts a permutation 1 -> a -> b ....
For instance, with the following matrix, starting with the marked entry
v
1 0 1 0 0 0 | 1
0 1 0 0 0 1 | 2
1 0 0 1 0 0 | 3
0 0 1 0 1 0 | 4
0 0 0 1 0 1 | 5
0 1 0 0 1 0 | 6
------------+--
1 2 3 4 5 6 |
we get permutation 1 -> 3 -> 5 -> 2 -> 6 -> 4 -> 1.
the cycles of such a permutation lead to the block matrix I mentioned earlier. I also mentioned scrambling the block matrix using arbitrary permutations on the rows and columns to rebuild a matrix compatible with the requirements.
But I was using any permutation, which led to some adjacent non-zero entries. To avoid that, I have to choose permutations that separate rows (and columns) that are adjacent in the block matrix. Actually, to be more precise, if two rows belong to a same block and are cyclically consecutive (the first and last rows of a block are considered consecutive too), then the permutation I want to apply has to move these rows into non-consecutive rows of the final matrix (I will call two rows incompatible in that case).
So the question becomes : How to build all such permutations ?
The simplest idea is to build a permutation progressively by randomly adding rows that are compatible with the previous one. As an example, consider the case n = 6 using partition 6 = 3 + 3 and the corresponding block matrix
1 1 0 0 0 0 | 1
0 1 1 0 0 0 | 2
1 0 1 0 0 0 | 3
0 0 0 1 1 0 | 4
0 0 0 0 1 1 | 5
0 0 0 1 0 1 | 6
------------+--
1 2 3 4 5 6 |
Here rows 1, 2 and 3 are mutually incompatible, as are 4, 5 and 6. Choose a random row, say 3.
We will write a permutation as an array: [2, 5, 6, 4, 3, 1] meaning 1 -> 2, 2 -> 5, 3 -> 6, ... This means that row 2 of the block matrix will become the first row of the final matrix, row 5 will become the second row, and so on.
Now let's build a suitable permutation by choosing randomly a row, say 3:
p = [3, ...]
The next row will then be chosen randomly among the remaining rows that are compatible with 3 : 4, 5and 6. Say we choose 4:
p = [3, 4, ...]
Next choice has to be made among 1 and 2, for instance 1:
p = [3, 4, 1, ...]
And so on: p = [3, 4, 1, 5, 2, 6].
Applying this permutation to the block matrix, we get:
1 0 1 0 0 0 | 3
0 0 0 1 1 0 | 4
1 1 0 0 0 0 | 1
0 0 0 0 1 1 | 5
0 1 1 0 0 0 | 2
0 0 0 1 0 1 | 6
------------+--
1 2 3 4 5 6 |
Doing so, we manage to vertically isolate all non-zero entries. Same has to be done with the columns, for instance by using permutation p' = [6, 3, 5, 1, 4, 2] to finally get
0 1 0 1 0 0 | 3
0 0 1 0 1 0 | 4
0 0 0 1 0 1 | 1
1 0 1 0 0 0 | 5
0 1 0 0 0 1 | 2
1 0 0 0 1 0 | 6
------------+--
6 3 5 1 4 2 |
So this seems to work quite efficiently, but building these permutations needs to be done with caution, because one can easily be stuck: for instance, with n=6 and partition 6 = 2 + 2 + 2, following the construction rules set up earlier can lead to p = [1, 3, 2, 4, ...]. Unfortunately, 5 and 6 are incompatible, so choosing one or the other makes the last choice impossible. I think I've found all situations that lead to a dead end. I will denote by r the set of remaining choices:
p = [..., x, ?], r = {y} with x and y incompatible
p = [..., x, ?, ?], r = {y, z} with y and z being both incompatible with x (no choice can be made)
p = [..., ?, ?], r = {x, y} with x and y incompatible (any choice would lead to situation 1)
p = [..., ?, ?, ?], r = {x, y, z} with x, y and z being cyclically consecutive (choosing x or z would lead to situation 2, choosing y to situation 3)
p = [..., w, ?, ?, ?], r = {x, y, z} with xwy being a 3-cycle (neither x nor y can be chosen, choosing z would lead to situation 3)
p = [..., ?, ?, ?, ?], r = {w, x, y, z} with wxyz being a 4-cycle (any choice would lead to situation 4)
p = [..., ?, ?, ?, ?], r = {w, x, y, z} with xyz being a 3-cycle (choosing w would lead to situation 4, choosing any other would lead to situation 4)
Now it seems that the following algorithm gives all suitable permutations:
As long as there are strictly more than 5 numbers to choose, choose randomly among the compatible ones.
If there are 5 numbers left to choose: if the remaining numbers contain a 3-cycle or a 4-cycle, break that cycle (i.e. choose a number belonging to that cycle).
If there are 4 numbers left to choose: if the remaining numbers contain three cyclically consecutive numbers, choose one of them.
If there are 3 numbers left to choose: if the remaining numbers contain two cyclically consecutive numbers, choose one of them.
I am quite sure that this allows me to generate all suitable permutations and, hence, all suitable matrices.
Unfortunately, every matrix will be obtained several times, depending on the partition that was chosen.
Intro
Here is some prototype-approach, trying to solve the more general task of
uniform combinatorial sampling, which for our approach here means: we can use this approach for everything which we can formulate as SAT-problem.
It's not exploiting your problem directly and takes a heavy detour. This detour to the SAT-problem can help in regards to theory (more powerful general theoretical results) and efficiency (SAT-solvers).
That being said, it's not an approach if you want to sample within seconds or less (in my experiments), at least while being concerned about uniformity.
Theory
The approach, based on results from complexity-theory, follows this work:
GOMES, Carla P.; SABHARWAL, Ashish; SELMAN, Bart. Near-uniform sampling of combinatorial spaces using XOR constraints. In: Advances In Neural Information Processing Systems. 2007. S. 481-488.
The basic idea:
formulate the problem as SAT-problem
add randomly generated xors to the problem (acting on the decision-variables only! that's important in practice)
this will reduce the number of solutions (some solutions will get impossible)
do that in a loop (with tuned parameters) until only one solution is left!
search for some solution is being done by SAT-solvers or #SAT-solvers (=model-counting)
if there is more than one solution: no xors will be added but a complete restart will be done: add random-xors to the start-problem!
The guarantees:
when tuning the parameters right, this approach achieves near-uniform sampling
this tuning can be costly, as it's based on approximating the number of possible solutions
empirically this can also be costly!
Ante's answer, mentioning the number sequence A001499 actually gives a nice upper bound on the solution-space (as it's just ignoring adjacency-constraints!)
The drawbacks:
inefficient for large problems (in general; not necessarily compared to the alternatives like MCMC and co.)
need to change / reduce parameters to produce samples
those reduced parameters lose the theoretical guarantees
but empirically: good results are still possible!
Parameters:
In practice, the parameters are:
N: number of xors added
L: minimum number of variables part of one xor-constraint
U: maximum number of variables part of one xor-constraint
N is important to reduce the number of possible solutions. Given N constant, the other variables of course also have some effect on that.
Theory says (if i interpret correctly), that we should use L = R = 0.5 * #dec-vars.
This is impossible in practice here, as xor-constraints hurt SAT-solvers a lot!
Here some more scientific slides about the impact of L and U.
They call xors of size 8-20 short-XORS, while we will need to use even shorter ones later!
Implementation
Final version
Here is a pretty hacky implementation in python, using the XorSample scripts from here.
The underlying SAT-solver in use is Cryptominisat.
The code basically boils down to:
Transform the problem to conjunctive normal-form
as DIMACS-CNF
Implement the sampling-approach:
Calls XorSample (pipe-based + file-based)
Call SAT-solver (file-based)
Add samples to some file for later analysis
Code: (i hope i did warn you already about the code-quality)
from itertools import count
from time import time
import subprocess
import numpy as np
import os
import shelve
import uuid
import pickle
from random import SystemRandom
cryptogen = SystemRandom()
""" Helper functions """
# K-ARY CONSTRAINT GENERATION
# ###########################
# SINZ, Carsten. Towards an optimal CNF encoding of boolean cardinality constraints.
# CP, 2005, 3709. Jg., S. 827-831.
def next_var_index(start):
next_var = start
while(True):
yield next_var
next_var += 1
class s_index():
def __init__(self, start_index):
self.firstEnvVar = start_index
def next(self,i,j,k):
return self.firstEnvVar + i*k +j
def gen_seq_circuit(k, input_indices, next_var_index_gen):
cnf_string = ''
s_index_gen = s_index(next_var_index_gen.next())
# write clauses of first partial sum (i.e. i=0)
cnf_string += (str(-input_indices[0]) + ' ' + str(s_index_gen.next(0,0,k)) + ' 0\n')
for i in range(1, k):
cnf_string += (str(-s_index_gen.next(0, i, k)) + ' 0\n')
# write clauses for general case (i.e. 0 < i < n-1)
for i in range(1, len(input_indices)-1):
cnf_string += (str(-input_indices[i]) + ' ' + str(s_index_gen.next(i, 0, k)) + ' 0\n')
cnf_string += (str(-s_index_gen.next(i-1, 0, k)) + ' ' + str(s_index_gen.next(i, 0, k)) + ' 0\n')
for u in range(1, k):
cnf_string += (str(-input_indices[i]) + ' ' + str(-s_index_gen.next(i-1, u-1, k)) + ' ' + str(s_index_gen.next(i, u, k)) + ' 0\n')
cnf_string += (str(-s_index_gen.next(i-1, u, k)) + ' ' + str(s_index_gen.next(i, u, k)) + ' 0\n')
cnf_string += (str(-input_indices[i]) + ' ' + str(-s_index_gen.next(i-1, k-1, k)) + ' 0\n')
# last clause for last variable
cnf_string += (str(-input_indices[-1]) + ' ' + str(-s_index_gen.next(len(input_indices)-2, k-1, k)) + ' 0\n')
return (cnf_string, (len(input_indices)-1)*k, 2*len(input_indices)*k + len(input_indices) - 3*k - 1)
# K=2 clause GENERATION
# #####################
def gen_at_most_2_constraints(vars, start_var):
constraint_string = ''
used_clauses = 0
used_vars = 0
index_gen = next_var_index(start_var)
circuit = gen_seq_circuit(2, vars, index_gen)
constraint_string += circuit[0]
used_clauses += circuit[2]
used_vars += circuit[1]
start_var += circuit[1]
return [constraint_string, used_clauses, used_vars, start_var]
def gen_at_least_2_constraints(vars, start_var):
k = len(vars) - 2
vars = [-var for var in vars]
constraint_string = ''
used_clauses = 0
used_vars = 0
index_gen = next_var_index(start_var)
circuit = gen_seq_circuit(k, vars, index_gen)
constraint_string += circuit[0]
used_clauses += circuit[2]
used_vars += circuit[1]
start_var += circuit[1]
return [constraint_string, used_clauses, used_vars, start_var]
# Adjacency conflicts
# ###################
def get_all_adjacency_conflicts_4_neighborhood(N, X):
conflicts = set()
for x in range(N):
for y in range(N):
if x < (N-1):
conflicts.add(((x,y),(x+1,y)))
if y < (N-1):
conflicts.add(((x,y),(x,y+1)))
cnf = '' # slow string appends
for (var_a, var_b) in conflicts:
var_a_ = X[var_a]
var_b_ = X[var_b]
cnf += '-' + var_a_ + ' ' + '-' + var_b_ + ' 0 \n'
return cnf, len(conflicts)
# Build SAT-CNF
#############
def build_cnf(N, verbose=False):
var_counter = count(1)
N_CLAUSES = 0
X = np.zeros((N, N), dtype=object)
for a in range(N):
for b in range(N):
X[a,b] = str(next(var_counter))
# Adjacency constraints
CNF, N_CLAUSES = get_all_adjacency_conflicts_4_neighborhood(N, X)
# k=2 constraints
NEXT_VAR = N*N+1
for row in range(N):
constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_most_2_constraints(X[row, :].astype(int).tolist(), NEXT_VAR)
N_CLAUSES += used_clauses
CNF += constraint_string
constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_least_2_constraints(X[row, :].astype(int).tolist(), NEXT_VAR)
N_CLAUSES += used_clauses
CNF += constraint_string
for col in range(N):
constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_most_2_constraints(X[:, col].astype(int).tolist(), NEXT_VAR)
N_CLAUSES += used_clauses
CNF += constraint_string
constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_least_2_constraints(X[:, col].astype(int).tolist(), NEXT_VAR)
N_CLAUSES += used_clauses
CNF += constraint_string
# build final cnf
CNF = 'p cnf ' + str(NEXT_VAR-1) + ' ' + str(N_CLAUSES) + '\n' + CNF
return X, CNF, NEXT_VAR-1
# External tools
# ##############
def get_random_xor_problem(CNF_IN_fp, N_DEC_VARS, N_ALL_VARS, s, min_l, max_l):
# .cnf not part of arg!
p = subprocess.Popen(['./gen-wff', CNF_IN_fp,
str(N_DEC_VARS), str(N_ALL_VARS),
str(s), str(min_l), str(max_l), 'xored'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result = p.communicate()
os.remove(CNF_IN_fp + '-str-xored.xor') # file not needed
return CNF_IN_fp + '-str-xored.cnf'
def solve(CNF_IN_fp, N_DEC_VARS):
seed = cryptogen.randint(0, 2147483647) # actually no reason to do it; but can't hurt either
p = subprocess.Popen(["./cryptominisat5", '-t', '4', '-r', str(seed), CNF_IN_fp], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
result = p.communicate()[0]
sat_line = result.find('s SATISFIABLE')
if sat_line != -1:
# solution found!
vars = parse_solution(result)[:N_DEC_VARS]
# forbid solution (DeMorgan)
negated_vars = list(map(lambda x: x*(-1), vars))
with open(CNF_IN_fp, 'a') as f:
f.write( (str(negated_vars)[1:-1] + ' 0\n').replace(',', ''))
# assume solve is treating last constraint despite not changing header!
# solve again
seed = cryptogen.randint(0, 2147483647)
p = subprocess.Popen(["./cryptominisat5", '-t', '4', '-r', str(seed), CNF_IN_fp], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
result = p.communicate()[0]
sat_line = result.find('s SATISFIABLE')
if sat_line != -1:
os.remove(CNF_IN_fp) # not needed anymore
return True, False, None
else:
return True, True, vars
else:
return False, False, None
def parse_solution(output):
# assumes there is one
vars = []
for line in output.split("\n"):
if line:
if line[0] == 'v':
line_vars = list(map(lambda x: int(x), line.split()[1:]))
vars.extend(line_vars)
return vars
# Core-algorithm
# ##############
def xorsample(X, CNF_IN_fp, N_DEC_VARS, N_VARS, s, min_l, max_l):
start_time = time()
while True:
# add s random XOR constraints to F
xored_cnf_fp = get_random_xor_problem(CNF_IN_fp, N_DEC_VARS, N_VARS, s, min_l, max_l)
state_lvl1, state_lvl2, var_sol = solve(xored_cnf_fp, N_DEC_VARS)
print('------------')
if state_lvl1 and state_lvl2:
print('FOUND')
d = shelve.open('N_15_70_4_6_TO_PLOT')
d[str(uuid.uuid4())] = (pickle.dumps(var_sol), time() - start_time)
d.close()
return True
else:
if state_lvl1:
print('sol not unique')
else:
print('no sol found')
print('------------')
""" Run """
N = 15
N_DEC_VARS = N*N
X, CNF, N_VARS = build_cnf(N)
with open('my_problem.cnf', 'w') as f:
f.write(CNF)
counter = 0
while True:
print('sample: ', counter)
xorsample(X, 'my_problem', N_DEC_VARS, N_VARS, 70, 4, 6)
counter += 1
Output will look like (removed some warnings):
------------
no sol found
------------
------------
no sol found
------------
------------
no sol found
------------
------------
sol not unique
------------
------------
FOUND
Core: CNF-formulation
We introduce one variable for every cell of the matrix. N=20 means 400 binary-variables.
Adjancency:
Precalculate all symmetry-reduced conflicts and add conflict-clauses.
Basic theory:
a -> !b
<->
!a v !b (propositional logic)
Row/Col-wise Cardinality:
This is tough to express in CNF and naive approaches need an exponential number
of constraints.
We use some adder-circuit based encoding (SINZ, Carsten. Towards an optimal CNF encoding of boolean cardinality constraints) which introduces new auxiliary-variables.
Remark:
sum(var_set) <= k
<->
sum(negated(var_set)) >= len(var_set) - k
These SAT-encodings can be put into exact model-counters (for small N; e.g. < 9). The number of solutions equals Ante's results, which is a strong indication for a correct transformation!
There are also interesting approximate model-counters (also heavily based on xor-constraints) like approxMC which shows one more thing we can do with the SAT-formulation. But in practice i have not been able to use these (approxMC = autoconf; no comment).
Other experiments
I did also build a version using pblib, to use more powerful cardinality-formulations
for the SAT-CNF formulation. I did not try to use the C++-based API, but only the reduced pbencoder, which automatically selects some best encoding, which was way worse than my encoding used above (which is best is still a research-problem; often even redundant-constraints can help).
Empirical analysis
For the sake of obtaining some sample-size (given my patience), i only computed samples for N=15. In this case we used:
N=70 xors
L,U = 4,6
I also computed some samples for N=20 with (100,3,6), but this takes a few mins and we reduced the lower bound!
Visualization
Here some animation (strengthening my love-hate relationship with matplotlib):
Edit: And a (reduced) comparison to brute-force uniform-sampling with N=5 (NXOR,L,U = 4, 10, 30):
(I have not yet decided on the addition of the plotting-code. It's as ugly as the above one and people might look too much into my statistical shambles; normalizations and co.)
Theory
Statistical analysis is probably hard to do as the underlying problem is of such combinatoric nature. It's even not entirely obvious how that final cell-PDF should look like. In the case of N=odd, it's probably non-uniform and looks like a chess-board (i did brute-force check N=5 to observe this).
One thing we can be sure about (imho): symmetry!
Given a cell-PDF matrix, we should expect, that the matrix is symmetric (A = A.T).
This is checked in the visualization and the euclidean-norm of differences over time is plotted.
We can do the same on some other observation: observed pairings.
For N=3, we can observe the following pairs:
0,1
0,2
1,2
Now we can do this per-row and per-column and should expect symmetry too!
Sadly, it's probably not easy to say something about the variance and therefore the needed samples to speak about confidence!
Observation
According to my simplified perception, current-samples and the cell-PDF look good, although convergence is not achieved yet (or we are far away from uniformity).
The more important aspect are probably the two norms, nicely decreasing towards 0.
(yes; one could tune some algorithm for that by transposing with prob=0.5; but this is not done here as it would defeat it's purpose).
Potential next steps
Tune parameters
Check out the approach using #SAT-solvers / Model-counters instead of SAT-solvers
Try different CNF-formulations, especially in regards to cardinality-encodings and xor-encodings
XorSample is by default using tseitin-like encoding to get around exponentially grow
for smaller xors (as used) it might be a good idea to use naive encoding (which propagates faster)
XorSample supports that in theory; but the script's work differently in practice
Cryptominisat is known for dedicated XOR-handling (as it was build for analyzing cryptography including many xors) and might gain something by naive encoding (as inferring xors from blown-up CNFs is much harder)
More statistical-analysis
Get rid of XorSample scripts (shell + perl...)
Summary
The approach is very general
This code produces feasible samples
It should be not hard to prove, that every feasible solution can be sampled
Others have proven theoretical guarantees for uniformity for some params
does not hold for our params
Others have empirically / theoretically analyzed smaller parameters (in use here)
(Updated test results, example run-through and code snippets below.)
You can use dynamic programming to calculate the number of solutions resulting from every state (in a much more efficient way than a brute-force algorithm), and use those (pre-calculated) values to create equiprobable random solutions.
Consider the example of a 7x7 matrix; at the start, the state is:
0,0,0,0,0,0,0
meaning that there are seven adjacent unused columns. After adding two ones to the first row, the state could be e.g.:
0,1,0,0,1,0,0
with two columns that now have a one in them. After adding ones to the second row, the state could be e.g.:
0,1,1,0,1,0,1
After three rows are filled, there is a possibility that a column will have its maximum of two ones; this effectively splits the matrix into two independent zones:
1,1,1,0,2,0,1 -> 1,1,1,0 + 0,1
These zones are independent in the sense that the no-adjacent-ones rule has no effect when adding ones to different zones, and the order of the zones has no effect on the number of solutions.
In order to use these states as signatures for types of solutions, we have to transform them into a canonical notation. First, we have to take into account the fact that columns with only 1 one in them may be unusable in the next row, because they contain a one in the current row. So instead of a binary notation, we have to use a ternary notation, e.g.:
2,1,1,0 + 0,1
where the 2 means that this column was used in the current row (and not that there are 2 ones in the column). At the next step, we should then convert the twos back into ones.
Additionally, we can also mirror the seperate groups to put them into their lexicographically smallest notation:
2,1,1,0 + 0,1 -> 0,1,1,2 + 0,1
Lastly, we sort the seperate groups from small to large, and then lexicographically, so that a state in a larger matrix may be e.g.:
0,0 + 0,1 + 0,0,2 + 0,1,0 + 0,1,0,1
Then, when calculating the number of solutions resulting from each state, we can use memoization using the canonical notation of each state as a key.
Creating a dictionary of the states and the number of solutions for each of them only needs to be done once, and a table for larger matrices can probably be used for smaller matrices too.
Practically, you'd generate a random number between 0 and the total number of solutions, and then for every row, you'd look at the different states you could create from the current state, look at the number of unique solutions each one would generate, and see which option leads to the solution that corresponds with your randomly generated number.
Note that every state and the corresponding key can only occur in a particular row, so you can store the keys in seperate dictionaries per row.
TEST RESULTS
A first test using unoptimized JavaScript gave very promising results. With dynamic programming, calculating the number of solutions for a 10x10 matrix now takes a second, where a brute-force algorithm took several hours (and this is the part of the algorithm that only needs to be done once). The size of the dictionary with the signatures and numbers of solutions grows with a diminishing factor approaching 2.5 for each step in size; the time to generate it grows with a factor of around 3.
These are the number of solutions, states, signatures (total size of the dictionaries), and maximum number of signatures per row (largest dictionary per row) that are created:
size unique solutions states signatures max/row
4x4 2 9 6 2
5x5 16 73 26 8
6x6 722 514 107 40
7x7 33,988 2,870 411 152
8x8 2,215,764 13,485 1,411 596
9x9 179,431,924 56,375 4,510 1,983
10x10 17,849,077,140 218,038 13,453 5,672
11x11 2,138,979,146,276 801,266 38,314 14,491
12x12 304,243,884,374,412 2,847,885 104,764 35,803
13x13 50,702,643,217,809,908 9,901,431 278,561 96,414
14x14 9,789,567,606,147,948,364 33,911,578 723,306 238,359
15x15 2,168,538,331,223,656,364,084 114,897,838 1,845,861 548,409
16x16 546,386,962,452,256,865,969,596 ... 4,952,501 1,444,487
17x17 155,420,047,516,794,379,573,558,433 12,837,870 3,754,040
18x18 48,614,566,676,379,251,956,711,945,475 31,452,747 8,992,972
19x19 17,139,174,923,928,277,182,879,888,254,495 74,818,773 20,929,008
20x20 6,688,262,914,418,168,812,086,412,204,858,650 175,678,000 50,094,203
(Additional results obtained with C++, using a simple 128-bit integer implementation. To count the states, the code had to be run using each state as a seperate signature, which I was unable to do for the largest sizes. )
EXAMPLE
The dictionary for a 5x5 matrix looks like this:
row 0: 00000 -> 16 row 3: 101 -> 0
1112 -> 1
row 1: 20002 -> 2 1121 -> 1
00202 -> 4 1+01 -> 0
02002 -> 2 11+12 -> 2
02020 -> 2 1+121 -> 1
0+1+1 -> 0
row 2: 10212 -> 1 1+112 -> 1
12012 -> 1
12021 -> 2 row 4: 0 -> 0
12102 -> 1 11 -> 0
21012 -> 0 12 -> 0
02121 -> 3 1+1 -> 1
01212 -> 1 1+2 -> 0
The total number of solutions is 16; if we randomly pick a number from 0 to 15, e.g. 13, we can find the corresponding (i.e. the 14th) solution like this:
state: 00000
options: 10100 10010 10001 01010 01001 00101
signature: 00202 02002 20002 02020 02002 00202
solutions: 4 2 2 2 2 4
This tells us that the 14th solution is the 2nd solution of option 00101. The next step is:
state: 00101
options: 10010 01010
signature: 12102 02121
solutions: 1 3
This tells us that the 2nd solution is the 1st solution of option 01010. The next step is:
state: 01111
options: 10100 10001 00101
signature: 11+12 1112 1+01
solutions: 2 1 0
This tells us that the 1st solution is the 1st solution of option 10100. The next step is:
state: 11211
options: 01010 01001
signature: 1+1 1+1
solutions: 1 1
This tells us that the 1st solutions is the 1st solution of option 01010. The last step is:
state: 12221
options: 10001
And the 5x5 matrix corresponding to randomly chosen number 13 is:
0 0 1 0 1
0 1 0 1 0
1 0 1 0 0
0 1 0 1 0
1 0 0 0 1
And here's a quick'n'dirty code example; run the snippet to generate the signature and solution count dictionary, and generate a random 10x10 matrix (it takes a second to generate the dictionary; once that is done, it generates random solutions in half a millisecond):
function signature(state, prev) {
var zones = [], zone = [];
for (var i = 0; i < state.length; i++) {
if (state[i] == 2) {
if (zone.length) zones.push(mirror(zone));
zone = [];
}
else if (prev[i]) zone.push(3);
else zone.push(state[i]);
}
if (zone.length) zones.push(mirror(zone));
zones.sort(function(a,b) {return a.length - b.length || a - b;});
return zones.length ? zones.join("2") : "2";
function mirror(zone) {
var ltr = zone.join('');
zone.reverse();
var rtl = zone.join('');
return (ltr < rtl) ? ltr : rtl;
}
}
function memoize(n) {
var memo = [], empty = [];
for (var i = 0; i <= n; i++) memo[i] = [];
for (var i = 0; i < n; i++) empty[i] = 0;
memo[0][signature(empty, empty)] = next_row(empty, empty, 1);
return memo;
function next_row(state, prev, row) {
if (row > n) return 1;
var solutions = 0;
for (var i = 0; i < n - 2; i++) {
if (state[i] == 2 || prev[i] == 1) continue;
for (var j = i + 2; j < n; j++) {
if (state[j] == 2 || prev[j] == 1) continue;
var s = state.slice(), p = empty.slice();
++s[i]; ++s[j]; ++p[i]; ++p[j];
var sig = signature(s, p);
var sol = memo[row][sig];
if (sol == undefined)
memo[row][sig] = sol = next_row(s, p, row + 1);
solutions += sol;
}
}
return solutions;
}
}
function random_matrix(n, memo) {
var matrix = [], empty = [], state = [], prev = [];
for (var i = 0; i < n; i++) empty[i] = state[i] = prev[i] = 0;
var total = memo[0][signature(empty, empty)];
var pick = Math.floor(Math.random() * total);
document.write("solution " + pick.toLocaleString('en-US') +
" from a total of " + total.toLocaleString('en-US') + "<br>");
for (var row = 1; row <= n; row++) {
var options = find_options(state, prev);
for (var i in options) {
var state_copy = state.slice();
for (var j in state_copy) state_copy[j] += options[i][j];
var sig = signature(state_copy, options[i]);
var solutions = memo[row][sig];
if (pick < solutions) {
matrix.push(options[i].slice());
prev = options[i].slice();
state = state_copy.slice();
break;
}
else pick -= solutions;
}
}
return matrix;
function find_options(state, prev) {
var options = [];
for (var i = 0; i < n - 2; i++) {
if (state[i] == 2 || prev[i] == 1) continue;
for (var j = i + 2; j < n; j++) {
if (state[j] == 2 || prev[j] == 1) continue;
var option = empty.slice();
++option[i]; ++option[j];
options.push(option);
}
}
return options;
}
}
var size = 10;
var memo = memoize(size);
var matrix = random_matrix(size, memo);
for (var row in matrix) document.write(matrix[row] + "<br>");
The code snippet below shows the dictionary of signatures and solution counts for a matrix of size 10x10. I've used a slightly different signature format from the explanation above: the zones are delimited by a '2' instead of a plus sign, and a column which has a one in the previous row is marked with a '3' instead of a '2'. This shows how the keys could be stored in a file as integers with 2×N bits (padded with 2's).
function signature(state, prev) {
var zones = [], zone = [];
for (var i = 0; i < state.length; i++) {
if (state[i] == 2) {
if (zone.length) zones.push(mirror(zone));
zone = [];
}
else if (prev[i]) zone.push(3);
else zone.push(state[i]);
}
if (zone.length) zones.push(mirror(zone));
zones.sort(function(a,b) {return a.length - b.length || a - b;});
return zones.length ? zones.join("2") : "2";
function mirror(zone) {
var ltr = zone.join('');
zone.reverse();
var rtl = zone.join('');
return (ltr < rtl) ? ltr : rtl;
}
}
function memoize(n) {
var memo = [], empty = [];
for (var i = 0; i <= n; i++) memo[i] = [];
for (var i = 0; i < n; i++) empty[i] = 0;
memo[0][signature(empty, empty)] = next_row(empty, empty, 1);
return memo;
function next_row(state, prev, row) {
if (row > n) return 1;
var solutions = 0;
for (var i = 0; i < n - 2; i++) {
if (state[i] == 2 || prev[i] == 1) continue;
for (var j = i + 2; j < n; j++) {
if (state[j] == 2 || prev[j] == 1) continue;
var s = state.slice(), p = empty.slice();
++s[i]; ++s[j]; ++p[i]; ++p[j];
var sig = signature(s, p);
var sol = memo[row][sig];
if (sol == undefined)
memo[row][sig] = sol = next_row(s, p, row + 1);
solutions += sol;
}
}
return solutions;
}
}
var memo = memoize(10);
for (var i in memo) {
document.write("row " + i + ":<br>");
for (var j in memo[i]) {
document.write(""" + j + "": " + memo[i][j] + "<br>");
}
}
Just few thoughts. Number of matrices satisfying conditions for n <= 10:
3 0
4 2
5 16
6 722
7 33988
8 2215764
9 179431924
10 17849077140
Unfortunatelly there is no sequence with these numbers in OEIS.
There is one similar (A001499), without condition for neighbouring one's. Number of nxn matrices in this case is 'of order' as A001499's number of (n-1)x(n-1) matrices. That is to be expected since number
of ways to fill one row in this case, position 2 one's in n places with at least one zero between them is ((n-1) choose 2). Same as to position 2 one's in (n-1) places without the restriction.
I don't think there is an easy connection between these matrix of order n and A001499 matrix of order n-1, meaning that if we have A001499 matrix than we can construct some of these matrices.
With this, for n=20, number of matrices is >10^30. Quite a lot :-/
This solution use recursion in order to set the cell of the matrix one by one. If the random walk finish with an impossible solution then we rollback one step in the tree and we continue the random walk.
The algorithm is efficient and i think that the generated data are highly equiprobable.
package rndsqmatrix;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.stream.IntStream;
public class RndSqMatrix {
/**
* Generate a random matrix
* #param size the size of the matrix
* #return the matrix encoded in 1d array i=(x+y*size)
*/
public static int[] generate(final int size) {
return generate(size, new int[size * size], new int[size],
new int[size]);
}
/**
* Build a matrix recursivly with a random walk
* #param size the size of the matrix
* #param matrix the matrix encoded in 1d array i=(x+y*size)
* #param rowSum
* #param colSum
* #return
*/
private static int[] generate(final int size, final int[] matrix,
final int[] rowSum, final int[] colSum) {
// generate list of valid positions
final List<Integer> positions = new ArrayList();
for (int y = 0; y < size; y++) {
if (rowSum[y] < 2) {
for (int x = 0; x < size; x++) {
if (colSum[x] < 2) {
final int p = x + y * size;
if (matrix[p] == 0
&& (x == 0 || matrix[p - 1] == 0)
&& (x == size - 1 || matrix[p + 1] == 0)
&& (y == 0 || matrix[p - size] == 0)
&& (y == size - 1 || matrix[p + size] == 0)) {
positions.add(p);
}
}
}
}
}
// no valid positions ?
if (positions.isEmpty()) {
// if the matrix is incomplete => return null
for (int i = 0; i < size; i++) {
if (rowSum[i] != 2 || colSum[i] != 2) {
return null;
}
}
// the matrix is complete => return it
return matrix;
}
// random walk
Collections.shuffle(positions);
for (int p : positions) {
// set '1' and continue recursivly the exploration
matrix[p] = 1;
rowSum[p / size]++;
colSum[p % size]++;
final int[] solMatrix = generate(size, matrix, rowSum, colSum);
if (solMatrix != null) {
return solMatrix;
}
// rollback
matrix[p] = 0;
rowSum[p / size]--;
colSum[p % size]--;
}
// we can't find a valid matrix from here => return null
return null;
}
public static void printMatrix(final int size, final int[] matrix) {
for (int y = 0; y < size; y++) {
for (int x = 0; x < size; x++) {
System.out.print(matrix[x + y * size]);
System.out.print(" ");
}
System.out.println();
}
}
public static void printStatistics(final int size, final int count) {
final int sumMatrix[] = new int[size * size];
for (int i = 0; i < count; i++) {
final int[] matrix = generate(size);
for (int j = 0; j < sumMatrix.length; j++) {
sumMatrix[j] += matrix[j];
}
}
printMatrix(size, sumMatrix);
}
public static void checkAlgorithm() {
final int size = 8;
final int count = 2215764;
final int divisor = 122;
final int sumMatrix[] = new int[size * size];
for (int i = 0; i < count/divisor ; i++) {
final int[] matrix = generate(size);
for (int j = 0; j < sumMatrix.length; j++) {
sumMatrix[j] += matrix[j];
}
}
int total = 0;
for(int i=0; i < sumMatrix.length; i++) {
total += sumMatrix[i];
}
final double factor = (double)total / (count/divisor);
System.out.println("Factor=" + factor + " (theory=16.0)");
}
public static void benchmark(final int size, final int count,
final boolean parallel) {
final long begin = System.currentTimeMillis();
if (!parallel) {
for (int i = 0; i < count; i++) {
generate(size);
}
} else {
IntStream.range(0, count).parallel().forEach(i -> generate(size));
}
final long end = System.currentTimeMillis();
System.out.println("rate="
+ (double) (end - begin) / count + "ms/matrix");
}
public static void main(String[] args) {
checkAlgorithm();
benchmark(8, 10000, true);
//printStatistics(8, 2215764/36);
printStatistics(8, 2215764);
}
}
The output is:
Factor=16.0 (theory=16.0)
rate=0.2835ms/matrix
552969 554643 552895 554632 555680 552753 554567 553389
554071 554847 553441 553315 553425 553883 554485 554061
554272 552633 555130 553699 553604 554298 553864 554028
554118 554299 553565 552986 553786 554473 553530 554771
554474 553604 554473 554231 553617 553556 553581 553992
554960 554572 552861 552732 553782 554039 553921 554661
553578 553253 555721 554235 554107 553676 553776 553182
553086 553677 553442 555698 553527 554850 553804 553444
Here is a very fast approach of generating the matrix row by row, written in Java:
public static void main(String[] args) throws Exception {
int n = 100;
Random rnd = new Random();
byte[] mat = new byte[n*n];
byte[] colCount = new byte[n];
//generate row by row
for (int x = 0; x < n; x++) {
//generate a random first bit
int b1 = rnd.nextInt(n);
while ( (x > 0 && mat[(x-1)*n + b1] == 1) || //not adjacent to the one above
(colCount[b1] == 2) //not in a column which has 2
) b1 = rnd.nextInt(n);
//generate a second bit, not equal to the first one
int b2 = rnd.nextInt(n);
while ( (b2 == b1) || //not the same as bit 1
(x > 0 && mat[(x-1)*n + b2] == 1) || //not adjacent to the one above
(colCount[b2] == 2) || //not in a column which has 2
(b2 == b1 - 1) || //not adjacent to b1
(b2 == b1 + 1)
) b2 = rnd.nextInt(n);
//fill the matrix values and increment column counts
mat[x*n + b1] = 1;
mat[x*n + b2] = 1;
colCount[b1]++;
colCount[b2]++;
}
String arr = Arrays.toString(mat).substring(1, n*n*3 - 1);
System.out.println(arr.replaceAll("(.{" + n*3 + "})", "$1\n"));
}
It essentially generates each a random row at a time. If the row will violate any of the conditions, it is generated again (again randomly). I believe this will satisfy condition 4 as well.
Adding a quick note that it will spin forever for N-s where there is no solutions (like N=3).

Maximum number achievable by converting two adjacent x to one (x+1)

Given a sequence of N integers where 1 <= N <= 500 and the numbers are between 1 and 50. In a step any two adjacent equal numbers x x can be replaced with a single x + 1. What is the maximum number achievable by such steps.
For example if given 2 3 1 1 2 2 then the maximum possible is 4:
2 3 1 1 2 2 ---> 2 3 2 2 2 ---> 2 3 3 2 ---> 2 4 2.
It is evident that I should try to do better than the maximum number available in the sequence. But I can't figure out a good algorithm.
Each substring of the input can make at most one single number (invariant: the log base two of the sum of two to the power of each entry). For every x, we can find the set of substrings that can make x. For each x, this is (1) every occurrence of x (2) the union of two contiguous substrings that can make x - 1. The resulting algorithm is O(N^2)-time.
An algorithm could work like this:
Convert the input to an array where every element has a frequency attribute, collapsing repeated consecutive values in the input into one single node. For example, this input:
1 2 2 4 3 3 3 3
Would be represented like this:
{val: 1, freq: 1} {val: 2, freq: 2} {val: 4, freq: 1} {val: 3, freq: 4}
Then find local minima nodes, like the node (3 3 3 3) in 1 (2 2) 4 (3 3 3 3) 4, i.e. nodes whose neighbours both have higher values. For those local minima that have an even frequency, "lift" those by applying the step. Repeat this until no such local minima (with even frequency) exist any more.
Start of the recursive part of the algorithm:
At both ends of the array, work inwards to "lift" values as long as the more inner neighbour has a higher value. With this rule, the following:
1 2 2 3 5 4 3 3 3 1 1
will completely resolve. First from the left side inward:
1 4 5 4 3 3 3 1 1
Then from the right side:
1 4 6 3 2
Note that when there is an odd frequency (like for the 3s above), there will be a "remainder" that cannot be incremented. The remainder should in this rule always be left on the outward side, so to maximise the potential towards the inner part of the array.
At this point the remaining local minima have odd frequencies. Applying the step to such a node will always leave a "remainder" (like above) with the original value. This remaining node can appear anywhere, but it only makes sense to look at solutions where this remainder is on the left side or the right side of the lift (not in the middle). So for example:
4 1 1 1 1 1 2 3 4
Can resolve to one of these:
4 2 2 1 2 3 4
Or:
4 1 2 2 2 3 4
The 1 in either second or fourth position, is the above mentioned "remainder". Obviously, the second way of resolving is more promising in this example. In general, the choice is obvious when on one side there is a value that is too high to merge with, like the left-most 4 is too high for five 1 values to get to. The 4 is like a wall.
When the frequency of the local minimum is one, there is nothing we can do with it. It actually separates the array in a left and right side that do not influence each other. The same is true for the remainder element discussed above: it separates the array into two parts that do not influence each other.
So the next step in the algorithm is to find such minima (where the choice is obvious), apply that kind of step and separate the problem into two distinct problems which should be solved recursively (from the top). So in the last example, the following two problems would be solved separately:
4
2 2 3 4
Then the best of both solutions will count as the overall solution. In this case that is 5.
The most challenging part of the algorithm is to deal with those local minima for which the choice of where to put the remainder is not obvious. For instance;
3 3 1 1 1 1 1 2 3
This can go to either:
3 3 2 2 1 2 3
3 3 1 2 2 2 3
In this example the end result is the same for both options, but in bigger arrays it would be less and less obvious. So here both options have to be investigated. In general you can have many of them, like 2 in this example:
3 1 1 1 2 3 1 1 1 1 1 3
Each of these two minima has two options. This seems like to explode into too many possibilities for larger arrays. But it is not that bad. The algorithm can take opposite choices in neighbouring minima, and go alternating like this through the whole array. This way alternating sections are favoured, and get the most possible value drawn into them, while the other sections are deprived of value. Now the algorithm turns the tables, and toggles all choices so that the sections that were previously favoured are now deprived, and vice versa. The solution of both these alternatives is derived by resolving each section recursively, and then comparing the two "grand" solutions to pick the best one.
Snippet
Here is a live JavaScript implementation of the above algorithm.
Comments are provided which hopefully should make it readable.
"use strict";
function Node(val, freq) {
// Immutable plain object
return Object.freeze({
val: val,
freq: freq || 1, // Default frequency is 1.
// Max attainable value when merged:
reduced: val + (freq || 1).toString(2).length - 1
});
}
function compress(a) {
// Put repeated elements in a single node
var result = [], i, j;
for (i = 0; i < a.length; i = j) {
for (j = i + 1; j < a.length && a[j] == a[i]; j++);
result.push(Node(a[i], j - i));
}
return result;
}
function decompress(a) {
// Expand nodes into separate, repeated elements
var result = [], i, j;
for (i = 0; i < a.length; i++) {
for (j = 0; j < a[i].freq; j++) {
result.push(a[i].val);
}
}
return result;
}
function str(a) {
return decompress(a).join(' ');
}
function unstr(s) {
s = s.replace(/\D+/g, ' ').trim();
return s.length ? compress(s.split(/\s+/).map(Number)) : [];
}
/*
The function merge modifies an array in-place, performing a "step" on
the indicated element.
The array will get an element with an incremented value
and decreased frequency, unless a join occurs with neighboring
elements with the same value: then the frequencies are accumulated
into one element. When the original frequency was odd there will
be a "remainder" element in the modified array as well.
*/
function merge(a, i, leftWards, stats) {
var val = a[i].val+1,
odd = a[i].freq % 2,
newFreq = a[i].freq >> 1,
last = i;
// Merge with neighbouring nodes of same value:
if ((!odd || !leftWards) && a[i+1] && a[i+1].val === val) {
newFreq += a[++last].freq;
}
if ((!odd || leftWards) && i && a[i-1].val === val) {
newFreq += a[--i].freq;
}
// Replace nodes
a.splice(i, last-i+1, Node(val, newFreq));
if (odd) a.splice(i+leftWards, 0, Node(val-1));
// Update statistics and trace: this is not essential to the algorithm
if (stats) {
stats.total_applied_merges++;
if (stats.trace) stats.trace.push(str(a));
}
return i;
}
/* Function Solve
Parameters:
a: The compressed array to be reduced via merges. It is changed in-place
and should not be relied on after the call.
stats: Optional plain object that will be populated with execution statistics.
Return value:
The array after the best merges were applied to achieve the highest
value, which is stored in the maxValue custom property of the array.
*/
function solve(a, stats) {
var maxValue, i, j, traceOrig, skipLeft, skipRight, sections, goLeft,
b, choice, alternate;
if (!a.length) return a;
if (stats && stats.trace) {
traceOrig = stats.trace;
traceOrig.push(stats.trace = [str(a)]);
}
// Look for valleys of even size, and "lift" them
for (i = 1; i < a.length - 1; i++) {
if (a[i-1].val > a[i].val && a[i].val < a[i+1].val && (a[i].freq % 2) < 1) {
// Found an even valley
i = merge(a, i, false, stats);
if (i) i--;
}
}
// Check left-side elements with always increasing values
for (i = 0; i < a.length-1 && a[i].val < a[i+1].val; i++) {
if (a[i].freq > 1) i = merge(a, i, false, stats) - 1;
};
// Check right-side elements with always increasing values, right-to-left
for (j = a.length-1; j > 0 && a[j-1].val > a[j].val; j--) {
if (a[j].freq > 1) j = merge(a, j, true, stats) + 1;
};
// All resolved?
if (i == j) {
while (a[i].freq > 1) merge(a, i, true, stats);
a.maxValue = a[i].val;
} else {
skipLeft = i;
skipRight = a.length - 1 - j;
// Look for other valleys (odd sized): they will lead to a split into sections
sections = [];
for (i = a.length - 2 - skipRight; i > skipLeft; i--) {
if (a[i-1].val > a[i].val && a[i].val < a[i+1].val) {
// Odd number of elements: if more than one, there
// are two ways to merge them, but maybe
// one of both possibilities can be excluded.
goLeft = a[i+1].val > a[i].reduced;
if (a[i-1].val > a[i].reduced || goLeft) {
if (a[i].freq > 1) i = merge(a, i, goLeft, stats) + goLeft;
// i is the index of the element which has become a 1-sized valley
// Split off the right part of the array, and store the solution
sections.push(solve(a.splice(i--), stats));
}
}
}
if (sections.length) {
// Solve last remaining section
sections.push(solve(a, stats));
sections.reverse();
// Combine the solutions of all sections into one
maxValue = sections[0].maxValue;
for (i = sections.length - 1; i >= 0; i--) {
maxValue = Math.max(sections[i].maxValue, maxValue);
}
} else {
// There is no more valley that can be resolved without branching into two
// directions. Look for the remaining valleys.
sections = [];
b = a.slice(0); // take copy
for (choice = 0; choice < 2; choice++) {
if (choice) a = b; // restore from copy on second iteration
alternate = choice;
for (i = a.length - 2 - skipRight; i > skipLeft; i--) {
if (a[i-1].val > a[i].val && a[i].val < a[i+1].val) {
// Odd number of elements
alternate = !alternate
i = merge(a, i, alternate, stats) + alternate;
sections.push(solve(a.splice(i--), stats));
}
}
// Solve last remaining section
sections.push(solve(a, stats));
}
sections.reverse(); // put in logical order
// Find best section:
maxValue = sections[0].maxValue;
for (i = sections.length - 1; i >= 0; i--) {
maxValue = Math.max(sections[i].maxValue, maxValue);
}
for (i = sections.length - 1; i >= 0 && sections[i].maxValue < maxValue; i--);
// Which choice led to the highest value (choice = 0 or 1)?
choice = (i >= sections.length / 2)
// Discard the not-chosen version
sections = sections.slice(choice * sections.length/2);
}
// Reconstruct the solution from the sections.
a = [].concat.apply([], sections);
a.maxValue = maxValue;
}
if (traceOrig) stats.trace = traceOrig;
return a;
}
function randomValues(len) {
var a = [];
for (var i = 0; i < len; i++) {
// 50% chance for a 1, 25% for a 2, ... etc.
a.push(Math.min(/\.1*/.exec(Math.random().toString(2))[0].length,5));
}
return a;
}
// I/O
var inputEl = document.querySelector('#inp');
var randEl = document.querySelector('#rand');
var lenEl = document.querySelector('#len');
var goEl = document.querySelector('#go');
var outEl = document.querySelector('#out');
goEl.onclick = function() {
// Get the input and structure it
var a = unstr(inputEl.value),
stats = {
total_applied_merges: 0,
trace: a.length < 100 ? [] : undefined
};
// Apply algorithm
a = solve(a, stats);
// Output results
var output = {
value: a.maxValue,
compact: str(a),
total_applied_merges: stats.total_applied_merges,
trace: stats.trace || 'no trace produced (input too large)'
};
outEl.textContent = JSON.stringify(output, null, 4);
}
randEl.onclick = function() {
// Get input (count of numbers to generate):
len = lenEl.value;
// Generate
var a = randomValues(len);
// Output
inputEl.value = a.join(' ');
// Simulate click to find the solution immediately.
goEl.click();
}
// Tests
var tests = [
' ', '',
'1', '1',
'1 1', '2',
'2 2 1 2 2', '3 1 3',
'3 2 1 1 2 2 3', '5',
'3 2 1 1 2 2 3 1 1 1 1 3 2 2 1 1 2', '6',
'3 1 1 1 3', '3 2 1 3',
'2 1 1 1 2 1 1 1 2 1 1 1 1 1 2', '3 1 2 1 4 1 2',
'3 1 1 2 1 1 1 2 3', '4 2 1 2 3',
'1 4 2 1 1 1 1 1 1 1', '1 5 1',
];
var res;
for (var i = 0; i < tests.length; i+=2) {
var res = str(solve(unstr(tests[i])));
if (res !== tests[i+1]) throw 'Test failed: ' + tests[i] + ' returned ' + res + ' instead of ' + tests[i+1];
}
Enter series (space separated):<br>
<input id="inp" size="60" value="2 3 1 1 2 2"><button id="go">Solve</button>
<br>
<input id="len" size="4" value="30"><button id="rand">Produce random series of this size and solve</button>
<pre id="out"></pre>
As you can see the program produces a reduced array with the maximum value included. In general there can be many derived arrays that have this maximum; only one is given.
An O(n*m) time and space algorithm is possible, where, according to your stated limits, n <= 500 and m <= 58 (consider that even for a billion elements, m need only be about 60, representing the largest element ± log2(n)). m is representing the possible numbers 50 + floor(log2(500)):
Consider the condensed sequence, s = {[x, number of x's]}.
If M[i][j] = [num_j,start_idx] where num_j represents the maximum number of contiguous js ending at index i of the condensed sequence; start_idx, the index where the sequence starts or -1 if it cannot join earlier sequences; then we have the following relationship:
M[i][j] = [s[i][1] + M[i-1][j][0], M[i-1][j][1]]
when j equals s[i][0]
j's greater than s[i][0] but smaller than or equal to s[i][0] + floor(log2(s[i][1])), represent converting pairs and merging with an earlier sequence if applicable, with a special case after the new count is odd:
When M[i][j][0] is odd, we do two things: first calculate the best so far by looking back in the matrix to a sequence that could merge with M[i][j] or its paired descendants, and then set a lower bound in the next applicable cells in the row (meaning a merge with an earlier sequence cannot happen via this cell). The reason this works is that:
if s[i + 1][0] > s[i][0], then s[i + 1] could only possibly pair with the new split section of s[i]; and
if s[i + 1][0] < s[i][0], then s[i + 1] might generate a lower j that would combine with the odd j from M[i], potentially making a longer sequence.
At the end, return the largest entry in the matrix, max(j + floor(log2(num_j))), for all j.
JavaScript code (counterexamples would be welcome; the limit on the answer is set at 7 for convenient visualization of the matrix):
function f(str){
var arr = str.split(/\s+/).map(Number);
var s = [,[arr[0],0]];
for (var i=0; i<arr.length; i++){
if (s[s.length - 1][0] == arr[i]){
s[s.length - 1][1]++;
} else {
s.push([arr[i],1]);
}
}
var M = [new Array(8).fill([0,0])],
best = 0;
for (var i=1; i<s.length; i++){
M[i] = new Array(8).fill([0,i]);
var temp = s[i][1],
temp_odd,
temp_start,
odd = false;
for (var j=s[i][0]; temp>0; j++){
var start_idx = odd ? temp_start : M[i][j-1][1];
if (start_idx != -1 && M[start_idx - 1][j][0]){
temp += M[start_idx - 1][j][0];
start_idx = M[start_idx - 1][j][1];
}
if (!odd){
M[i][j] = [temp,start_idx];
temp_odd = temp;
} else {
M[i][j] = [temp_odd,-1];
temp_start = start_idx;
}
if (!odd && temp & 1 && temp > 1){
odd = true;
temp_start = start_idx;
}
best = Math.max(best,j + Math.floor(Math.log2(temp)));
temp >>= 1;
temp_odd >>= 1;
}
}
return [arr, s, best, M];
}
// I/O
var button = document.querySelector('button');
var input = document.querySelector('input');
var pre = document.querySelector('pre');
button.onclick = function() {
var val = input.value;
var result = f(val);
var text = '';
for (var i=0; i<3; i++){
text += JSON.stringify(result[i]) + '\n\n';
}
for (var i in result[3]){
text += JSON.stringify(result[3][i]) + '\n';
}
pre.textContent = text;
}
<input value ="2 2 3 3 2 2 3 3 5">
<button>Solve</button>
<pre></pre>
Here's a brute force solution:
function findMax(array A, int currentMax)
for each pair (i, i+1) of indices for which A[i]==A[i+1] do
currentMax = max(A[i]+1, currentMax)
replace A[i],A[i+1] by a single number A[i]+1
currentMax = max(currentMax, findMax(A, currentMax))
end for
return currentMax
Given the array A, let currentMax=max(A[0], ..., A[n])
print findMax(A, currentMax)
The algorithm terminates because in each recursive call the array shrinks by 1.
It's also clear that it is correct: we try out all possible replacement sequences.
The code is extremely slow when the array is large and there's lots of options regarding replacements, but actually works reasonbly fast on arrays with small number of replaceable pairs. (I'll try to quantify the running time in terms of the number of replaceable pairs.)
A naive working code in Python:
def findMax(L, currMax):
for i in range(len(L)-1):
if L[i] == L[i+1]:
L[i] += 1
del L[i+1]
currMax = max(currMax, L[i])
currMax = max(currMax, findMax(L, currMax))
L[i] -= 1
L.insert(i+1, L[i])
return currMax
# entry point
if __name__ == '__main__':
L1 = [2, 3, 1, 1, 2, 2]
L2 = [2, 3, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2]
print findMax(L1, max(L1))
print findMax(L2, max(L2))
The result of the first call is 4, as expected.
The result of the second call is 5 as expected; the sequence that gives the result: 2,3,1,1,2,2,2,2,2,2,2,2, -> 2,3,1,1,3,2,2,2,2,2,2 -> 2,3,1,1,3,3,2,2,2,2, -> 2,3,1,1,3,3,3,2,2 -> 2,3,1,1,3,3,3,3 -> 2,3,1,1,4,3, -> 2,3,1,1,4,4 -> 2,3,1,1,5

Maximum Devastation to be caused if a building with height h causes all h-1 buildings to its right to collapse

In a recent interview question I got the following problem.
In a particular city we have a row of buildings with varying heights.
The collapse of a building with height h causes the next h-1 buildings on its right to collapse too.
The height of the buildings can be between 1 and 5000. Given the heights of all the buildings (arranged from left to right ie; for leftmost building index=1 and for rightmost building index=N) we needed to find out the index of the building which would cause the maximum devastation.
For example:
Input:
Number of buildings : 6
Height of Buildings:
2 1 3 3 1 6
Answer should be building at the index 3
The solution I tried was using the brute force technique with a complexity of O(N^2).
What I did was for each building in the list I found out the number of buildings that it would destroy.
Could a better solution for this question be constructed?
Simply go from the left, collapse the first building, and calculate how much total(*) damage it did.
Do this again and again from the very next building (which hasn't collapsed).
From these, pick the maximum.
Complexity: O(n).
This greedy algorithm works because the whole process is a chain reaction, if building A forces the collapse of B, then you cannot achieve better score starting from B.
(*) You can do this by maintaining one counter which stores how many buildings to the right should be collapsed. counter = max(counter - 1, height of next building).
some areas of the city function as "firewalls" - collapse stops at that point. a little thought shows that these are sections to the left of a value of 1 where height increases (to the left) no more than once per step (if you can have 0 heights that complicates things very slightly).
and the highest scoring region must start just after a firewall (since if it didn't there would be a higher region just to the left).
so scan from the right, finding these firewalls, and then find which section to the right of a firewall has the largest damage. this is O(n) because it's just linear scans (once from right to left and then once for each section, with no overlap).
actually, Karoly's answer is equivalent and simpler to implement.
Start with rightmost index.
The last building shall cause a devastation value of 1.
Iterate leftwards.
Something like (devastation from building i)
D[i] = 1 + min( N-i, max( index[i]-1, 0+D[i+1],1+D[i+2],... to index[i]-1 terms ) )
Same approach as #Karoly's answer. In ruby:
def find_max_damage_index(buildings)
max_damage = 0
max_start_index = nil
current_start_index = nil
current_h = 0
current_damage = 0
buildings.each_with_index{|h,i|
if current_h == 0 #end of batch
if current_damage > max_damage
max_damage = current_damage
max_start_index = current_start_index
end
#start new batch
current_h = h
current_damage = 1
current_start_index = i
else
current_h = h if h > current_h
current_damage += 1
end
current_h -= 1
}
#last batch
if current_damage > max_damage
max_damage = current_damage
max_start_index = current_start_index
end
return max_start_index
end
In Java, without considering subsequent collapses:
public static int collapse(int[] buildings) {
int i, maxDamage, index, currentDamage;
// do not consider the last building, it will cause only its own fall
maxDamage = 1;
index = buildings.length - 1;
for(i = buildings.length - 1; i >= 0; i--) {
// update maximum damage as the mimimum value between the building[i] (the height of the building at index i) and the remaining number of elements from i to the end of the array
currentDamage = Math.min(buildings[i], buildings.length - i);
System.out.println(currentDamage);
if(currentDamage > maxDamage) {
maxDamage = currentDamage;
index = i;
}
}
return index;
}
My final solution is different from the accepted one, which by the way I didn't fully understand.
The idea is to count starting from the rightmost position the number of buildings that the current index will collapse.
index: 7 6 5 4 3 2 1 0
height: 1 3 1 2 4 1 3 2
damage: 1 2 1 2 4 1 3 2
Then I just make a cumulative sum, starting from the rightmost position again. I add to the number of buildings the current position collapses the number of buildings that were collapsed staring from the next building to the right that didn't collapse until the end.
index: 7 6 5 4 3 2 1 0
height: 1 3 1 2 4 1 3 2
damage: 1 2 1 2 5 1 7 8
In the end, I just return the index with the maximum damage.
This solution runs in O(n) but uses an extra O(n) space.
The next code is the complete version (also works for subsequent collapses):
public static int collapse(int[] buildings) {
int i, maxIndex, max;
int damage[] = new int[buildings.length];
for(i = buildings.length - 1; i >= 0; i--) {
// compute damage for each position
damage[i] = Math.min(buildings[i], buildings.length - i);
}
for(i = buildings.length - 1; i >= 0; i--) {
// update total accumulated damage for each position
if(damage[i] > 1) {
if(damage[i] + i - 1 < buildings.length && i != (i + damage[i] - 1) ) {
damage[i] += damage[i + damage[i] - 1] - 1;
}
}
}
max = damage[0];
maxIndex = 0;
for(i = 1; i < buildings.length; i++) {
// find the maximum damage
if(damage[i] > max) {
max = damage[i];
maxIndex = i;
}
}
return maxIndex;
}

Reasonable optimized chart scaling

I need to make a chart with an optimized y axis maximum value.
The current method I have of making charts simply uses the maximum value of all the graphs, then divides it by ten, and uses that as grid lines. I didn't write it.
Update Note: These graphs have been changed. As soon as I fixed the code, my dynamic graphs started working, making this question nonsensical (because the examples no longer had any errors in them). I've updated these with static images, but some of the answers refrence different values. Keep that in mind.
There were between 12003 and 14003 inbound calls so far in February. Informative, but ugly.
I'd like to avoid charts that look like a monkey came up with the y-axis numbers.
Using the Google charts API helps a little bit, but it's still not quite what I want.
The numbers are clean, but the top of the y value is always the same as the maximum value on the chart. This chart scales from 0 to 1357. I need to have calculated the proper value of 1400, problematically.
I'm throwing in rbobby's defanition of a 'nice' number here because it explains it so well.
A "nice" number is one that has 3 or fewer non-zero digits (eg. 1230000)
A "nice" number has the same or few non-zero digits than zero digits (eg 1230 is not nice, 1200 is nice)
The nicest numbers are ones with multiples of 3 zeros (eg. "1,000", "1,000,000")
The second nicest numbers are onces with multples of 3 zeros plus 2 zeros (eg. "1,500,000", "1,200")
Solution
I found the way to get the results that I want using a modified version of Mark Ransom's idea.
Fist, Mark Ransom's code determines the optimum spacing between ticks, when given the number of ticks. Sometimes this number ends up being more than twice what the highest value on the chart is, depending on how many grid lines you want.
What I'm doing is I'm running Mark's code with 5, 6, 7, 8, 9, and 10 grid lines (ticks) to find which of those is the lowest. With a value of 23, the height of the chart goes to 25, with a grid line at 5, 10, 15, 20, and 25. With a value of 26, the chart's height is 30, with grid lines at 5, 10, 15, 20, 25, and 30. It has the same spacing between grid lines, but there are more of them.
So here's the steps to just-about copy what Excel does to make charts all fancy.
Temporarily bump up the chart's highest value by about 5% (so that there is always some space between the chart's highest point and the top of the chart area. We want 99.9 to round up to 120)
Find the optimum grid line placement
for 5, 6, 7, 8, 9, and 10 grid
lines.
Pick out the lowest of those numbers. Remember the number of grid lines it took to get that value.
Now you have the optimum chart height. The lines/bar will never butt up against the top of the chart and you have the optimum number of ticks.
PHP:
function roundUp($maxValue){
$optiMax = $maxValue * 2;
for ($i = 5; $i <= 10; $i++){
$tmpMaxValue = bestTick($maxValue,$i);
if (($optiMax > $tmpMaxValue) and ($tmpMaxValue > ($maxValue + $maxValue * 0.05))){
$optiMax = $tmpMaxValue;
$optiTicks = $i;
}
}
return $optiMax;
}
function bestTick($maxValue, $mostTicks){
$minimum = $maxValue / $mostTicks;
$magnitude = pow(10,floor(log($minimum) / log(10)));
$residual = $minimum / $magnitude;
if ($residual > 5){
$tick = 10 * $magnitude;
} elseif ($residual > 2) {
$tick = 5 * $magnitude;
} elseif ($residual > 1){
$tick = 2 * $magnitude;
} else {
$tick = $magnitude;
}
return ($tick * $mostTicks);
}
Python:
import math
def BestTick(largest, mostticks):
minimum = largest / mostticks
magnitude = 10 ** math.floor(math.log(minimum) / math.log(10))
residual = minimum / magnitude
if residual > 5:
tick = 10 * magnitude
elif residual > 2:
tick = 5 * magnitude
elif residual > 1:
tick = 2 * magnitude
else:
tick = magnitude
return tick
value = int(input(""))
optMax = value * 2
for i in range(5,11):
maxValue = BestTick(value,i) * i
print maxValue
if (optMax > maxValue) and (maxValue > value + (value*.05)):
optMax = maxValue
optTicks = i
print "\nTest Value: " + str(value + (value * .05)) + "\n\nChart Height: " + str(optMax) + " Ticks: " + str(optTicks)
This is from a previous similar question:
Algorithm for "nice" grid line intervals on a graph
I've done this with kind of a brute
force method. First, figure out the
maximum number of tick marks you can
fit into the space. Divide the total
range of values by the number of
ticks; this is the minimum
spacing of the tick. Now calculate
the floor of the logarithm base 10 to
get the magnitude of the tick, and
divide by this value. You should end
up with something in the range of 1 to
10. Simply choose the round number greater than or equal to the value and
multiply it by the logarithm
calculated earlier. This is your
final tick spacing.
Example in Python:
import math
def BestTick(largest, mostticks):
minimum = largest / mostticks
magnitude = 10 ** math.floor(math.log(minimum) / math.log(10))
residual = minimum / magnitude
if residual > 5:
tick = 10 * magnitude
elif residual > 2:
tick = 5 * magnitude
elif residual > 1:
tick = 2 * magnitude
else:
tick = magnitude
return tick
You could round up to two significant figures. The following pseudocode should work:
// maxValue is the largest value in your chart
magnitude = floor(log10(maxValue))
base = 10^(magnitude - 1)
chartHeight = ceiling(maxValue / base) * base
For example, if maxValue is 1357, then magnitude is 3 and base is 100. Dividing by 100, rounding up, and multiplying by 100 has the result of rounding up to the next multiple of 100, i.e. rounding up to two significant figures. In this case, the result if 1400 (1357 ⇒ 13.57 ⇒ 14 ⇒ 1400).
In the past I've done this in a brute force-ish sort of way. Here's a chunk of C++ code that works well... but for a hardcoded lower and upper limits (0 and 5000):
int PickYUnits()
{
int MinSize[8] = {20, 20, 20, 20, 20, 20, 20, 20};
int ItemsPerUnit[8] = {5, 10, 20, 25, 50, 100, 250, 500};
int ItemLimits[8] = {20, 50, 100, 250, 500, 1000, 2500, 5000};
int MaxNumUnits = 8;
double PixelsPerY;
int PixelsPerAxis;
int Units;
//
// Figure out the max from the dataset
// - Min is always 0 for a bar chart
//
m_MinY = 0;
m_MaxY = -9999999;
m_TotalY = 0;
for (int j = 0; j < m_DataPoints.GetSize(); j++) {
if (m_DataPoints[j].m_y > m_MaxY) {
m_MaxY = m_DataPoints[j].m_y;
}
m_TotalY += m_DataPoints[j].m_y;
}
//
// Give some space at the top
//
m_MaxY = m_MaxY + 1;
//
// Figure out the size of the range
//
double yRange = (m_MaxY - m_MinY);
//
// Pick the initial size
//
Units = MaxNumUnits;
for (int k = 0; k < MaxNumUnits; k++)
{
if (yRange < ItemLimits[k])
{
Units = k;
break;
}
}
//
// Adjust it upwards based on the space available
//
PixelsPerY = m_rcGraph.Height() / yRange;
PixelsPerAxis = (int)(PixelsPerY * ItemsPerUnit[Units]);
while (PixelsPerAxis < MinSize[Units]){
Units += 1;
PixelsPerAxis = (int)(PixelsPerY * ItemsPerUnit[Units]);
if (Units == 5)
break;
}
return ItemsPerUnit[Units];
}
However something in what you've said tweaked me. To pick nice axis numbers a definition of "nice number" would help:
A "nice" number is one that has 3 or fewer non-zero digits (eg. 1230000)
A "nice" number has the same or few non-zero digits than zero digits (eg 1230 is not nice, 1200 is nice)
The nicest numbers are ones with multiples of 3 zeros (eg. "1,000", "1,000,000")
The second nicest numbers are onces with multples of 3 zeros plus 2 zeros (eg. "1,500,000", "1,200")
Not sure if the above definition is "right" or actually helpful (but with the definition in hand it then becomes a simpler task to devise an algorithm).
A slight refinement and tested... (works for fractions of units and not just integers)
public void testNumbers() {
double test = 0.20000;
double multiple = 1;
int scale = 0;
String[] prefix = new String[]{"", "m", "u", "n"};
while (Math.log10(test) < 0) {
multiple = multiple * 1000;
test = test * 1000;
scale++;
}
double tick;
double minimum = test / 10;
double magnitude = 100000000;
while (minimum <= magnitude){
magnitude = magnitude / 10;
}
double residual = test / (magnitude * 10);
if (residual > 5) {
tick = 10 * magnitude;
} else if (residual > 2) {
tick = 5 * magnitude;
} else if (residual > 1) {
tick = 2 * magnitude;
} else {
tick = magnitude;
}
double curAmt = 0;
int ticks = (int) Math.ceil(test / tick);
for (int ix = 0; ix < ticks; ix++) {
curAmt += tick;
BigDecimal bigDecimal = new BigDecimal(curAmt);
bigDecimal.setScale(2, BigDecimal.ROUND_HALF_UP);
System.out.println(bigDecimal.stripTrailingZeros().toPlainString() + prefix[scale] + "s");
}
System.out.println("Value = " + test + prefix[scale] + "s");
System.out.println("Tick = " + tick + prefix[scale] + "s");
System.out.println("Ticks = " + ticks);
System.out.println("Scale = " + multiple + " : " + scale);
}
If you want 1400 at the top, how about adjusting the last two parameters to 1400 instead of 1357:
You could use div and mod. For example.
Let's say you want your chart to round up by increments of 20 (just to make it more a more arbitrary number than your typical "10" value).
So I would assume that 1, 11, 18 would all round up to 20. But 21, 33, 38 would round to 40.
To come up with the right value do the following:
Where divisor = your rounding increment.
divisor = 20
multiple = maxValue / divisor; // Do an integer divide here.
if (maxValue modulus divisor > 0)
multiple++;
graphMax = multiple * maxValue;
So now let's plugin real numbers:
divisor = 20;
multiple = 33 / 20; (integer divide)
so multiple = 1
if (33 modulus 20 > 0) (it is.. it equals 13)
multiple++;
so multiple = 2;
graphMax = multiple (2) * maxValue (20);
graphMax = 40;

Algorithm for "nice" grid line intervals on a graph

I need a reasonably smart algorithm to come up with "nice" grid lines for a graph (chart).
For example, assume a bar chart with values of 10, 30, 72 and 60. You know:
Min value: 10
Max value: 72
Range: 62
The first question is: what do you start from? In this case, 0 would be the intuitive value but this won't hold up on other data sets so I'm guessing:
Grid min value should be either 0 or a "nice" value lower than the min value of the data in range. Alternatively, it can be specified.
Grid max value should be a "nice" value above the max value in the range. Alternatively, it can be specified (eg you might want 0 to 100 if you're showing percentages, irrespective of the actual values).
The number of grid lines (ticks) in the range should be either specified or a number within a given range (eg 3-8) such that the values are "nice" (ie round numbers) and you maximise use of the chart area. In our example, 80 would be a sensible max as that would use 90% of the chart height (72/80) whereas 100 would create more wasted space.
Anyone know of a good algorithm for this? Language is irrelevant as I'll implement it in what I need to.
I've done this with kind of a brute force method. First, figure out the maximum number of tick marks you can fit into the space. Divide the total range of values by the number of ticks; this is the minimum spacing of the tick. Now calculate the floor of the logarithm base 10 to get the magnitude of the tick, and divide by this value. You should end up with something in the range of 1 to 10. Simply choose the round number greater than or equal to the value and multiply it by the logarithm calculated earlier. This is your final tick spacing.
Example in Python:
import math
def BestTick(largest, mostticks):
minimum = largest / mostticks
magnitude = 10 ** math.floor(math.log(minimum, 10))
residual = minimum / magnitude
if residual > 5:
tick = 10 * magnitude
elif residual > 2:
tick = 5 * magnitude
elif residual > 1:
tick = 2 * magnitude
else:
tick = magnitude
return tick
Edit: you are free to alter the selection of "nice" intervals. One commenter appears to be dissatisfied with the selections provided, because the actual number of ticks can be up to 2.5 times less than the maximum. Here's a slight modification that defines a table for the nice intervals. In the example, I've expanded the selections so that the number of ticks won't be less than 3/5 of the maximum.
import bisect
def BestTick2(largest, mostticks):
minimum = largest / mostticks
magnitude = 10 ** math.floor(math.log(minimum, 10))
residual = minimum / magnitude
# this table must begin with 1 and end with 10
table = [1, 1.5, 2, 3, 5, 7, 10]
tick = table[bisect.bisect_right(table, residual)] if residual < 10 else 10
return tick * magnitude
There are 2 pieces to the problem:
Determine the order of magnitude involved, and
Round to something convenient.
You can handle the first part by using logarithms:
range = max - min;
exponent = int(log(range)); // See comment below.
magnitude = pow(10, exponent);
So, for example, if your range is from 50 - 1200, the exponent is 3 and the magnitude is 1000.
Then deal with the second part by deciding how many subdivisions you want in your grid:
value_per_division = magnitude / subdivisions;
This is a rough calculation because the exponent has been truncated to an integer. You may want to tweak the exponent calculation to handle boundary conditions better, e.g. by rounding instead of taking the int() if you end up with too many subdivisions.
I use the following algorithm. It's similar to others posted here but it's the first example in C#.
public static class AxisUtil
{
public static float CalcStepSize(float range, float targetSteps)
{
// calculate an initial guess at step size
var tempStep = range/targetSteps;
// get the magnitude of the step size
var mag = (float)Math.Floor(Math.Log10(tempStep));
var magPow = (float)Math.Pow(10, mag);
// calculate most significant digit of the new step size
var magMsd = (int)(tempStep/magPow + 0.5);
// promote the MSD to either 1, 2, or 5
if (magMsd > 5)
magMsd = 10;
else if (magMsd > 2)
magMsd = 5;
else if (magMsd > 1)
magMsd = 2;
return magMsd*magPow;
}
}
CPAN provides an implementation here (see source link)
See also Tickmark algorithm for a graph axis
FYI, with your sample data:
Maple: Min=8, Max=74, Labels=10,20,..,60,70, Ticks=10,12,14,..70,72
MATLAB: Min=10, Max=80, Labels=10,20,,..,60,80
Here's another implementation in JavaScript:
var calcStepSize = function(range, targetSteps)
{
// calculate an initial guess at step size
var tempStep = range / targetSteps;
// get the magnitude of the step size
var mag = Math.floor(Math.log(tempStep) / Math.LN10);
var magPow = Math.pow(10, mag);
// calculate most significant digit of the new step size
var magMsd = Math.round(tempStep / magPow + 0.5);
// promote the MSD to either 1, 2, or 5
if (magMsd > 5.0)
magMsd = 10.0;
else if (magMsd > 2.0)
magMsd = 5.0;
else if (magMsd > 1.0)
magMsd = 2.0;
return magMsd * magPow;
};
I am the author of "Algorithm for Optimal Scaling on a Chart Axis". It used to be hosted on trollop.org, but I have recently moved domains/blogging engines.
Please see my answer to a related question.
Taken from Mark above, a slightly more complete Util class in c#. That also calculates a suitable first and last tick.
public class AxisAssists
{
public double Tick { get; private set; }
public AxisAssists(double aTick)
{
Tick = aTick;
}
public AxisAssists(double range, int mostticks)
{
var minimum = range / mostticks;
var magnitude = Math.Pow(10.0, (Math.Floor(Math.Log(minimum) / Math.Log(10))));
var residual = minimum / magnitude;
if (residual > 5)
{
Tick = 10 * magnitude;
}
else if (residual > 2)
{
Tick = 5 * magnitude;
}
else if (residual > 1)
{
Tick = 2 * magnitude;
}
else
{
Tick = magnitude;
}
}
public double GetClosestTickBelow(double v)
{
return Tick* Math.Floor(v / Tick);
}
public double GetClosestTickAbove(double v)
{
return Tick * Math.Ceiling(v / Tick);
}
}
With ability to create an instance, but if you just want calculate and throw it away:
double tickX = new AxisAssists(aMaxX - aMinX, 8).Tick;
I wrote an objective-c method to return a nice axis scale and nice ticks for given min- and max values of your data set:
- (NSArray*)niceAxis:(double)minValue :(double)maxValue
{
double min_ = 0, max_ = 0, min = minValue, max = maxValue, power = 0, factor = 0, tickWidth, minAxisValue = 0, maxAxisValue = 0;
NSArray *factorArray = [NSArray arrayWithObjects:#"0.0f",#"1.2f",#"2.5f",#"5.0f",#"10.0f",nil];
NSArray *scalarArray = [NSArray arrayWithObjects:#"0.2f",#"0.2f",#"0.5f",#"1.0f",#"2.0f",nil];
// calculate x-axis nice scale and ticks
// 1. min_
if (min == 0) {
min_ = 0;
}
else if (min > 0) {
min_ = MAX(0, min-(max-min)/100);
}
else {
min_ = min-(max-min)/100;
}
// 2. max_
if (max == 0) {
if (min == 0) {
max_ = 1;
}
else {
max_ = 0;
}
}
else if (max < 0) {
max_ = MIN(0, max+(max-min)/100);
}
else {
max_ = max+(max-min)/100;
}
// 3. power
power = log(max_ - min_) / log(10);
// 4. factor
factor = pow(10, power - floor(power));
// 5. nice ticks
for (NSInteger i = 0; factor > [[factorArray objectAtIndex:i]doubleValue] ; i++) {
tickWidth = [[scalarArray objectAtIndex:i]doubleValue] * pow(10, floor(power));
}
// 6. min-axisValues
minAxisValue = tickWidth * floor(min_/tickWidth);
// 7. min-axisValues
maxAxisValue = tickWidth * floor((max_/tickWidth)+1);
// 8. create NSArray to return
NSArray *niceAxisValues = [NSArray arrayWithObjects:[NSNumber numberWithDouble:minAxisValue], [NSNumber numberWithDouble:maxAxisValue],[NSNumber numberWithDouble:tickWidth], nil];
return niceAxisValues;
}
You can call the method like this:
NSArray *niceYAxisValues = [self niceAxis:-maxy :maxy];
and get you axis setup:
double minYAxisValue = [[niceYAxisValues objectAtIndex:0]doubleValue];
double maxYAxisValue = [[niceYAxisValues objectAtIndex:1]doubleValue];
double ticksYAxis = [[niceYAxisValues objectAtIndex:2]doubleValue];
Just in case you want to limit the number of axis ticks do this:
NSInteger maxNumberOfTicks = 9;
NSInteger numberOfTicks = valueXRange / ticksXAxis;
NSInteger newNumberOfTicks = floor(numberOfTicks / (1 + floor(numberOfTicks/(maxNumberOfTicks+0.5))));
double newTicksXAxis = ticksXAxis * (1 + floor(numberOfTicks/(maxNumberOfTicks+0.5)));
The first part of the code is based on the calculation I found here to calculate nice graph axis scale and ticks similar to excel graphs. It works excellent for all kind of data sets. Here is an example of an iPhone implementation:
Another idea is to have the range of the axis be the range of the values, but put the tick marks at the appropriate position.. i.e. for 7 to 22 do:
[- - - | - - - - | - - - - | - - ]
10 15 20
As for selecting the tick spacing, I would suggest any number of the form 10^x * i / n, where i < n, and 0 < n < 10. Generate this list, and sort them, and you can find the largest number smaller than value_per_division (as in adam_liss) using a binary search.
Using a lot of inspiration from answers already availible here, here's my implementation in C. Note that there's some extendibility built into the ndex array.
float findNiceDelta(float maxvalue, int count)
{
float step = maxvalue/count,
order = powf(10, floorf(log10(step))),
delta = (int)(step/order + 0.5);
static float ndex[] = {1, 1.5, 2, 2.5, 5, 10};
static int ndexLenght = sizeof(ndex)/sizeof(float);
for(int i = ndexLenght - 2; i > 0; --i)
if(delta > ndex[i]) return ndex[i + 1] * order;
return delta*order;
}
In R, use
tickSize <- function(range,minCount){
logMaxTick <- log10(range/minCount)
exponent <- floor(logMaxTick)
mantissa <- 10^(logMaxTick-exponent)
af <- c(1,2,5) # allowed factors
mantissa <- af[findInterval(mantissa,af)]
return(mantissa*10^exponent)
}
where range argument is max-min of domain.
Here is a javascript function I wrote to round grid intervals (max-min)/gridLinesNumber to beautiful values. It works with any numbers, see the gist with detailed commets to find out how it works and how to call it.
var ceilAbs = function(num, to, bias) {
if (to == undefined) to = [-2, -5, -10]
if (bias == undefined) bias = 0
var numAbs = Math.abs(num) - bias
var exp = Math.floor( Math.log10(numAbs) )
if (typeof to == 'number') {
return Math.sign(num) * to * Math.ceil(numAbs/to) + bias
}
var mults = to.filter(function(value) {return value > 0})
to = to.filter(function(value) {return value < 0}).map(Math.abs)
var m = Math.abs(numAbs) * Math.pow(10, -exp)
var mRounded = Infinity
for (var i=0; i<mults.length; i++) {
var candidate = mults[i] * Math.ceil(m / mults[i])
if (candidate < mRounded)
mRounded = candidate
}
for (var i=0; i<to.length; i++) {
if (to[i] >= m && to[i] < mRounded)
mRounded = to[i]
}
return Math.sign(num) * mRounded * Math.pow(10, exp) + bias
}
Calling ceilAbs(number, [0.5]) for different numbers will round numbers like that:
301573431.1193228 -> 350000000
14127.786597236991 -> 15000
-63105746.17236853 -> -65000000
-718854.2201183736 -> -750000
-700660.340487957 -> -750000
0.055717507097870114 -> 0.06
0.0008068701205775142 -> 0.00085
-8.66660070605576 -> -9
-400.09256079792976 -> -450
0.0011740548815578223 -> 0.0015
-5.3003294346854085e-8 -> -6e-8
-0.00005815960629843176 -> -0.00006
-742465964.5184875 -> -750000000
-81289225.90985894 -> -85000000
0.000901771713513881 -> 0.00095
-652726598.5496342 -> -700000000
-0.6498901364393532 -> -0.65
0.9978325804695487 -> 1
5409.4078950583935 -> 5500
26906671.095639467 -> 30000000
Check out the fiddle to experiment with the code. Code in the answer, the gist and the fiddle is slightly different I'm using the one given in the answer.
If you are trying to get the scales looking right on VB.NET charts, then I've used the example from Adam Liss, but make sure when you set the min and max scale values that you pass them in from a variable of type decimal (not of type single or double) otherwise the tick mark values end up being set to like 8 decimal places.
So as an example, I had 1 chart where I set the min Y Axis value to 0.0001 and the max Y Axis value to 0.002.
If I pass these values to the chart object as singles I get tick mark values of 0.00048000001697801, 0.000860000036482233 ....
Whereas if I pass these values to the chart object as decimals I get nice tick mark values of 0.00048, 0.00086 ......
In python:
steps = [numpy.round(x) for x in np.linspace(min, max, num=num_of_steps)]
Answer that can dynamically always plot 0, handle positive and negatives, and small and large numbers, gives the tick interval size and how many to plot; written in Go
forcePlotZero changes how the max values are rounded so it'll always make a nice multiple to then get back to zero. Example:
if forcePlotZero == false then 237 --> 240
if forcePlotZero == true then 237 --> 300
Intervals are calculated by getting the multiple of 10/100/1000 etc for max and then subtracting till the cumulative total of these subtractions is < min
Here's the output from the function, along with showing forcePlotZero
Force to plot zero
max and min inputs
rounded max and min
intervals
forcePlotZero=false
min: -104 max: 240
minned: -160 maxed: 240
intervalCount: 5 intervalSize: 100
forcePlotZero=true
min: -104 max: 240
minned: -200 maxed: 300
intervalCount: 6 intervalSize: 100
forcePlotZero=false
min: 40 max: 1240
minned: 0 maxed: 1300
intervalCount: 14 intervalSize: 100
forcePlotZero=false
min: 200 max: 240
minned: 190 maxed: 240
intervalCount: 6 intervalSize: 10
forcePlotZero=false
min: 0.7 max: 1.12
minned: 0.6 maxed: 1.2
intervalCount: 7 intervalSize: 0.1
forcePlotZero=false
min: -70.5 max: -12.5
minned: -80 maxed: -10
intervalCount: 8 intervalSize: 10
Here's the playground link https://play.golang.org/p/1IhiX_hRQvo
func getMaxMinIntervals(max float64, min float64, forcePlotZero bool) (maxRounded float64, minRounded float64, intervalCount float64, intervalSize float64) {
//STEP 1: start off determining the maxRounded value for the axis
precision := 0.0
precisionDampener := 0.0 //adjusts to prevent 235 going to 300, instead dampens the scaling to get 240
epsilon := 0.0000001
if math.Abs(max) >= 0 && math.Abs(max) < 2 {
precision = math.Floor(-math.Log10(epsilon + math.Abs(max) - math.Floor(math.Abs(max)))) //counting number of zeros between decimal point and rightward digits
precisionDampener = 1
precision = precision + precisionDampener
} else if math.Abs(max) >= 2 && math.Abs(max) < 100 {
precision = math.Ceil(math.Log10(math.Abs(max)+1)) * -1 //else count number of digits before decimal point
precisionDampener = 1
precision = precision + precisionDampener
} else {
precision = math.Ceil(math.Log10(math.Abs(max)+1)) * -1 //else count number of digits before decimal point
precisionDampener = 2
if forcePlotZero == true {
precisionDampener = 1
}
precision = precision + precisionDampener
}
useThisFactorForIntervalCalculation := 0.0 // this is needed because intervals are calculated from the max value with a zero origin, this uses range for min - max
if max < 0 {
maxRounded = (math.Floor(math.Abs(max)*(math.Pow10(int(precision)))) / math.Pow10(int(precision)) * -1)
useThisFactorForIntervalCalculation = (math.Floor(math.Abs(max)*(math.Pow10(int(precision)))) / math.Pow10(int(precision))) + ((math.Ceil(math.Abs(min)*(math.Pow10(int(precision)))) / math.Pow10(int(precision))) * -1)
} else {
maxRounded = math.Ceil(max*(math.Pow10(int(precision)))) / math.Pow10(int(precision))
useThisFactorForIntervalCalculation = maxRounded
}
minNumberOfIntervals := 2.0
maxNumberOfIntervals := 19.0
intervalSize = 0.001
intervalCount = minNumberOfIntervals
//STEP 2: get interval size (the step size on the axis)
for {
if math.Abs(useThisFactorForIntervalCalculation)/intervalSize < minNumberOfIntervals || math.Abs(useThisFactorForIntervalCalculation)/intervalSize > maxNumberOfIntervals {
intervalSize = intervalSize * 10
} else {
break
}
}
//STEP 3: check that intervals are not too large, safety for max and min values that are close together (240, 220 etc)
for {
if max-min < intervalSize {
intervalSize = intervalSize / 10
} else {
break
}
}
//STEP 4: now we can get minRounded by adding the interval size to 0 till we get to the point where another increment would make cumulative increments > min, opposite for negative in
minRounded = 0.0
if min >= 0 {
for {
if minRounded < min {
minRounded = minRounded + intervalSize
} else {
minRounded = minRounded - intervalSize
break
}
}
} else {
minRounded = maxRounded //keep going down, decreasing by the interval size till minRounded < min
for {
if minRounded > min {
minRounded = minRounded - intervalSize
} else {
break
}
}
}
//STEP 5: get number of intervals to draw
intervalCount = (maxRounded - minRounded) / intervalSize
intervalCount = math.Ceil(intervalCount) + 1 // include the origin as an interval
//STEP 6: Check that the intervalCount isn't too high
if intervalCount-1 >= (intervalSize * 2) && intervalCount > maxNumberOfIntervals {
intervalCount = math.Ceil(intervalCount / 2)
intervalSize *= 2
}
return}
This is in python and for base 10.
Doesn't cover all your questions but I think you can build on it
import numpy as np
def create_ticks(lo,hi):
s = 10**(np.floor(np.log10(hi - lo)))
start = s * np.floor(lo / s)
end = s * np.ceil(hi / s)
ticks = [start]
t = start
while (t < end):
ticks += [t]
t = t + s
return ticks

Resources