I need to sort an n-thousand size array of random unique positive integers into groups of consecutive integers, each of group size k or larger, and then further grouped into dividends of some arbitrary positive integer j.
In other words, let's say I work at Chuck E. Cheese and we sometimes give away free tickets. I have a couple hundred thousand tickets on the floor and want to find out what employee handed out what but only for ticket groupings of consecutive integers that are larger than 500. Each employee has a random number from 0 to 100 assigned to them. That number corresponds to what "batch" of tickets where handed out, i.e. tickets from #000000 to #001499 where handed out by employee 1, tickets from #001500 to #002999 were handed out by employee 2, and so on. A large number of tickets are lost or are missing. I only care about groups of consecutive ticket numbers larger than 500.
What is the fastest way for me to sort through this pile?
Edit:
As requested by #trincot, here is a worked out example:
I have 150,000 unique tickets on the floor ranging from ticket #000000 to #200000 (i.e. missing 50,001 random tickets from the pile)
Step 1: sort each ticket from smallest to largest using an introsort algorithm.
Step 2: go through the list of tickets one by one and gather only tickets with "consecutiveness" greater than 500. i.e. I keep a tally of how many consecutive values I have found and only keep those with tallys 500 or higher. If I have tickets #409 thru #909 but not #408 or #1000 then I would keep that group but if that group had missed a ticket anywhere from #409 to #909, I would have thrown out the group and moved on.
Step 3: combine all my newly sorted groups together, each of which are size 500 or larger.
Step 4: figure out what tickets belong to who by going through the final numbers one by one again, dividing each by 1500, rounding down to nearest whole number, and putting them in their respective pile where each pile represents an employee.
The end result is a set of piles telling me which employees gave out more than 500 tickets at a time, how many times they did so, and what tickets they did so with.
Sample with numbers:
where k == 3 and j = 1500; k is minimum consecutive integer grouping size, j is final ticket interval grouping size i.e. 5,6, and 7 fall into the 0th group of intervals of size 1500 and 5996, 5997, 5998, 5999 fall into the third group of intervals of size 1500.
Input: [5 , 5996 , 8111 , 1000 , 1001, 5999 , 8110 , 7 , 5998 , 2500 , 1250 , 6 , 8109 , 5997]
Output:[ 0:[5, 6, 7] , 3:[5996, 5997, 5998, 5999] , 5:[8109, 8110, 8111] ]
Here is how you could do it in Python:
from collections import defaultdict
def partition(data, k, j):
data = sorted(data)
start = data[0] # assuming data is not an empty list
count = 0
output = defaultdict(list) # to automatically create a partition when referenced
for value in data:
bucket = value // j # integer division
if value % j == start % j + count: # in same partition & consecutive?
count += 1
if count == k:
# Add the k entries that we skipped so far:
output[bucket].extend(list(range(start, start+count)))
elif count > k:
output[bucket].append(value)
else:
start = value
count = 1
return dict(output)
# The example given in the question:
data = [5, 5996, 8111, 1000, 1001, 5999, 8110, 7, 5998, 2500, 1250, 6, 8109, 5997]
print(partition(data, k=3, j=1500))
# outputs {0: [5, 6, 7], 3: [5996, 5997, 5998, 5999], 5: [8109, 8110, 8111]}
Here is untested Python for the fastest approach that I can think of. It will return just pairs of first/last ticket for each range of interest found.
def grouped_tickets (tickets, min_group_size, partition_size):
tickets = sorted(tickets)
answer = {}
min_ticket = -1
max_ticket = -1
next_partition = 0
for ticket in tickets:
if next_partition <= ticket or max_ticket + 1 < ticket:
if min_group_size <= max_ticket - min_ticket + 1:
partition = min_ticket // partition_size
if partition in answer:
answer[partition].append((min_ticket, max_ticket))
else:
answer[partition] = [(min_ticket, max_ticket)]
# Find where the next partition is.
next_partition = (ticket // partition_size) * partition_size + partition_size
min_ticket = ticket
max_ticket = ticket
else:
max_ticket = ticket
# And don't lose the last group!
if min_group_size <= max_ticket - min_ticket + 1:
partition = min_ticket // partition_size
if partition in answer:
answer[partition].append((min_ticket, max_ticket))
else:
answer[partition] = [(min_ticket, max_ticket)]
return answer
I have a Scala application and have the following use case. Given a numberOfDates: Int and an optimalFrequencyInDays: Int I need to find the frequency in days closest to this optimal frequency in days that will give me evenly spaced triggers within this number of days. As extra conditions the trigger also has to happen at the beginning and at the end ; furthermore, the number of days between any two triggers can not be smaller than the optimal frequency e.g.
val numberOfDays = 260
val optimalFrequencyInDays = 2
// equally spaced answer is 3 i.e. 87 triggers Seq(0, 3, 6, 9, .. , 255, 259)
val numberOfDays = 260
val optimalFrequencyInDays = 124
// equally spaced answer is 130 i.e. 3 triggers Seq(0, 130, 259)
I think the rule to solve this is:
val solution = (numberOfDates % optimalFrequencyInDays ) match {
case 0 => numberOfDates / (((numberOfDates / optimalFrequencyInDays) / 2) + 1)
case _ => numberOfDates / (((numberOfDates / optimalFrequencyInDays + 1) / 2) + 1)
}
In words, the formula (length / 2 + 1) gives me the range of odd numbers that will produce the number of triggers I need for an evenly spaced solution e.g. for 20 would be 20 / 2 + 1 = 11, 9, 7, 5, 3, 2 If I divide the length by the result of that formula I get the evenly spaced frequency I need.
The output of this use-case is encoded in an array of Booleans of the form Array(1, 0, 0, 1, ..., 1, 0, 0, 1) meaning whether there was a trigger at the day of that index. What is the idiomatic Scala way to test that the triggers are equally spaced except the last that can be evenly spaced +- 1 because there is no perfect fit.
You have a collection of 1s and 0s, and you want to test if the 1s are evenly spaced except for the final spacing which could be an outlier.
triggers.mkString // one long string of 0's and 1's
.split("(?=1)") // multiple strings, all starting with '1'
.dropRight(2) // drop the final `1` and the possible outlier
.sliding(2) // pair up the rest
.forall{ // all the same?
case Array(a,b) => a == b
case Array(_) => true // too few to matter
}
This will handle an empty triggers collection as well as a collection of one or more 1s (no 0s).
update
This will work with an Array[Boolean], either by mapping it to an Array[Int] or by changing the split() pattern to split("(?=true)").
You can test the "outlier" for its off-by-one condition by saving the intermediate collection after the split() and testing its head against its init.last.
You can take indexes of elements with ones, and then calculate the difference between each pair of elements to calculate intervals. Then you just have to check if all elements but last are equal and if the last element is equal +/- 1.
val triggers = Vector(1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1);
val intervals = triggers.zipWithIndex.filter(_._1 == 1).map(_._2).sliding(2).map { case Vector(a, b) => b - a }.toVector
val allButAllAreEqual = intervals.init.forall(_ == intervals.head)
val lastIsEqualToAllPlusMinusOne = intervals.last == intervals.head || intervals.last + 1 == intervals.head
First off, I'm not even sure the terminology is the right one, as I havent found anything similar (especially since I dont even know what keywords to use)
The problem:
There is a population of people, and I want to assign them into groups. I have a set of rules to give each assignation a score. I want to find the best one (or at least a very good one).
For example, with a population of four {A,B,C,D} and assigning to two groups of two, the possible assignations are:
{A,B},{C,D}
{A,C},{B,D}
{A,D},{B,C}
And for example, {B,A},{C,D} and {C,D},{A,B} are both the same as the first one (I don't care about the order inside the groups and the order of the groups themselves).
The number of people, the amount of groups and how many people fit in each group are all inputs.
My idea was to list each possible assignation, calculate their score and keep track of the best one. That is, to brute force it. As the population can be big, I was thinking of going through them in a random order and return the best one found when time runs out (probably when the user gets bored or thinks it is a good enough find). The population can vary from very small (the four listed) to really big (maybe 200+) so just trying random ones without caring about repeats breaks down with the small ones, where a brute force is possible (plus I wouldn't know when to stop if I used plain random permutations).
The population is big enough that listing all the assignations to be able to shuffle them doesn't fit into memory. So I need either a method to find all the possible assignations in a random order, or a method to, given an index, generate the corresponding assignation, and use an index array and shuffle that (the second would be better because I can then easily distribute the tasks into multiple servers).
A simple recursive algorithm for generating these pairings is to pair the first element with each of the remaining elements, and for each of those couplings, recursively generate all the pairings of the remaining elements. For groups, generate all the groups made up of the first element and all the combinations of the remaining elements, then recurse for the remainders.
You can compute how many possible sets of groups there are like this:
public static int numGroupingCombinations(int n, int groupSize)
{
if(n % groupSize != 0)
return 0; // n must be a multiple of groupSize
int count = 1;
while(n > groupSize)
{
count *= nCr(n - 1, groupSize - 1);
n -= groupSize;
}
return count;
}
public static int nCr(int n, int r)
{
int ret = 1;
for (int k = 0; k < r; k++) {
ret = ret * (n-k) / (k+1);
}
return ret;
}
So I need either a method to find all the possible assignations in a random order, or a method to, given an index, generate the corresponding assignation, and use an index array and shuffle that (the second would be better because I can then easily distribute the tasks into multiple servers).
To generate a grouping from an index, choose a combination of items to group with the first element by taking the modulo of the index with the number of possible combinations, and generating the combination from the result using this algorithm. Then divide the index by that same number and recursively generate the rest of the set.
public static void generateGrouping(String[] elements, int groupSize, int start, int index)
{
if(elements.length % groupSize != 0)
return;
int remainingSize = elements.length - start;
if(remainingSize == 0)
{
// output the elements:
for(int i = 0; i < elements.length; i += groupSize)
{
System.out.print("[");
for(int j = 0; j < groupSize; j++)
System.out.print(((j==0)?"":",")+elements[i+j]);
System.out.print("]");
}
System.out.println("");
return;
}
int combinations = nCr(remainingSize - 1, groupSize - 1);
// decide which combination of remaining elements to pair the first element with:
int[] combination = getKthCombination(remainingSize - 1, groupSize - 1, index % combinations);
// swap elements into place
for(int i = 0; i < groupSize - 1; i++)
{
String temp = elements[start + 1 + i];
elements[start + 1 + i] = elements[start + 1 + combination[i]];
elements[start + 1 + combination[i]] = temp;
}
generateGrouping(elements, groupSize, start + groupSize, index / combinations);
// swap them back:
for(int i = groupSize - 2; i >= 0; i--)
{
String temp = elements[start + 1 + i];
elements[start + 1 + i] = elements[start + 1 + combination[i]];
elements[start + 1 + combination[i]] = temp;
}
}
public static void getKthCombination(int n, int r, int k, int[] c, int start, int offset)
{
if(r == 0)
return;
if(r == n)
{
for(int i = 0; i < r; i++)
c[start + i] = i + offset;
return;
}
int count = nCr(n - 1, r - 1);
if(k < count)
{
c[start] = offset;
getKthCombination(n-1, r-1, k, c, start + 1, offset + 1);
return;
}
getKthCombination(n-1, r, k-count, c, start, offset + 1);
}
public static int[] getKthCombination(int n, int r, int k)
{
int[] c = new int[r];
getKthCombination(n, r, k, c, 0, 0);
return c;
}
Demo
The start parameter is just how far along the list you are, so pass zero when calling the function at the top level. The function could easily be rewritten to be iterative. You could also pass an array of indices instead of an array of objects that you want to group, if swapping the objects is a large overhead.
What you call "assignations" are partitions with a fixed number of equally sized parts. Well, mostly. You didn't specify what should happen if (# of groups) * (size of each group) is less than or greater than your population size.
Generating every possible partition in a non-specific order is not too difficult, but it is only good for small populations or for filtering and finding any partition that matches some independent criteria. If you need to optimize or minimize something, you'll end up looking at the whole set of partitions, which may not be feasible.
Based on the description of your actual problem, you want to read up on local search and optimization algorithms, of which the aforementioned simulated annealing is one such technique.
With all that said, here is a simple recursive Python function that generates fixed-length partitions with equal-sized parts in no particular order. It is a specialization of my answer to a similar partition problem, and that answer is itself a specialization of this answer. It should be fairly straightforward to translate into JavaScript (with ES6 generators).
def special_partitions(population, num_groups, group_size):
"""Yields all partitions with a fixed number of equally sized parts.
Each yielded partition is a list of length `num_groups`,
and each part a tuple of length `group_size.
"""
assert len(population) == num_groups * group_size
groups = [] # a list of lists, currently empty
def assign(i):
if i >= len(population):
yield list(map(tuple, groups))
else:
# try to assign to an existing group, if possible
for group in groups:
if len(group) < group_size:
group.append(population[i])
yield from assign(i + 1)
group.pop()
# assign to an entirely new group, if possible
if len(groups) < num_groups:
groups.append([population[i]])
yield from assign(i + 1)
groups.pop()
yield from assign(0)
for partition in special_partitions('ABCD', 2, 2):
print(partition)
print()
for partition in special_partitions('ABCDEF', 2, 3):
print(partition)
When executed, this prints:
[('A', 'B'), ('C', 'D')]
[('A', 'C'), ('B', 'D')]
[('A', 'D'), ('B', 'C')]
[('A', 'B', 'C'), ('D', 'E', 'F')]
[('A', 'B', 'D'), ('C', 'E', 'F')]
[('A', 'B', 'E'), ('C', 'D', 'F')]
[('A', 'B', 'F'), ('C', 'D', 'E')]
[('A', 'C', 'D'), ('B', 'E', 'F')]
[('A', 'C', 'E'), ('B', 'D', 'F')]
[('A', 'C', 'F'), ('B', 'D', 'E')]
[('A', 'D', 'E'), ('B', 'C', 'F')]
[('A', 'D', 'F'), ('B', 'C', 'E')]
[('A', 'E', 'F'), ('B', 'C', 'D')]
Let's say we have a total of N elements that we want to organize in G groups of E (with G*E = N). Neither the order of the groups nor the order of the elements within groups matter. The end goal is to produce every solution in a random order, knowing that we cannot store every solution at once.
First, let's think about how to produce one solution. Since order doesn't matter, we can normalize any solution by sorting the elements within groups as well as the groups themselves, by their first element.
For instance, if we consider the population {A, B, C, D}, with N = 4, G = 2, E = 2, then the solution {B,D}, {C,A} can be normalized as {A,C}, {B,D}. The elements are sorted within each group (A before C), and the groups are sorted (A before B).
When the solutions are normalized, the first element of the first group is always the first element of the population. The second element is one of the N-1 remaining, the third element is one of the N-2 remaining, and so on, except these elements must remain sorted. So there are (N-1)!/((N-E)!*(E-1)!) possibilities for the first group.
Similarly, the first element of the next groups are fixed : they are the first of the remaining elements after each group has been created. Thus, the number of possibilities for the (n+1)th group (n from 0 to G-1) is (N-nE-1)!/((N-(n+1)E)!*(E-1)!) = ((G-n)E-1)!/(((G-n-1)E)!*(E-1)!).
This gives us one possible way of indexing a solution. The index is not a single integer, but rather an array of G integers, the integer n (still from 0 to G-1) being in the range 1 to (N-nE-1)!/((N-nE-E)!*(E-1)!), and representing the group n (or "(n+1)th group") of the solution. This is easy to produce randomly and to check for duplicates.
The last thing we need to find is a way to produce a group from a corresponding integer, n. We need to choose E-1 elements from the N-nE-1 remaining. At this point, you can imagine listing every combination and choosing the (n+1)th one. Of course, this can be done without generating every combination : see this question.
For curiosity, the total number of solutions is (GE)!/(G!*(E!)^G).
In your example, it is (2*2)!/(2!*(2!)^2) = 3.
For N = 200 and E = 2, there are 6.7e186 solutions.
For N = 200 and E = 5, there are 6.6e243 solutions (the maximum I found for 200 elements).
Additionally, for N = 200 and E > 13, the number of possibilities for the first group is greater than 2^64 (so it cannot be stored in a 64-bit integer), which is problematic for representing an index. But as long as you don't need groups with more than 13 elements, you can use arrays of 64-bit integers as indices.
Perhaps a simulated annealing approach might work. You can start with a non-optimal initial solution and iterate using heuristics to improve.
Your scoring criteria may help you choose the initial solution, e.g. make the best scoring first group you can, then with what's left make the best scoring second group, and so on.
Good choices of "neighboring states" may be implied by your scoring criteria, but at the very least, you could consider two states neighboring if they differ by a single swap.
So the iteration portion of the algorithm would be to try a bunch of swaps, sampled randomly, and choose the one that improves the global score according to the annealing schedule.
I'm hoping you can find a better choice of the adjacent states! That is, I'm hoping you can find better rules for iteratively improving based on your scoring criteria.
If you have a sufficiently large population that you can't fit all assignations in memory and are unlikely to ever test all possible assignations, then the simplest method will just be to choose test assignations randomly. For example:
repeat
randomly shuffle population
put 1st n/2 members of the shuffled pop into assig1 and 2nd n/2 into assig2
score assignation and record it if best so far
until bored
If you have a large population it is unlikely that there will be much loss of efficiency due to duplicating a test as it is unlikely that you would chance on the same assignation again.
Depending on your scoring rules it may be more efficient to choose the next assignation to be tested by, for example swapping a pair of members between the best assignation so far found, but you haven't provided enough information to determine if that is the case.
Here is an approach targeting your optimization-problem (and ignoring your permutation-based approach).
I formulate the problem as mixed-integer-problem and use specialized solvers to calculate good solutions.
As your problem is not well-formulated, it might need some modifications. But the general message is: this approach will be hard to beat!.
Code
import numpy as np
from cvxpy import *
""" Parameters """
N_POPULATION = 50
GROUPSIZES = [3, 6, 12, 12, 17]
assert sum(GROUPSIZES) == N_POPULATION
N_GROUPS = len(GROUPSIZES)
OBJ_FACTORS = [0.4, 0.1, 0.15, 0.35] # age is the most important
""" Create fake data """
age_vector = np.clip(np.random.normal(loc=35.0, scale=10.0, size=N_POPULATION).astype(int), 0, np.inf)
height_vector = np.clip(np.random.normal(loc=180.0, scale=15.0, size=N_POPULATION).astype(int), 0, np.inf)
weight_vector = np.clip(np.random.normal(loc=85, scale=20, size=N_POPULATION).astype(int), 0, np.inf)
skill_vector = np.random.randint(0, 100, N_POPULATION)
""" Calculate a-priori stats """
age_mean, height_mean, weight_mean, skill_mean = np.mean(age_vector), np.mean(height_vector), \
np.mean(weight_vector), np.mean(skill_vector)
""" Build optimization-model """
# Variables
X = Bool(N_POPULATION, N_GROUPS) # 1 if part of group
D = Variable(4, N_GROUPS) # aux-var for deviation-norm
# Constraints
constraints = []
# (1) each person is exactly in one group
for p in range(N_POPULATION):
constraints.append(sum_entries(X[p, :]) == 1)
# (2) each group has exactly n (a-priori known) members
for g_ind, g_size in enumerate(GROUPSIZES):
constraints.append(sum_entries(X[:, g_ind]) == g_size)
# Objective: minimize deviation from global-statistics within each group
# (ugly code; could be improved a lot!)
group_deviations = [[], [], [], []] # age, height, weight, skill
for g_ind, g_size in enumerate(GROUPSIZES):
group_deviations[0].append((sum_entries(mul_elemwise(age_vector, X[:, g_ind])) / g_size) - age_mean)
group_deviations[1].append((sum_entries(mul_elemwise(height_vector, X[:, g_ind])) / g_size) - height_mean)
group_deviations[2].append((sum_entries(mul_elemwise(weight_vector, X[:, g_ind])) / g_size) - weight_mean)
group_deviations[3].append((sum_entries(mul_elemwise(skill_vector, X[:, g_ind])) / g_size) - skill_mean)
for i in range(4):
for g in range(N_GROUPS):
constraints.append(D[i,g] >= abs(group_deviations[i][g]))
obj_parts = [sum_entries(OBJ_FACTORS[i] * D[i, :]) for i in range(4)]
objective = Minimize(sum(obj_parts))
""" Build optimization-problem & solve """
problem = Problem(objective, constraints)
problem.solve(solver=GUROBI, verbose=True, TimeLimit=120) # might need to use non-commercial solver here
print('Min-objective: ', problem.value)
""" Evaluate solution """
filled_groups = [[] for g in range(N_GROUPS)]
for g_ind, g_size in enumerate(GROUPSIZES):
for p in range(N_POPULATION):
if np.isclose(X[p, g_ind].value, 1.0):
filled_groups[g_ind].append(p)
for g_ind, g_size in enumerate(GROUPSIZES):
print('Group: ', g_ind, ' of size: ', g_size)
print(' ' + str(filled_groups[g_ind]))
group_stats = []
for g in range(N_GROUPS):
age_mean_in_group = age_vector[filled_groups[g]].mean()
height_mean_in_group = height_vector[filled_groups[g]].mean()
weight_mean_in_group = weight_vector[filled_groups[g]].mean()
skill_mean_in_group = skill_vector[filled_groups[g]].mean()
group_stats.append((age_mean_in_group, height_mean_in_group, weight_mean_in_group, skill_mean_in_group))
print('group-assignment solution means: ')
for g in range(N_GROUPS):
print(np.round(group_stats[g], 1))
""" Compare with input """
input_data = np.vstack((age_vector, height_vector, weight_vector, skill_vector))
print('input-means')
print(age_mean, height_mean, weight_mean, skill_mean)
print('input-data')
print(input_data)
Output (time-limit of 2 minutes; commercial solver)
Time limit reached
Best objective 9.612058823514e-01, best bound 4.784117647059e-01, gap 50.2280%
('Min-objective: ', 0.961205882351435)
('Group: ', 0, ' of size: ', 3)
[16, 20, 27]
('Group: ', 1, ' of size: ', 6)
[26, 32, 34, 45, 47, 49]
('Group: ', 2, ' of size: ', 12)
[0, 6, 10, 12, 15, 21, 24, 30, 38, 42, 43, 48]
('Group: ', 3, ' of size: ', 12)
[2, 3, 13, 17, 19, 22, 23, 25, 31, 36, 37, 40]
('Group: ', 4, ' of size: ', 17)
[1, 4, 5, 7, 8, 9, 11, 14, 18, 28, 29, 33, 35, 39, 41, 44, 46]
group-assignment solution means:
[ 33.3 179.3 83.7 49. ]
[ 33.8 178.2 84.3 49.2]
[ 33.9 178.7 83.8 49.1]
[ 33.8 179.1 84.1 49.2]
[ 34. 179.6 84.7 49. ]
input-means
(33.859999999999999, 179.06, 84.239999999999995, 49.100000000000001)
input-data
[[ 22. 35. 28. 32. 41. 26. 25. 37. 32. 26. 36. 36.
27. 34. 38. 38. 38. 47. 35. 35. 34. 30. 38. 34.
31. 21. 25. 28. 22. 40. 30. 18. 32. 46. 38. 38.
49. 20. 53. 32. 49. 44. 44. 42. 29. 39. 21. 36.
29. 33.]
[ 161. 158. 177. 195. 197. 206. 169. 182. 182. 198. 165. 185.
171. 175. 176. 176. 172. 196. 186. 172. 184. 198. 172. 162.
171. 175. 178. 182. 163. 176. 192. 182. 187. 161. 158. 191.
182. 164. 178. 174. 197. 156. 176. 196. 170. 197. 192. 171.
191. 178.]
[ 85. 103. 99. 93. 71. 109. 63. 87. 60. 94. 48. 122.
56. 84. 69. 162. 104. 71. 92. 97. 101. 66. 58. 69.
88. 69. 80. 46. 74. 61. 25. 74. 59. 69. 112. 82.
104. 62. 98. 84. 129. 71. 98. 107. 111. 117. 81. 74.
110. 64.]
[ 81. 67. 49. 74. 65. 93. 25. 7. 99. 34. 37. 1.
25. 1. 96. 36. 39. 41. 33. 28. 17. 95. 11. 80.
27. 78. 97. 91. 77. 88. 29. 54. 16. 67. 26. 13.
31. 57. 84. 3. 87. 7. 99. 35. 12. 44. 71. 43.
16. 69.]]
Solution remarks
This solution looks quite nice (regarding mean-deviation) and it only took 2 minutes (we decided on the time-limit a-priori)
We also got tight bounds: 0.961 is our solution; we know it can't be lower than 4.784
Reproducibility
The code uses numpy and cvxpy
An commercial solver was used
You might need to use a non-commercial MIP-solver (supporting time-limit for early abortion; take current best-solution)
The valid open-source MIP-solvers supported in cvxpy are: cbc (no chance of setting time-limits for now) and glpk (check the docs for time-limit support)
Model decisions
The code uses L1-norm penalization, which results in an MIP-problem
Depending on your problem, it might be wise to use L2-norm penalization (one big deviation hurts more than many smaller ones), which will result in a harder problem (MIQP / MISOCP)
I'm trying to solve this problem in Python3. I know how to find min1 and min2, but I cannot guess how to search 5 elements in a single pass.
Problem Statement
The input program serves measurements performed by a device at intervals of 1 minute. All data are in natural numbers not exceeding 1000. The problem is to find the smallest sum of the squares of two measurements performed at intervals not less than 5 minutes apart. The first line will contain one natural number -- the number of measurements N. It is guaranteed that 5 < N <= 10000. Each of the following N lines contains one natural number -- the result of the next measurement.
Your program should output a single number, the lowest sum of the squares of two measurements performed at intervals not less than 5 minutes apart.
Sample input:
9
12
45
5
4
21
20
10
12
26
Expected output: 169
I like this question. Fun brain-teaser. :)
I noticed your sample input was all integers in range(1, 100) with some repetition, so I generated sample lists like so:
>>> import random
>>> sample_list = [random.choice(range(1, 100)) for i in range(10)]
>>> sample_list
[74, 68, 57, 18, 36, 8, 89, 73, 77, 80]
According to the problem statement, these numbers represent data measured at one-minute intervals, and one of our constraints is that our result must represent data gathered at least five minutes apart. Ultimately, that means the indices of the data in the original list must differ by at least five. In other words, for any two inputs v1 and v2:
abs(sample_list.index(v1) - sample_list.index(v2)) >= 5
must be true. We also know that we're searching for the smallest sum, so it will be helpful to look at the smallest numbers first.
Thus, I started by mapping the values in the sample_list to the indices where they occur, then sorting them:
>>> occurrences = {}
>>> for index, value in enumerate(sample_list):
... try:
... occurrences[value].append(index)
... except KeyError:
... occurrences[value] = [index]
...
>>> occurrences
{80: [9], 18: [3], 68: [1], 73: [7], 89: [6], 8: [5], 57: [2], 74: [0], 77: [8], 36: [4]}
>>> sorted_occurrences = sorted(occurrences)
>>> sorted_occurrences
[8, 18, 36, 57, 68, 73, 74, 77, 80, 89]
After a whole lot of trial and error, here's what I finally came up with in function form (including some of the earlier-discussed pieces):
def smallest_sum_of_squares_five_apart(sample):
occurrences = {}
for index, value in enumerate(sample):
try:
occurrences[value].append(index)
except KeyError:
occurrences[value] = [index]
sorted_occurrences = sorted(occurrences)
least_sum = 0
for index, v1 in enumerate(sorted_occurrences):
if least_sum and v1**2 > least_sum:
return least_sum
for v2 in sorted_occurrences[:index+1]:
if (abs(max(occurrences[v1]) - min(occurrences[v2])) >= 5 or
abs(max(occurrences[v2]) - min(occurrences[v1])) >= 5):
print('Found candidates:', str((v1, v2)))
sum_of_squares = v1**2 + v2**2
if not least_sum or sum_of_squares < least_sum:
least_sum = sum_of_squares
return least_sum
The idea here is to:
Start by looking at the smallest values first.
Compare them one by one with all the values smaller, up to themselves.
Check each against our criteria. Notice we do this by checking the extremes of each, where these two numbers occur the farthest away from one another in the original sample.
Break out when checking becomes pointless.
Unfortunately, it is not sufficient to find the first one. Depending how the list is constructed, it will not always find the smallest pair first this way. In fact, it does not for your own sample input. However, once v1**2 (the square of the larger value) is larger than the sum, we know since all numbers are natural numbers it is pointless to continue looking.
I have included a full runnable implementation of this below. It takes a command line argument (default 10) indicating the number of items you want in the randomly generated sample. It will print the randomly generated sample as well as all candidate pairs it checked, and finally the sum itself. I have checked this on 10-sized inputs several times and it seems to be working in general. However, feedback is welcome if it is not correct. Note also you can uncomment your sample list from the question to see how it works (and that it gets the right answer) for it.
import random
import sys
def smallest_sum_of_squares_five_apart(sample):
occurrences = {}
for index, value in enumerate(sample):
try:
occurrences[value].append(index)
except KeyError:
occurrences[value] = [index]
sorted_occurrences = sorted(occurrences)
least_sum = 0
for index, v1 in enumerate(sorted_occurrences):
if least_sum and v1**2 > least_sum:
return least_sum
for v2 in sorted_occurrences[:index+1]:
if (abs(max(occurrences[v1]) - min(occurrences[v2])) >= 5 or
abs(max(occurrences[v2]) - min(occurrences[v1])) >= 5):
print('Found candidates:', str((v1, v2)))
sum_of_squares = v1**2 + v2**2
if not least_sum or sum_of_squares < least_sum:
least_sum = sum_of_squares
return least_sum
if __name__ == '__main__':
try:
r = int(sys.argv[1])
except IndexError:
r = 10
sample_list = [random.choice(range(1, 100)) for i in range(r)]
#sample_list = [9, 12, 45, 5, 4, 21, 20, 10, 12, 26]
print(sample_list)
print(smallest_sum_of_squares_five_apart(sample_list))
Try this:
#!/usr/bin/env python3
import queue
inp = [9,12,45,5,4,21,20,10,12,26]
q = queue.Queue() #Make a new queue
smallest = False #No smallest number, yet
best = False #No best sum of squares, yet
for x in inp:
q.put(x) #Place current element on queue
#If there's an item from more than five minutes ago, consider it
if q.qsize()>5:
temp = q.get() #Pop oldest item from queue into temporary variable
if not smallest: #If this is the first item more than 5 minutes old
smallest = temp #it is the smallest item by default
else: #otherwise...
smallest = min(temp,smallest) #only store it if it is the smallest yet
#If we have no best sum of squares or the current item produces one, then
#save it as the best
if (not best) or (x*x+smallest*smallest<best):
best = x*x+smallest*smallest
print(best)
The idea is to walk through the queue keeping track of the smallest element we have seen yet which is older than five minutes and comparing it against the newest element keeping track of the smallest sum of squares as we go.
I think you'll find the answer to be pretty intuitive if you think about it.
The algorithm operates in O(N) time.