You can find the explanation of Algorithm 4.3.1D, as it appears in the book Art of The Computer Programming Vol. 2 (pages 272-273) by D. Knuth in the appendix of this question.
It appears that, in the step D.6, qhat is expected to be off by one at most.
Lets assume base is 2^32 (i.e we are working with unsigned 32 bit digits). Let u = [238157824, 2354839552, 2143027200, 0] and v = [3321757696, 2254962688]. Expected output of this division is 4081766756 Link
Both u and v is already normalized as described in D.1(v[1] > b / 2 and u is zero padded).
First iteration of the loop D.3 through D.7 is no-op because qhat = floor((0 * b + 2143027200) / (2254962688)) = 0 in the first iteration.
In the second iteration of the loop, qhat = floor((2143027200 * b + 2354839552) / (2254962688)) = 4081766758 Link.
We don't need to calculate steps D.4 and D.5 to see why this is a problem. Since qhat will be decreased by one in D.6, result of the algorithm will come out as 4081766758 - 1 = 4081766757, however, result should be 4081766756 Link.
Am I right to think that there is a bug in the algorithm, or is there a fallacy in my reasoning?
Appendix
There is no bug; you're ignoring the loop inside Step D3:
In your example, as a result of this test, the value of q̂, which was initially set to 4081766758, is decreased two times, first to 4081766757 and then to 4081766756, before proceeding to Step D4.
(Sorry I did not have the time to make a more detailed / “proper” answer.)
Let's say I have a given length c and I need to cut out several pieces of different lengths a{i}, where i is the index of a specific piece. The length of every piece is smaller or equal to the length c. I need to find all possible permutations of cutting patterns.
Does someone has a smart approach for such tasks or an algorithm to solve this?
The function could look something similar to this:
Pattern[] getPatternList(double.. a, double c);
The input is hence a list of different sizes and the total available space. My goal is to optimize/minimize the trim loss.
I'll use the simplex algorithm for that but to create an linear programming model, I need a smart way to determine all the cutting patterns.
There are exponentially many cutting-patterns in general. So it might not be feasible to construct them all (time and memory)
If you need to optimize some cutting based on some objective, enumerating all possible cuttings is a bad approach (like #harold mentioned)
A bad analogy (which does not exactly apply here as your base-problem is np-hard):
solving 2-SAT is possible in polynomial-time
enumerating all 2-SAT solutions is Sharp-P-complete (an efficient algorithm would imply P=NP, so there might be none!)
A simple approach (to generate all valid cutting-patterns):
Generate all permutations if items = ordering of items (bounded by !n)
Place them one after another and stop if c is exceeded
(It would be a good idea to do this incrementally; build one permutation after another)
Assumption: each item can only be selected once
Assumption: moving/shifting a cut within a free range does not generate a new solution. It it would: solution-space is possibly an uncountably infinite set
edit
Code
Here is a more powerful approach handling the problem with the same assumptions as described above. It uses integer-programming to minimize the trim-loss, implemented in python with the use of cvxpy (and a commercial-solver; can be replaced by an open-source solver like cbc):
import numpy as np
from cvxpy import *
np.random.seed(1)
# random problem
SPACE = 25000
N_ITEMS = 10000
items = np.random.randint(0, 10, size=N_ITEMS)
def minimize_loss(items, space):
N = items.shape[0]
X = Bool(N)
constraint = [sum_entries(mul_elemwise(items, X)) <= space]
objective = Minimize(space - sum_entries(mul_elemwise(items, X)))
problem = Problem(objective, constraint)
problem.solve(solver=GUROBI, verbose=True)
print('trim-loss: ', problem.value)
print('validated trim-loss: ', space - sum(np.dot(X.value.flatten(), items)))
print('# selected items: ', np.count_nonzero(np.round(X.value)))
print('items: ', items)
print('space: ', SPACE)
minimize_loss(items, SPACE)
Output
items: [5 8 9 ..., 5 3 5]
space: 25000
Parameter OutputFlag unchanged
Value: 1 Min: 0 Max: 1 Default: 1
Changed value of parameter QCPDual to 1
Prev: 0 Min: 0 Max: 1 Default: 0
Optimize a model with 1 rows, 10000 columns and 8987 nonzeros
Coefficient statistics:
Matrix range [1e+00, 9e+00]
Objective range [1e+00, 9e+00]
Bounds range [1e+00, 1e+00]
RHS range [2e+04, 2e+04]
Found heuristic solution: objective -25000
Presolve removed 1 rows and 10000 columns
Presolve time: 0.01s
Presolve: All rows and columns removed
Explored 0 nodes (0 simplex iterations) in 0.01 seconds
Thread count was 1 (of 4 available processors)
Optimal solution found (tolerance 1.00e-04)
Best objective -2.500000000000e+04, best bound -2.500000000000e+04, gap 0.0%
trim-loss: 0.0
validated trim-loss: [[ 0.]]
# selected items: 6516
edit v2
After read your new comments, it is clear, that your model-description was incomplete/imprecise and nothing above tackles the problem you want to solve. It's a bit sad.
You will need to enumerate all permutations of a, and then take the longest prefix that has length less than or equal to c.
This sounds like a version of the knapsack problem (https://en.wikipedia.org/wiki/Knapsack_problem), and nobody knows an efficient way to do this.
I am currently working with multi-objective problem with 3 objectives and I am using weighted sum approach, where weights on all objectives together should sum up to 1, I am also using 0.1 as a step, for example if I have two objectives my weights would be:
0.9 0.1
0.8 0.2
0.7 0.3
But currently I'm stuck with 3 objectives and trying to figure out an algorithm that could make a similar distribution of weights across 3 objectives like this:
1obj 2obj 3obj
1.0 0.0 0.0
0.9 (0 ; 0.1) (0.1 ; 0)
0.8 (0 ; 0.1 ; 0.2) (0.2 ; 0.1 ; 0)
Can you pls suggest an algorithm that will get all possible combinations
The simplest way IMHO to get all combinations in this case is to model the problem as a graph and run an graph discovery algorithm, such as BFS.
The graph will be G=(V,E) where V = { (x1,x2,x3) | x1 + x2 + x3 = 1 } and E = { (v1,v2) | can get from v1 to v2 within a single change} (For example ((1,0,0.0,0.0),(0.9,0.0,0.1)) is an edge - because you can get from one to the other with a single change).
Now, since the graph is strongly connected (convince yourself why) - BFS will discover all vertices in this graph, and thus all possible combinations.
Caution: The number of possibilities is exponential, so for large number of objects or small step - discovering the entire graph will take a long time.
Small optimization: (Not needed if the graph size is not expected to be too large): One can optimize the space consumption of this solution by modeling the graph as a DAG (not allowing "back edges" as a structural restriction). This will also yield the correct result because all vertices are still accessible from a single (unique) source.
This saves space consumption because when running BFS in a DAG - a visited set does not have to be maintained.
I have got this assignment question on HMM and I have solved it. I would like to know if I am correct. The problem is:
Suppose a dishonest dealer has two coins, one fair and one biased; the biased coin
has heads probability 1/4. Assume that the dealer never switches the coins. Which
coin is more likely to have generated the sequence HTTTHHHTTTTHTHHTT? It may
be useful to know that log2(3) = 1.585
I calculated the P for fair coin and biased coin.
The P for fair coin is 7.6*10-6 where as P for biased coin is 3.43*10-6. I didn't use log term, which can be used if I solve it the other way. So, I concluded that it is more likely that the given sequence is generated by a fair coin.
Am I right?
Any help is greatly appreciated.
So you are given the following.
P(H|Fake) = 1/4 P(T|Fake) = 3/4
P(H|Fair) = 1/2 P(T|Fair) = 1/2
P(Fair) = 1/2 P(Fake) = 1/2
To answer the question you need to answer P(Fake/HTTTHHHTTTTHTHHTT) and P(Fair/HTTTHHHTTTTHTHHTT) for which you need to apply bayes:
Let X be HTTTHHHTTTTHTHHTT
P(Fake|X) = (P(X|Fake) * P(Fake)) / P(X)
P(Fair|X) = (P(X|Fair) * P(Fair)) / P(X)
Where
P(X) = P(X|Fake) * P(Fake) + P(X|Fair) * P(Fair)
P(X) = (3.43710e-6 * 0.5) + (7.629e-6 * 0.5) = 5.533e-6
And therefore
P(Fake|X) = (3.43710e-6 * 0.5) / 5.533e-6 = 0.3106
P(Fair|X) = (7.629e-6 * 0.5) / 5.533e-6 = 0.6894
So therefore, is more likely that the used coin is the FAIR one. Even though intuitively one might think that the selected coin is the Fake it seems that this is not the case. The given distribution is closer to 0.5 tail 0.5 heads than to 0.25 heads 0.75 tails. For example, in the case of tails 10/17 is 0.58 that is closer to P(T|Fair)=.5 than to P(T|Fake)=.75
HMM is a bit of an overkill for this example. The probability of getting heads in binomially distributed, with p = 0.5 for the fair coin and p = 0.25 for the other one. For both of them, the number of trials n = 17 (if my counting is correct). From the 17 samples you got 7 successes (7 heads). Using Wolfram Alpha, the probability of the fair coin generating this sample is approx 0.15, as opposed to approx 0.07 for the unfair coin. Note I did not bother calculating the exact numbers, just looked at the plots. The formula is there for you to work with if you want to.
EDIT
If you absolutely must use a HMM, set the set of hidden states to be {fair; unfair} . The transition probabilities are: from a hidden state "fair" to a hidden state "fair"= 1, from a fair to unfair 0, etc, because the dealer is not allowed to change coins halfway through the trial. The emission probability from a hidden state "fair" are 0.5 for observable state "heads" and 0.5 for observable state "tails" (0.25 and 0.75 from "unfair"). You can assume at time t=0 hidden state "fair" and "unfair" are equally likely.
Let's say that I know the probability of a "success" is P. I run the test N times, and I see S successes. The test is akin to tossing an unevenly weighted coin (perhaps heads is a success, tails is a failure).
I want to know the approximate probability of seeing either S successes, or a number of successes less likely than S successes.
So for example, if P is 0.3, N is 100, and I get 20 successes, I'm looking for the probability of getting 20 or fewer successes.
If, on the other hadn, P is 0.3, N is 100, and I get 40 successes, I'm looking for the probability of getting 40 our more successes.
I'm aware that this problem relates to finding the area under a binomial curve, however:
My math-fu is not up to the task of translating this knowledge into efficient code
While I understand a binomial curve would give an exact result, I get the impression that it would be inherently inefficient. A fast method to calculate an approximate result would suffice.
I should stress that this computation has to be fast, and should ideally be determinable with standard 64 or 128 bit floating point computation.
I'm looking for a function that takes P, S, and N - and returns a probability. As I'm more familiar with code than mathematical notation, I'd prefer that any answers employ pseudo-code or code.
Exact Binomial Distribution
def factorial(n):
if n < 2: return 1
return reduce(lambda x, y: x*y, xrange(2, int(n)+1))
def prob(s, p, n):
x = 1.0 - p
a = n - s
b = s + 1
c = a + b - 1
prob = 0.0
for j in xrange(a, c + 1):
prob += factorial(c) / (factorial(j)*factorial(c-j)) \
* x**j * (1 - x)**(c-j)
return prob
>>> prob(20, 0.3, 100)
0.016462853241869437
>>> 1-prob(40-1, 0.3, 100)
0.020988576003924564
Normal Estimate, good for large n
import math
def erf(z):
t = 1.0 / (1.0 + 0.5 * abs(z))
# use Horner's method
ans = 1 - t * math.exp( -z*z - 1.26551223 +
t * ( 1.00002368 +
t * ( 0.37409196 +
t * ( 0.09678418 +
t * (-0.18628806 +
t * ( 0.27886807 +
t * (-1.13520398 +
t * ( 1.48851587 +
t * (-0.82215223 +
t * ( 0.17087277))))))))))
if z >= 0.0:
return ans
else:
return -ans
def normal_estimate(s, p, n):
u = n * p
o = (u * (1-p)) ** 0.5
return 0.5 * (1 + erf((s-u)/(o*2**0.5)))
>>> normal_estimate(20, 0.3, 100)
0.014548164531920815
>>> 1-normal_estimate(40-1, 0.3, 100)
0.024767304545069813
Poisson Estimate: Good for large n and small p
import math
def poisson(s,p,n):
L = n*p
sum = 0
for i in xrange(0, s+1):
sum += L**i/factorial(i)
return sum*math.e**(-L)
>>> poisson(20, 0.3, 100)
0.013411150012837811
>>> 1-poisson(40-1, 0.3, 100)
0.046253037645840323
I was on a project where we needed to be able to calculate the binomial CDF in an environment that didn't have a factorial or gamma function defined. It took me a few weeks, but I ended up coming up with the following algorithm which calculates the CDF exactly (i.e. no approximation necessary). Python is basically as good as pseudocode, right?
import numpy as np
def binomial_cdf(x,n,p):
cdf = 0
b = 0
for k in range(x+1):
if k > 0:
b += + np.log(n-k+1) - np.log(k)
log_pmf_k = b + k * np.log(p) + (n-k) * np.log(1-p)
cdf += np.exp(log_pmf_k)
return cdf
Performance scales with x. For small values of x, this solution is about an order of magnitude faster than scipy.stats.binom.cdf, with similar performance at around x=10,000.
I won't go into a full derivation of this algorithm because stackoverflow doesn't support MathJax, but the thrust of it is first identifying the following equivalence:
For all k > 0, sp.misc.comb(n,k) == np.prod([(n-k+1)/k for k in range(1,k+1)])
Which we can rewrite as:
sp.misc.comb(n,k) == sp.misc.comb(n,k-1) * (n-k+1)/k
or in log space:
np.log( sp.misc.comb(n,k) ) == np.log(sp.misc.comb(n,k-1)) + np.log(n-k+1) - np.log(k)
Because the CDF is a summation of PMFs, we can use this formulation to calculate the binomial coefficient (the log of which is b in the function above) for PMF_{x=i} from the coefficient we calculated for PMF_{x=i-1}. This means we can do everything inside a single loop using accumulators, and we don't need to calculate any factorials!
The reason most of the calculations are done in log space is to improve the numerical stability of the polynomial terms, i.e. p^x and (1-p)^(1-x) have the potential to be extremely large or extremely small, which can cause computational errors.
EDIT: Is this a novel algorithm? I've been poking around on and off since before I posted this, and I'm increasingly wondering if I should write this up more formally and submit it to a journal.
I think you want to evaluate the incomplete beta function.
There's a nice implementation using a continued fraction representation in "Numerical Recipes In C", chapter 6: 'Special Functions'.
I can't totally vouch for the efficiency, but Scipy has a module for this
from scipy.stats.distributions import binom
binom.cdf(successes, attempts, chance_of_success_per_attempt)
An efficient and, more importantly, numerical stable algorithm exists in the domain of Bezier Curves used in Computer Aided Design. It is called de Casteljau's algorithm used to evaluate the Bernstein Polynomials used to define Bezier Curves.
I believe that I am only allowed one link per answer so start with Wikipedia - Bernstein Polynomials
Notice the very close relationship between the Binomial Distribution and the Bernstein Polynomials. Then click through to the link on de Casteljau's algorithm.
Lets say I know the probability of throwing a heads with a particular coin is P.
What is the probability of me throwing
the coin T times and getting at least
S heads?
Set n = T
Set beta[i] = 0 for i = 0, ... S - 1
Set beta[i] = 1 for i = S, ... T
Set t = p
Evaluate B(t) using de Casteljau
or at most S heads?
Set n = T
Set beta[i] = 1 for i = 0, ... S
Set beta[i] = 0 for i = S + 1, ... T
Set t = p
Evaluate B(t) using de Casteljau
Open source code probably exists already. NURBS Curves (Non-Uniform Rational B-spline Curves) are a generalization of Bezier Curves and are widely used in CAD. Try openNurbs (the license is very liberal) or failing that Open CASCADE (a somewhat less liberal and opaque license). Both toolkits are in C++, though, IIRC, .NET bindings exist.
If you are using Python, no need to code it yourself. Scipy got you covered:
from scipy.stats import binom
# probability that you get 20 or less successes out of 100, when p=0.3
binom.cdf(20, 100, 0.3)
>>> 0.016462853241869434
# probability that you get exactly 20 successes out of 100, when p=0.3
binom.pmf(20, 100, 0.3)
>>> 0.0075756449257260777
From the portion of your question "getting at least S heads" you want the cummulative binomial distribution function. See http://en.wikipedia.org/wiki/Binomial_distribution for the equation, which is described as being in terms of the "regularized incomplete beta function" (as already answered). If you just want to calculate the answer without having to implement the entire solution yourself, the GNU Scientific Library provides the function: gsl_cdf_binomial_P and gsl_cdf_binomial_Q.
The DCDFLIB Project has C# functions (wrappers around C code) to evaluate many CDF functions, including the binomial distribution. You can find the original C and FORTRAN code here. This code is well tested and accurate.
If you want to write your own code to avoid being dependent on an external library, you could use the normal approximation to the binomial mentioned in other answers. Here are some notes on how good the approximation is under various circumstances. If you go that route and need code to compute the normal CDF, here's Python code for doing that. It's only about a dozen lines of code and could easily be ported to any other language. But if you want high accuracy and efficient code, you're better off using third party code like DCDFLIB. Several man-years went into producing that library.
Try this one, used in GMP. Another reference is this.
import numpy as np
np.random.seed(1)
x=np.random.binomial(20,0.6,10000) #20 flips of coin,probability of
heads percentage and 10000 times
done.
sum(x>12)/len(x)
The output is 41% of times we got 12 heads.