What is the bottleneck in this primes related predicate? - performance

Question

So here it is : I'm trying to calculate the sum of all primes below two millions (for this problem), but my program is very slow. I do know that the algorithm in itself is terribly bad and a brute force one, but it seems way slower than it should to me.
Here I limit the search to 20,000 so that the result isn't waited too long.
I don't think that this predicate is difficult to understand but I'll explain it anyway : I calculate the list of all the primes below 20,000 and then sum them. The sum part is fine, the primes part is really slow.
problem_010(R) :-
p010(3, [], Primes),
sumlist([2|Primes], R).
p010(20001, Primes, Primes) :- !.
p010(Current, Primes, Result) :-
(
prime(Current, Primes)
-> append([Primes, [Current]], NewPrimes)
; NewPrimes = Primes
),
NewCurrent is Current + 2,
p010(NewCurrent, NewPrimes, Result).
prime(_, []) :- !.
prime(N, [Prime|_Primes]) :- 0 is N mod Prime, !, fail.
prime(ToTest, [_|Primes]) :- prime(ToTest, Primes).
I'd like some insight about why it is so slow. Is it a good implementation of the stupid brute force algorithm, or is there some reason that makes Prolog fall?
EDIT : I already found something, by appending new primes instead of letting them in the head of the list, I have primes that occur more often at start so it's ~3 times faster. Still need some insight though :)

Answer 1

First, Prolog does not fail here.
There are very smart ways how to generate prime numbers. But as a cheap start simply accumulate the primes in reversed order! (7.9s -> 2.6s) In this manner the smaller ones are tested sooner. Then, consider to test only against primes up to 141. Larger primes cannot be a factor.
Then, instead of stepping only through numbers not divisible by 2, you might add 3, 5, 7.
There are people writing papers on this "problem". See, for example this paper, although it's a bit of a sophistic discussion what the "genuine" algorithm actually was, 22 centuries ago when the latest release of the abacus was celebrated as Salamis tablets.

Answer 2

Consider using for example a sieve method ("Sieve of Eratosthenes"): First create a list [2,3,4,5,6,....N], using for example numlist/3. The first number in the list is a prime, keep it. Eliminate its multiples from the rest of the list. The next number in the remaining list is again a prime. Again eliminate its multiples. And so on. The list will shrink quite rapidly, and you end up with only primes remaining.

Answer 3

First of all, appending at the end of a list using append/3 is quite slow. If you must, then use difference lists instead. (Personally, I try to avoid append/3 as much as possible)
Secondly, your prime/2 always iterates over the whole list when checking a prime. This is unnecessarily slow. You can instead just check id you can find an integral factor up to the square root of the number you want to check.
problem_010(R) :-
p010(3, 2, R).
p010(2000001, Primes, Primes) :- !.
p010(Current, In, Result) :-
( prime(Current) -> Out is In+Current ; Out=In ),
NewCurrent is Current + 2,
p010(NewCurrent, Out, Result).
prime(2).
prime(3).
prime(X) :-
integer(X),
X > 3,
X mod 2 =\= 0,
\+is_composite(X, 3). % was: has_factor(X, 3)
is_composite(X, F) :- % was: has_factor(X, F)
X mod F =:= 0, !.
is_composite(X, F) :-
F * F < X,
F2 is F + 2,
is_composite(X, F2).
Disclaimer: I found this implementation of prime/1 and has_factor/2 by googling.
This code gives:
?- problem_010(R).
R = 142913828922
Yes (12.87s cpu)
Here is even faster code:
problem_010(R) :-
Max = 2000001,
functor(Bools, [], Max),
Sqrt is integer(floor(sqrt(Max))),
remove_multiples(2, Sqrt, Max, Bools),
compute_sum(2, Max, 0, R, Bools).
% up to square root of Max, remove multiples by setting bool to 0
remove_multiples(I, Sqrt, _, _) :- I > Sqrt, !.
remove_multiples(I, Sqrt, Max, Bools) :-
arg(I, Bools, B),
(
B == 0
->
true % already removed: do nothing
;
J is 2*I, % start at next multiple of I
remove(J, I, Max, Bools)
),
I1 is I+1,
remove_multiples(I1, Sqrt, Max, Bools).
remove(I, _, Max, _) :- I > Max, !.
remove(I, Add, Max, Bools) :-
arg(I, Bools, 0), % remove multiple by setting bool to 0
J is I+Add,
remove(J, Add, Max, Bools).
% sum up places that are not zero
compute_sum(Max, Max, R, R, _) :- !.
compute_sum(I, Max, RI, R, Bools) :-
arg(I, Bools, B),
(B == 0 -> RO = RI ; RO is RI + I ),
I1 is I+1,
compute_sum(I1, Max, RO, R, Bools).
This runs an order of magnitude faster than the code I gave above:
?- problem_010(R).
R = 142913828922
Yes (0.82s cpu)

Answer 4

OK, before the edit the problem was just the algorithm (imho).
As you noticed, it's more efficient to check if the number is divided by the smaller primes first; in a finite set, there are more numbers divisible by 3 than by 32147.
Another algorithm improvement is to stop checking when the primes are greater than the square root of the number.
Now, after your change there are indeed some prolog issues:
you use append/3. append/3 is quite slow since you have to traverse the whole list to place the element at the end.
Instead, you should use difference lists, which makes placing the element at the tail really fast.
Now, what is a difference list? Instead of creating a normal list [1,2,3] you create this one [1,2,3|T]. Notice that we leave the tail uninstantiated. Then, if we want to add one element (or more) at the end of the list we can simply say T=[4|NT]. awesome?
The following solution (accumulate primes in reverse order, stop when prime>sqrt(N), difference lists to append) takes 0.063 for 20k primes and 17sec for 2m primes while your original code took 3.7sec for 20k and the append/3 version 1.3sec.
problem_010(R) :-
p010(3, Primes, Primes),
sumlist([2|Primes], R).
p010(2000001, _Primes,[]) :- !. %checking for primes till 2mil
p010(Current, Primes,PrimesTail) :-
R is sqrt(Current),
(
prime(R,Current, Primes)
-> PrimesTail = [Current|NewPrimesTail]
; NewPrimesTail = PrimesTail
),
NewCurrent is Current + 2,
p010(NewCurrent, Primes,NewPrimesTail).
prime(_,_, Tail) :- var(Tail),!.
prime(R,_N, [Prime|_Primes]):-
Prime>R.
prime(_R,N, [Prime|_Primes]) :-0 is N mod Prime, !, fail.
prime(R,ToTest, [_|Primes]) :- prime(R,ToTest, Primes).
also, considering adding the numbers while you generate them to avoid the extra o(n) because of sumlist/2
in the end, you can always implement the AKS algorithm that runs in polynomial time (XD)