Related
I'm trying to benefit from an optimisation that sometimes occurs in GHC (8.4.3), where a "build" of a large volume of data is put in PINNED memory. (I may not have all the terminology correct here). Here's a simple example:
Pinned1.hs:
main = print $ sum $ tail ([1..100000000] :: [Int])
then:
ghc -O2 Pinned1.hs -prof -rtsopts
Pinned1 +RTS -hc -p -xt
hp2ps -e8in -c Pinned1.hp
shows ~40K PINNED and virtually no STACK usage, and Pinned1 +RTS -hd -p -xt shows the ~40K is ARR_WORDS.
The Pinned1.prof shows:
total time = 2.14 secs (2137 ticks # 1000 us, 1 processor)
total alloc = 8,000,046,088 bytes (excludes profiling overheads)
Having looked at the -sdump-simpl, I can see the kind of code that leads to this. Here's a slightly more complex example, back-translated from Core into Haskell code, where the same thing happens:
Pinned2.hs:
main = print $ sum $ snd $ wgoC 1 0
wgoC :: Int -> Int -> (Int, [Int])
wgoC n finalState =
let (nxt, ys') = case n of 100000000 -> (finalState, [])
_ -> wgoC (n+1) finalState
in (n, n + nxt * 9: ys')
wgoC passes the next n back, which is used in the calculation of the values in the list. It reports ~40K PINNED/ARR_WORDS memory, and almost no STACK, and this profile output:
total time = 5.50 secs (5500 ticks # 1000 us, 1 processor)
total alloc = 16,800,046,112 bytes (excludes profiling overheads)
However, this:
Pinned3.hs:
main = print $ sum $ snd $ wgoD 1 0
wgoD :: Int -> Int -> (Int, [Int])
wgoD n finalState =
let (ttl', ys') = case n of 100000000 -> (finalState, [])
_ -> wgoD (n+1) finalState
in (ttl' + n, n + (ttl' + n) * 9 : ys')
doesn't complete after 2 mins. It does complete with a value of only 1000000, and I see no PINNED memory and STACK usage (~100M) instead. (I think it is the STACK usage that is making it run much more slowly, somehow).
The main difference I see between Pinned2 and Pinned3 is that Pinned3 includes information from the recursive call in the returned state (the fst of the returned pair: the cumulative sum of subsequent values), but Pinned2 only includes the parameter to wgoC.
So my questions are:
Q1) Where (in the compiler pipeline) does the decision to use PINNED memory happen? -ddump-simpl shows no obvious difference, nor -ddump-cmm (though it's a little complicated so maybe I'm missing something).
Q2) What is the PINNED/STACK decision based on? (The only references I can find to PINNED, such as this, say it's useful for FFI calls, but it seems it's also been adopted for this "optimisation" as well).
Q3) Is there some way to modify Pinned3 so that it does use PINNED?
Q4) (as a last resort) Is there some other tweak to Pinned3 so that there is enough STACK space, and it runs in a reasonable time? (Naïvely, I would expect similar performance to Pinned2).
[Note that I'm only trying to understand the PINNED/STACK mechanism here. I'm sure there are other ways to write Pinned3 so it fuses nicely and needs hardly any memory, but this question is not about that.]
Thanks!
Pinned memory doesn't play a role here.
This program:
main = print $ sum $ tail ([1..100000000] :: [Int])
does not use any pinned memory directly. The pinned memory you're seeing is from the initialization of the runtime system itself. Pinned memory is allocated by GHC's byte array primitives; in user code, you're most likely to see pinned memory usage when using Data.Text or Data.ByteString both of which use byte arrays for their internal implementation. For this program, I'm going to guess that the I/O buffers for stdin and stdout are pinned, but maybe it's something else. Anyway, lists of Ints won't pin anything.
Like (almost) all Haskell programs, Pinned1.hs uses tons of heap and tons of stack (gigabytes of each) but, critically, frees it as quickly as it allocates it (or if you prefer when talking about the stack, "pops" it as quickly as it "pushes" it). The same is true of Pinned2.hs. These programs are functioning correctly.
The problem with Pinned3.hs is not that it uses stack instead of pinned memory, but rather that it uses even more stack than Pinned1 and Pinned2 and fails to pop it as quickly as it pushes it, so stack accumulates.
So, why does Pinned3 accumulate stack?
Generally speaking, stack accumulates in a recursive call if some part of the result of the recursive call is the target of a function application when it returns AND evaluating that part of the result itself requires another recursive call. Consider the program:
eatStack 100000000 = 1
eatStack n = 1 + eatStack (n + 1)
main = print $ eatStack 1
which, compiled and run with:
stack ghc -- -O2 -prof -rtsopts EatStack.hs
./EatStack +RTS -hd -p -xt
stack exec hp2ps -- -e8in -c EatStack.hp
produces the usual sort of pyramid-shaped stack accumulation (with a peak of 1.4G or so). The problem here is that the return value of the recursive eatStack (n+1) call is subject to the function application \x -> 1 + x when it returns, and calculating that result itself requires further recursion. That is, calculating eatStack 0 requires pushing \x -> 1 + x onto the stack before calling eatStack 1 which can only return its result after pushing \x -> 1 + x onto the stack before calling eatStack 2, and so on. The result is stack accumulation.
Notably, constructor applications are handled differently. The following program:
noStack 100000000 = []
noStack n = 1 : noStack (n + 1)
main = print $ last (noStack 1)
which applies the partially applied constructor (:) 1 to the recursive result noStack (n+1) uses no stack. (It appears to use 40k of pinned, but again that's really the runtime system. EatStack uses 40k of pinned, too.) In some cases (not here), constructor application like this can cause heap accumulation, but it doesn't generally accumulate stack.
For your Pinned2 and Pinned3 examples, something similar is going on, though it's obviously a little more complicated. Let's look at Pinned2 first, and consider evaluating wgoC 1 0. Matching the case and substituting in the arguments, the evaluation is equivalent to:
wgoC 1 0 =
let (nxt, ys') = wgoC 2 0
in (1, 1 + nxt * 9 : ys')
When sum . snd demands the first element of the list, namely the thunk 1 + nxt * 9, this forces nxt to be evaluated via the recursive call. Because this return value is subject to a function application (namely \x -> 1 + x * 9), this uses a bit of stack, but the evaluation of the recursive call:
wgoC 2 0 =
let (nxt, ys') = wgoC 3 0
in (2, 2 + nxt * 9 : ys')
immediately yields the required value for the locally bound nxt in the wgoC 1 0 call, namely the first element of the returned tuple fst (wgoC 2 0) = 2, without requiring further recursion. So, we take that value 2, pop off the continuation \x -> 1 + x * 9 and pass the value to the continuation to yield 1 + 2 * 9 = 19. That gives the list's first element, with no net stack usage. The rest of the list, namely the locally bound ys' in the wgoC 1 0 call, is still in a thunk 2 + nxt * 9 : ys' as closed by the wgoC 2 0 call.
When the next element is demanded, we'll need some stack to apply the continuation \x -> 2 + x * 9 to the result nxt in the recursive (nxt, ys') = wgoC 3 0, but this will be evaluated the same way, immediately returning nxt = 3 and a thunk for ys', so the continuation \x -> 2 + x * 9 will be popped off the stack and applied to nxt = 3 without further recursion, yielding 2 + 3 * 9 = 29 and a thunk 3 + nxt * 9 : ys' as closed by the wgoC 3 0 call.
Each element can be forced with no net stack use. We push a continuation and then immediately pop it and apply it to a part of the return value from the recursive call that doesn't itself require further recursion. The result is no net stack accumulation.
Now, consider Pinned3, and the call wgoD 1 0:
wgoD 1 0 =
let (ttl', ys') = wgoD 2 0
in (ttl' + 1, 1 + (ttl' + 1) * 9 : ys')
When sum . snd demands the first element of the list, namely the thunk 1 + (ttl' + 1) * 9, this forces ttl' to be evaluated via the recursive call. Because there's a pending function application \x -> 1 + (ttl' + 1) * 9, this will use a bit of stack. The recursive call:
wgoD 2 0 =
let (ttl', ys') = wgoD 3 0
in (ttl' + 2, 2 + (ttl' + 2) * 9 : ys')
can only provide the required value for the locally bound ttl' in the wgoC 1 0 call by evaluating the first component of the return tuple ttl' + 2, but this requires forcing ttl' via the recursive wgoD 3 0 call. Because ttl' is subject to a function application \x -> x + 2 when it returns, we push a little more stack and proceed to evaluate:
wgoD 3 0 =
let (ttl', ys') = wgoD 4 0
in (ttl' + 3, 3 + (ttl' + 3) * 9 : ys')
To get the required ttl' as locally bound in the wgoD 2 0 call, we need to evaluate the first component of return tuple from wgoD 3 0, namely ttl' + 3. This is a function application \x -> x + 3 which we push on the stack, applied to ttl' returned from the recursive call wgoD 4 0.
So, Pinned3 pushes a sequence of continuations \x -> x + 2, \x -> x + 3, \x -> x + 4', etc. onto the stack, all in an effort to evaluate the first component of the tuple returned by wgoD 2 0, never getting an opportunity to pop anything until it gets to wgoD 100000000 0, and then it finally gets a number finalState = 0 as the first tuple component, if there's enough stack to get that far. Then all that stack will be popped off as the continuations are applied, and we'll have the first element of the list!
Once it gets through that, things aren't so bad. All of the expressions ttl' + n have been evaluated at this point, and they can be reused in calculating the expressions n + (ttl' + n) * 9, so the remaining elements can be generated relatively quickly, though -- since their values must be kept somewhere -- you'll also get accumulating heap usage at roughly the same rate as stack usage.
You can swap out the 100000000 for something like 10000000 (seven zeros), and that runs in a reasonable amount of time and shows accumulation of both stack and heap in a pyramid shape; it peaks at 1.4 gigs or so before dropping back down to zero.
I don't see any really straightforward way of "fixing" this while still keeping the algorithmic structure of Pinned3 intact.
I want to go through a matrix and check if any block of it is the same as a predefined unit. Here is my code. 'sd5' is the 2 by 2 predefined unit.
ALLOCATE (fList((n-1)**2,3))
fList = 0
p = 1
DO i = 1, n-1, 1
DO j = 1, n-1, 1
IF (TEST(i:i+1, j:j+1) == sd5) THEN
fList(p,1:3) = (i, j+1, 101) ! 101 should be replaced by submatrix number
END IF
p = p+1
END DO
END DO
The problem seems to be in the IF statement as four logical statements are returned in TEST(i:i+1, j:j+1) == sd5. I get this error:
Error: IF clause at (1) requires a scalar LOGICAL expression
I get another error:
fList(p,1:3) = (i, j+1, 101) ! 101 should be replaced by sub matrix number
1
Error: Expected PARAMETER symbol in complex constant at (1)
I do not understand this error, as all variables are integer which I defined.
First, if statements require scalar clauses.
(TEST(i:i+1, j:j+1) == sd5)
results in a 2x2 matrix containing .true. or .false.. Since you want to check all entries, the statement should read
IF ( all( TEST(i:i+1, j:j+1) == sd5) ) THEN
[ You could also use any if only one matching entry is sufficient. ]
The second statement is a little tricky, since you do not state what you want to achieve. As it is, it is not what you would expect. My guess is that you are trying to store a vector of length three, and the assignment should read
fList(p,1:3) = (/ i, j+1, 101 /)
or
fList(p,1:3) = [ i, j+1, 101 ]
The syntax you provided is in fact used to specify complex constants:
( Real, Imag )
In this form, Real and Imag need to be constants or literals themselves, cf. the Fortran 2008 Standard, R417.
I have this bit of code:
let rec random_list = function
| 0 -> []
| n -> ( Random.int max_int ) :: ( random_list (n-1) )
It compiles okay, but when I execute it, this error shows up:
exception Invalid_argument("Random.int")
What is the workaround for this issue ?
The documentation says:
Random.int bound returns a random integer between 0 (inclusive) and bound (exclusive). bound must be greater than 0 and less than 2^30.
So the closest to what you want is:
let my_max_int = (1 lsl 30) - 1 in
Random.int my_max_int
As gsg suggested, using Random.bits () is cleaner to get almost the same result (it can also return 2^30 - 1).
If you really want to get any positive native integer, maybe you could use Random.nativeint, but this means you would have to use the module Nativeint and Nativeint.t instead of int.
This is an interview problem I came across yesterday, I can think of a recursive solution but I wanna know if there's a non-recursive solution.
Given a number N, starting with number 1, you can only multiply the result by 5 or add 3 to the result. If there's no way to get N through this method, return "Can't generate it".
Ex:
Input: 23
Output: (1+3)*5+3
Input: 215
Output: ((1*5+3)*5+3)*5
Input: 12
Output: Can't generate it.
The recursive method can be obvious and intuitive, but are there any non-recursive methods?
I think the quickest, non recursive solution is (for N > 2):
if N mod 3 == 1, it can be generated as 1 + 3*k.
if N mod 3 == 2, it can be generated as 1*5 + 3*k
if N mod 3 == 0, it cannot be generated
The last statement comes from the fact that starting with 1 (= 1 mod 3) you can only reach numbers which are equals to 1 or 2 mod 3:
when you add 3, you don't change the value mod 3
a number equals to 1 mod 3 multiplied by 5 gives a number equals to 2 mod 3
a number equals to 2 mod 3 multiplied by 5 gives a number equals to 1 mod 3
The key here is to work backwards. Start with the number you want to reach and if it's divisible by 5 then divide by 5 because multiplication by 5 results in a shorter solution than addition by 3. The only exceptions are if the value equals 10, because dividing by 5 would yield 2 which is insolvable. If the number is not divisible by 5 or is equal to 10, subtract 3. This produces the shortest string
Repeat until you reach 1
Here is python code:
def f(x):
if x%3 == 0 or x==2:
return "Can't generate it"
l = []
while x!=1:
if x%5 != 0 or x==10:
l.append(3)
x -= 3
else:
l.append(5)
x /=5
l.reverse()
s = '1'
for v in l:
if v == 3:
s += ' + 3'
else:
s = '(' + s + ')*5'
return s
Credit to the previous solutions for determining whether a given number is possible
Model the problem as a graph:
Nodes are numbers
Your root node is 1
Links between nodes are *5 or +3.
Then run Dijkstra's algorithm to get the shortest path. If you exhaust all links from nodes <N without getting to N then you can't generate N. (Alternatively, use #obourgain's answer to decide in advance whether the problem can be solved, and only attempt to work out how to solve the problem if it can be solved.)
So essentially, you enqueue the node (1, null path). You need a dictionary storing {node(i.e. number) => best path found so far for that node}. Then, so long as the queue isn't empty, in each pass of the loop you
Dequeue the head (node,path) from the queue.
If the number of this node is >N, or you've already seen this node before with fewer steps in the path, then don't do any more on this pass.
Add (node => path) to the dictionary.
Enqueue nodes reachable from this node with *5 and +3 (together with the paths that get you to those nodes)
When the loop terminates, look up N in the dictionary to get the path, or output "Can't generate it".
Edit: note, this is really Breadth-first search rather than Dijkstra's algorithm, as the cost of traversing a link is fixed at 1.
You can use the following recursion (which is indeed intuitive):
f(input) = f(input/5) OR f(input -3)
base:
f(1) = true
f(x) = false x is not natural positive number
Note that it can be done using Dynamic Programming as well:
f[-2] = f[-1] = f[0] = false
f[1] = true
for i from 2 to n:
f[i] = f[i-3] or (i%5 == 0? f[i/5] : false)
To get the score, you need to get on the table after building it from f[n] and follow the valid true moves.
Time and space complexity of the DP solution is O(n) [pseudo-polynomial]
All recursive algorithms can also be implemented using a stack. So, something like this:
bool canProduce(int target){
Stack<int> numStack;
int current;
numStack.push(1);
while(!numStack.empty){
current=numStack.top();
numStack.pop();
if(current==target)
return true;
if(current+3 < target)
numStack.push(current+3);
if(current*5 < target)
numStack.push(current*5);
}
return false;
}
In Python:
The smart solution:
def f(n):
if n % 3 == 1:
print '1' + '+3' * (n // 3)
elif n % 3 == 2:
print '1*5' + '+3' * ((n - 5) // 3)
else:
print "Can't generate it."
A naive but still O(n) version:
def f(n):
d={1:'1'}
for i in range(n):
if i in d:
d[i*5] = '(' + d[i] + ')*5'
d[i+3] = d[i] + '+3'
if n in d:
print d[n]
else:
print "Can't generate it."
And of course, you could also use a stack to reproduce the behavior of the recursive calls.
Which gives:
>>> f(23)
(1)*5+3+3+3+3+3+3
>>> f(215)
(1)*5+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3+3
>>> f(12)
Can't generate it.
Let me start with an example -
I have a range of numbers from 1 to 9. And let's say the target number that I want is 29.
In this case the minimum number of operations that are required would be (9*3)+2 = 2 operations. Similarly for 18 the minimum number of operations is 1 (9*2=18).
I can use any of the 4 arithmetic operators - +, -, / and *.
How can I programmatically find out the minimum number of operations required?
Thanks in advance for any help provided.
clarification: integers only, no decimals allowed mid-calculation. i.e. the following is not valid (from comments below): ((9/2) + 1) * 4 == 22
I must admit I didn't think about this thoroughly, but for my purpose it doesn't matter if decimal numbers appear mid-calculation. ((9/2) + 1) * 4 == 22 is valid. Sorry for the confusion.
For the special case where set Y = [1..9] and n > 0:
n <= 9 : 0 operations
n <=18 : 1 operation (+)
otherwise : Remove any divisor found in Y. If this is not enough, do a recursion on the remainder for all offsets -9 .. +9. Offset 0 can be skipped as it has already been tried.
Notice how division is not needed in this case. For other Y this does not hold.
This algorithm is exponential in log(n). The exact analysis is a job for somebody with more knowledge about algebra than I.
For more speed, add pruning to eliminate some of the search for larger numbers.
Sample code:
def findop(n, maxlen=9999):
# Return a short postfix list of numbers and operations
# Simple solution to small numbers
if n<=9: return [n]
if n<=18: return [9,n-9,'+']
# Find direct multiply
x = divlist(n)
if len(x) > 1:
mults = len(x)-1
x[-1:] = findop(x[-1], maxlen-2*mults)
x.extend(['*'] * mults)
return x
shortest = 0
for o in range(1,10) + range(-1,-10,-1):
x = divlist(n-o)
if len(x) == 1: continue
mults = len(x)-1
# We spent len(divlist) + mults + 2 fields for offset.
# The last number is expanded by the recursion, so it doesn't count.
recursion_maxlen = maxlen - len(x) - mults - 2 + 1
if recursion_maxlen < 1: continue
x[-1:] = findop(x[-1], recursion_maxlen)
x.extend(['*'] * mults)
if o > 0:
x.extend([o, '+'])
else:
x.extend([-o, '-'])
if shortest == 0 or len(x) < shortest:
shortest = len(x)
maxlen = shortest - 1
solution = x[:]
if shortest == 0:
# Fake solution, it will be discarded
return '#' * (maxlen+1)
return solution
def divlist(n):
l = []
for d in range(9,1,-1):
while n%d == 0:
l.append(d)
n = n/d
if n>1: l.append(n)
return l
The basic idea is to test all possibilities with k operations, for k starting from 0. Imagine you create a tree of height k that branches for every possible new operation with operand (4*9 branches per level). You need to traverse and evaluate the leaves of the tree for each k before moving to the next k.
I didn't test this pseudo-code:
for every k from 0 to infinity
for every n from 1 to 9
if compute(n,0,k):
return k
boolean compute(n,j,k):
if (j == k):
return (n == target)
else:
for each operator in {+,-,*,/}:
for every i from 1 to 9:
if compute((n operator i),j+1,k):
return true
return false
It doesn't take into account arithmetic operators precedence and braces, that would require some rework.
Really cool question :)
Notice that you can start from the end! From your example (9*3)+2 = 29 is equivalent to saying (29-2)/3=9. That way we can avoid the double loop in cyborg's answer. This suggests the following algorithm for set Y and result r:
nextleaves = {r}
nops = 0
while(true):
nops = nops+1
leaves = nextleaves
nextleaves = {}
for leaf in leaves:
for y in Y:
if (leaf+y) or (leaf-y) or (leaf*y) or (leaf/y) is in X:
return(nops)
else:
add (leaf+y) and (leaf-y) and (leaf*y) and (leaf/y) to nextleaves
This is the basic idea, performance can be certainly be improved, for instance by avoiding "backtracks", such as r+a-a or r*a*b/a.
I guess my idea is similar to the one of Peer Sommerlund:
For big numbers, you advance fast, by multiplication with big ciphers.
Is Y=29 prime? If not, divide it by the maximum divider of (2 to 9).
Else you could subtract a number, to reach a dividable number. 27 is fine, since it is dividable by 9, so
(29-2)/9=3 =>
3*9+2 = 29
So maybe - I didn't think about this to the end: Search the next divisible by 9 number below Y. If you don't reach a number which is a digit, repeat.
The formula is the steps reversed.
(I'll try it for some numbers. :) )
I tried with 2551, which is
echo $((((3*9+4)*9+4)*9+4))
But I didn't test every intermediate result whether it is prime.
But
echo $((8*8*8*5-9))
is 2 operations less. Maybe I can investigate this later.