I would like to know how I can limit the maximum memory (heap) a process can use to allocate objects.
I looked into spawn_opt with max_heap_size as mentioned here but I keep getting a badarg error. Is this even possible now? I want to avoid using a gen server to kill the process when heapsize goes over a limit.
-module(maxMemCheck).
-export([fib/1,printfib/1]).
-export([main/0]).
printfib(N) ->
Res = maxMemCheck:fib(N),
io:fwrite("~w ~w~n", [N, Res]).
fib(0) -> 0 ;
fib(1) -> 1 ;
fib(N) when N > 0 -> fib(N-1) + fib(N-2) .
main () ->
spawn_opt(maxMemCheck,printfib,[10],[{max_heap_size,#{size => 300, kill => true, error_logger => true}}]).
This results in the error:
{"init terminating in do_boot",{badarg,[{erlang,spawn_opt,[maxMemCheck,fib,"\n",[{max_heap_size,#{error_logger=>true,kill=>true,size=>300}}]],[]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
Crash dump is being written to: erl_crash.dump...done
init terminating in do_boot ()
The syntax is correct, maybe you are using an "old" erlang version. This syntax uses a map variable which was introduced in OTP18 or 19.
Related
Issue
Following is a minimal, contrived example:
read :: FilePath -> Aff String
read f = do
log ("File: " <> f) -- (1)
readTextFile UTF8 f -- (2)
I would like to do some debug logging in (1), before a potential error on (2) occurs. Executing following code in Spago REPL works for success cases so far:
$ spago repl
> launchAff_ $ read "test/data/tree/root.txt"
File: test/data/tree/root.txt
unit
Problem: If there is an error with (2) - file is directory here - , (1) seems to be not executed at all:
$ spago repl
> launchAff_ $ read "test/data/tree"
~/purescript-book/exercises/chapter9/.psci_modules/node_modules/Effect.Aff/foreign.js:532
throw util.fromLeft(step);
^
[Error: EISDIR: illegal operation on a directory, read] {
errno: -21,
code: 'EISDIR',
syscall: 'read'
}
The original problem is more complex including several layers of recursions (see E-Book exercise 3), where I need logging to debug above error.
Questions
How can I properly log regardless upcoming errors here?
(Optional) Is there a more sophisticated, well-established debugging alternative - purescript-debugger? A decicated VS Code debug extension/functionality would be the cherry on the cake.
First of all, the symptoms you observe do not mean that the first line doesn't execute. It does always execute, you're just not seeing output from it due to how console works in the PureScript REPL. The output gets swallowed. Not the only problem with REPL, sadly.
You can verify that the first line is always executed by replacing log with throwError and observing that the error always gets thrown. Or, alternatively, you can make the first line modify a mutable cell instead of writing to the console, and then examine the cell's contents.
Finally, this only happens in REPL. If you put that launchAff_ call inside main and run the program, you will always get the console output.
Now to the actual question at hand: how to debug trace.
Logging to console is fine if you can afford it, but there is a more elegant way: Debug.trace.
This function has a hidden effect - i.e. its type says it's pure, but it really produces an effect when called. This little lie lets you use trace in a pure setting and thus debug pure code. No need for Effect! This is ok as long as used for debugging only, but don't put it in production code.
The way it works is that it takes two parameters: the first one gets printed to console and the second one is a function to be called after printing, and the result of the whole thing is whatever that function returns. For example:
calculateSomething :: Int -> Int -> Int
calculateSomething x y =
trace ("x = " <> show x) \_ ->
x + y
main :: Effect Unit
main =
log $ show $ calculateSomething 37 5
> npx spago run
'x = 37'
42
The first parameter can be anything at all, not just a string. This lets you easily print a lot of stuff:
calculateSomething :: Int -> Int -> Int
calculateSomething x y =
trace { x, y } \_ ->
x + y
> npx spago run
{ x: 37, y: 5 }
42
Or, applying this to your code:
read :: FilePath -> Aff String
read f = trace ("File: " <> f) \_ -> do
readTextFile UTF8 f
But here's a subtle detail: this tracing happens as soon as you call read, even if the resulting Aff will never be actually executed. If you need tracing to happen on effectful execution, you'll need to make the trace call part of the action, and be careful not to make it the very first action in the sequence:
read :: FilePath -> Aff String
read f = do
pure unit
trace ("File: " <> f) \_ -> pure unit
readTextFile UTF8 f
It is, of course, a bit inconvenient to do this every time you need to trace in an effectful context, so there is a special function that does it for you - it's called traceM:
read :: FilePath -> Aff String
read f = do
traceM ("File: " <> f)
readTextFile UTF8 f
If you look at its source code, you'll see that it does exactly what I did in the example above.
The sad part is that trace won't help you in REPL when an exception happens, because it's still printing to console, so it'll still get swallowed for the same reasons.
But even when it doesn't get swallowed, the output is a bit garbled, because trace actually outputs in color (to help you make it out among other output), and PureScript REPL has a complicated relationship with color:
> calculateSomething 37 5
←[32m'x = 37'←[39m
42
In addition to Fyodor Soikin's great answer, I found a variant using VS Code debug view.
1.) Make sure to build with sourcemaps:
spago build --purs-args "-g sourcemaps"
2.) Add debug configuration to VS Code launch.json:
{
"version": "0.2.0",
"configurations": [
{
"type": "pwa-node",
"request": "launch",
"name": "Launch Program",
"skipFiles": ["<node_internals>/**"],
"runtimeArgs": ["-e", "require('./output/Main/index.js').main()"],
"smartStep": true // skips files without (valid) source map
}
]
}
Replace "./output/Main/index.js" / .main() with the compiled .js file / function to be debugged.
3.) Set break points and step through the .purs file via sourcemap support.
The debugger can be programmatically invoked by executing (break). For example, the debugging banner then displays what caused the interrupt, the HELP line, the available restarts, some related info, and finally the source of the interrupt:
debugger invoked on a SIMPLE-CONDITION in thread
#<THREAD "main thread" RUNNING {10010B0523}>:
break
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [CONTINUE] Return from BREAK.
1: [ABORT ] Exit debugger, returning to top level.
#(
NODE: STATE=<NIL NIL NIL 0.0 0.0
( )> DEPTH=0)
#(
NODE: STATE=<NIL NIL NIL 0.0 0.0
((ACTIVE GATE1) (ACTIVE GATE2) (COLOR RECEIVER1 BLUE) (COLOR RECEIVER2 RED) (COLOR TRANSMITTER1 BLUE) (COLOR TRANSMITTER2 RED) (FREE ME) (LOC CONNECTOR1 AREA5) (LOC CONNECTOR2 AREA7) (LOC ME AREA5))> DEPTH=0)
(DF-BNB1 )
source: (BREAK)
0]
I don't understand the related info between the restarts and the source. Can this info be suppressed, as
sometimes it is many lines long in my application. I've tried changing the debug & safety optimization settings, but to no effect.
The output you are confused with is related to the place in the code where break was invoked. When I call it from the vanilla Lisp REPL (without SLIME), it displays:
(SB-INT:SIMPLE-EVAL-IN-LEXENV (BREAK) #<NULL-LEXENV>)
However, if I do something wrong in the debugger, here's what happens:
0] q
; in: PROGN (PRINT 1)
; (PROGN Q)
;
; caught WARNING:
; undefined variable: COMMON-LISP-USER::Q
;
; compilation unit finished
; Undefined variable:
; Q
; caught 1 WARNING condition
debugger invoked on a UNBOUND-VARIABLE in thread
#<THREAD "main thread" RUNNING {10005204C3}>:
The variable Q is unbound.
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [CONTINUE ] Retry using Q.
1: [USE-VALUE ] Use specified value.
2: [STORE-VALUE] Set specified value and use it.
3: [ABORT ] Reduce debugger level (to debug level 1).
4: Return from BREAK.
5: Exit debugger, returning to top level.
((LAMBDA (#:G498)) #<unused argument>)
source: (PROGN Q)
You can see that the last line resembles the output you got with the line starting at source:. Actually, the output we saw consists of 3 main parts:
1. Description of the condition
2. Listing of the available restarts
3. Debug REPL prompt printed by debug-loop-fun
The last output is part of the prompt and it is generated by the invocation of:
(print-frame-call *current-frame* *debug-io* :print-frame-source t)
So, you can recompile the call providing :print-frame-source nil or try to understand why your current frame looks this way...
I noticed that the base package use errorWithoutStackTrace to implement lots of functions. Is there some performance different between the following two definition?
head :: [a] -> a
head (x:_) = x
head [] = errorWithoutStackTrace ("Prelude.head: empty list")
head :: [a] -> a
head (x:_) = x
head [] = withFrozenCallStack $ error ("Prelude.head: empty list")
error means something bad happened so for most, if not all purposes, it does not matter how fast it is, because it indicates a program that's not working.
That said, a quick glance at the code is enough to reasonably guess that error does strictly more work than errorWithoutStackTrace (and that is compounded by the addition of withFrozenCallStack to the error variant of your code). Confirming that with benchmarks is left as an exercise to the reader.
Here's the definition of error and errorWithoutStackTrace:
https://hackage.haskell.org/package/base-4.12.0.0/docs/src/GHC.Err.html#error
error s = raise# (errorCallWithCallStackException s ?callStack)
errorWithoutStackTrace s = raise# (errorCallException s)
Now those two internal functions are defined as follows:
errorCallException :: String -> SomeException
errorCallException s = toException (ErrorCall s)
errorCallWithCallStackException :: String -> CallStack -> SomeException
errorCallWithCallStackException s stk = unsafeDupablePerformIO $ do
...
return $ toException (ErrorCallWithLocation s stack)
Note that both essentially do toException (something s), but errorCallWithCallStackException also has a whole lot more code to handle the stack (in "...").
In Julia, I want to use addprocs and pmap inside a function that is defined inside a module. Here's a silly example:
module test
using Distributions
export g, f
function g(a, b)
a + rand(Normal(0, b))
end
function f(A, b)
close = false
if length(procs()) == 1 # If there are already extra workers,
addprocs() # use them, otherwise, create your own.
close = true
end
W = pmap(x -> g(x, b), A)
if close == true
rmprocs(workers()) # Remove the workers you created.
end
return W
end
end
test.f(randn(5), 1)
This returns a very long error
WARNING: Module test not defined on process 4
WARNING: Module test not defined on process 3
fatal error on fatal error on WARNING: Module test not defined on process 2
43: : WARNING: Module test not defined on process 5
fatal error on fatal error on 5: 2: ERROR: UndefVarError: test not defined
in deserialize at serialize.jl:504
in handle_deserialize at serialize.jl:477
in deserialize at serialize.jl:696
...
in message_handler_loop at multi.jl:878
in process_tcp_streams at multi.jl:867
in anonymous at task.jl:63
Worker 3 terminated.
Worker 2 terminated.ERROR (unhandled task failure): EOFError: read end of file
WARNING: rmprocs: process 1 not removed
Worker 5 terminated.ERROR (unhandled task failure): EOFError: read end of file
4-element Array{Any,1}:Worker 4 terminated.ERROR (unhandled task failure): EOFError: read end of file
ERROR (unhandled task failure): EOFError: read end of file
ProcessExitedException()
ProcessExitedException()
ProcessExitedException()
ProcessExitedException()
What I'm trying to do is write a package that contains functions that perform operations that can be optionally parallelized at the user's discretion. So a function like f might take an argument par::Bool that does something like I've shown above if the user calls f with par = true and loops otherwise. So from within the definition of f (and within the definition of the module test), I want to create workers and broadcast the Distributions package and the function g to them.
What's wrong with using #everywhere in your function? The following, for example, works fine on my computer.
function f(A, b)
close = false
if length(procs()) == 1 # If there are already extra workers,
addprocs() # use them, otherwise, create your own.
#everywhere begin
using Distributions
function g(a, b)
a + rand(Normal(0, b))
end
end
close = true
end
W = pmap(x -> g(x, b), A)
if close == true
rmprocs(workers()) # Remove the workers you created.
end
return W
end
f(randn(5), 1)
Note: when I first ran this, I needed to recompile the Distributions package since it had been updated since I had last used it. When I first tried the above script right after the recompiling, it failed. But, I then quit Julia and reopened it and it worked fine. Perhaps that was what is causing your error?
Is there any way to view the reduction steps in haskell, i.e trace the recursive function calls made? For example, chez scheme provides us with trace-lambda. Is there an equivalent form in Haskell?
You could try inserting Debug.Trace.trace in places you want to trace, but this has the tendency of (a) producing wildly out-of-order output, as your trace statement may belong to a thunk that isn't evaluated until far far away from the original call, and (b) changing the runtime behavior of your program, if tracing requires evaluating things that wouldn't otherwise have been evaluated (yet).
Is this for debugging? If so...
Hat modifies your source code to output tracing which can be viewed after running. The output should be pretty close to what you want: the example on their homepage is
For example, the computation of the faulty program
main = let xs :: [Int]
xs = [4*2,5 `div` 0,5+6]
in print (head xs,last' xs)
last' (x:xs) = last' xs
last' [x] = x
gives the result
(8, No match in pattern.
and the Hat viewing tools can be used to explore its behaviour as follows:
Hat-stack
For aborted computations, that is computations that terminated with an error message or were interrupted, hat-stack shows in which function call the computation was aborted. It does so by showing a virtual stack of function calls (redexes). Thus, every function call shown on the stack caused the function call above it. The evaluation of the top stack element caused the error (or during its evaluation the computation was interrupted). The stack shown is virtual, because it does not correspond to the actual runtime stack. The actual runtime stack enables lazy evaluation whereas the virtual stack corresponds to a stack that would be used for eager (strict) evaluation.
Using the same example program as above, hat-stack shows
$ hat-stack Example
Program terminated with error:
No match in pattern.
Virtual stack trace:
(Last.hs:6) last' []
(Last.hs:6) last' [_]
(Last.hs:6) last' [_,_]
(Last.hs:4) last' [8,_,_]
(unknown) main
$
These days, GHCi (≥6.8.1) also comes with a debugger:
$ ghci -fbreak-on-exception
GHCi, version 6.10.1: http://www.haskell.org/ghc/ :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer ... linking ... done.
Loading package base ... linking ... done.
Prelude> :l Example.hs
[1 of 1] Compiling Main ( Example.hs, interpreted )
Example.hs:5:0:
Warning: Pattern match(es) are overlapped
In the definition of `last'': last' [x] = ...
Ok, modules loaded: Main.
*Main> :trace main
(8,Stopped at <exception thrown>
_exception :: e = _
[<exception thrown>] *Main> :back
Logged breakpoint at Example.hs:(5,0)-(6,12)
_result :: t
[-1: Example.hs:(5,0)-(6,12)] *Main> :hist
-1 : last' (Example.hs:(5,0)-(6,12))
-2 : last' (Example.hs:5:15-22)
-3 : last' (Example.hs:(5,0)-(6,12))
-4 : last' (Example.hs:5:15-22)
-5 : last' (Example.hs:(5,0)-(6,12))
-6 : last' (Example.hs:5:15-22)
-7 : last' (Example.hs:(5,0)-(6,12))
-8 : main (Example.hs:3:25-32)
-9 : main (Example.hs:2:17-19)
-10 : main (Example.hs:2:16-34)
-11 : main (Example.hs:3:17-23)
-12 : main (Example.hs:3:10-33)
<end of history>
[-1: Example.hs:(5,0)-(6,12)] *Main> :force _result
*** Exception: Example.hs:(5,0)-(6,12): Non-exhaustive patterns in function last'
[-1: Example.hs:(5,0)-(6,12)] *Main> :back
Logged breakpoint at Example.hs:5:15-22
_result :: t
xs :: [t]
[-2: Example.hs:5:15-22] *Main> :force xs
xs = []
While not as nice, it has the benefit of being easily available, and being usable without recompiling your code.
There's a reduction count in hugs, if that helps?
Alternatively, could you use something like the hugs hood to wrap your code, to get more detail around what it's doing at each step?
Nothing of the kind is built into the Haskell standard.
I would hope that the Helium graphical interpreter would offer something like this, but the web page is silent on the topic.
A partial solution is to use vacuum to visualize data structures.
I've seen some gif animations of fold, scan and others, but I can't find them at the moment. I think Cale Gibbard made the animations.