How to implement breakpoints in Lua using the standard hooks? - debugging

Question

This question is motivated by Exercise 25.7 on p. 264 of Programming in Lua (4th ed.), and more specifically, the optimization proposed in the hint (I've emphasized it in the quote below):
Exercise 25.7: Write a library for breakpoints. It should offer at least two functions
setbreakpoint(function, line) --> returns handle
removebreakpoint(handle)
We specify a breakpoint by a function and a line inside that function. When the program hits a breakpoint, the library should call debug.debug. (Hint: for a basic implementation, use a line hook that checks whether it is in a breakpoint; to improve performance, use a call hook to trace program execution and only turn on the line hook when the program is running the target function.)
I can't figure out how to implement the optimization described in the hint.
Consider the following code (this is, of course, an artificial example concocted only for the sake of this question):
function tweedledum ()
while true do
local ticket = math.random(1000)
if ticket % 5 == 0 then tweedledee() end
if ticket % 17 == 0 then break end
end
end
function tweedledee ()
while true do
local ticket = math.random(1000)
if ticket % 5 == 0 then tweedledum() end
if ticket % 17 == 0 then break end
end
end
function main ()
tweedledum()
end
Function main is supposed to represent the program's entrypoint. Functions tweedledum and tweedledee are almost identical to each other, and do little more than invoke each other repeatedly.
Suppose I set a breakpoint on tweedledum's assignment line. I can implement a call hook can check whether tweedledum has been invoked, and then sets a line hook that will check when the desired line is being invoked1.
More likely than not, tweedledum will invoke tweedledee before it breaks out of its loop. Suppose that this happens. The currently enabled line hook can detect that it is no longer in tweedledum, and re-install the call hook.
At this point the execution can switch from tweedledee to tweedledum in one of two ways:
tweedledee can invoke tweedledum (yet again);
tweedledee can return to its invoker, which happens to be tweedledum.
And here's the problem: the call hook can detect the event in (1) but it cannot detect the event in (2).
Granted, this example is very artificial, but it's the simplest way I could come up with to illustrate the problem.
The best approach I can think of (and it's very weak!) is to keep track of the stack depth N at the first invokation of tweedledum, have the line hook reinstall the call hook only when the stack depth sinks below N. Thus, the line hook will be in force as long as tweedledee is in the stack, whether it is being executed or not.
Is it possible to implement the optimization described in the hint using only the standard hooks available in Lua?2
1 My understanding is that, by installing the line hook, the call hook essentially uninstalls itself. AFAICT, only one hook can be active per coroutine. Please do correct me if I am wrong.
2 Namely: call, line, return, and count hooks.

Answer 1

And here's the problem: the call hook can detect the event in (1) but it cannot detect the event in (2).
And that's where you're wrong: There's three possible hook events: l for line, c for call and r for return.
Inside your hook function you can treat return and call events as almost the same, except when the return event is fired, you're still inside the called function, so the target function is one place higher in the stack.
debug.sethook(function(event, line)
if event == "call" or event == "return" then
if debug.getinfo(event=='call' and 2 or 3).func == target then
debug.sethook(debug.gethook(), 'crl')
else
debug.sethook(debug.gethook(), 'cr')
end
elseif event == 'line' then
-- Check if the line is right and possibly call debug.debug() here
end
end, 'cr')
It's all in the manual ;)
Note that, when setting the hook, you may need to check if you're currently inside the target function; otherwise you may skip a break point unless you call (and return from) another function before reaching it.