I feel that understanding this subtlety might help me to understand how scope works in Scheme.
So why does Scheme error out if you try to do something like:
(define (func n)
(define n (+ 1 n))
n)
It only errors out at runtime when calling the function.
The reason I find it strange is because Scheme does allow re-definitions, even inside functions. For example this gives no error and will always return the value 5 as expected:
(define (func n)
(define n 5)
n)
Additionally, Scheme also seems to support self-re-definition in global space. For instance:
(define a 5)
(define a (+ 1 a))
gives no error and results in "a" displaying "6" as expected.
So why does the same thing give a runtime error when it occurs inside a function (which does support re-definition)? In other words: Why does self-re-definition only not work when inside of a function?
global define
First off, top level programs are handled by a different part of the implementation than in a function and defining an already defined variable is not allowed.
(define a 10)
(define a 20) ; ERROR: duplicate definition for identifier
It might happen that it works in a REPL since it's common to redefine stuff, but when running programs this is absolutely forbidden. In R5RS and before what happened is underspecified and didn't care since be specification by violating it it no longer is a Scheme program and implementers are free to do whatever they want. The result is of course that a lot of underspecified stuff gets implementation specific behaviour which are not portable or stable.
Solution:
set! mutates bindings:
(define a 10)
(set! a 20)
define in a lambda (function, let, ...)
A define in a lambda is something completely different, handled by completely different parts of the implementation. It is handled by the macro/special form lambda so that it is rewritten to a letrec*
A letrec* or letrec is used for making functions recursive so the names need to be available at the time the expressions are evaluated. Because of that when you define n it has already shadowed the n you passed as argument. In addition from R6RS implementers are required to signal an error when a binding is evaluated that is not yet initialized and that is probably what happens. Before R6RS implementers were free to do whatever they wanted:
(define (func n)
(define n (+ n 1)) ; illegal since n hasn't got a value yet!
n)
This actually becomes:
(define func
(lambda (n)
(let ((n 'undefined-blow-up-if-evaluated))
(let ((tmpn (+ n 1)))
(set! n tmpn))
n)))
Now a compiler might see that it violates the spec at compile time, but many implementations doesn't know before it runs.
(func 5) ; ==> 42
Perfectly fine result in R5RS if the implementers have good taste in books.
The difference in the version you said works is that this does not violate the rule of evaluating n before the body:
(define (func n)
(define n 5)
n)
becomes:
(define func
(lambda (n)
(let ((n 'undefined-blow-up-if-evaluated))
(let ((tmpn 5)) ; n is never evaluated here!
(set! n tmpn))
n)))
Solutions
Use a non conflicting name
(define (func n)
(define new-n (+ n 1))
new-n)
Use let. It does not have its own binding when the expression gets evaluated:
(define (func n)
(let ((n (+ n 1)))
n))
Related
Footnote #28 of SICP says the following:
Embedded definitions must come first in a procedure body. The management is not responsible for the consequences of running programs that intertwine definition and use.
What exactly does this mean? I understand:
"definitions" to be referring to both procedure creations and value assignments.
"a procedure body" to be as defined in section 1.1.4. It's the code that comes after the list of parameters in a procedure's definition.
"Embedded definitions must come first in a procedure body" to mean 'any definitions that are made and used in a procedure's body must come before anything else'.
"intertwine definition and use" to mean 'calling a procedure or assigned value before it has been defined;'
However, this understanding seems to contradict the answers to this question, which has answers that I can summarise as 'the error that your quote is referring to is about using definitions at the start of a procedure's body that rely on definitions that are also at the start of that body'. This has me triply confused:
That interpretation clearly contradicts what I've said above, but seems to have strong evidence - compiler rules - behind it.
SICP seems happy to put definition in a body with other definitions that use them. Just look at the sqrt procedure just above the footnote!
At a glance, it looks to me that the linked question's author's real error was treating num-prod like a value in their definition of num rather than as a procedure. However, the author clearly got it working, so I'm probably wrong.
So what's really going on? Where is the misunderstanding?
In a given procedure's definition / code,
"internal definitions" are forms starting with define.
"a procedure body" is all the other forms after the forms which start with define.
"embedded definitions must come first in a procedure body" means all the internal define forms must go first, then all the other internal forms. Once a non-define internal form appears, there can not appear an internal define form after that.
"no intertwined use" means no use of any name before it's defined. Imagine all internal define forms are gathered into one equivalent letrec, and follow its rules.
Namely, we have,
(define (proc args ...)
;; internal, or "embedded", definitions
(define a1 ...init1...)
(define a2 ...init2...)
......
(define an ...initn...)
;; procedure body
exp1 exp2 .... )
Any ai can be used in any of the initj expressions, but only inside a lambda expression.(*) Otherwise it would refer to the value of ai while aj is being defined, and that is forbidden, because any ai names are considered not yet defined while any of the initj expressions are being evaluated.
(*) Remember that (define (foo x) ...x...) is the same as (define foo (lambda (x) ...x...)). That's why the definitions in that sqrt procedure in the book you mention are OK -- they are all lambda expressions, and any name's use inside a lambda expression will only actually refer to that name's value when the lambda expression's value -- a lambda function -- will be called, not when that lambda expression is being evaluated, producing that lambda function which is its value.
The book is a bit vague at first with the precise semantics of its language but in Scheme the above code is equivalent to
(define proc
(lambda (args ...)
;; internal, or "embedded", definitions
(letrec ( (a1 ...init1...)
(a2 ...init2...)
......
(an ...initn...) )
;; procedure body
exp1 exp2 ....
)))
as can be seen explained, for instance, here, here or here.
For example,
;; or, equivalently,
(define (my-proc x) (define my-proc
(lambda (x)
(define (foo) a) (letrec ( (foo (lambda () a))
(define a x) (a x) )
;; my-proc's body ;; letrec's body
(foo)) (foo))))
first evaluates the lambda expression, (lambda () a), and binds the name foo to the result, a function; that function will refer to the value of a when (foo) will be called, so it's OK that there's a reference to a inside that lambda expression -- because when that lambda expression is evaluated, no value of a is immediately needed, just the reference to its future value, under the name a, is present there; i.e. the value that a will have after all the names in that letrec are initialized, and the body of letrec is entered. Or in other words, when all the internal defines are completed and the body proper of the procedure my-proc is entered.
So we see that foo is defined, but is not used during the initializations; a is defined but is not used during the initializations; thus all is well. But if we had e.g.
(define (my-proc x)
(define (foo) x) ; `foo` is defined as a function
(define a (foo)) ; `foo` is used by being called as a function
a)
then here foo is both defined and used during the initializations of the internal, or "embedded", defines; this is forbidden in Scheme. This is what the book is warning about: the internal definitions are only allowed to define stuff, but its use should be delayed for later, when we're done with the internal defines and enter the full procedure's body.
This is subtle, and as the footnote and the question you reference imply, the subtlties can vary depending on a particular language's implementation.
These issues will be covered in far more detail later in the book (Chapters 3 and 4) and, generally, the text avoids using internal definitions so that these issues can be avoided until they are examined in detail.
A key difference between between the code above the footnote:
(define (sqrt x)
(define (good-enough? guess)
(< (abs (- (square guess) x)) 0.001))
(define (improve guess)
(average guess (/ x guess)))
(define (sqrt-iter guess)
(if (good-enough? guess)
guess
(sqrt-iter (improve guess))))
(sqrt-iter 1.0))
and the code in the other question:
(define (pi-approx n)
(define (square x) (* x x))
(define (num-prod ind) (* (* 2 ind) (* 2 (+ ind 1)))) ; calculates the product in the numerator for a certain term
(define (denom-prod ind) (square (+ (* ind 2 ) 1))) ;Denominator product at index ind
(define num (product num-prod 1 inc n))
(define denom (product denom-prod 1 inc n))
is that all the defintions in former are procedure definitions, whereas num and denom are value defintions. The body of a procedure is not evaluated until that procedures is called. But a value definition is evaluated when the value is assigned.
With a value definiton:
(define sum (add 2 2))
(add 2 2) will evaluated when the definition is evaluated, if so add must already be defined. But with a procedure definition:
(define (sum n m) (add n m))
a procedure object will be assigned to sum but the procedure body is not yet evaluated so add does not need be defined when sum is defined, but must be by the time sum is called:
(sum 2 2)
As I say there is a lot sublty and a lot of variation so I'm not sure if the following is always true for every variation of scheme, but within 'SICP scheme' you can say..
Valid (order of evaluation of defines not significant):
;procedure body
(define (sum m n) (add m n))
(define (add m n) (+ m n))
(sum 2 2)
Also valid:
;procedure body
(define (sum) (add 2 2))
(define (add m n) (+ m n))
(sum)
Usually invalid (order of evaluation of defines is significant):
;procedure body
(define sum (add 2 2))
(define (add m n) (+ m n))
Whether the following is valid depends on the implementation:
;procedure body
(define (add m n) (+ m n))
(define sum (add 2 2))
And finally an example of intertwining definition and use, whether this works also depends on the implementation. IIRC, this would work with Scheme described in Chapter 4 of the book if the scanning out has been implemented.
;procedure body
(sum)
(define (sum) (add 2 2))
(define (add m n) (+ m n))
It is complicated and subtle, but the key points are:
value definitions are evaluated differently from procedure definitions,
behaviour inside blocks may be different from outside blocks,
there is variation between scheme implementations,
the book is designed so you don't have to worry much about this until Chapter 3,
the book will cover this in detail in Chapter 4.
You discovered one of the difficulties of scheme. And of lisp. Due to this kind of chicken and egg issue there is no single lisp, but lots of lisps appeared.
If the binding forms are not present in code in a letrec-logic in R5RS and letrec*-logic in R6RS, the semantics is undefined. Which means, everything depend on the will of the implementor of scheme.
See the paper Fixing Letrec: A Faithful Yet Efficient Implementation of Scheme’s Recursive Binding Construct.
Also, you can read the discussions from the mailing list from 1986, when there was no general consensus among different implementors of scheme.
Also, at MIT there were developed 2 versions of scheme -- the student version and the researchers' development version, and they behaved differently concerning the order of define forms.
I'm trying to understand the scheme code of make-counter procedure. It's a higher order procedure (a procedure outputs another procedure) and I'm stuck with it.
(define make-counter
(lambda (n)
(lambda ()
(set! n (+ n 1))
n)))
(define ca (make-counter 0))
(ca)
(ca)
This outputs 1 and 2 respectively as expected. Why do we need 2 nested procedures here? What are their functions individually?
I'd be appreciated if someone explains in details. Thanks from now on.
Indented properly, this is:
(define make-counter
(lambda (n)
(lambda ()
(set! n (+ n 1))
n)))
By the way, you can use a different syntax:
(define (make-counter n)
(lambda ()
(set! n (+ n 1))
n))
make-counter is a function that accepts a number n and returns an object called closure, which acts like a function but contains a state. Different invocations of make-counter will produce different closures, even when given the same n in argument. A closure can be called using the function-call syntax, as you experimented.
When you call the closure, the code that is contained within is executed. In your example, the closure accepts zero arguments, and mutates the variable named n. Again, the binding from n to a value is local to the closure and different for all instances of counters. But inside a particular counter, n always refer to the same memory location.
A call to the set! function changes what n evaluates to, and replaces the previous value with (+ n 1), incrementing the local counter variable.
I would like to know if there is a style guideline, published by ANSI or implementation authors or another influential authority, for Lisp functions which are implemented using recursive helper functions which take additional parameters that the person calling the function doesn't need to think about. Here is the simplest example I can think of. Which of these three, if any, is preferred in a standardized style guide for common lisp (if there is one)?
(defun factorial (n)
(defun tail-factorial (n m)
(if (= n 1)
m
(tail-factorial (- n 1) (* n m))))
(tail-factorial n 1))
This seems unpleasant because of the function declaration within the function declaration. I would use a lambda to avoid naming the helper function and convoluting things, but I am not clear how to recurse within a lambda expression by having it call itself. Even if there is a way to have an anonymous function call itself, it seems like it would get messy, especially if the helper needs a helper needs a helper ...
Another alternative is to declare the tail guy first (or after, but that makes sbcl complain):
(defun tail-factorial (n m)
(if (= n 1)
n
(tail-factorial (- n 1) (* n m))))
(defun factorial (n)
(tail-factorial n 1))
This seems unpleasant because somebody reading my code would see tail-factorial and might feel the need to understand it, even though it's just a helper function for factorial and won't be used elsewhere. In my own code, I keep writing functions analogous to this and I really struggle to come up with comments that will make me re-understand what I did months or years from now. Again, the situation gets really bad here when the helper needs a helper needs a ...
Another alternative uses optionals:
(defun factorial (n &optional (m 1))
(if (= n 1)
m
(factorial (- n 1) (* n m))))
This seems unpleasant because we expect only one argument for a factorial. Nobody outside this function would have a reason to call this with that second argument. I find it annoying to try to understand code with optional arguments I will never use.
Now, I'm clear that asking what you think is best is the kind of subjective conversation stackoverflow dislikes, so my objective question is whether or not there is some kind of standardized style guideline about which of these alternatives is preferred. I use SBCL and have not found a guideline for this sort of thing on their site, and I am not aware of such a guide published by ANSI or any other standardizing body, including other implementors.
Perhaps there is another alternative altogether, and I welcome your suggestions. I keep running into this situation of needing a little helper function (who needs a helper function, etc) and I want to get used to writing in the way most people will find clear. Somebody asked a similar question at Recursing in a lambda function , and several people recommended some favorite pet projects but nobody mentioned a style guideline.
Thanks in advance!
defun only makes global functions. If you use defun inside defun you are actually making a new global function at each time you call it. To make a local lexical function to only be used inside factorial the standard way is with labels
(defun factorial (n)
(labels ((tail-factorial (n m)
(if (= n 1)
m
(tail-factorial (- n 1) (* n m)))))
(tail-factorial n 1)))
Common Lisp has no guarantee to do tail call optimization so this might blow the stack. For serious work you would have used the loop macro to do this.
Optional arguments works, but you really shouldn't make internals optional if it's not the intention that the user would ever want to supply it. Then it's just leaking abstraction and making confusing documentation.
I think that some of the answers provide more general answers, but I do want to point out that some of the most common uses of tail-recursive helper functions are often very cleanly expressed using a do loop. With a do loop, you declare the variables, their initial values, a step form (the value for the next iteration), a termination condition, and a result form. (Some of those are optional, but we'll use all of them here.) In that sense, a do is more like Scheme's named let, except that you don't end up putting a recursive call in the body. In fact, many times, you end up not needing a body at all. For the factorial function, your code could be expressed as:
(defun factorial (n)
(do ((n n (1- n)) ; n starts as n, and is (1- n) on next iteration
(m 1 (* n m))) ; m starts at 1, and is (* m n) on next iteration
((= 1 n) m))) ; if (= 1 n), then return m, else go to next iteration
CL-USER> (factorial 11)
39916800
Speaking of named let, you can implement it pretty easily in terms of labels in Common Lisp:
(defmacro nlet (name bindings &body body)
`(labels ((,name ,(mapcar 'first bindings)
,#body))
(,name ,#(mapcar 'second bindings))))
CL-USER> (macroexpand-1 '(nlet factorial ((n n) (m 1))
(if (= n 1) m
(factorial (1- n) (* m n)))))
(LABELS ((FACTORIAL (N M)
(IF (= N 1)
M
(FACTORIAL (1- N) (* M N)))))
(FACTORIAL N 1))
That still has the issue that Common Lisp implementations don't have to support tail recursion, though, but it can be a useful utility if you're coming from a Scheme background. You could implement a nlet type macro that makes the recursive call go to the next iteration, but that wouldn't be quite the same, since you only want the "reuse stack frame" behavior in a tail position; in a non-tail position, you'd still want the proper return.
A very subtle bug in the implementation above
As Sylwester pointed out in the comments, there's a very subtle bug in the implementation of nlet above. Because the call to the local function happens within the lexical scope where it's defined, doing something like
(nlet frob ((f #'frob))
...)
makes the initializer #'foo a reference to the new local function. That's probably not what you'd want, and it's not what Scheme's named let does. You can get around this by returning the local function from the labels form and calling it outside of that scope:
(defmacro nlet (name bindings &body body)
`(funcall (labels ((,name ,(mapcar 'first bindings)
,#body))
#',name)
,#(mapcar 'second bindings)))
CL-USER> (macroexpand-1 '(nlet factorial ((n n) (m 1))
(if (= n 1) m
(factorial (1- n) (* m n)))))
(FUNCALL
(LABELS ((FACTORIAL (N M)
(IF (= N 1)
M
(FACTORIAL (1- N) (* M N)))))
#'FACTORIAL)
N 1)
(defmacro nlet (name bindings &body body)
`(funcall (labels ((,name ,(mapcar 'first bindings)
,#body))
#',name)
,#(mapcar 'second bindings)))
Good style is:
useful function name
documentation string
argument type checking
no sources of control stack overflows in portable code -> mostly avoid tail-recursive code
Example:
(defun factorial (n &aux (result 1))
"Calculates the factorial N! for N >= 0"
(check-type n (integer 0 *))
(loop for i from 1 to n
do (setf result (* result i)))
result)
Using:
CL-USER 94 > (apropos "factorial")
FACTORIAL (defined)
CL-USER 95 > (documentation 'factorial 'function)
"Calculates the factorial N! for N >= 0"
CL-USER 96 > (factorial 7)
5040
In an effort to find a simple example of CPS which doesn't give me a headache , I came across this Scheme code (Hand typed, so parens may not match) :
(define fact-cps
(lambda(n k)
(cond
((zero? n) (k 1))
(else
(fact-cps (- n 1)
(lambda(v)
(k (* v n))))))))
(define fact
(lambda(n)
(fact-cps n (lambda(v)v)))) ;; (for giggles try (lambda(v)(* v 2)))
(fact 5) => 120
Great, but Scheme isn't Common Lisp, so I took a shot at it:
(defun not-factorial-cps(n k v)
(declare (notinline not-factorial-cps)) ;; needed in clisp to show the trace
(cond
((zerop n) (k v))
((not-factorial-cps (1- n) ((lambda()(setq v (k (* v n))))) v))))
;; so not that simple...
(defun factorial(n)
(not-factorial-cps n (lambda(v)v) 1))
(setf (symbol-function 'k) (lambda(v)v))
(factorial 5) => 120
As you can see, I'm having some problems, so although this works, this has to be wrong. I think all I've accomplished is a convoluted way to do accumulator passing style. So other than going back to the drawing board with this, I had some questions: Where exactly in the Scheme example is the initial value for v coming from? Is it required that lambda expressions only be used? Wouldn't a named function accomplish more since you could maintain the state of each continuation in a data structure which can be manipulated as needed? Is there in particular style/way of continuation passing style in Common Lisp with or without all the macros? Thanks.
The problem with your code is that you call the anonymous function when recurring instead of passing the continuation like in the Scheme example. The Scheme code can easily be made into Common Lisp:
(defun fact-cps (n &optional (k #'values))
(if (zerop n)
(funcall k 1)
(fact-cps (- n 1)
(lambda (v)
(funcall k (* v n))))))
(fact-cps 10) ; ==> 3628800
Since the code didn't use several terms or the implicit progn i switched to if since I think it's slightly more readable. Other than that and the use of funcall because of the LISP-2 nature of Common Lisp it's the identical code to your Scheme version.
Here's an example of something you cannot do tail recursively without either mutation or CPS:
(defun fmapcar (fun lst &optional (k #'values))
(if (not lst)
(funcall k lst)
(let ((r (funcall fun (car lst))))
(fmapcar fun
(cdr lst)
(lambda (x)
(funcall k (cons r x)))))))
(fmapcar #'fact-cps '(0 1 2 3 4 5)) ; ==> (1 1 2 6 24 120)
EDIT
Where exactly in the Scheme example is the initial value for v coming
from?
For every recursion the function makes a function that calls the previous continuation with the value from this iteration with the value from the next iteration, which comes as an argument v. In my fmapcar if you do (fmapcar #'list '(1 2 3)) it turns into
;; base case calls the stacked lambdas with NIL as argument
((lambda (x) ; third iteration
((lambda (x) ; second iteration
((lambda (x) ; first iteration
(values (cons (list 1) x)))
(cons (list 2) x)))
(cons (list 3) x))
NIL)
Now, in the first iteration the continuation is values and we wrap that in a lambda together with consing the first element with the tail that is not computed yet. The next iteration we make another lambda where we call the previous continuation with this iterations consing with the tail that is not computed yet.. At the end we call this function with the empty list and it calls all the nested functions from end to the beginning making the resulting list in the correct order even though the iterations were in oposite order from how you cons a list together.
Is it required that lambda expressions only be used? Wouldn't a named
function accomplish more since you could maintain the state of each
continuation in a data structure which can be manipulated as needed?
I use a named function (values) to start it off, however every iteration of fact-cps has it's own free variable n and k which is unique for that iteration. That is the data structure used and for it to be a named function you'd need to use flet or labels in the very same scope as the anonymous lambda functions are made. Since you are applying previous continuation in your new closure you need to build a new one every time.
Is there in particular style/way of continuation passing style in
Common Lisp with or without all the macros?
It's the same except for the dual namespace. You need to either funcall or apply. Other than that you do it as in any other language.
I have to define a variadic function in Scheme that takes the following form:
(define (n-loop procedure [a list of pairs (x,y)]) where the list of pairs can be any length.
Each pair specifies a lower and upper bound. That is, the following function call: (n-loop (lambda (x y) (inspect (list x y))) (0 2) (0 3)) produces:
(list x y) is (0 0)
(list x y) is (0 1)
(list x y) is (0 2)
(list x y) is (1 0)
(list x y) is (1 1)
(list x y) is (1 2)
Obviously, car and cdr are going to have to be involved in my solution. But the stipulation that makes this difficult is the following. There are to be no assignment statements or iterative loops (while and for) used at all.
I could handle it using while and for to index through the list of pairs, but it appears I have to use recursion. I don't want any code solutions, unless you feel it is necessary for explanation, but does anyone have a suggestion as to how this might be attacked?
The standard way to do looping in Scheme is to use tail recursion. In fact, let's say you have this loop:
(do ((a 0 b)
(b 1 (+ a b))
(i 0 (+ i 1)))
((>= i 10) a)
(eprintf "(fib ~a) = ~a~%" i a))
This actually get macro-expanded into something like the following:
(let loop ((a 0)
(b 1)
(i 0))
(cond ((>= i 10) a)
(else (eprintf "(fib ~a) = ~a~%" i a)
(loop b (+ a b) (+ i 1)))))
Which, further, gets macro-expanded into this (I won't macro-expand the cond, since that's irrelevant to my point):
(letrec ((loop (lambda (a b i)
(cond ((>= i 10) a)
(else (eprintf "(fib ~a) = ~a~%" i a)
(loop b (+ a b) (+ i 1)))))))
(loop 0 1 0))
You should be seeing the letrec here and thinking, "aha! I see recursion!". Indeed you do (specifically in this case, tail recursion, though letrec can be used for non-tail recursions too).
Any iterative loop in Scheme can be rewritten as that (the named let version is how loops are idiomatically written in Scheme, but if your assignment won't let you use named let, expand one step further and use the letrec). The macro-expansions I've described above are straightforward and mechanical, and you should be able to see how one gets translated to the other.
Since your question asked how about variadic functions, you can write a variadic function this way:
(define (sum x . xs)
(if (null? xs) x
(apply sum (+ x (car xs)) (cdr xs))))
(This is, BTW, a horribly inefficient way to write a sum function; I am just using it to demonstrate how you would send (using apply) and receive (using an improper lambda list) arbitrary numbers of arguments.)
Update
Okay, so here is some general advice: you will need two loops:
an outer loop, that goes through the range levels (that's your variadic stuff)
an inner loop, that loops through the numbers in each range level
In each of these loops, think carefully about:
what the starting condition is
what the ending condition is
what you want to do at each iteration
whether there is any state you need to keep between iterations
In particular, think carefully about the last point, as that is how you will nest your loops, given an arbitrary number of nesting levels. (In my sample solution below, that's what the cur variable is.)
After you have decided on all these things, you can then frame the general structure of your solution. I will post the basic structure of my solution below, but you should have a good think about how you want to go about solving the problem, before you look at my code, because it will give you a good grasp of what differences there are between your solution approach and mine, and it will help you understand my code better.
Also, don't be afraid to write it using an imperative-style loop first (like do), then transforming it to the equivalent named let when it's all working. Just reread the first section to see how to do that transformation.
All that said, here is my solution (with the specifics stripped out):
(define (n-loop proc . ranges)
(let outer ((cur ???)
(ranges ranges))
(cond ((null? ranges) ???)
(else (do ((i (caar ranges) (+ i 1)))
((>= i (cadar ranges)))
(outer ??? ???))))))
Remember, once you get this working, you will still need to transform the do loop into one based on named let. (Or, you may have to go even further and transform both the outer and inner loops into their letrec forms.)