How to analyze unknown common lisp source code, in order to understand it?
Given, I have Common Lisp source code of a function. That imaginary function operates on arbitrary and complex data. Now I'd like to analyze such a function.
So, I'm interestend in working methods to analyze some given common lisp source code. I.e. by stepping through source code, like it can be done with elisp source code in Emacs by using edebug-defun.
My tools are slime, sbcl and Emacs.
End of actual question, things below are written only to give a better overview, what this question is about.
Notes about questions background:
I'm not interested in what the above given function does, since
I already know basic common lisp like if, dolist, defun and basics. Also I also know how to query CLHS.
I could probably go through code and data with pen and paper, but why? When I'm sitting in front of a computer.
I already read CL Cookbook - Debugging and Malispers debugging tutorial
Here is an example of such a function, the answer does not need to refer to this function:
(defun wins (grid color)
(declare (optimize (speed 0)
(debug 3)))
(dolist (phase *phase*)
(dolist (start (car phase))
(if (= (* 4 color)
(reduce #'+ (cadr phase)
:key
(lambda (offset)
(aref grid (+ start offset)))))
(return-from wins t)))))
Given above code, for me, it is a mystery when which parts of the arbitrary data is processed and how. It would probably be useful to see it in action.
With that special example, I tried to use (step (wins ...)), but this only stops at (* 4 color), (reduce ... and (* start offset), which is not enough (for me) to understand the function.
Also (trace wins) did not help much.
I took this function from an common lisp teaching book, which already explains it. So there is no need to explain this function.
Related
I am a undergraduate who wants to go through "The Scheme programming language" as a self-study.
Here is a simple program and I named it as "reciprocal.ss"
(define reciprocal
(lambda (n)
(if(= n 0)
"oops!"
(/ 1 n))))
Then I wanted to load my procedure:
(load "reciprocal.ss")
It produces this error:
reciprocal.ss:1:0: #%top-interaction: unbound identifier;
also, no #%app syntax transformer is bound in: #%top-interaction
I did each parts as what the book says. Perhaps I am just making a rookie mistake. Any insight would be appreciated.
Since load uses eval, using it outside of a REPL generally will not work — for reasons described in Namespaces
Using racket/load can work for you here however:
loader.ss
#lang racket/load
(load "reciprocal.ss")
(display (reciprocal 10))
reciprocal.ss
(define reciprocal
(lambda (n)
(if (= n 0) "oops!"
(/ 1 n))))
In Racket (and Scheme at large) has a more complex story than the average language regarding running external code. In general, you should use import when you want to directly 'include' a file, you should use provide/require when you want to establish module boundaries and you should use load when you are sophisticated enough to be stretching the limits of either.
The simplest approach is not to use load at all.
In "reciprocal.ss" make the first lines:
#lang racket
(provide (all-defined-out))
(define reciprocal
(lambda (n)
(if (= n 0)
"oops!"
(/ 1 n))))
Then use (require "reciprocal.ss") in the file where you need to use the function reciprocal.
The load mechanism was used back in the good old days before module systems had arrived. Writing (load "foo.ss") basically works as if you manually pasted the contents of foo.ss into the repl and excecuted it. This means that the result of your program is dependent of the order of loading files (if you are using side effects). Module systems handle this (and other things too) much better.
Given an expression in the form of : (* 3 (+ x y)), how can I evaluate the expression so as to put it in the form (+ (* 3 x) (* 3 y))? (note: in the general case, 3 is any constant, and "x" or "y" could be terms of single variables or other s-expressions (e.g. (+ 2 x)).
How do I write a lambda expression that will recursively evaluate the items (atoms?) in the original expression and determine whether they are a constant or a term? In the case of a term, it would then need to be evaluated again recursively to determine the type of each item in that term's list.
Again, the crux of the issue for me is the recursive "kernel" of the definition.
I would obviously need a base case that would determine once I have reached the last, single atom in the deepest part of the expression. Then recursively work "back up", building the expression in the proper form according to rules.
Coming from a Java / C++ background I am having great difficulty in understanding how to do this syntactically in Scheme.
Let's take a quick detour from the original problem to something slightly related. Say that you're given the following: you want to write an evaluator that takes "string-building" expressions like (* 3 "hello") and "evaluates" it to "hellohellohello". Other examples that we'd like to make work include things like
(+ "rock" (+ (* 5 "p") "aper")) ==> "rockpppppaper"
(* 3 (+ "scis" "sors")) ==> "scissorsscissorsscissors"
To tackle a problem like this, we need to specify exactly what the shape of the inputs are. Essentially, we want to describe a data-type. We'll say that our inputs are going to be "string-expressions". Let's call them str-exprs for short. Then let's define what it means to be a str-expr.
A str-expr is either:
<string>
(+ <str-expr> <str-expr>)
(* <number> <str-expr>)
By this notation, we're trying to say that str-exprs will all fit one of those three shapes.
Once we have a good idea of what the shape of the data is, we have a better guide to write functions that process str-exprs: they must case-analyze those three alternatives!
;; A str-expr is either:
;; a plain string, or
;; (+ str-expr str-expr), or
;; (* number str-expr)
;; We want to write a definition to "evaluate" such string-expressions.
;; evaluate: str-expr -> string
(define (evaluate expr)
(cond
[(string? expr)
...]
[(eq? (first expr) '+)
...]
[(eq? (first expr) '*)
...]))
where the '...'s are things that we'll be filling in.
Actually, we know how to fill in a little more about the '...': we know that in the second and third cases, the inner parts are themselves str-exprs. Those are prime spots where recurrence will probably happen: since our data is described in terms of itself, the programs that process them will also probably need to refer to themselves. In short, programs that process str-exprs will almost certainly follow this shape:
(define (evaluate expr)
(cond
[(string? expr)
... expr
...]
[(eq? (first expr) '+)
... (evaluate (second expr))
... (evaluate (third expr))
...]
[(eq? (first expr) '*)
... (second expr)
... (evaluate (third expr))
...]))
That's all without even doing any real work: we can figure this part out just purely because that's what the data's shape tells us. Filling in the remainder of the '...'s to make this all work out is actually not too bad, especially when we also consider the test cases we've cooked up. (Code)
It's this kind of standard data-analysis/case-analysis that's at the heart of your question, and it's one that's covered extensively by curricula such as HTDP. This is not Scheme or Racket specific: you'd do the same kind of data analysis in Java, and you see the same kind of approach in many other places. In Java, the low-mechanism used for the case analysis might be done differently, perhaps with dynamic dispatch, but the core ideas are all the same. You need to describe the data. Once you have a data definition, use it to help you sketch out what the code needs to look like to process that data. Use test cases to triangulate how to fill in the sketch.
You need to break down your problem. I would start by following the HtDP (www.htdp.org) approach. What are your inputs? Can you specify them precisely? In this case, those inputs are going to be self-referential.
Then, specify the output form. In fact, your text above is a little fuzzy on this: I think I know what your output form looks like, but I'm not entirely sure.
Next, write a set of tests. These should be based on the structure of your input terms; start with the simplest ones, and work upward from there.
Once you have a good set of tests, implementing the function should be pretty straightforward. I'd be glad to help if you get stuck!
I've looked through On Lisp, Practical Common Lisp and the SO archives in order to answer this on my own, but those attempts were frustrated by my inability to name the concept I'm interested in. I would be grateful if anyone could just tell me the canonical term for this sort of thing.
This question is probably best explained by an example. Let's say I want to implement Python-style list comprehensions in Common Lisp. In Python I would write:
[x*2 for x in range(1,10) if x > 3]
So I begin by writing down:
(listc (* 2 x) x (range 1 10) (> x 3))
and then defining a macro that transforms the above into the correct comprehension. So far so good.
The interpretation of that expression, however, would be opaque to a reader not already familiar with Python list comprehensions. What I'd really like to be able to write is the following:
(listc (* 2 x) for x in (range 1 10) if (> x 3))
but I haven't been able to track down the Common Lisp terminology for this. It seems that the loop macro does exactly this sort of thing. What is it called, and how can I implement it? I tried macro-expanding a sample loop expression to see how it's put together, but the resulting code was unintelligible. Could anyone guide me in the right direction?
Thanks in advance.
Well, what for does is essentially, that it parses the forms supplied as its body. For example:
(defmacro listc (expr &rest forms)
;;
;;
;; (listc EXP for VAR in GENERATOR [if CONDITION])
;;
;;
(labels ((keyword-p (thing name)
(and (symbolp thing)
(string= name thing))))
(destructuring-bind (for* variable in* generator &rest tail) forms
(unless (and (keyword-p for* "FOR") (keyword-p in* "IN"))
(error "malformed comprehension"))
(let ((guard (if (null tail) 't
(destructuring-bind (if* condition) tail
(unless (keyword-p if* "IF") (error "malformed comprehension"))
condition))))
`(loop
:for ,variable :in ,generator
:when ,guard
:collecting ,expr)))))
(defun range (start end &optional (by 1))
(loop
:for k :upfrom start :below end :by by
:collecting k))
Apart from the hackish "parser" I used, this solution has a disadvantage, which is not easily solved in common lisp, namely the construction of the intermediate lists, if you want to chain your comprehensions:
(listc x for x in (listc ...) if (evenp x))
Since there is no moral equivalent of yield in common lisp, it is hard to create a facility, which does not require intermediate results to be fully materialized. One way out of this might be to encode the knowledge of possible "generator" forms in the expander of listc, so the expander can optimize/inline the generation of the base sequence without having to construct the entire intermediate list at run-time.
Another way might be to introduce "lazy lists" (link points to scheme, since there is no equivalent facility in common lisp -- you had to build that first, though it's not particularily hard).
Also, you can always have a look at other people's code, in particular, if they tries to solve the same or a similar problem, for example:
Iterate
Loop in SBCL
Pipes (which does the lazy list thing)
Macros are code transformers.
There are several ways of implementing the syntax of a macro:
destructuring
Common Lisp provides a macro argument list which also provides a form of destructuring. When a macro is used, the source form is destructured according to the argument list.
This limits how macro syntax looks like, but for many uses of Macros provides enough machinery.
See Macro Lambda Lists in Common Lisp.
parsing
Common Lisp also gives the macro the access to the whole macro call form. The macro then is responsible for parsing the form. The parser needs to be provided by the macro author or is part of the macro implementation done by the author.
An example would be an INFIX macro:
(infix (2 + x) * (3 + sin (y)))
The macro implementation needs to implement an infix parser and return a prefix expression:
(* (+ 2 x) (+ 3 (sin y)))
rule-based
Some Lisps provide syntax rules, which are matched against the macro call form. For a matching syntax rule the corresponding transformer will be used to create the new source form. One can easily implement this in Common Lisp, but by default it is not a provided mechanism in Common Lisp.
See syntax case in Scheme.
LOOP
For the implementation of a LOOP-like syntax one needs to write a parser which is called in the macro to parse the source expression. Note that the parser does not work on text, but on interned Lisp data.
In the past (1970s) this has been used in Interlisp in the so-called 'Conversational Lisp', which is a Lisp syntax with a more natural language like surface. Iteration was a part of this and the iteration idea has then brought to other Lisps (like Maclisp's LOOP, from where it then was brought to Common Lisp).
See the PDF on 'Conversational Lisp' by Warren Teitelmann from the 1970s.
The syntax for the LOOP macro is a bit complicated and it is not easy to see the boundaries between individual sub-statements.
See the extended syntax for LOOP in Common Lisp.
(loop for i from 0 when (oddp i) collect i)
same as:
(loop
for i from 0
when (oddp i)
collect i)
One problem that the LOOP macro has is that the symbols like FOR, FROM, WHEN and COLLECT are not the same from the "COMMON-LISP" package (a namespace). When I'm now using LOOP in source code using a different package (namespace), then this will lead to new symbols in this source namespace. For that reason some like to write:
(loop
:for i :from 0
:when (oddp i)
:collect i)
In above code the identifiers for the LOOP relevant symbols are in the KEYWORD namespace.
To make both parsing and reading easier it has been proposed to bring parentheses back.
An example for such a macro usage might look like this:
(iter (for i from 0) (when (oddp i) (collect i)))
same as:
(iter
(for i from 0)
(when (oddp i)
(collect i)))
In above version it is easier to find the sub-expressions and to traverse them.
The ITERATE macro for Common Lisp uses this approach.
But in both examples, one needs to traverse the source code with custom code.
To complement Dirk's answer a little:
Writing your own macros for this is entirely doable, and perhaps a nice exercise.
However there are several facilities for this kind of thing (albeit in a lisp-idiomatic way) out there of high quality, such as
Loop
Iterate
Series
Loop is very expressive, but has a syntax not resembling the rest of common lisp. Some editors don't like it and will indent poorly. However loop is defined in the standard. Usually it's not possible to write extentions to loop.
Iterate is even more expressive, and has a familiar lispy syntax. This doesn't require any special indentation rules, so all editors indenting lisp properly will also indent iterate nicely. Iterate isn't in the standard, so you'll have to get it yourself (use quicklisp).
Series is a framework for working on sequences. In most cases series will make it possible not to store intermediate values.
Does anyone have a good guide as to how it works? Something with visual aids would be nice, every guide I've come across all seem to say the same thing I need a fresh take on it.
Here's the diagram that was left on our CS lab's whiteboard. So you're going to fetch some apples, and you grab a continuation before you begin. You wander through the forest, collecting apples, when at the end you apply your continuation on your apples. Suddenly, you find yourself where you were before you went into the forest, except with all of your apples.
(display
(call/cc (lambda (k)
(begin
(call-with-forest
(lambda (f)
(k (collect-apples f))))
(get-eaten-by-a-bear)))))
=> some apples (and you're not eaten by a bear)
I think a Bar Mitzvah and buried gold might have been involved.
Have a look at the continuation part of PLAI -- it's very "practical
oriented", and it uses a "black-hole" visualization for continuations that can help you
understand it.
There is no shortcut in learning call/cc. Read the chapters in The Scheme Programming Language or Teach Yourself Scheme in Fixnum Days.
I found that it helps to visualize the call stack. When evaluating an expression, keep track of the call stack at every step. (See for example http://4.flowsnake.org/archives/602) This may be non-intuitive at first, because in most languages the call stack is implicit; you don't get to manipulate it directly.
Now think of a continuation as a function that saves the call stack. When that function is called (with a value X), it restores the saved call stack, then passes X to it.
Never likes visual representation of call/cc as I can't reflect it back to the code (yes, poor imagination) ;)
Anyway, I think it is easier start not with call/cc but with call/ec (escape continuation) if you already familiar with exceptions in other languages.
Here is some code which should evaluate to value:
(lambda (x) (/ 1 x))
What if x will be equal '0'? In other languages we can throw exception, what about scheme?
We can throw it too!
(lambda (x) (call/ec (cont)
(if (= x 0) (cont "Oh noes!") (/ 1 x))))
call/ec (as well as call/cc) is works like "try" here. In imperative languages you can easily jump out of function simply returning value or throwing exception.
In functional you can't jump out, you should evaluate something. And call/* comes to rescue.
What it does it represent expression under "call/ec" as function (this named "cont" in my case) with one argument. When this function is called it replaces the WHOLE call/* to it's argument.
So, when (cont "Oh noes!") replaces (call/ec (cont) (if (= x 0) (cont "Oh noes!") (/ 1 x))) to "Oh noes!" string.
call/cc and call/ec are almost equals to each other except ec simplier to implement. It allows only jump up, whil cc may be jumped down from outside.
I recently started reading through Paul Graham's On Lisp with a friend, and we realized that we have very different opinions of reduce: I think it expresses a certain kind of recursive form very clearly and concisely; he prefers to write out the recursion very explicitly.
I suspect we're each right in some context and wrong in another, but we don't know where the line is. When do you choose one form over the other, and what do you think about when making that choice?
To be clear about what I mean by reduce vs. explicit recursion, here's the same function implemented twice:
(defun my-remove-if (pred lst)
(fold (lambda (left right)
(if (funcall pred left)
right
(cons left right)))
lst :from-end t))
(defun my-remove-if (pred lst)
(if lst
(if (funcall pred (car lst))
(my-remove-if pred (cdr lst))
(cons (car lst) (my-remove-if pred (cdr lst))))
'()))
I'm afraid I started out a Schemer (now we're Racketeers?) so please let me know if I've botched the Common Lisp syntax. Hopefully the point will be clear even if the code is incorrect.
If you have a choice, you should always express your computational intent in the most abstract terms possible. This makes it easier for a reader to figure out your intentions, and it makes it easier for the compiler to optimize your code. In your example, when the compiler trivially knows you are doing a fold operation by virtue of you naming it, it also trivially knows that it could possibly parallelize the leaf operations. It would be much harder for a compiler to figure that out when you write extremely low level operations.
I'm going to take a slightly-subjective question and give a highly-subjective answer, since Ira already gave a perfectly pragmatic and logical one. :-)
I know writing things out explicitly is highly valued in some circles (the Python guys make it part of their "zen"), but even when I was writing Python I never understood it. I want to write at the highest level possible, all the time. When I want to write things out explicitly, I use assembly language. The point of using a computer (and a HLL) is to get it to do these things for me!
For your my-remove-if example, the reduce one looks fine to me (apart from the Scheme-isms like fold and lst :-)). I'm familiar with the concept of reduce, so all I need to understand it is figure out your f(x,y) -> z. For the explicit variant, I had to think it for a second: I have to figure out the loop myself. Recursion isn't the hardest concept out there, but I think it is harder than "a function of two arguments".
I also don't care for a whole line being repeated -- (my-remove-if pred (cdr lst)). I think I like Lisp in part because I'm absolutely ruthless at DRY, and Lisp allows me to be DRY on axes that other languages don't. (You could put in another LET at the top to avoid this, but then it's longer and more complex, which I think is another reason to prefer the reduction, though at this point I might just be rationalizing.)
I think maybe the contexts in which the Python guys, at least, dislike implicit functionality would be:
when no-one could be expected to guess the behavior (like frobnicate("hello, world", True) -- what does True mean?), or:
cases when it's reasonable for implicit behavior to change (like when the True argument gets moved, or removed, or replaced with something else, since there's no compile-time error in most dynamic languages)
But reduce in Lisp fails both of these criteria: it's a well-understood abstraction that everybody knows, and that isn't going to change, at least not on any timescale I care about.
Now, I absolutely believe there are some cases where it'd be easier for me to read an explicit function call, but I think you'd have to be pretty creative to come up with them. I can't think of any offhand, because reduce and mapcar and friends are really good abstractions.
In Common Lisp one prefers the higher-order functions for data structure traversal, filtering, and other related operations over recursion. That's also to see from many provided functions like REDUCE, REMOVE-IF, MAP and others.
Tail recursion is a) not supported by the standard, b) maybe invoked differently with different CL compilers and c) using tail recursion may have side effects on the generated machine code for surrounding code.
Often, for certain data structures, many of these above operations are implemented with LOOP or ITERATE and provided as higher-order function. There is a tendency to prefer new language extensions (like LOOP and ITERATE) for iterative code over using recursion for iteration.
(defun my-remove-if (pred list)
(loop for item in list
unless (funcall pred item)
collect item))
Here is also a version that uses the Common Lisp function REDUCE:
(defun my-remove-if (pred list)
(reduce (lambda (left right)
(if (funcall pred left)
right
(cons left right)))
list
:from-end t
:initial-value nil))