Recently, I started to study formal verification techniques. In literature, model checker and solver are used somehow interchangeably.
But, how model checker and solver are connected with each other?
p.s. I would appreciate if some papers or links are suggested.
In model checking, you have a model and a specification (or property), and you check if the model meets the specification.
In SAT solving, you have a formula and you try to find a satisfying assignment to it.
Now, in model checking, you can conjunct the model and the negation of the property to give you one formula. Use a solver to solve for this formula. If it gives you a solution, it would mean the property is sometimes violated (since you conjuncted the negated property). Getting unsat would mean your model satisfies the property/specification.
To perform model checking a reachability analysis is needed and to do this the program transitions are often executed symbolically. The solution to the resulting satisfaction problem is created by a solver. A very basic and very good introduction is found in this free text book (Part III: Analysis and Verification):
http://leeseshia.org
Edward A. Lee and Sanjit A. Seshia, Introduction to Embedded Systems, A Cyber-Physical Systems Approach, Second Edition, MIT Press, ISBN 978-0-262-53381-2, 2017
Related
I'm looking for research, algorithms or even terminology for this area of research that take a Prolog program and a query I want to be true and attempt to find the facts that would need to be asserted to make it true. For example:
% Program
hasProperty(Object, Property) :-
property(Object, hasProperty, Property).
property(apple, hasProperty, red).
property(car, hasProperty, drivable).
% Magic function that determines what Facts would make
% query 'hasProperty(lemon, sour)' true
% in the program above
?- whatFacts(hasProperty(lemon, sour), Facts).
Facts = [property(lemon, sour)]
I'm sure research has been done on this, and certainly it seems unsolvable in the general case, but I'm curious what has been done but am having trouble finding the right terminology to find the work.
Would love any pointers to actual algorithms or names for the area or problem I'm describing.
This is called "abduction".
For the view from philosophical logic, Stanford Encyclopedia of Philosophy offers this entry: Abduction.
For the view from logic programming, Wikipedia offers this entry: Abductive Logic Programming.
A paper that uses Prolog and CHR (Constraint Handling Rules) for Abductive reasoning:
Henning Christiansen: Abductive reasoning in Prolog and CHR (PDF): A short introduction for the KIIS course, Autumn 2005.
Christiansen refers to the book
Abduction and Induction: Essays on their Relation and Integration, edited by Peter A. Flach and Antonis Hadjiantonis (Kluwer Academic Publishers, April 2000), Amazon Link, first chapter at researchgate
and provides this introductory explainer:
Deduction, reasoning within the knowledge we have already, i.e.,from those
facts we know and those rules and regularities of the world that we are
familiar with. E.g., reasoning from causes to effects: If you make a fire
here, you will burn down the house.
In Prolog, the language is structured so as to most naturally find the premise "make a fire here" if your goal happens to be "burn the house down".
Induction, finding general rules from the regularities that we have
experienced in the facts that we know; these rules can be used later for
prediction: Every time I made a fire in my living room, the house burnt
down ... Aha, the next time I make a fire in my living room, the house
will burn down too.
Abduction, reasoning from observed results to the basic facts from which
they follow, quite often it means from an observed effect to produce a
qualified guess for a possible cause: The house burnt down, perhaps my cousin
has made a fire in the living room again.
"Abductive Logic Programming" (ALP) is (used to be?) an active area of research.
Here is a Sprinker Link with search result.
ALP is a common problem in commonsense reasoning and planning. Examples that come to mind:
The CIFF Proof Procedure for
Abductive Logic Programming with Constraints: Theory, Implementation and
Experiments
Robert Kowalski, Fariba Sadri et al. have worked on "LPS" (Logic Production
System), which uses ALP (but not by name?) in the
context of the event calculus
to decide what actions to take to make facts about the world true (wishing
for more details here, I do hope they are editing a book on this).
Contrariwise, Raymond Reiter does not use Prolog but
Answer Set Programming
(which may be more adapted to ALP than the SLDNF approach of Prolog) for
(among others) abductive reasoning in the
Situation Calculus.
More on this in the book Knowledge in Action (MIT Press, July 2001).
As said in the comments this is called abductive reasoning, a good guide is in simply logical: https://too.simply-logical.space/src/text/3_part_iii/8.0.html
While doing my studies on propositional logic i came up with the following question:
Is a software bug always a logical contradiction between the program and the specification?
Consider the following example:
Our specification tells us that "we do action C iff premise A and B are true".
Which is implemented as follows:
main ()
if A then C
if B then C
Clearly one can see that the specification does not fit the implementation since (consider the program above) "we do C iff the premise A or the premise B is true".
Expressing our specification and our program as propositional formulas we get the following equation:
We transform our specification to CNF and apply the resolution calculus and now we can easily see that the very first clause contradicts with the very last clause. Therefore this formula is not satisfiable and therefore our specification contradicts our implementation.
My questions are now (since the above was only an example):
Is this true for every software bug assuming a complete documentation?
and if so:
If we convert a complete specification to propositional formulas could we automate the process of software bug finding?
To answer my own question: This is called "Model Checking" and is very common in computer science in large companies like Intel to check whether hardware is actually doing what it is supposed to do.
Recently model checking started to occur more and more in software development as well. For example NASA and Microsoft are using this technology to quite some extend.
In its base form it works as follows: The specification is converted to logical statements and a compiler translates a given software program to a tree-like structure called "Kripke Structure". Model checkers take these as input and give either a counterexample for which the specification is incomplete or emit a truth value.
https://en.wikipedia.org/wiki/Model_checking
I want to learn more about the internals of Prolog and understand how this works.
I know how to use it. But not how it works internally. What are the names of the algorithms and concepts used in Prolog?
Probably it builds some kind of tree structure or directed object graph, and then upon queries it traveres that graph with a sophisticated algorithm. A Depth First Search maybe. There might be some source code around but it would be great to read about it from a high level perspective first.
I'm really new to AI and understanding Prolog seems to be a great way to start, imho. My idea is to try to rebuild something similar and skipping the parser part completely. I need to know the directions in which I have to do my research efforts.
What are the names of the algorithms and concepts used in Prolog?
Logic programming
Depth-first, backtracking search
Unification
See Sterling & Shapiro, The Art of Prolog (MIT Press) for the theory behind Prolog.
Probably it builds some kind of tree structure or directed object graph, and then upon queries it traveres that graph with a sophisticated algorithm. A Depth First Search maybe.
It doesn't build the graph explicitly, that wouldn't even be possible with infinite search spaces. Check out the first chapters of Russell & Norvig for the concept of state-space search. Yes, it does depth-first search with backtracking, but no, that isn't very sophisticated. It's just very convenient and programming alternative search strategies isn't terribly hard in Prolog.
understanding Prolog seems to be a great way to start, imho.
Depends on what you want to do, but knowing Prolog certainly doesn't hurt. It's a very different way of looking at programming. Knowing Prolog helped me understand functional programming very quickly.
My idea is to try to rebuild something similar and skipping the parser part completely
You mean skipping the Prolog syntax? If you happen to be familiar with Scheme or Lisp, then check out section 4.4 of Abelson & Sussman where they explain how to implement a logic programming variant of Scheme, in Scheme.
AI is a wide field, Prolog only touches symbolic AI. As for Prolog, the inner workings are too complex to explain here, but googling will give you plenty of resources. E.g. http://www.amzi.com/articles/prolog_under_the_hood.htm .
Check also Wikipedia articles to learn about the other areas of AI.
You might also want to read about the Warren Abstract Machine
typically, prolog code is translated to WAM instructions and then executed more efficiently.
I would add:
Programming Languages: An interpreter based approach by Samuel N. Kamin. The book is out of print, but you may find it in a University Library. It contains a Prolog implementation in Pascal.
Tim Budd's "The Kamin Interpreters in C++" (in postscript)
The book by Sterling and Shapiro, mentioned by larsmans, actually contains an execution model of Prolog. It's quite nice and explains clearly "how Prolog works". And it's an excellent book!
There are also other sources you could try. Most notably, some Lisp books build pedagogically-oriented Prolog interpreters:
On Lisp by paul Graham (in Common Lisp, using -- and perhaps abusing -- macros)
Paradigms of Artificial Intelligence Programming by Peter Norvig (in Common Lisp)
Structure and Interpretation of Computer Programs by Abelson and Sussman (in Scheme).
Of these, the last one is the clearest (in my humble opinion). However, you'd need to learn some Lisp (either Common Lisp or Scheme) to understand those.
The ISO core standard for Prolog also contains an execution model. The execution model is of interest since it gives a good model of control constructs such as cut !/0, if-then-else (->)/2, catch/3 and throw/1. It also explains how to conformantly deal with naked variables.
The presentation in the ISO core standard is not that bad. Each control construct is described in a form of a prose use case with a reference to an abstract Prolog machine consisting of a stack, etc.. Then there are pictures that show the stack before and after execution of the control construct.
The cheapest source is ANSI:
http://webstore.ansi.org/RecordDetail.aspx?sku=INCITS%2FISO%2FIEC+13211-1-1995+%28R2007%29
In addition to the many good answers already posted, I add some historical facts on Prolog.
Wikipedia on Prolog: Prolog was created around 1972 by Alain Colmerauer with Philippe Roussel, based on Robert Kowalski's procedural interpretation of Horn clauses.
Alain was a French computer scientist and professor at Aix-Marseille University from 1970 to 1995. Retired in 2006, he remained active until he died in 2017. He was named Chevalier de la Legion d’Honneur by the French government in 1986.
The inner works of Prolog can best be explained by its inventor in this article Prolog in 10 figures. It was published in Communications of the ACM, vol. 28, num. 12, December. 1985.
Prolog uses a subset of first order predicate logic, called Horn logic. The algorithm used to derive answers is called SLD resolution.
Could you please explain me what is the basic connection between the fundamentals of logical programming and the phenomenon of syntactic similarity between type systems and conventional logic?
The Curry-Howard correspondence is not about logic programming, but functional programming. The fundamental mechanic of Prolog is justified in proof theory by John Robinson's resolution technique, which shows how it is possible to check whether logical formulae expressed as Horn clauses are satisfiable, that is, whether you can find terms to substitue for their logic variables that make them true.
Thus logic programming is about specifying programs as logical formulae, and the calculation of the program is some form of proof inference, in Prolog reolution, as I have said. By contrast the Curry-Howard correspondence shows how proofs in a special formulasition of logic, called natural deduction, correspond to programs in the lambda calculus, with the type of the program corresponding to the formula that the proof proves; computation in the lambda calculus corresponds to an important phenomenon in proof theory called normalisation, which transforms proofs into new, more direct proofs. So logic programming and functional programming correspond to different levels in these logics: logic programs match formulae of a logic, whilst functional programs match proofs of formulae.
There's another difference: the logics used are generally different. Logic programming generally uses simpler logics — as I said, Prolog is founded on Horn clauses, which are a highly restricted class of formulae where implications may not be nested, and there are no disjunctions, although Prolog recovers the full strength of classical logic using the cut rule. By contrast, functional programming languages such as Haskell make heavy use of programs whose types have nested implications, and are decorated by all kinds of forms of polymorphism. They are also based on intuitionistic logic, a class of logics that forbids use of the principle of the excluded middle, which Robinson's computational mechanism is based on.
Some other points:
It is possible to base logic programming on more sophisticated logics than Horn clauses; for example, Lambda-prolog is based on intuitionistic logic, with a different computation mechanism than resolution.
Dale Miller has called the proof-theoretic paradigm behind logic programming the proof search as programming metaphor, to contrast with the proofs as programs metaphor that is another term used for the Curry-Howard correspondence.
Logic programming is fundamentally about goal directed searching for proofs. The structural relationship between typed languages and logic generally involves functional languages, although sometimes imperative and other languages - but not logic programming languages directly. This relationship relates proofs to programs.
So, logic programming proof search can be used to find proofs that are then interpreted as functional programs. This seems to be the most direct relationship between the two (as you asked for).
Building whole programs this way isn't practical, but it can be useful for filling in tedious details in programs, and there's some important examples of this in practice. A basic example of this is structural subtyping - which corresponds to filling in a few proof steps via a simple entailment proof. A much more sophisticated example is the type class system of Haskell, which involves a particular kind of goal directed search - in the extreme this involves a Turing-complete form of logic programming at compile time.
First of all, is this only possible on algorithms which have no side effects?
Secondly, where could I learn about this process, any good books, articles, etc?
COQ is a proof assistant that produces correct ocaml output. It's pretty complicated though. I never got around to looking at it, but my coworker started and then stopped using it after two months. It was mostly because he wanted to get things done quicker, but if you need to verify an algorithm this might be a good idea.
Here is a course that uses COQ and talks about proving algorithms.
And here is a tutorial about writing academic papers in COQ.
It's generally a lot easier to verify/prove correctness when no side effects are involved, but it's not an absolute requirement.
You might want to look at some of the documentation for a formal specification language like Z. A formal specification isn't a proof itself, but is often the basis for one.
I think that verifying the correctness of an algorithm would be validating its conformance with a specification. There is a branch of theoretical Computer Science called Formal Methods which may be what you are looking for if you need to get as close to proof as you can. From wikipedia,
Formal Methods are a particular kind
of mathematically-based techniques for
the specification, development and
verification of software and hardware
systems
You will be able to find many learning resources and tools from the multitude of links on the linked Wikipedia page and from the Formal Methods wiki.
Usually proofs of correctness are very specific to the algorithm at hand.
However, there are several well known tricks that are used and re-used again. For example, with recursive algorithms you can use loop invariants.
Another common trick is reducing the original problem to a problem for which your algorithm's proof of correctness is easier to show, then either generalizing the easier problem or showing that the easier problem can be translated to a solution to the original problem. Here is a description.
If you have a particular algorithm in mind, you may do better in asking how to construct a proof for that algorithm rather than a general answer.
Buy these books: http://www.amazon.com/Science-Programming-Monographs-Computer/dp/0387964800
The Gries book, Scientific Programming is great stuff. Patient, thorough, complete.
Logic in Computer Science, by Huth and Ryan, gives a reasonably readable overview of modern systems for verifying systems. Once upon a time people talked about proving programs correct - with programming languages which may or may not have side effects. The impression I get from this book and elsewhere is that real applications are different - for instance proving that a protocol is correct, or that a chip's floating point unit can divide correctly, or that a lock-free routine for manipulating linked lists is correct.
ACM Computing Surveys Vol 41 Issue 4 (October 2009) is a special issue on software verification. It looks like you can get to at least one of the papers without an ACM account by searching for "Formal Methods: Practice and Experience".
The tool Frama-C, for which Elazar suggests a demo video in the comments, gives you a specification language, ACSL, for writing function contracts and various analyzers for verifying that a C function satisfies its contract and safety properties such as the absence of run-time errors.
An extended tutorial, ACSL by example, shows examples of actual C algorithms being specified and verified, and separates the side-effect-free functions from the effectful ones (the side-effect-free ones are considered easier and come first in the tutorial). This document is also interesting in that it was not written by the designers of the tools it describe, so it gives a fresher and more didactic look at these techniques.
If you are familiar with LISP then you should definitely check out ACL2: http://www.cs.utexas.edu/~moore/acl2/acl2-doc.html
Dijkstra's Discipline of Programming and his EWDs lay the foundation for formal verification as a science in programming. A simpler work is Wirth's Systematic Programming, which begins with the simple approach to using verification. Wirth uses pre-ISO Pascal for the language; Dijkstra uses an Algol-68-like formalism called Guarded (GCL). Formal verification has matured since Dijkstra and Hoare, but these older texts may still be a good starting point.
PVS tool developed by Stanford guys is a specification and verification system. I worked on it and found it very useful for Theoram Proving.
WRT (1), you will probably have to create a model of the algorithm in a way that "captures" the side-effects of the algorithm in a program variable intended to model such state-based side-effects.