I am learning about compilers, specifically looking at 2-phase compilers, and am confused on the certain phases where errors are detected. Let's say we have something like:
int x, y;
x = x + y + z;
Where we are trying to reference a variable that has not been declared. I think this is an error that would be detected in the front-end of the compiler. But I don't know which sub-area of the front-end would detect this error.
The three parts to the front-end are: the scanner, the parser, and the context-free analyzer. The scanner reads every single character in a statement and splits the statement up into tokens. So, I could be wrong, but I don't think the error would be detected here. The parser checks to see if the statement is syntactically correct. Here is where I start to get confused. Even though z is undeclared, the syntax of the statement is technically correct, so would the error not be detected here either? The context-free analyzer uses the symbol table and syntax tree to check the program to see if it is semantically consistent with the language definition. Here it also does type checking. Would it be here that the error would be detected? Because at this point the compiler would look in the symbol table and notice that z doesn't have a type (or that it's not in there at all?). Or is this something that would be detected by the back-end of the compiler? If it is the back-end, I don't understand why that is the case. Any clarification would be highly appreciated. Thanks.
This is ultimately compiler-dependent, but typically this would come up at the semantic analysis level, which is still in the compiler front-end.
With a traditional compiler, this couldn't be done in the scanning phase because scanners use finite automata and the language of "strings that represent proper variable scoping" isn't regular. This also typically wouldn't be done as part of parsing, since parsing usually is about building up an AST and, if it were done bottom-up, the scoping information wouldn't be available at the time that the parser determined the structure of the code.
However, the semantic analyzer has all the information necessary to find this error - it has the AST and can use that to build a symbol table, walk through all the expressions in the code, and notice that z isn't anywhere in that symbol table.
Background:
I am implementing a language similar to Ruby, called Sapphire, as a way to try out some Ideas I have on concurrency in programming languages. I am trying to copy Ruby's double quoted strings with embedded code which I find very useful as a programmer.
Question:
How do any of the Ruby interpreters turn a double quotes string with embedded code into and AST?
eg:
puts "The value of foo is #{#foo}."
puts "this is an example of unmatched braces in code: #{ foo.go('}') }"
Details:
The problem I have is how to decide which } closes the code block. Code blocks can have other braces within them and with a little effort they can be unmatched. The lexer can find the beginning of a code block in a string, but without the aid of the parser, it cannot know for sure which character is the end of that block.
It looks like Ruby's parse.y file does both the lexing and parsing steps, but reading that thing is a nightmare it is 11628 lines long with no comments and lots of abbr.
True, Yacc files can be a bit daunting to read at first and parse.y is not the best file to start with. Have you looked at the various string production rules? Do you have any specific questions?
As for the actual parsing, it's indeed not uncommon that lexers do also parse numeric literals and strings, see e.g. the accepted answer to a similar question here on SO. If you approach things this way, it's not too hard to see how to go about it. Hitting #{ inside a string, basically starts a new parsing context that gets parsed as an expression again. This means that the first } in your example can't be the terminating one for the interpolation, since it's part of a literal string within the expression. Once you reach the end of the expression (keep in mind expression separators like ;), the next } is the one you need.
This is not a complete answer, but I leave it in hopes that it might be useful either to me or one who follows me.
Matz gives a pretty detailed rundown of the yylex() function of parse.y in chapter 11 of his book. It does not directly mention strings, but it does describe how the lexer uses lex_state to resolve several locally ambiguous constructs in Ruby.
A reproduction of an English translation of this chapter can be found here.
Please bear in mind that they don't have to (create an AST at compile time).
Ruby strings can be assembled at runtime and will interpolate correctly. Therefore all the parsing and evaluation machinery has to be available at runtime. Any work done at compile time in that sense could be considered an optimisation.
So why does this matter? Because there are very effective stack-based techniques for parsing and evaluating expressions that do not create or decorate an AST. The string is read (parsed) from left to right, and as embedded tokens are encountered they are either evaluated or pushed on a stack, or cause stack contents to be popped and evaluated.
This is a simple technique to implement provided the expressions are relatively simple. If you really want the full power of the language inside every string, then you need the full compiler at runtime. Not everyone does.
Disclosure: I wrote a commercial language product that does exactly this.
Dart also supports expressions interpolated into strings like Ruby, and I've skimmed a few parsers for it. I believe what they do is define separate tokens for a string literal preceding interpolation and a string literal at the end. So if you tokenize:
"before ${the + expression} after"
You would get tokens like:
STRING_START "before "
IDENTIFIER the
PLUS
IDENTIFIER expression
STRING " after"
Then in your parser, it's a pretty straightforward process of handling STRING_START to parse the interpolated expression(s) following it.
Our Ruby parser (see my bio) treats Ruby "strings" as complex objects having lots of substructures, including string start and end tokens, bare string literal fragments, lots of funny punctuation sequences representing the various regexp operators, and of course, recursively, most of Ruby itself for expressions nested inside such strings.
This is accomplished by allowing the lexer to detect and generate such string fragments in a (for Ruby, many) special lexing modes. The parser has a (sub)grammar that defines valid sequences of tokens. And that kind of parsing solves OP's original problem; the parser knows whether a curly brace matches other curly braces from the regexp content, and/or if the regexp has been completely assembled and the curly brace is a matching block end.
Yes, it builds an AST of the Ruby code, and of the regexps.
The purpose of all this is to allow us to build analyzers and transformers of Ruby code. See https://softwarerecs.stackexchange.com/q/11779/101
I looked over the internet for the lexical errors for Pascal programming language and there still this case which I don't know whether it's lexical error or semantic.
does the case two symbols (operators) after each other count as a lexical error or as a semantic error ?
example: "+-", "<>", ";;" or something like that.
thanks in advance.
I would say it is part of the parsing process, semantic check comes only after that. (if you assume the basic sequence lexing,parsing,semantic check).
Btw, <> is a valid operation oin Pascal. (not equal).
What is the difference between syntax and semantics in programming languages (like C, C++)?
TL; DR
In summary, syntax is the concept that concerns itself only whether or not the sentence is valid for the grammar of the language. Semantics is about whether or not the sentence has a valid meaning.
Long answer:
Syntax is about the structure or the grammar of the language. It answers the question: how do I construct a valid sentence? All languages, even English and other human (aka "natural") languages have grammars, that is, rules that define whether or not the sentence is properly constructed.
Here are some C language syntax rules:
separate statements with a semi-colon
enclose the conditional expression of an IF statement inside parentheses
group multiple statements into a single statement by enclosing in curly braces
data types and variables must be declared before the first executable statement (this feature has been dropped in C99. C99 and latter allow mixed type declarations.)
Semantics is about the meaning of the sentence. It answers the questions: is this sentence valid? If so, what does the sentence mean? For example:
x++; // increment
foo(xyz, --b, &qrs); // call foo
are syntactically valid C statements. But what do they mean? Is it even valid to attempt to transform these statements into an executable sequence of instructions? These questions are at the heart of semantics.
Consider the ++ operator in the first statement. First of all, is it even valid to attempt this?
If x is a float data type, this statement has no meaning (according to the C language rules) and thus it is an error even though the statement is syntactically correct.
If x is a pointer to some data type, the meaning of the statement is to "add sizeof(some data type) to the value at address x and store the result into the location at address x".
If x is a scalar, the meaning of the statement is "add one to the value at address x and store the result into the location at address x".
Finally, note that some semantics can not be determined at compile-time and therefore must be evaluated at run-time. In the ++ operator example, if x is already at the maximum value for its data type, what happens when you try to add 1 to it? Another example: what happens if your program attempts to dereference a pointer whose value is NULL?
Syntax refers to the structure of a language, tracing its etymology to how things are put together.
For example you might require the code to be put together by declaring a type then a name and then a semicolon, to be syntactically correct.
Type token;
On the other hand, the semantics is about meaning.
A compiler or interpreter could complain about syntax errors. Your co-workers will complain about semantics.
Semantics is what your code means--what you might describe in pseudo-code. Syntax is the actual structure--everything from variable names to semi-colons.
Wikipedia has the answer. Read syntax (programming languages) & semantics (computer science) wikipages.
Or think about the work of any compiler or interpreter. The first step is lexical analysis where tokens are generated by dividing string into lexemes then parsing, which build some abstract syntax tree (which is a representation of syntax). The next steps involves transforming or evaluating these AST (semantics).
Also, observe that if you defined a variant of C where every keyword was transformed into its French equivalent (so if becoming si, do becoming faire, else becoming sinon etc etc...) you would definitely change the syntax of your language, but you won't change much the semantics: programming in that French-C won't be easier!
You need correct syntax to compile.
You need correct semantics to make it work.
Late to the party - but to me, the answers here seem correct but incomplete.
Pragmatically, I would distinguish between three levels:
Syntax
Low level semantics
High level semantics
1. SYNTAX
Syntax is the formal grammar of the language, which specifies a well-formed statement the compiler will recognise.
So in C, the syntax of variable initialisation is:
data_type variable_name = value_expression;
Example:
int volume = 66 * 22 * 55;
While in Go, which offers type inference, one form of initialisation is:
variable_name := value_expression
Example:
volume := 66 * 22 * 55
Clearly, a Go compiler won't recognise the C syntax, and vice versa.
2. LOW LEVEL SEMANTICS
Where syntax is concerned with form, semantics is concerned with meaning.
In natural languages, a sentence can be syntactically correct but semantically meaningless. For example:
The man bought the infinity from the store.
The sentence is grammatically correct but doesn't make real-world sense.
At the low level, programming semantics is concerned with whether a statement with correct syntax is also consistent with the semantic rules as expressed by the developer using the type system of the language.
For example, this is a syntactically correct assignment statement in Java, but semantically it's an error as it tries to assign an int to a String
String firstName = 23;
So type systems are intended to protect the developer from unintended slips of meaning at the low level.
Loosely typed languages like JavaScript or Python provide very little semantic protection, while languages like Haskell or F# with expressive type systems provide the skilled developer with a much higher level of protection.
For example, in F# your ShoppingCart type can specify that the cart must be in one of three states:
type ShoppingCart =
| EmptyCart // no data
| ActiveCart of ActiveCartData
| PaidCart of PaidCartData
Now the compiler can check that your code hasn't tried to put the cart into an illegal state.
In Python, you would have to write your own code to check for valid state.
3. HIGH LEVEL SEMANTICS
Finally, at a higher level, semantics is concerned with what the code is intended to achieve - the reason that the program is being written.
This can be expressed as pseudo-code which could be implemented in any complete language. For example:
// Check for an open trade for EURUSD
// For any open trade, close if the profit target is reached
// If there is no open trade for EURUSD, check for an entry signal
// For an entry signal, use risk settings to calculate trade size
// Submit the order.
In this (heroically simplified) scenario, you are making a high-level semantic error if your system enters two trades at once for EURUSD, enters a trade in the wrong direction, miscalculates the trade size, and so on.
TL; DR
If you screw up your syntax or low-level semantics, your compiler will complain.
If you screw up your high-level semantics, your program isn't fit for purpose and your customer will complain.
Syntax is the structure or form of expressions, statements, and program units but Semantics is the meaning of those expressions, statements, and program units. Semantics follow directly from syntax.
Syntax refers to the structure/form of the code that a specific programming language specifies but Semantics deal with the meaning assigned to the symbols, characters and words.
Understanding how the compiler sees the code
Usually, syntax and semantics analysis of the code is done in the 'frontend' part of the compiler.
Syntax: Compiler generates tokens for each keyword and symbols: the token contains the information- type of keyword and its location in the code.
Using these tokens, an AST(short for Abstract Syntax Tree) is created and analysed.
What compiler actually checks here is whether the code is lexically meaningful i.e. does the 'sequence of keywords' comply with the language rules? As suggested in previous answers, you can see it as the grammar of the language(not the sense/meaning of the code).
Side note: Syntax errors are reported in this phase.(returns tokens with the error type to the system)
Semantics: Now, the compiler will check whether your code operations 'makes sense'.
e.g. If the language supports Type Inference, sematic error will be reported if you're trying to assign a string to a float. OR declaring the same variable twice.
These are errors that are 'grammatically'/ syntaxially correct, but makes no sense during the operation.
Side note: For checking whether the same variable is declared twice, compiler manages a symbol table
So, the output of these 2 frontend phases is an annotated AST(with data types) and symbol table.
Understanding it in a less technical way
Considering the normal language we use; here, English:
e.g. He go to the school. - Incorrect grammar/syntax, though he wanted to convey a correct sense/semantic.
e.g. He goes to the cold. - cold is an adjective. In English, we might say this doesn't comply with grammar, but it actually is the closest example to incorrect semantic with correct syntax I could think of.
He drinks rice (wrong semantic- meaningless, right syntax- grammar)
Hi drink water (right semantic- has meaning, wrong syntax- grammar)
Syntax: It is referring to grammatically structure of the language.. If you are writing the c language . You have to very care to use of data types, tokens [ it can be literal or symbol like "printf()". It has 3 tokes, "printf, (, )" ]. In the same way, you have to very careful, how you use function, function syntax, function declaration, definition, initialization and calling of it.
While semantics, It concern to logic or concept of sentence or statements. If you saying or writing something out of concept or logic, then you are semantically wrong.
I'm just starting to use antlr, with antlr for ruby. The version is 3.2.1
I'm trying to create a parser for the chef language, and the grammar is giving me a real headache :P I'm sure I'm missing some fundamental concept, but I just couldn't figure it out.
I created 3 grammars. The main one is the recipe parser, which (of course) parses the recipes. Once a recipe is parsed, I used the other 2 grammars, that parse ingredients and instructions (the method section).
My problem is with the last one, the one that parses the instructions, such as "put ... into the mixing bowl", "liquefy ...", etc. Everything works great except for a few rules. I've posted the Instructions.g source here, at paste.bin because of its length.
Here's what's happening:
When I uncomment the rules combine_ingredient_into_mixing_bowl or divide_ingredient_into_mixing_bowl, the parser stops recognizing almost all of the other rules (such as put_ingredient_into_mixing_bowl). This seems strange to me, because they don't seem to override each other (of course they are, somehow). I get the error: "line 0:-1 mismatched input "" expecting WS"
stir_mixing_bowl does not match anything, but it's really no different from the other rules that do work ok. I get the error: "line 0:-1 mismatched input "" expecting set nil"
Is it possible to include the rules verb_the_ingredient and liquefy_ingredient without making them conflict with the other rules? The former will actually conflict with everything else I guess, and the latter will conflict with liquefy_mixing_bowl. What would be the best way to deal with such a nasty grammar?
By the way, I haven't set the WS (whitespace such as space and tab) to the ignore channel because since an ingredient can consiste of one or more words (such as dijon mustard or just zuchinnis) I found that it is easier to specify the grammar by using the WS token as separators.
Also, running the antlr4ruby command to generate the parsers/lexers code shows no warnings at all.
Any tips, hints, or enlightening is really appreciated here :)
Thanks in advance.