JavaCC: A LOOKAHEAD of 2 or greater make my compiler crash? - compilation

Question

I am using the Grammar defined in the official Java 8 Language Specification to write a Parser for Java.
In my .jj file I have all of the usual kinds of choice conflicts such as
Warning: Choice conflict involving two expansions at
line 25, column 3 and line 31, column 3 respectively.
A common prefix is:
Consider using a lookahead of 2 for earlier expansion.
or
Warning: Choice conflict in (...)* construct at line 25, column 8.
I did carefully read the Lookahead tutorial from JavaCC but my problem is that whenever I set a LOOKAHEAD(n) where n > 1 and I compile the .jj file the compilation gets stuck and I need to kill the java process.
Why?
CODE
Since I am unable to localize the code which causes my problem I am also not possible to isolate the corresponding code portions.
I was able to restrict the search for the erroneous code fragments as follows:
I have uploaded the code at scribd here.
Please note:
The first rules have a leading // OK comment. This means that when I only have these rules I do get the warnings from the compiler that I have choice conflicts but when I add
LOOKAHEAD(3)
at the corresponding position the warnings disappear.
When I add all successive rules (at once) I am not able to add the
LOOKAHEAD(3) statement anymore. When I do my Eclipse IDE freezes and the javaw.exe process seems get deadlocked or run into an infinite loop when I try to compile the file with JavaCC (which is my actual problem).

Answer 1

Your grammar is so far from LL(1) that it is hard to know where to begin. Let's look at types. After correcting it to follow the grammar in the JLS 8, you have
void Type() :
{ }
{
PrimitiveType() |
ReferenceType()
}
where
void PrimitiveType() :
{ }
{
(Annotation())* NumericType() |
(Annotation())* <KW_boolean>
}
void ReferenceType() :
{ }
{
ClassOrInterfaceType() |
TypeVariable() |
ArrayType()
}
void ClassOrInterfaceType() :
{ }
{
(Annotation())* <Identifier> (TypeArguments())? |
(Annotation())* <Identifier> (TypeArguments())? M()
}
And the error for Type is
Warning: Choice conflict involving two expansions at
line 796, column 3 and line 797, column 3 respectively.
A common prefix is: "#" <Identifier>
Consider using a lookahead of 3 or more for earlier expansion.
The error message tells you exactly what the problem is. There can be annotations at the start of both alternatives in Type. One way to deal with this is to factor out what's common, which is annotations.
Now you have
void Type() :
{ }
{
( Annotation() )*
( PrimitiveType() | ReferenceType() )
}
void PrimitiveType() :
{ }
{
NumericType() |
<KW_boolean>
}
void ReferenceType() :
{ }
{
ClassOrInterfaceType() |
TypeVariable() |
ArrayType()
}
void ClassOrInterfaceType() :
{ }
{
<Identifier> (TypeArguments())? |
<Identifier> (TypeArguments())? M()
}
That fixes the problem with Type. There are still lots of problems, but now there is one less.
For example, all three choices in ReferenceType can start with an identifier. In the end you will want something like this
void Type() :
{ }
{
( Annotation() )*
( PrimitiveType() | ReferenceTypesOtherThanArrays() )
( Dims() )?
}
void PrimitiveType() :
{ }
{
NumericType() | <KW_boolean>
}
void ReferenceTypesOtherThanArrays() :
{ }
{
<Identifier>
( TypeArguments() )?
(
<Token_Dot>
( Annotation() )*
<Identifier>
( TypeArguments() )?
)*
}
Notice that TypeVariable is gone. This is because there is no way to syntactically distinguish a type variable from a class (or interface) name. Thus the grammar just above will accept, say T.x, where T is a type variable, whereas the JLS grammar does not. This is the kind of error you can only rule out using a symbol table. There are a few of situations like this in Java; for example, without a symbol table, you can't tell a package name from a class name or a class name from a variable name; in an expression a.b.c, a could be a package name, a class name, an interface name, a type variable, a variable, or a field name.
You can handle these sorts of issues in one of two ways: you can deal with the problem after parsing, i.e. in a later phase, or you can have a symbol table present during the parsing phase and use the symbol table to guide the parser using semantic lookahead. The latter option is not a good one for Java, however; it is best to parse first and deal with all issues that need a symbol table later. This is because, in Java a symbol can be declared after it is used. It might even be declared in another file. What we did in the Java compiler for the Teaching Machine was to parse all files first. Then build a symbol table. Then do semantic analysis. Of course if your application does not require diagnosing all errors, then these considerations can largely be ignored.