ANTL4 Parser (Flat parser vs Structor parse) for Language Translator - performance

Question

Over the last couple of months, with the help of members from this site, I have been able to write (Phase 1) a Lexer and Parser to translate Lang X to Java. Because I was new to this topic, I opted for a simple line by line, parser and now it's able to parse around 1000 language files in 15 minutes with a small number of errors/exceptions and circa 1M lines of code, with the problems being isolated to the source files not the parser. I will refer to this a flat parsing, for want of a better expression.
Now for Phase 2, the translation to Java. Like any language, mine has Data Structures, Procedures, Sub-routines, etc and I thought it best to alter the parser from below (for simplicity I have focussed on the Data Structure (called TABLE)):
// Main entry point of the program
program
: executableUnit+ EOF
;
// Execution units (line by line)
executableUnit:
| itemBlockStart
| itemBlockEnd
| itemStatement
| tableHeader
;
itemBlockStart: BEGIN;
itemBlockEnd: END;
tableHeader: // A TABLE declaration statement
TABLE atom LETTER (atom)*
;
// Item statement
itemStatement:
// Tables with Item statements
ITEM atom+
// Base atom lowest of the low
atom:
MINUS? INT #IntegerAtom
| REAL_FORMAT #RealAtom
| FIX_POINT #FixPointAtom
| (MINUS | EQUALS)? NAME DOT? #NameAtom
| LETTER #LetterAtom
| keywords DOT? #KeywordAtom
| DOLLAR atom DOLLAR #DollarAtom
| hex_assign #HexItem
;
to this:
// Execution units (by structure)
executableUnit:
tableStatement
| itemStatement
;
// Table statement, header and body
tableStatement:
tableHeader (itemBlockStart | itemBlockEnd | itemStatement)*;
Before we go any futher, TABLE and individual ITEM statements can occur anywhere in the code, on their own (Java output would be public) or inside a Procedure (Jave output would be private)
Imagine my dismay (if you will) when the parser produced the same number of errors, but took 10 times longer to parser the input. I kind of understand the increased time period, in terms of selecting the right path. My questions for the group are:
Is there a way to force the parser down the TABLE structure early to reduce the time period?
Whether having this logical tree structure grouping is worth the increased time?
My desire to move in this direction was to have a Listener callback with a mini tree with all the relevant items accessable to walk. I.e. If the mini tree wasnt inside a Procedure statement is was public in Java.

Answer 1

It's not entirely clear to me what performance difference you are referring to (presumably, the difference between the "line by line" parser, and this, full file, parser. (???)
A few things that "jump out" about your grammar, and could have some performance impact:
1 - itemBlockStart: BEGIN; and itemBlockEnd: END;. There's no point in have a rule that is a single Token. Just use the token in the rule definition.
2 - You are, probably unintentionally, being VERY relaxed in the acceptance of itemStartBlock and itemStopBlock in this rule (tableStatement: tableHeader (itemBlockStart | itemBlockEnd | itemStatement)*;). This could also have performance implications. I'm assuming in the rest of this response that BEGIN should appear at the beginning of an itemStatement and END should appear at the end (not that the three can appear in any order willy-nilly).
Try this refactoring:
// Main entry point of the program
program
: executableUnit+ EOF
;
// Execution units (line by line)
executableUnit:
| itemStatement # ItemStmt
| tableHeader # TableHeader
;
tableHeader: // A TABLE declaration statement
TABLE atom LETTER atom*
;
// Item statement
itemStatement: // Tables with Item statements
BEGIN ITEM atom+ END
;
// Base atom lowest of the low
atom: MINUS? INT #IntegerAtom
| REAL_FORMAT #RealAtom
| FIX_POINT #FixPointAtom
| (MINUS | EQUALS)? NAME DOT? #NameAtom
| LETTER #LetterAtom
| keywords DOT? #KeywordAtom
| DOLLAR atom DOLLAR #DollarAtom
| hex_assign #HexItem
;
admittedly, I can't quite make out what your intention is, but this should be a step in the right direction.
As Kaby76 points out, the greedy operator at the end of the tableHeader is quite likely to "gobble up" a lot of input. This is partly because of the lack of a terminator token (which would, no doubt, stop the token consumption earlier than not having a termination token. However, your atom rule seems to be something of a "kitchen sink" rule that can match all manner of input. Couple that with the use of atom+ and atom* and there's quite the likelihood of consuming a long stream of tokens. Is it really your intention that any of the atoms can appear one after the other with no structure? They appear to be pieces/parts of expressions. If that's the case, you will want to define your grammar for expressions. This added structure will both help performance and give you a MUCH more useful parse tree to act upon.
Much like the structure for tableStatement in your question's grammar, it doesn't really represent any structure (see my recommendation to change it to BEGIN ITEM atom+ END rather than accepting any combination in any order. The same thought process needs to be applied to atom. Both of these approaches let ANTLR march through your code consuming a LOT of tokens without any clue as to whether the order is actually correct (which is then very expensive to attempt to "back out of" when a problem is encountered).