ANTLR4. Comments not being swallowed by the lexer - comments

Question

I have the following Antlr4 grammar:
grammar Smarty;
/*
* Parser Rules
*/
parse: smartyBody
;
smartyBody: include
| bodyText?
;
include: INCLUDE fileName CB
;
bodyText: BODY_TEXT
;
fileName: FILENAME
;
/*
* Lexer Rules
*/
COMMENT: '{*' .*? '*}' -> skip;
INCLUDE: '{include' SPACE+ 'file='?;
BODY_TEXT: ANY_CHAR+;
CB: SPACE? '}';
FILENAME: '\'' FILE_NAME '\''
| '"' FILE_NAME '"';
SPACE: [ \t];
NEW_LINE: [\r\n]+ -> skip;
fragment FILE_NAME: PATH_CHARACTERS+;
fragment PATH_CHARACTERS: ~[\u0000-\u001f"*:<>?\\/|\u007f-\uffff];
fragment ANY_CHAR: [ -~];
Now if i try to use comments they are passed back and not swallowed.
Text passed to the parser: {* {include file='xxxxx'} *} zzzzzzzz
If I override EnterBodyText and pass the text string to the parser the Text variable in the listener get the original text including the comments.
internal partial class BaseListener : SmartyBaseListener
{
public override void EnterBodyText( SmartyParser.BodyTextContext context )
{
var Text = context.BODY_TEXT().GetText();
// Text gets: {* {include file='xxxxx'} *} zzzzzzzz
// not just zzzzzzzz
}
}
Any help appreciated.

Answer 1

ANTLR's lexer will try to match as much as possible. So the input {* {include file='xxxxx'} *} zzzzzzzz is matched entirely by BODY_TEXT.
One way to solve this is make BODY_TEXT a parser rule instead:
body_text: ANY_CHAR+;
...
COMMENT: '{*' .*? '*}' -> skip;
...
// No fragment anymore. And keep this as the last lexer rule!
ANY_CHAR: [ -~];