I am trying to do basic ANTLR-based scanning. I have a problem with a lexer not matching wanted tokens.
lexer grammar DefaultLexer;
ALPHANUM : (LETTER | DIGIT)+;
ACRONYM : LETTER '.' (LETTER '.')+;
HOST : ALPHANUM (('.' | '-') ALPHANUM)+;
fragment
LETTER : UNICODE_CLASS_LL | UNICODE_CLASS_LM | UNICODE_CLASS_LO | UNICODE_CLASS_LT | UNICODE_CLASS_LU;
fragment
DIGIT : UNICODE_CLASS_ND | UNICODE_CLASS_NL;
For the grammar above, hello. world string given as an input results in world only. Whereas I would expect to get both hello and world. What am I missing? Thanks.
ADDED:
Ok, I learned that input hello. world matches more characters using rule HOST than ALPHANUM, therefore lexer will choose to use it. Then, when it fails to match input to the HOST rule, it does not "look back" to , because that's how lexer works.
How I get around it?
As a foreword, ANTLR 4 would not behave in a strange manner here. Both ANTLR 3 and ANTLR 4 should be matching ALPHANUM, then giving 2 syntax errors, then matching another ALPHANUM, and I can state with confidence that ANTLR 4 will behave that way.
It looks like your HOST rule might be better suited to be host, a parser rule.
You need to make sure and provide a lexer rule that can match . (either together or as two separate tokens).
Related
I have these two rules from a jflex code:
Bool = true
Ident = [:letter:][:letterdigit:]*
if I try for example to analyse the word "trueStat", it gets recognnized as an Ident expression and not Bool.
How can I avoid this type of ambiguity in Jflex?
In almost all languages, a keyword is only recognised as such if it is a complete word. Otherwise, you would end up banning identifiers like format, downtime and endurance (which would instead start with the keywords for, do and end, respectively). That's quite confusing for programmers, although it's not unheard-of. Lexical scanner generators, like Flex and JFlex generally try to make the common case easy; thus, the snippet you provide, which recognises trueStat as an identifier. But if you really want to recognise it as a keyword followed by an identifier, you can accomplish that by adding trailing context to all your keywords:
Bool = true/[:letterdigit:]*
Ident = [:letter:][:letterdigit:]*
With that pair of patterns, true will match the Bool rule, even if it occurs as trueStat. The pattern matches true and any alphanumeric string immediately following it, and then rewinds the input cursor so that the token matched is just true.
Note that like Lex and Flex, JFlex accepts the longest match at the current input position; if more than one rule accepts this match, the action corresponding to the first such rule is executed. (See the manual section "How the Input is Matched" for a slightly longer explanation of the matching algorithm.) Trailing context is considered part of the match for the purposes of this rule (but, as noted above, is then removed from the match).
The consequence of this rule is that you should always place more specific patterns before the general patterns they might override, whether or not the specific pattern uses trailing context. So the Bool rule must precede the Ident rule.
This is a dummy example, my actual language is more complicated:
grammar wordasnumber;
WS: [ \t\n] -> skip;
AS: [Aa] [Ss];
ID: [A-Za-z]+;
NUMBER: [0-9]+;
wordAsNumber: (ID AS NUMBER)* EOF;
In this language, these two strings are legal:
seven as 7 eight as 8
seven as 7eight as8
Which is exactly what I told it to do, but not what I want. Because ID and AS are both strings of letters, white space is required between them, I would like that second phrase
to be a syntax error. I could add some other rule to try and match theses mashed up things ...
fragment LETTER: [A-Za-z];
fragment DIGIT: [0-9];
BAD_THING: ( LETTER+ DIGIT (LETTER|DIGIT)* ) | ( DIGIT+ LETTER (LETTER|DIGIT)* );
ID: LETTER+;
NUMBER: DIGIT+;
... to make the lexer return a different token for these smashed up things, but this feels like a weird bandaid which sort of found the need for accidentally and maybe there are more if I really stared at my lexer very carefully.
Is there a better way to do this? My actual grammar is much larger so, for example, making WS NOT be skipped and placing it explicitly between the tokens where it is required is non starter.
There was an older question on this list, which I could not find, which I think is the same question, in that case someone who was parsing white space separated numbers was surprised that 1.2.3 was parsing as 1.2 and .3 and not as a syntax error.
Add another rule for the wrong input, but don't use that in your parser. It will then cause a syntax error when matched:
INVALID: (ID | NUMBER)+;
This additional rule will change the parse tree output, for the input in the question, to:
This trick works because ANTLR4's lexing approach tries to match the longest input in on go, and that INVALID rule matches more than ID and NUMBER alone. But you have to place it after these 2 rules, to make use of another lexing rule: "If two lexer rules would match the same input, pick the first one.". This way, you get the correct tokens for single appearances of ID and NUMBER.
I have been starting to use ANTLR and have noticed that it is pretty fickle with its lexer rules. An extremely frustrating example is the following:
grammar output;
test: FILEPATH NEWLINE TITLE ;
FILEPATH: ('A'..'Z'|'a'..'z'|'0'..'9'|':'|'\\'|'/'|' '|'-'|'_'|'.')+ ;
NEWLINE: '\r'? '\n' ;
TITLE: ('A'..'Z'|'a'..'z'|' ')+ ;
This grammar will not match something like:
c:\test.txt
x
Oddly if I change TITLE to be TITLE: 'x' ; it still fails this time giving an error message saying "mismatched input 'x' expecting 'x'" which is highly confusing. Even more oddly if I replace the usage of TITLE in test with FILEPATH the whole thing works (although FILEPATH will match more than I am looking to match so in general it isn't a valid solution for me).
I am highly confused as to why ANTLR is giving such extremely strange errors and then suddenly working for no apparent reason when shuffling things around.
This seems to be a common misunderstanding of ANTLR:
Language Processing in ANTLR:
The Language Processing is done in two strictly separated phases:
Lexing, i.e. partitioning the text into tokens
Parsing, i.e. building a parse tree from the tokens
Since lexing must preceed parsing there is a consequence: The lexer is independent of the parser, the parser cannot influence lexing.
Lexing
Lexing in ANTLR works as following:
all rules with uppercase first character are lexer rules
the lexer starts at the beginning and tries to find a rule that matches best to the current input
a best match is a match that has maximum length, i.e. the token that results from appending the next input character to the maximum length match is not matched by any lexer rule
tokens are generated from matches:
if one rule matches the maximum length match the corresponding token is pushed into the token stream
if multiple rules match the maximum length match the first defined token in the grammar is pushed to the token stream
Example: What is wrong with your grammar
Your grammar has two rules that are critical:
FILEPATH: ('A'..'Z'|'a'..'z'|'0'..'9'|':'|'\\'|'/'|' '|'-'|'_'|'.')+ ;
TITLE: ('A'..'Z'|'a'..'z'|' ')+ ;
Each match, that is matched by TITLE will also be matched by FILEPATH. And FILEPATH is defined before TITLE: So each token that you expect to be a title would be a FILEPATH.
There are two hints for that:
keep your lexer rules disjunct (no token should match a superset of another).
if your tokens intentionally match the same strings, then put them into the right order (in your case this will be sufficient).
if you need a parser driven lexer you have to change to another parser generator: PEG-Parsers or GLR-Parsers will do that (but of course this can produce other problems).
This was not directly OP's problem, but for those who have the same error message, here is something you could check.
I had the same Mismatched Input 'x' expecting 'x' vague error message when I introduced a new keyword. The reason for me was that I had placed the new key word after my VARNAME lexer rule, which assigned it as a variable name instead of as the new keyword. I fixed it by putting the keywords before the VARNAME rule.
Below is a snippet of a FLEX program
%%
a |
ab |
abc |
abcd ECHO; REJECT;
.|\n printf("xx%c", *yytext);
%%
Input:
abcd
Output:
abcdabcabaxxaxxbxxcxxdxx
Can someone explain how to get this output?
REJECT effectively causes flex to backup to the next-better match, bearing in mind the precedence rules for flex:
Match the longest possible token.
Of the tokens with the same length, prefer the pattern earlier in the source file.
In your case, the string abcd will match a, ab, abc or abcd; the preferred one is the longest one (abcd).
Since you have an ECHO action before the REJECT action, the ECHO happens even though the match will later be rejected. Eventually, flex will fall back to the default rule (which also matches a but is later in the source file), which will print xxa and accept the character. Now, nothing matches except for the default rule, so the next three characters get matched one at a time against it.
This would probably have been much clearer if you'd written '\n's to stdout.
I'm trying to write a data validation rule that ensures Shipping Street is one line (doesn't contain line breaks)
I've tried things like
CONTAINS( ShippingStreet , BR() ), and
CONTAINS( ShippingStreet , "\n" ),
but I can't get the rule to trigger.
Any help?
This will do it:
REGEX(ShippingStreet,'.*\\n.*')
There are 2 things to learn from this question about SFDC REGEX parsing:
(1) As per Java SE 6 Pattern syntax, you need to double-escape the new-line character (\n), along with various other special characters, when used in a string that gets compiled to a regular expression, that is, use '\n'.
(2) The Salesforce Regular Expression parser matches the entire phrase by default. To match on just part of the phrase, you have to surround your pattern with .*
Examples:
1. REGEX('Marc Benioff','Marc Benioff') -> TRUE
2. REGEX('Marc Benioff is a CEO','Marc Benioff') -> FALSE
3. REGEX('Marc Benioff','.*Marc Benioff.*') -> TRUE
4. REGEX('Marc Benioff is a CEO','.*Marc Benioff.*') -> TRUE
For more info, read the 'Tips' section of the SFDC REGEX Help docs.
The following in a validation rule should mean that only ShippingStreet values containing characters only (and thereby no line breaks and the like) are accepted.
NOT(REGEX(ShippingStreet, '.*'))