I have these lexer rules in my ANTLR3 grammar:
INTEGER: DIGITS;
FLOAT: DIGITS? DOT_SYMBOL DIGITS ('E' (MINUS_OPERATOR | PLUS_OPERATOR)? DIGITS)?;
HEXNUMBER: '0X' HEXDIGIT+;
HEXSTRING: 'X' '\'' HEXDIGIT+ '\'';
BITNUMBER: '0B' ('0' | '1')+;
BITSTRING: 'B' '\'' ('0' | '1')+ '\'';
NCHAR_TEXT: 'N' SINGLE_QUOTED_TEXT;
IDENTIFIER: LETTER_WHEN_UNQUOTED+;
fragment LETTER_WHEN_UNQUOTED:
'0'..'9'
| 'A'..'Z' // Only upper case, as we use a case insensitive parser (insensitive only for ASCII).
| '$'
| '_'
| '\u0080'..'\uffff'
;
and
qualified_identifier:
IDENTIFIER ( options { greedy = true; }: DOT_SYMBOL IDENTIFIER)?
;
This works mostly fine except for very specific situations like the input t1.1_d which is supposed to be parsed as 2 identifiers connected with a dot. What happens is that .1 is matched as float even though it's followed by underscore and letter(s).
It's clear where that comes from: LETTER_WHEN_UNQUOTED includes digits so '1' can be both an integer and an identifier. But the rule order should take care to resolve this to an integer, as intented (and usually does).
However, I'm perplexed the t1.1_d input causes the float rule to kick in and would appreciate some pointers to resolve this problem. As soon as I add a space after the dot all is fine, but that is obviously not a real solution.
When I move the IDENTIFIER rule before the others I get new trouble because several other rules can no longer be matched then. Moving the FLOAT rule after the IDENTIFIER rule doesn't fix the problem either (but at least doesn't produce new problems). In this case we see the actual problem: the dot is always matched by the FLOAT rule if directly followed by a digit. What can I do to make it not match in my case?
The problem is that the lexer operates independently of the parser. When faced with the input string t1.1_d, the lexer will first consume an IDENTIFIER, leaving .1_d. You now want it to match DOT_SYMBOL, followed by IDENTIFIER. However, the lexer will always match the longest possible token, resulting in FLOAT matching .1.
Moving IDENTIFIER before FLOAT doesn't help, because '.' isn't a valid IDENTIFIER symbol and so can't match the input at all when it starts with ..
Note that Java and co. don't allow identifiers to start with numbers, probably to avert these kinds of problems.
One possible solution would be to change the FLOAT rule to require digits before the dot: FLOAT: DIGITS '.' DIGITS ...
Related
I need to allow only set of characters i.e.,
a to z A to Z 0 to 9 . !##$% *()_=+|[]{}"'';:?/.,-
but When I add dash(-) character to below query it is not working please help me at earliest.
SELECT :p_string FROM dual
WHERE NOT REGEXP_LIKE (translate(:p_string,chr(10)||chr(11)||chr(13), ' '),'[^]^A-Z^a-z^0-9^[^.^{^}^!^#^#^$^%^*^(^)^_^=^+^|^\^{^}^"^''^;^:^?^/^,^-^ ]' );
[.-.] will work fine on this query .
The extra ^ symbols inside the bracket expression in your pattern are not, as I think you expect, negations; only the first ^ inside the brackets does that.
The main issue that is causing, apart from allowing that actual circumflex symbol to be matched when you didn't seem to want it, is that you end up with ^-^ being treated as a range.
To include a literal - it has to be the first or last thing in the brackets; from the docs:
To specify a right bracket (]) in the bracket expression, place it first in the list (after the initial circumflex (^), if any).
To specify a hyphen in the bracket expression, place it first in the list (after the initial circumflex (^), if any), last in the list, or as an ending range point in a range expression.
So as you need to do both, make the hyphen last; you can change your pattern to:
'[^]A-Za-z0-9[.{}!##$%*()_=+|\{}"'';:?/, -]'
You could also skip the tralsnate step by including those special characters in the pattern too:
'[^]A-Za-z0-9[.{}!##$%*()_=+|\{}"'';:?/, '||chr(10)||chr(11)||chr(13)||'-]'
Looks like you need to permit only (7-bit) ASCII characters with exception of ~ and ^
In this case I would try it like this:
WHERE CONVERT(p_string, 'US7ASCII') = p_string
AND NOT REGEXP_LIKE(p_string, '~|\^')
Instead of CONVERT(p_string, 'US7ASCII') = p_string you can also use ASCIISTR(REPLACE(p_string, '\', '/')) = REPLACE(p_string, '\', '/')
I'm working on a lexer for Ruby. Such a lexer needs to clearly
distinguish divide '/' operators from regex /..../ operands.
Lexers are nicest to build when they are context free (stateless)
with respect to lexing-the-next token.
Some program text that starts with "/" might be:
... / abc*(foo(def,bar[q-z]*)+sam) / ...
You can't tell if the '/' symbol is a divide or the start of regexp.
So clearly Ruby must be looking at the context, or it must have rule
to decide when it is ambiguous. What's the rule?
[one possibility: it only allows them where divide cannot occur, e.g, after
when [ ( , #{ { if elseif != = !~ + , << and or not
(Edit 8/24/2015: extended the above list)
Does that cover everything? Or it is something entirely different?]
The Ruby lexer emits completely different tokens for a division operator and for the start of a regex (one is '/', the other tREGEXP_BEG). So the parser has no idea that the two actually use the same source text.
How does the lexer know which token to emit? See parse.y:8451 from the Ruby source.
The parser_params struct which is passed to the lexer has a member called lex.state. This is a bitfield, with each bit indicating something about the lexer state. The individual bits are called BEG, END, ENDARG, ENDFN, ARG, CMDARG, MID, FNAME, DOT, CLASS, LABEL, and LABELED.
When the lexer sees a '/' character, it emits tREGEXP_BEG if...
The lexer state is true for both ARG and LABELED, or
The lexer state is true for any one of BEG, MID, or CLASS.
Otherwise, it emits a division operator token.
So what do the states actually mean? The Ruby source contains the following comments on them:
EXPR_BEG_bit, /* ignore newline, +/- is a sign. */
EXPR_END_bit, /* newline significant, +/- is an operator. */
EXPR_ENDARG_bit, /* ditto, and unbound braces. */
EXPR_ENDFN_bit, /* ditto, and unbound braces. */
EXPR_ARG_bit, /* newline significant, +/- is an operator. */
EXPR_CMDARG_bit, /* newline significant, +/- is an operator. */
EXPR_MID_bit, /* newline significant, +/- is an operator. */
EXPR_FNAME_bit, /* ignore newline, no reserved words. */
EXPR_DOT_bit, /* right after `.' or `::', no reserved words. */
EXPR_CLASS_bit, /* immediate after `class', no here document. */
EXPR_LABEL_bit, /* flag bit, label is allowed. */
EXPR_LABELED_bit, /* flag bit, just after a label. */
Whenever the lexer emits a token, depending on the current lexer state, the token which was lexed, and possibly what the lexer sees next in the source text (it does look ahead in a number of places), it may move to a new state.
Some of the states are only entered after lexing a reserved keyword. For example, EXPR_MID is entered after lexing break, next, rescue, or return.
This is because of the way how the parser is defined. Having a look at BNF definition of Ruby you can see that the division operation (in the ARGS section) is defined before the definition of a REGEXP. That's why the division operation has a higher precedence than a regexp.
Meaning, if the ruby parser stumbles upon a section that resolves to
ARG / ARG
it will treat it as a division and goes further.
Walking trough a flex/bison tutorial will enlighten you! (Plus it is a fun)
Is there a way to parse words that start with a specific character?
I've been trying the following but i couldn't get any promising results:
//This one is working it accepts AD CD and such
example1
:
.'D'
;
//This one is not, it expects character D, then any ws character then any character
example2
:
'D'.
;
//These two are not working either
example3
:
'D'.*
;
//Doesn't accept input due to error: "line 1:3 missing 'D' at '<EOF>'"
example4
:
.*'D'
;
//just in case my WS rule:
/** WhiteSpace Characters (HIDDEN)*/
WS : ( ' '
| '\t'
)+ {$channel=HIDDEN;}
;
I am using ANTLR 3.4
Thanks in advance
//This one is not, it expects character D, then any ws character then any character
example2
:
'D'.
;
No, it does not it accept the token (not character!) 'D' followed by a space and then any character. Since example2 is a parser rule, it does not match characters, but matches tokens (there's a big difference!). And since you put spaces on a separate channel, the spaces are not matched by this rule either. At the end, the . (DOT) matches any token (again: not any character!).
More info on meta chars (like the . (DOT)) whose meaning differ inside lexer- and parser rules: Negating inside lexer- and parser rules
//These two are not working either
example3
:
'D'.*
;
//Doesn't accept input due to error: "line 1:3 missing 'D' at '<EOF>'"
example4
:
.*'D'
;
Unless you know exactly what you're doing, don't use .*: they gobble up too much in your case (especially when placed at the start or end of a rule).
It looks like you're trying to tokenize things inside the parser (all your example rules are parser rules). As far as I can see, these should be lexer rules instead. More on the difference between parser- and lexer rules, see: Practical difference between parser rules and lexer rules in ANTLR?
I have a short question:
// Lexer
LOOP_NAME : (LETTER|DIGIT)+;
OTHERCHARS : ~('>' | '}')+;
LETTER : ('A'..'Z')|('a'..'z');
DIGIT : ('0'..'9');
A_ELEMENT
: (LETTER|'_')*(LETTER|DIGIT|'_'|'.');
// Parser-Konfiguration
WS : ( ' '
| '\t'
| '\r'
| '\n'
) {$channel=HIDDEN;}
;
My problem is that this is impossible due to:
As a result, alternative(s) 2 were disabled for that input [14:55:32]
error(208): ltxt2.g:61:1: The following token definitions can never be
matched because prior tokens match the same input:
LETTER,DIGIT,A_ELEMENT,WS
My issue is that I also need to catch UTF8 with OTHERCHARS... and I cannot put all special UTF8 chars into a Lexer rule since I cannot range like ("!".."?").
So I need the NOT (~). The OTHERCHARS here can be everything but ">" or "}". These two close a literal context and are forbidden within.
It doesn't seem such cases are referenced very well, so I'd be happy if someone knew a workaround. The NOT operator here creates the ambivalence I need to solve.
Thanks in advance.
Best,
wishi
Move OTHERCHARS to the very end of the lexer and define it like this:
OTHERCHARS : . ;
In the Java target, this will match a single UTF-16 code point which is not matched by a previous rule. I typically name the rule ANY_CHAR and treat it as a fall-back. By using . instead of .+, the lexer will only use this rule if no other rule matches.
If another rule matches more than one character, that rule will have priority over ANY_CHAR due to matching a larger number of characters from the input.
If another rule matches exactly one character, that rule will have priority over ANY_CHAR due to appearing earlier in the grammar.
Edit: To exclude } and > from the ANY_CHAR rule, you'll want to create rules for them so they are covered under point 2.
RBRACE : '}' ;
GT : '>' ;
ANY_CHAR : . ;
I need to be able to match a certain string ('[' then any number of equals signs or none then '['), then i need to match a matching close bracket (']' then the same number of equals signs then ']') after some other match rules. ((options{greedy=false;}:.)* if you must know). I have no clue how to do this in ANTLR, how can i do it?
An example: I need to match [===[whatever arbitrary text ]===] but not [===[whatever arbitrary text ]==].
I need to do it for an arbitrary number of equals signs as well, so therein lies the problem: how do i get it to match an equal number of equals signs in the open as in the close? The supplied parser rules so far dont seem to make sense as far as helping.
You can't easely write a lexer for it, you need parsing rules. Two rules should be sufficient. One is responsible for matching the braces, one for matching the equal signs.
Something like this:
braces : '[' ']'
| '[' equals ']'
;
equals : '=' equals '='
| '=' braces '='
;
This should cover the use case you described. Not absolute shure but maybe you have to use a predicate in the first rule of 'equals' to avoid ambiguous interpretations.
Edit:
It is hard to integrate your greedy rule and at the same time avoid a lexer context switch or something similar (hard in ANTLR). But if you are willing to integrate a little bit of java in your grammer you can write an lexer rule.
The following example grammar shows how:
grammar TestLexer;
SPECIAL : '[' { int counter = 0; } ('=' { counter++; } )+ '[' (options{greedy=false;}:.)* ']' ('=' { counter--; } )+ { if(counter != 0) throw new RecognitionException(input); } ']';
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*
;
WS : ( ' '
| '\t'
| '\r'
| '\n'
) {$channel=HIDDEN;}
;
rule : ID
| SPECIAL
;
Your tags mention lexing, but your question itself doesn't. What you're trying to do is non-regular, so I don't think it can be done as part of lexing (though I don't remember if ANTLR's lexer is strictly regular -- it's been a couple of years since I last used ANTLR).
What you describe should be possible in parsing, however. Here's the grammar for what you described:
thingy : LBRACKET middle RBRACKET;
middle : EQUAL middle EQUAL
| LBRACKET RBRACKET;