R7RS-small says that all identifiers must be terminated by a delimiter, but at the same time it defines pretty elaborate rules for what can be in an identifier. So, which one is it?
Is an identifier supposed to start with an initial character and then continue until a delimiter, or does it start with an initial character and continue following the syntax defined in 7.1.1.
Here are a couple of obvious cases. Are these valid identifiers?
a#a
b,b
c'c
d[d]
If they are not supposed to be valid, what is the purpose of saying that an identifier must be terminated by a delimiter?
|..ident..| are delimiters for symbols in R7RS, to allow any character that you cannot insert in an old style symbol (| is the delimiter).
However, in R6RS the "official" grammar was incorrect, as it did not allow to define symbols such that 1+, which led all implementations define their own rules to overcome this illness of the official grammar.
Unless you need to read the source code of a given implementation and see how it defines the symbols, you should not care too much about these rules and use classical symbols.
In the section 7.1.1 you find the backus-naur form that defines the lexical structure of R7RS identifiers but I doubt the implementations follow it.
I quote from here
As with identifiers, different implementations of Scheme use slightly
different rules, but it is always the case that a sequence of
characters that contains no special characters and begins with a
character that cannot begin a number is taken to be a symbol
In other words, an implementation will use a function like read-atom and after that it will classify an atom by backtracking with read-number and if number? fails it will be a symbol.
I have been starting to use ANTLR and have noticed that it is pretty fickle with its lexer rules. An extremely frustrating example is the following:
grammar output;
test: FILEPATH NEWLINE TITLE ;
FILEPATH: ('A'..'Z'|'a'..'z'|'0'..'9'|':'|'\\'|'/'|' '|'-'|'_'|'.')+ ;
NEWLINE: '\r'? '\n' ;
TITLE: ('A'..'Z'|'a'..'z'|' ')+ ;
This grammar will not match something like:
c:\test.txt
x
Oddly if I change TITLE to be TITLE: 'x' ; it still fails this time giving an error message saying "mismatched input 'x' expecting 'x'" which is highly confusing. Even more oddly if I replace the usage of TITLE in test with FILEPATH the whole thing works (although FILEPATH will match more than I am looking to match so in general it isn't a valid solution for me).
I am highly confused as to why ANTLR is giving such extremely strange errors and then suddenly working for no apparent reason when shuffling things around.
This seems to be a common misunderstanding of ANTLR:
Language Processing in ANTLR:
The Language Processing is done in two strictly separated phases:
Lexing, i.e. partitioning the text into tokens
Parsing, i.e. building a parse tree from the tokens
Since lexing must preceed parsing there is a consequence: The lexer is independent of the parser, the parser cannot influence lexing.
Lexing
Lexing in ANTLR works as following:
all rules with uppercase first character are lexer rules
the lexer starts at the beginning and tries to find a rule that matches best to the current input
a best match is a match that has maximum length, i.e. the token that results from appending the next input character to the maximum length match is not matched by any lexer rule
tokens are generated from matches:
if one rule matches the maximum length match the corresponding token is pushed into the token stream
if multiple rules match the maximum length match the first defined token in the grammar is pushed to the token stream
Example: What is wrong with your grammar
Your grammar has two rules that are critical:
FILEPATH: ('A'..'Z'|'a'..'z'|'0'..'9'|':'|'\\'|'/'|' '|'-'|'_'|'.')+ ;
TITLE: ('A'..'Z'|'a'..'z'|' ')+ ;
Each match, that is matched by TITLE will also be matched by FILEPATH. And FILEPATH is defined before TITLE: So each token that you expect to be a title would be a FILEPATH.
There are two hints for that:
keep your lexer rules disjunct (no token should match a superset of another).
if your tokens intentionally match the same strings, then put them into the right order (in your case this will be sufficient).
if you need a parser driven lexer you have to change to another parser generator: PEG-Parsers or GLR-Parsers will do that (but of course this can produce other problems).
This was not directly OP's problem, but for those who have the same error message, here is something you could check.
I had the same Mismatched Input 'x' expecting 'x' vague error message when I introduced a new keyword. The reason for me was that I had placed the new key word after my VARNAME lexer rule, which assigned it as a variable name instead of as the new keyword. I fixed it by putting the keywords before the VARNAME rule.
Let a programming language PL. Its variable definition part is described as follows:
At the beginning we have to start with the keyword "var". After writing this keyword we have to write the variable names(one or more) separated by commas ",". Then a colon ":" is inserted and after that we must write the variable type(say real, boolean, integer or char in my example) followed by a semicolon ";". After doing the previous steps there is the potentiality to declare into a new line new variables(variable names separrated by commas "," followed by colon ":" followed by variable type followed by a semicolon ";"), but we must not use the "var" keyword again at the beginning of the new line( the "var" keyword is written once!!!)
E.g.
var number_of_attendants, sum: integer;
ticket_price: real;
symbols: char;
I would like the EBNF of the above descripted programming language PL to understand it better. Ok, I may have read some texts about EBNF, but in my opinion an example is always something very useful.
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
What is this?
This is a collection of common Q&A. This is also a Community Wiki, so everyone is invited to participate in maintaining it.
Why is this?
regex is suffering from give me ze code type of questions and poor answers with no explanation. This reference is meant to provide links to quality Q&A.
What's the scope?
This reference is meant for the following languages: php, perl, javascript, python, ruby, java, .net.
This might be too broad, but these languages share the same syntax. For specific features there's the tag of the language behind it, example:
What are regular expression Balancing Groups? .net
The Stack Overflow Regular Expressions FAQ
See also a lot of general hints and useful links at the regex tag details page.
Online tutorials
RegexOne ↪
Regular Expressions Info ↪
Quantifiers
Zero-or-more: *:greedy, *?:reluctant, *+:possessive
One-or-more: +:greedy, +?:reluctant, ++:possessive
?:optional (zero-or-one)
Min/max ranges (all inclusive): {n,m}:between n & m, {n,}:n-or-more, {n}:exactly n
Differences between greedy, reluctant (a.k.a. "lazy", "ungreedy") and possessive quantifier:
Greedy vs. Reluctant vs. Possessive Quantifiers
In-depth discussion on the differences between greedy versus non-greedy
What's the difference between {n} and {n}?
Can someone explain Possessive Quantifiers to me? php, perl, java, ruby
Emulating possessive quantifiers .net
Non-Stack Overflow references: From Oracle, regular-expressions.info
Character Classes
What is the difference between square brackets and parentheses?
[...]: any one character, [^...]: negated/any character but
[^] matches any one character including newlines javascript
[\w-[\d]] / [a-z-[qz]]: set subtraction .net, xml-schema, xpath, JGSoft
[\w&&[^\d]]: set intersection java, ruby 1.9+
[[:alpha:]]:POSIX character classes
[[:<:]] and [[:>:]] Word boundaries
Why do [^\\D2], [^[^0-9]2], [^2[^0-9]] get different results in Java? java
Shorthand:
Digit: \d:digit, \D:non-digit
Word character (Letter, digit, underscore): \w:word character, \W:non-word character
Whitespace: \s:whitespace, \S:non-whitespace
Unicode categories (\p{L}, \P{L}, etc.)
Escape Sequences
Horizontal whitespace: \h:space-or-tab, \t:tab
Newlines:
\r, \n:carriage return and line feed
\R:generic newline php java-8
Negated whitespace sequences: \H:Non horizontal whitespace character, \V:Non vertical whitespace character, \N:Non line feed character pcre php5 java-8
Other: \v:vertical tab, \e:the escape character
Anchors
anchor
matches
flavors
^
Start of string
Common*
^
Start of line
Commonm
$
End of line
Commonm
$
End of text
Common* except javascript
$
Very end of string
javascript*, phpD
\A
Start of string
Common except javascript
\Z
End of text
Common except javascript python
\Z
Very end of string
python
\z
Very end of string
Common except javascript python
\b
Word boundary
Common
\B
Not a word boundary
Common
\G
End of previous match
Common except javascript, python
Term
Definition
Start of string
At the very start of the string.
Start of line
At the very start of the string, andafter a non-terminal line terminator.
Very end of string
At the very end of the string.
End of text
At the very end of the string, andat a terminal line terminator.
End of line
At the very end of the string, andat a line terminator.
Word boundary
At a word character not preceded by a word character, andat a non-word character not preceded by a non-word character.
End of previous match
At a previously set position, usually where a previous match ended.At the very start of the string if no position was set.
"Common" refers to the following: icu java javascript .net objective-c pcre perl php python swift ruby
* Default |
m Multi-line mode. |
D Dollar end only mode.
Groups
(...):capture group, (?:):non-capture group
Why is my repeating capturing group only capturing the last match?
\1:backreference and capture-group reference, $1:capture group reference
What's the meaning of a number after a backslash in a regular expression?
\g<1>123:How to follow a numbered capture group, such as \1, with a number?: python
What does a subpattern (?i:regex) mean?
What does the 'P' in (?P<group_name>regexp) mean?
(?>):atomic group or independent group, (?|):branch reset
Equivalent of branch reset in .NET/C# .net
Named capture groups:
General named capturing group reference at regular-expressions.info
java: (?<groupname>regex): Overview and naming rules (Non-Stack Overflow links)
Other languages: (?P<groupname>regex) python, (?<groupname>regex) .net, (?<groupname>regex) perl, (?P<groupname>regex) and (?<groupname>regex) php
Lookarounds
Lookaheads: (?=...):positive, (?!...):negative
Lookbehinds: (?<=...):positive, (?<!...):negative
Lookbehind limits in:
Lookbehinds need to be constant-length php, perl, python, ruby
Lookarounds of limited length {0,n} java
Variable length lookbehinds are allowed .net
Lookbehind alternatives:
Using \K php, perl (Flavors that support \K)
Alternative regex module for Python python
The hacky way
JavaScript negative lookbehind equivalents External link
Modifiers
flag
modifier
flavors
a
ASCII
python
c
current position
perl
e
expression
php perl
g
global
most
i
case-insensitive
most
m
multiline
php perl python javascript .net java
m
(non)multiline
ruby
o
once
perl ruby
r
non-destructive
perl
S
study
php
s
single line
ruby
U
ungreedy
php r
u
unicode
most
x
whitespace-extended
most
y
sticky ↪
javascript
How to convert preg_replace e to preg_replace_callback?
What are inline modifiers?
What is '?-mix' in a Ruby Regular Expression
Other:
|:alternation (OR) operator, .:any character, [.]:literal dot character
What special characters must be escaped?
Control verbs (php and perl): (*PRUNE), (*SKIP), (*FAIL) and (*F)
php only: (*BSR_ANYCRLF)
Recursion (php and perl): (?R), (?0) and (?1), (?-1), (?&groupname)
Common Tasks
Get a string between two curly braces: {...}
Match (or replace) a pattern except in situations s1, s2, s3...
How do I find all YouTube video ids in a string using a regex?
Validation:
Internet: email addresses, URLs (host/port: regex and non-regex alternatives), passwords
Numeric: a number, min-max ranges (such as 1-31), phone numbers, date
Parsing HTML with regex: See "General Information > When not to use Regex"
Advanced Regex-Fu
Strings and numbers:
Regular expression to match a line that doesn't contain a word
How does this PCRE pattern detect palindromes?
Match strings whose length is a fourth power
How does this regex find triangular numbers?
How to determine if a number is a prime with regex?
How to match the middle character in a string with regex?
Other:
How can we match a^n b^n?
Match nested brackets
Using a recursive pattern php, perl
Using balancing groups .net
“Vertical” regex matching in an ASCII “image”
List of highly up-voted regex questions on Code Golf
How to make two quantifiers repeat the same number of times?
An impossible-to-match regular expression: (?!a)a
Match/delete/replace this except in contexts A, B and C
Match nested brackets with regex without using recursion or balancing groups?
Flavor-Specific Information
(Except for those marked with *, this section contains non-Stack Overflow links.)
Java
Official documentation: Pattern Javadoc ↪, Oracle's regular expressions tutorial ↪
The differences between functions in java.util.regex.Matcher:
matches()): The match must be anchored to both input-start and -end
find()): A match may be anywhere in the input string (substrings)
lookingAt(): The match must be anchored to input-start only
(For anchors in general, see the section "Anchors")
The only java.lang.String functions that accept regular expressions: matches(s), replaceAll(s,s), replaceFirst(s,s), split(s), split(s,i)
*An (opinionated and) detailed discussion of the disadvantages of and missing features in java.util.regex
.NET
How to read a .NET regex with look-ahead, look-behind, capturing groups and back-references mixed together?
Official documentation:
Boost regex engine: General syntax, Perl syntax (used by TextPad, Sublime Text, UltraEdit, ...???)
JavaScript general info and RegExp object
.NET MySQL Oracle Perl5 version 18.2
PHP: pattern syntax, preg_match
Python: Regular expression operations, search vs match, how-to
Rust: crate regex, struct regex::Regex
Splunk: regex terminology and syntax and regex command
Tcl: regex syntax, manpage, regexp command
Visual Studio Find and Replace
General information
(Links marked with * are non-Stack Overflow links.)
Other general documentation resources: Learning Regular Expressions, *Regular-expressions.info, *Wikipedia entry, *RexEgg, Open-Directory Project
DFA versus NFA
Generating Strings matching regex
Books: Jeffrey Friedl's Mastering Regular Expressions
When to not use regular expressions:
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. (blog post written by Stack Overflow's founder)*
Do not use regex to parse HTML:
Don't. Please, just don't
Well, maybe...if you're really determined (other answers in this question are also good)
Examples of regex that can cause regex engine to fail
Why does this regular expression kill the Java regex engine?
Tools: Testers and Explainers
(This section contains non-Stack Overflow links.)
Online (* includes replacement tester, + includes split tester):
Debuggex (Also has a repository of useful regexes) javascript, python, pcre
*Regular Expressions 101 php, pcre, python, javascript, java
Regex Pal, regular-expressions.info javascript
Rubular ruby RegExr Regex Hero dotnet
*+ regexstorm.net .net
*RegexPlanet: Java java, Go go, Haskell haskell, JavaScript javascript, .NET dotnet, Perl perl php PCRE php, Python python, Ruby ruby, XRegExp xregexp
freeformatter.com xregexp
*+regex.larsolavtorvik.com php PCRE and POSIX, javascript
Offline:
Microsoft Windows: RegexBuddy (analysis), RegexMagic (creation), Expresso (analysis, creation, free)
MySQL 8.0: Various syntax changes were made. Note especially the doubling of backslashes in some contexts. (This Answer need further editing to reflect the differences.)
This question was asked to me in an interview question:
Write a code to generate the parse tree like compilers do internally for any given expression. For example:
a+(b+c*(e/f)+d)*g
Start by defining the language. No one can implement a parser or a compiler to a language that isn't very well defined. You give an example: 'a+(b+c*(e/f)+d)*g', which should trigger the following questions:
Is the language a single expression, or there may be multiple statements (separated by ';' maybe?
what are the 'a', 'b', ... 'g' tokens? Is it variable? What is the syntax of variables? Is it a C-like variable, or is it a single alphanumeric character as your example may imply.
There are 3 binary expression in your example. Is that all? Does the language also support '-'. Does your language support logical, and bit-wise operators?
Does the language support number literals? integer only? double? Does the language support string literals? Do you quote string literals?
Syntax for comments?
Which operator has precedence? Does '*' operator has precedence over '+' as in the example? Does operands evaluated right to left or left to right?
Any Pre-processing?
Once you are equipped with a good definition of the language syntax, start with implementing a tokenizer. A tokenizer gets a stream of characters and generates a list of tokens. In the example above, each character is a token, but in var*12 (var power 12) there are 3 tokens: 'var', '*' and '12'. If regular expression is permitted, it is possible you can do this part of the parsing with regular expressions.
Next, have a function that identify each token by type: is it an operator, is it a variable, a number literal, string literal, etc. Package all in a method called NextToken that returns a token and its type.
Finally, start parsing. In your sample above the root of the parsing tree will be a node with the '+' operator (which has precedence over the ''). The left child is a variable token 'a' and the right child is a tree with a root element the '' token. Work recursively.
Simple way out is to convert your expression into postfix notation (abcef/*++) & then refer to the answer to this question (http://stackoverflow.com/questions/423898/postfix-notation-to-expression-tree) for converting the postfix expression to a Tree.
This is what the interviewer expected :)
Whenever you intend to write a parser, the main question to ask is if you want to do it manually, or to use a parser generator framework.
In this case, I would say that it's a good exercise to write it all yourself.
Start with a good representation for the tree itself. This will be be output of your algorithm. For example, this could be a collection of objects, where one object kind could represent a "label" like a, b, and c in your example. Others could represent numbers. You could then defined a representation of operators, for example + is a binary operator, which would have two subobjects, representing the left and right subexpression.
Next step is the actual parser, I would suggest a classical recursive decent parser. One text describing this, and provides a standard pseudo-code implementation is this text by Theodore Norvell
I'd start with a simple grammar, something like those used by ANTLR and JavaCC.