In section 2.3 ("Token Recognition") of this document: https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/utilities/V3_chap02.html
the following sentence appears:
"If it is indicated that a token is delimited, and no characters have been included in a token, processing shall continue until an actual token is delimited"
What's this supposed to mean? How can a token be marked as delimited before any character has been included in it?
That sentence does appear ambiguous. Reworded it says "If a token is empty, processing continues until a non-empty token is given."
A token is 'marked as delimited' when its beginning and end are known. So from the last delimiter to the newly found delimiter.
A token can be empty if there are 2 delimiters next to each other.
For example, say the delimiter is comma.
fashl,gasdf,agasdf,aasdf,,ghask
Of the six tokens, the fifth token is empty. Its beginning and end are established by the fourth and fifth commas, therefore it is 'delimited'. But no characters were included in it.
However the sentence does go on to be ambiguous. 'Processing shall continue' does not specify what happens to the empty token. Or rather, it does not specify what logical action or path should be taken. So it could either mean:
If an empty token is found, continue to read until the next delimiter and consider that the token.
If an empty token is found, ignore the token and continue reading.
Though the end effect of that may not make a difference.
Related
I have been starting to use ANTLR and have noticed that it is pretty fickle with its lexer rules. An extremely frustrating example is the following:
grammar output;
test: FILEPATH NEWLINE TITLE ;
FILEPATH: ('A'..'Z'|'a'..'z'|'0'..'9'|':'|'\\'|'/'|' '|'-'|'_'|'.')+ ;
NEWLINE: '\r'? '\n' ;
TITLE: ('A'..'Z'|'a'..'z'|' ')+ ;
This grammar will not match something like:
c:\test.txt
x
Oddly if I change TITLE to be TITLE: 'x' ; it still fails this time giving an error message saying "mismatched input 'x' expecting 'x'" which is highly confusing. Even more oddly if I replace the usage of TITLE in test with FILEPATH the whole thing works (although FILEPATH will match more than I am looking to match so in general it isn't a valid solution for me).
I am highly confused as to why ANTLR is giving such extremely strange errors and then suddenly working for no apparent reason when shuffling things around.
This seems to be a common misunderstanding of ANTLR:
Language Processing in ANTLR:
The Language Processing is done in two strictly separated phases:
Lexing, i.e. partitioning the text into tokens
Parsing, i.e. building a parse tree from the tokens
Since lexing must preceed parsing there is a consequence: The lexer is independent of the parser, the parser cannot influence lexing.
Lexing
Lexing in ANTLR works as following:
all rules with uppercase first character are lexer rules
the lexer starts at the beginning and tries to find a rule that matches best to the current input
a best match is a match that has maximum length, i.e. the token that results from appending the next input character to the maximum length match is not matched by any lexer rule
tokens are generated from matches:
if one rule matches the maximum length match the corresponding token is pushed into the token stream
if multiple rules match the maximum length match the first defined token in the grammar is pushed to the token stream
Example: What is wrong with your grammar
Your grammar has two rules that are critical:
FILEPATH: ('A'..'Z'|'a'..'z'|'0'..'9'|':'|'\\'|'/'|' '|'-'|'_'|'.')+ ;
TITLE: ('A'..'Z'|'a'..'z'|' ')+ ;
Each match, that is matched by TITLE will also be matched by FILEPATH. And FILEPATH is defined before TITLE: So each token that you expect to be a title would be a FILEPATH.
There are two hints for that:
keep your lexer rules disjunct (no token should match a superset of another).
if your tokens intentionally match the same strings, then put them into the right order (in your case this will be sufficient).
if you need a parser driven lexer you have to change to another parser generator: PEG-Parsers or GLR-Parsers will do that (but of course this can produce other problems).
This was not directly OP's problem, but for those who have the same error message, here is something you could check.
I had the same Mismatched Input 'x' expecting 'x' vague error message when I introduced a new keyword. The reason for me was that I had placed the new key word after my VARNAME lexer rule, which assigned it as a variable name instead of as the new keyword. I fixed it by putting the keywords before the VARNAME rule.
I am using antlr3. I have my parser and lexer files.I want to print the incoming token(As given by user input) in the parser file.I have tried input.LT(1).
It prints the input token(given by user) nicely. But then for future analysis I can't use this lookahead token. So is there any other commands or instruction that can print the incoming tokens?
(For example if my input is 1+2+; my token for '+' is 'PLUS', then I must print '+' not 'PLUS').
To get the actual text of the token ("+"), use token.getText().
To get the text representing the token name ("PLUS"), you'll have to ask the parser: <YourParserClass>.tokenNames[token.getType()].
I need create a regular expression for validate the first and second name of a person. The second name is optional because there are people without second name. The space character can be between the two names, but it can not be the end of string
example
"Juan Perez" is valid
"Juan Perez " is invalid because there is a space character the end of the string
You could use the below regex which uses an optional group.
^[A-Za-z]+(?:\s[A-Za-z]+)?$
(?:\s[A-Za-z]+)? optional group will do a match if there is a space followed by one or more alphabets. It won't match the string if there is only a single space exists. And also it asserts that there must be an alphabet present at the last.
DEMO
Update:
If the user name contains not only the firstname but also middlename,lastname ... then you could use the below regex. * repeats the previous token zero or more times where ? after a token (not of * or +) will turn the previous token as an optional one.
^[A-Za-z]+(?:\s[A-Za-z]+)*$
How about a way that doesn't require repeating the char class:
^\b(\s?[[:alpha:]]+)+$
I am new to cucumber with capybara. I got an application to test whose flow is:'after submitting a form, an email will be sent to the user which contains the link to another app. In order to access the app we have to open the mail and click the link, which will redirect to the app.'. I don't have access to the mail Id. Is there any way to extract that link and continue with the flow?
Please, give some possible way to do it.
Regards,
Abhisek Das
In your test, use whatever means you need in order to trigger the sending of the email by your application. Once the email is sent, use a regular expression to find the URL from the link within the email body (note this will work only for an email that contains a single link), and then visit the path from that URL with Capybara to continue with your test:
path_regex = /(?:"https?\:\/\/.*?)(\/.*?)(?:")/
email = ActionMailer::Base.deliveries.last
path = email.body.match(path_regex)[1]
visit(path)
Regular expression explained
A regular expression (regex) itself is demarcated by forward slashes, and this regex in particular consists of three groups, each demarcated by pairs of parentheses. The first and third groups both begin with ?:, indicating that they are non-capturing groups, while the second is a capturing group (no ?:). I will explain the significance of this distinction below.
The first group, (?:"https?\:\/\/.*?), is a:
non-capturing group, ?:
that matches a single double quote, "
we match a quote since we anticipate the URL to be in the href="..." attribute of a link tag
followed by the string http
optionally followed by a lowercase s, s?
the question mark makes the preceding match, in this case s, optional
followed by a colon and two forward slashes, \:\/\/
note the backslashes, which are used to escape characters that otherwise have a special meaning in a regex
followed by a wildcard, .*?, which will match any character any number of times up until the next match in the regex is reached
the period, or wildcard, matches any character
the asterisk, *, repeats the preceding match up to an unlimited number of times, depending on the successive match that follows
the question mark makes this a lazy match, meaning the wildcard will match as few characters as possible while still allowing the next match in the regex to be satisfied
The second group, (\/.*?) is a capturing group that:
matches a single forward slash, \/
this will match the first forward slash after the host portion of the URL (e.g. the slash at the end of http://www.example.com/) since the slashes in http:// were already matched by the first group
followed by another lazy wildcard, .*?
The third group, (?:"), is:
another non-capturing group, ?:
that matches a single double quote, "
And thus, our second group will match the portion of the URL starting with the forward slash after the host and going up to, but not including, the double quote at the end of our href="...".
When we call the match method using our regex, it returns an instance of MatchData, which behaves much like an array. The element at index 0 is a string containing the entire matched string (from all of the groups in the regex), while elements at subsequent indices contain only the portions of the string matched by the regex's capturing groups (only our second group, in this case). Thus, to get the corresponding match of our second group—which is the path we want to visit using Capybara—we grab the element at index 1.
You can use Nokogiri to parse the email body and find the link you want to click.
Imagine you want to click a link Change my password:
email = ActionMailer::Base.deliveries.last
html = Nokogiri::HTML(email.html_part.body.to_s)
target_url = html.at("a:contains('Change my password')")['href']
visit target_url
I think this is more semantic and robust that using regular expressions. For example, this would work if the email has many links.
If you're using or willing to use the capybara-email gem, there's now a simpler way of doing this. Let's say you've generated an email to recipient#email.com, which contains the link 'fancy link'.
Then you can just do this in your test suite:
open_email('recipient#email.com') # Allows the current_email method
current_email.click_link 'fancy link'
For some reason I need to fetch emails from /var/mail/username file. It seems like an append only file.
My question is, is it safe to parse the content of the /var/email/username file depending on the first line From username#host Mon Jun 20 16:50:15 2011? What if the similar pattern found inside the email body?
Furthermore, is there any opensource ruby script available for reference?
Yes, that seems like more or less the right way to parse the mbox format - from a quick scan of the RFC specification:
The structure of the separator lines
vary across implementations, but
usually contain the exact character
sequence of "From", followed by a
single Space character (0x20), an
email address of some kind, another
Space character, a timestamp sequence
of some kind, and an end-of- line
marker.
And...
Many implementations are also known
to escape message body lines that
begin with the character sequence of
"From ", so as to prevent confusion
with overly-liberal parsers that do
not search for full separator
lines. In the common case, a leading
Greater-Than symbol (0x3E) is used
for this purpose (with "From "
becoming ">From "). However, other
implementations are known not to
escape such lines unless they are
immediately preceded by a blank line
or if they also appear to contain
an email address and a timestamp.
Other implementations are also
known to perform secondary escapes
against these lines if they are
already escaped or quoted, while
others ignore these mechanisms
altogether.
Update:
There's also this: https://github.com/meh/ruby-mbox