Treetop infinite recursion with negative rule - ruby

I have the following treetop grammar:
grammar TestGrammar
rule body
text / expression
end
rule text
not_delimiter*
end
rule expression
delimiter text delimiter
end
rule delimiter
'$'
end
rule not_delimiter
!delimiter
end
end
When I try to parse an expression, eg 'hello world $test$', the script goes in an infinite loop.
The problem seems to come from the not_delimiter rule, as when I remove it the expression get parsed.
What is the problem with this grammar?
Thanks in advance.

The problem seems to be where you are attempting to match:
rule text
not_delimiter*
end
Since the * will also match nothing you have the possibility of matching [^$]*, which I think is what is causing the infinite loop.
Also, you need to match multiple bodies at the starting rule, otherwise it will return nil, since you will only ever match either a text rule or an expression rule but not both.
rule bodies
body+
end
This will parse:
require 'treetop'
Treetop.load_from_string DATA.read
parser = TestGrammarParser.new
p parser.parse "hello world $test$"
__END__
grammar TestGrammar
rule bodies
body+
end
rule body
expression / text
end
rule expression
delimiter text delimiter
end
rule text
not_delimiter+
end
rule not_delimiter
[^$]
end
rule delimiter
'$'
end
end

Related

How to trigger functions in subnodes in Ruby Treetop tree. (was:How to prevent ruby Treetop doing AST squashing)

I am been using treetop for a while. I wrote rules following
http://thingsaaronmade.com/blog/a-quick-intro-to-writing-a-parser-using-treetop.html
I can parse my whole input string but i none of the other to_array function gets triggered other than the initial one.
Then I found https://whitequark.org/blog/2011/09/08/treetop-typical-errors/ which talk about AST squashing and I figured out that my rule is doing the same.
The first rule I have is
rule bodies
blankLine* interesting:(body+) blankLine* <Bodies>
end
and everything is getting gobbled up by body.
Can someone suggest me what can I do to fix this?
Edit
Adding code snippet:
grammar Sexp
rule bodies
blankLine* interesting:(body+) blankLine* <Bodies>
end
rule body
commentPortString (ifdef_blocks / interface)+ (blankLine / end_of_file) <Body>
end
rule interface
space? (intf / intfWithSize) space? newLine <Interface>
end
rule commentPortString
space? '//' space portString space? <CommentPortString>
end
rule portString
'Port' space? '.' newLine <PortString>
end
rule expression
space? '(' body ')' space? <Expression>
end
rule intf
(input / output) space wire:wireName space? ';' <Intf>
end
rule intfWithSize
(input / output) space? width:ifWidth space? wire:wireName space? ';' <IntfWithSize>
end
rule input
'input' <InputDir>
end
rule output
'output' <OutputDir>
end
rule ifdef_blocks
ifdef_line (interface / ifdef_block)* endif_line <IfdefBlocks>
end
rule ifdef_block
ifdef_line interface* endif_line <IfdefBlocks>
end
rule ifdef_line
space? (ifdef / ifndef) space+ allCaps space? newLine <IfdefLine>
end
rule endif_line
space? (endif) space? newLine <EndifLine>
end
rule ifdef
'`ifdef' <Ifdef>
end
rule ifndef
'`ifndef' <Ifndef>
end
rule endif
'`endif' <Endif>
end
rule ifWidth
'[' space? msb:digits space? ':' space? lsb:digits ']' <IfWidth>
end
rule digits
[0-9]+ <Digits>
end
rule integer
('+' / '-')? [0-9]+ <IntegerLiteral>
end
rule float
('+' / '-')? [0-9]+ (('.' [0-9]+) / ('e' [0-9]+)) <FloatLiteral>
end
rule string
'"' ('\"' / !'"' .)* '"' <StringLiteral>
end
rule identifier
[a-zA-Z\=\*] [a-zA-Z0-9_\=\*]* <Identifier>
end
rule allCaps
[A-Z] [A-Z0-9_]*
end
rule wireName
[a-zA-Z] [a-zA-Z0-9_]* <WireName>
end
rule non_space
!space .
end
rule space
[^\S\n]+
end
rule non_space
!space .
end
rule blankLine
space* newLine
end
rule not_newLine
!newLine .
end
rule newLine
[\n\r]
end
rule end_of_file
!.
end
end
Test string
// Port.
input CLK;
// Port.
input REFCLK;
// Port.
input [ 41:0] mem_power_ctrl;
output data;
EDIT: Adding more details
The test code is checked into:
https://github.com/justrajdeep/treetop_ruby_issue.
As you would see in my node_extensions.rb all the nodes except the Bodies raise an exception in the method to_array. But none of the exceptions trigger.
You call to_array on tree, which is a Bodies. That is the only thing you ever call to_array on, so no other to_array method will be called.
If you want to_array to be called on child nodes of the Bodies node, Bodies#to_array needs to call to_array on those child nodes. So if you want to call it on the Body nodes you labelled interesting, you should iterate over interesting and call .to_array on each element.
Try breaking (body+) into a new rule like this:
rule bodies
blankLine* interesting:interesting blankLine* <Bodies>
end
rule interesting
body+ <Interesting>
end
Otherwise, it would be helpful to see the SyntaxNode classes.

Treetop parser : how to handle spaces?

Good morning everyone,
I'm currently trying to describe some basic Ruby grammar but I'm now stuck with parse space?
I can handle x = 1 + 1,
but can't parser x=1+1,
how can I parser space?
I have tried add enough space after every terminal.
but it can't parse,give a nil.....
How can I fix it?
Thank you very much, have a nice day.
grammar Test
rule main
s assign
end
rule assign
name:[a-z]+ s '=' s expression s
{
def to_ast
Assign.new(name.text_value.to_sym, expression.to_ast)
end
}
end
rule expression
add
end
rule add
left:brackets s '+' s right:add s
{
def to_ast
Add.new(left.to_ast, right.to_ast)
end
}
/
minus
end
rule minus
left:brackets s '-' s right:minus s
{
def to_ast
Minus.new(left.to_ast, right.to_ast)
end
}
/
brackets
end
rule brackets
'(' s expression ')' s
{
def to_ast
expression.to_ast
end
}
/
term
end
rule term
number / variable
end
rule number
[0-9]+ s
{
def to_ast
Number.new(text_value.to_i)
end
}
end
rule variable
[a-z]+ s
{
def to_ast
Variable.new(text_value.to_sym)
end
}
end
rule newline
s "\n"+ s
end
rule s
[ \t]*
end
end
this code works
problem Solved!!!!
It's not enough to define the space rule, you have to use it anywhere there might be space. Because this occurs often, I usually use a shorter rule name S for mandatory space, and the lowercase version s for optional space.
Then, as a principle, I skip optional space first in my top rule, and again after every terminal that can be followed by space. Terminals here are strings, character sets, etc. So at the start of assign, and before the {} block on variable, boolean, number, and also after your '=', '-' and '+' literals, add a call to the rule s to skip any spaces.
This policy works well for me. It's a good idea to have a test case which has minimum space, and another case that has maximum space (in all possible places).

simplest rules in treetop not working

I have a treetop grammar with only two rules:
grammar RCFAE
rule num
[0-9]+ <Num>
end
rule identifier
[a-zA-Z] [a-zA-Z]* <ID>
end
end
I'm trying to parse simple strings ("A" and "5"). The "5" is recognized as a Num if I put that rule first, and returns nil if i put that rule second. Similarly, "A" is recognized as an ID if I put that rule first, and returns nil if I put that rule second. I can't understand how these two rules overlap in any way. It's driving me crazy!
Is there something I'm missing or don't understand about treetop or regular expressions? Thanks in advance for your help.
Treetop expects the first rule to be the "main rule". It doesn't try to apply all the rules you defined until one matches - it only applies the main rule and if that does not match, it fails.
To do what you want, you need to add a main rule which might be a num or an identifier, like this:
grammar RCFAE
rule expression
num / identifier
end
rule num
[0-9]+ <Num>
end
rule identifier
[a-zA-Z] [a-zA-Z]* <ID>
end
end

matching tag pairs in Treetop grammar

I don't want a repeat of the Cthulhu answer, but I want to match up pairs of opening and closing HTML tags using Treetop. Using this grammar, I can match opening tags and closing tags, but now I want a rule to tie them both together. I've tried the following, but using this makes my parser go on forever (infinite loop):
rule html_tag_pair
html_open_tag (!html_close_tag (html_tag_pair / '' / text / newline /
whitespace))+ html_close_tag <HTMLTagPair>
end
I was trying to base this off of the recursive parentheses example and the negative lookahead example on the Treetop Github page. The other rules I've referenced are as follows:
rule newline
[\n\r] {
def content
:newline
end
}
end
rule tab
"\t" {
def content
:tab
end
}
end
rule whitespace
(newline / tab / [\s]) {
def content
:whitespace
end
}
end
rule text
[^<]+ {
def content
[:text, text_value]
end
}
end
rule html_open_tag
"<" html_tag_name attribute_list ">" <HTMLOpenTag>
end
rule html_empty_tag
"<" html_tag_name attribute_list whitespace* "/>" <HTMLEmptyTag>
end
rule html_close_tag
"</" html_tag_name ">" <HTMLCloseTag>
end
rule html_tag_name
[A-Za-z0-9]+ {
def content
text_value
end
}
end
rule attribute_list
attribute* {
def content
elements.inject({}){ |hash, e| hash.merge(e.content) }
end
}
end
rule attribute
whitespace+ html_tag_name "=" quoted_value {
def content
{elements[1].content => elements[3].content}
end
}
end
rule quoted_value
('"' [^"]* '"' / "'" [^']* "'") {
def content
elements[1].text_value
end
}
end
I know I'll need to allow for matching single opening or closing tags, but if a pair of HTML tags exist, I'd like to get them together as a pair. It seemed cleanest to do this by matching them with my grammar, but perhaps there's a better way?
Here is a really simple grammar that uses a semantic predicate to match the closing tag to the starting tag.
grammar SimpleXML
rule document
(text / tag)*
end
rule text
[^<]+
end
rule tag
"<" [^>]+ ">" (text / tag)* "</" [^>]+ &{|seq| seq[1].text_value == seq[5].text_value } ">"
end
end
You can only do this using either a separate rule for each HTML tag pair, or using a semantic predicate. That is, by saving the opening tag (in a sempred), then accepting (in another sempred) a closing tag only if it is the same tag. This is much harder to do in Treetop than it should be, because there's no convenient place to save the context and you can't peek up the parser stack, but it is possible.
BTW, the same problem occurs in parsing MIME boundaries (and in Markdown). I haven't checked Mikel's implementation in ActionMailer (probably he uses a nested Mime parser for that), but it is possible in Treetop.
In http://github.com/cjheath/activefacts/blob/master/lib/activefacts/cql/parser.rb I save context in a fake input stream - you can see what methods it has to support - because "input" is available on all SyntaxNodes. I have a different kind of reason for using sempreds there, but some of the techniques are applicable.

Treetop grammar issues using regular expressions

I have a simple grammar setup like so:
grammar Test
rule line
(adjective / not_adjective)* {
def content
elements.map{|e| e.content }
end
}
end
rule adjective
("good" / "bad" / "excellent") {
def content
[:adjective, text_value]
end
}
end
rule not_adjective
!adjective {
def content
[:not_adjective, text_value]
end
}
end
end
Let's say my input is "this is a good ball. let's use it". This gives an error, which I'm not mentioning right now because I want to understand the theory about why its wrong first.
So, how do I create rule not_adjective so that it matches anything that is not matched by rule adjective? In general, how to I write I rule (specifically in Treetop) that "doesnt" match another named rule?
Treetop is a parser generator that generates parsers out of a special class of grammars called Parsing Expression Grammars or PEG.
The operational interpretation of !expression is that it succeeds if expression fails and fails if expression succeeds but it consumes NO input.
To match anything that rule expression does not match use the dot operator (that matches anything) in conjunction with the negation operator to avoid certain "words":
( !expression . )* ie. "match anything BUT expression"
The previous answer is incorrect for the OP's question, since it will match any sequence of individual characters up to any adjective. So if you see the string xyzgood, it'll match xyz and a following rule will match the "good" part as an adjective. Likewise, the adjective rule of the OP will match the first three characters of "badge" as the adjective "bad", which isn't what they want.
Instead, the adjective rule should look something like this:
rule adjective
a:("good" / "bad" / "excellent") ![a-z] {
def content
[:adjective, a.text_value]
end
}
end
and the not_adjective rule like this:
rule not_adjective
!adjective w:([a-z]+) {
def content
[:not_adjective, w.text_value]
end
}
end
include handling for upper-case, hyphenation, apostrophes, etc, as necessary. You'll also need white-space handling, of course.

Resources