My first version of this question was rich with misunderstandings. My answer below suits my needs. But I kept at it to understand what could be done with with<>. What I get is that it intended to inject context into a parser. Then the parser is called from with_directive::parse (in x3's with.hpp) In the following code, that is just what happens.
#include <boost/spirit/home/x3.hpp>
using namespace boost::spirit::x3;
struct eol_parser_cnt : parser<eol_parser_cnt>
{
struct context {
int line = 0;
std::string::iterator iter_pos;
};
template <typename Iterator, typename Context, typename Attribute>
bool parse(Iterator& first, Iterator const& last
, Context const& context, unused_type, Attribute& attr) const
{
//std::cout << context.line;
auto& ctx = context;
return boost::spirit::x3::parse(first, last, lit(' ') | (lit("//") >> *(char_ - eol) >> eol));
}
};
const auto& our_skipper = eol_parser_cnt{};
eol_parser_cnt::context lines;
auto with_skipper = with<eol_parser_cnt::context>(lines)[our_skipper];
int main()
{
std::string str("12 word");
auto first = str.begin();
phrase_parse(first, str.end(), int_ >> *char_("a-z"), with_skipper);
}
Putting a break point in eol_parser_cnt::parse and I see it working. The debugger consistently shows the context is there and that it is the structure of eol_parser_cnt::context. I can change the value of line in the debugger and the next hit shows that value, it is a real object. But, try to uncomment the line std::cout << context.line; and the compiler complains that is it an unused_type. So, I just don't get it.
test.cpp(15,30): error C2039: 'line': is not a member of 'boost::spirit::x3::context<ID,T,Context>'
with
[
ID=eol_parser_cnt::context,
T=eol_parser_cnt::context,
Context=boost::spirit::x3::unused_type
]
F:\cpp\boost_1_76_0\boost\spirit\home\x3\support\context.hpp(18): message : see declaration of 'boost::spirit::x3::context<ID,T,Context>'
with
[
ID=eol_parser_cnt::context,
T=eol_parser_cnt::context,
Context=boost::spirit::x3::unused_type
]
F:\cpp\boost_1_76_0\boost\spirit\home\x3\directive\with.hpp(62): message : see reference to function template instantiation 'bool eol_parser_cnt::parse<Iterator,boost::spirit::x3::context<ID,T,Context>,Attribute>(Iterator &,const Iterator &,const boost::spirit::x3::context<ID,T,Context> &,boost::spirit::x3::unused_type,Attribute &) const' being compiled
with
[
Iterator=std::_String_iterator<std::_String_val<std::_Simple_types<char>>>,
ID=eol_parser_cnt::context,
T=eol_parser_cnt::context,
Context=boost::spirit::x3::unused_type,
Attribute=const boost::spirit::x3::unused_type
]
Well, it took a while but I was going to understand this. I re-read C++ Template Metaprogramming seriously this time. Then looking at the x3 code I finally understood that with<> is just another parser wrapper. It is interesting to debug an optimized build and see just how much code disappears. VS shows all the disappeared stuff on the stack as inlined and what was 9 layers of parser calls becomes 2 into the likes of with_error_handling::on_error. Everything from my call to parse_rhs_main (in rule.hpp, line 232), is gone.
So because with<> is just another parser wrapper and if the skipper is wrapped, the context is only avaliable to the skipper. But no big deal as the skipper is in the context of the main parser. The difference is that in the skipper object a get<skipper_eol_cnt>(context) returns a reference to the skipper_eol_cnt::context{}. Whereas in the main parser we have to us get<skipper_tag>(context) and this returns the with<> parser object. val is a member of that parser, the reference to the skipper_eol_cnt::context{}. So get<skipper_tag>(context).val retrieves the context we are looking for.
So here it is, using with<> applied to a skipper.
#include<iostream>
#include <iomanip>
#include <vector>
//#define BOOST_SPIRIT_X3_DEBUG
#include <boost/spirit/home/x3.hpp>
#include <boost/fusion/adapted.hpp>
using namespace boost::spirit::x3;
struct skipper_eol_cnt : parser<skipper_eol_cnt>
{
struct context {
int line = 1;
std::string::iterator iter_pos;
};
template <typename Iterator, typename Context, typename Attribute>
bool parse(Iterator& first, Iterator const& last
, Context const& context, unused_type, Attribute& attr) const
{
const char* start_cmt = "/*";
const char* end_cmt = "*/";
if (first == last)
return false;
bool matched = false;
//here we are getting from the 'with<>' wrapper
auto& ctx = get<skipper_eol_cnt>(context);
//skip: space | '//comment'
boost::spirit::x3::parse(first, last, lit(' ') | (lit("//") >> *(char_ - eol)));//eol counted below
//skip: '/*comment*/'
if (detail::string_parse(start_cmt, first, last, unused, case_compare<Iterator>())) {
for (; first != last; ++first) {
if (detail::string_parse(end_cmt, first, last, unused, case_compare<Iterator>()))
break;
if (*first == '\n')
++ctx.line, ctx.iter_pos = first;
}
}
Iterator iter = first;
for (; iter != last && (*iter == '\r' || *iter == '\n'); ++iter) {
matched = true;
if (*iter == '\n') // LF
++ctx.line, ctx.iter_pos = iter;
}
//{static int pos = 0; if (pos < ctx.line) { pos = ctx.line; std::cout << pos << std::endl; }}
if (matched) first = iter;
return matched;
}
};
auto const& skip_eol_cnt = skipper_eol_cnt{};
struct with_error_handling {
template<typename It, typename Ctx>
error_handler_result on_error(It f, It l, expectation_failure<It> const& ef, Ctx const& ctx) const {
It erit = f + std::distance(f, ef.where());
//here we are getting the wrapped skipper so need the 'with<>.val'
const auto& sctx = get<skipper_tag>(ctx).val;
It bit = erit;
for (; *bit != '\n'/* && bit != f*/; --bit)
;
It eit = erit;
for (; *eit != '\n' && eit != l; ++eit)
;
int str_pos = erit - bit - 1;
std::ostringstream oss;
oss << "Expecting " << ef.which() << "\n at line: " << sctx.line
<< "\n\t" << std::string(bit + 1, eit)
<< "\n\t" << std::setw(str_pos) << std::setfill('-') << "" << "^";
get<with_error_handling>(ctx).push_back(oss.str());;
return error_handler_result::fail;
}
};
//attr sections
struct section_type {
std::string name;
int line;
std::string::iterator iter_pos;
};
BOOST_FUSION_ADAPT_STRUCT(section_type, name)
using sections_type = std::vector<section_type>;
struct parser_find_sections : parser<parser_find_sections> {
template<typename Iterator, typename Context, typename RContext, typename Attribute>
bool parse(Iterator& first, Iterator const& last, Context const& context, RContext const& rcontext, Attribute& section) const {
const auto& sssctx = get<skipper_eol_cnt>(context); //now here this doesn't work, unused_type, but
const auto& sctx = get<skipper_tag>(context).val;
auto to_line = [&sctx, first](auto& ctx) {
_attr(ctx).line = sctx.line;
//_attr(ctx).iter_pos = first; // this one will get at 'section color(x,x)'
_attr(ctx).iter_pos = _where(ctx).begin(); // this one is '(x,x)'
};
static_assert(BOOST_VERSION / 100 % 1000 >= 77);
////NOTE!!! if you have a boost version of less than 1.77, x3::seek will fail here
////quick fix, copy from: https://github.com/boostorg/spirit/blob/boost-1.78.0/include/boost/spirit/home/x3/directive/seek.hpp
////and paste to your boost file...
return phrase_parse(first, last, *(seek["section"] >> (*alpha)[to_line]), get<skipper_tag>(context), section);
}
};
auto const parse_section = rule<with_error_handling, std::pair<int, int>>("the_sec_parser") = [] {
return '(' > int_ > ',' > int_ > ')';
}();
template<typename T>
std::ostream& operator << (std::ostream& os, std::pair<T, T>& t) {
return os << t.first << ',' << t.second;
}
//errors
std::vector<std::string> errors;
auto with_errors = with<with_error_handling>(errors)[parse_section];
auto test_section = [](auto& content, auto& section) {
//attr
std::pair<int, int> attr;
skipper_eol_cnt::context ctx{ section.line, section.iter_pos };
auto with_skip_cnt = with< skipper_eol_cnt>(ctx)[skip_eol_cnt];
auto first(section.iter_pos);
return std::tuple(phrase_parse(first, content.end(), with_errors, with_skip_cnt, attr), attr);
};
int main() {
std::string str(R"(//line 1
section red(5, 6)
section green( 7, 8) //line 3
section blue(9, 10) //no error
/*comment
bunch of lines of stuff....
*/ section white(11, a 12) //error on line 7
section black( 13,14)
)");
//get the list of sections
auto with_skip_cnt = with<skipper_eol_cnt>(skipper_eol_cnt::context{})[skip_eol_cnt];
sections_type secs;
auto first(str.begin());
phrase_parse(first, str.end(), parser_find_sections(), with_skip_cnt, secs);
for (auto& item : secs)
std::cout << item.name << "\t at line: " << item.line << std::endl;
//section 'blue', at 2, has no error
auto [r, attr] = test_section(str, secs.at(2));
if (r)
std::cout << "\nthe " << secs.at(2).name << " hase vals: " << attr << "\n\n";
//section 'white', at 3, has an error
test_section(str, secs.at(3));
if (errors.size())
std::cout << errors.front() << std::endl;
return 0;
}
I'm using the Spirit error handling code from the article "Dispatching on Expectation Point Failures" at http://boost-spirit.com/home/2011/02/28/dispatching-on-expectation-point-failures/comment-page-1/, the last example there. My "diagnostics" and "error_handler_impl" classes are pretty much the same as is found in the article, but I can post them if someone thinks it is necessary.
In the code below, the start rule in question
comma = lit(',');
comma.name(",");
start = lit("begin") >> ident > comma >> ident;
has an expectation point after the "begin" keyword. In accordance with the article, I expected that a missing comma would cause error handler to be passed the qi::_4 value to be ",", which is the name of the comma rule. Instead, it is passing "sequence", causing the error handler to print a default message rather than the informative "Missing comma after begin keyword".
An idea what I am missing?
template <typename Iterator, typename Skipper = ascii::space_type>
class Grammar1 : boost::spirit::qi::grammar<Iterator, Skipper>
{
public:
typedef boost::spirit::qi::rule<Iterator, Skipper> rule_nil_T;
typedef boost::spirit::qi::rule<Iterator, string()> rule_str_T;
// structs from Rob Stewart's above-mentioned article
diagnostics<10> d1;
boost::phoenix::function<error_handler_impl> error_handler;
rule_str_T ident;
rule_nil_T comma;
rule_nil_T start;
Grammar1(void) : Grammar1::base_type(start)
{
ident %= lexeme [ qi::raw [ (qi::alpha | '_') >> *(qi::alnum | '_') ] ];
comma = lit(',');
comma.name(",");
start = lit("begin") >> ident > comma >> ident;
d1.add(",", "Missing comma after begin keyword");
on_error<fail>(start,
error_handler(ref(d1), _1, _2, _3, _4));
}
~Grammar1(void) { };
void parseInputFile(Iterator itr, Iterator itr_end)
{
bool r = phrase_parse(itr, itr_end, start, ascii::space);
if (r && itr == itr_end)
{
std::cout << "Parsing succeeded\n";
} else
{
string rest(itr, itr_end);
std::cout << "stopped at: \": " << rest << "\"\n";
}
}
};
What is the most idiomatic way to do post-skipping? More specific I want to ensure there is no "non-skippable" (garbage) characters in my input after matching my top rule.
auto const blankOrComment
= ascii::space
| x3::lexeme ['#' >> *(x3::char_ - x3::eol) >> -x3::eol ]
;
auto const program = rule<AstProgram>("program")
= *(as<AstDefinition> (definition > ";"))
;
auto const programEntry = x3::skip(blankOrComment) [program];
One idea, I consider quite ugly was to do a separate parse call for the blankOrComment afterwards, if the main iterator position is not the end iterator. The current better idea I have is to change the root rule:
auto const programEntry = x3::skip(blankOrComment) [program >> x3::omit[*blankOrComment]];
Is there a more idiomatic way?
The simplest hack is to tack on >> eps: Live On Coliru
Note I'd strive to make the skipper more self-descriptive:
auto const skipper
= space
| '#' >> *(char_ - eol) >> (eol|eoi)
;
Likewise you can make that postskip hack more self-descriptive:
auto const post_skip = eps;
auto const program = "program" >> post_skip;
Live On Coliru
#include <iostream>
#define BOOST_SPIRIT_X3_DEBUG
#include <boost/spirit/home/x3.hpp>
namespace Parser {
namespace x3 = boost::spirit::x3;
namespace rules {
using namespace x3;
auto const skipper
= space
| '#' >> *(char_ - eol) >> (eol|eoi)
;
auto const post_skip = eps;
auto const program = "program" >> post_skip;
}
auto const programEntry = x3::skip(rules::skipper) [rules::program];
}
int main() {
using It = std::string::const_iterator;
for (std::string const input : {
"",
" program ",
"#hello\n program # comment\n",
}) {
It f = input.begin(), l = input.end();
if(parse(f, l, Parser::programEntry)) {
std::cout << "Parse success\n";
} else {
std::cout << "Parse failed\n";
}
std::cout << "Remaining: '" << std::string(f,l) << "'\n";
}
}
Prints
Parse failed
Remaining: ''
Parse success
Remaining: ''
Parse success
Remaining: ''
Imagine we want to parse and generate simple C++ member function declarations with Boost.Spirit.
The Qi grammar might look like this:
function_ %= type_ > id_ > "()" > matches["const"];
That means, whether the function is const is stored in a bool.
How to write the corresponding generator with Karma?
function_ %= type_ << ' ' << id_ << "()" << XXX[" const"];
Here, we want a directive that consumes a boolean attribute, executes the embedded generator if the attribute is true and does nothing otherwise. We want something that makes the following tests succeed.
test_generator_attr("abc", XXX["abc"], true);
test_generator_attr("", XXX["abc"], false);
Is such a directive already available in Boost.Spirit?
The first thing that enters my mind at the moment is
bool const_qualifier = true;
std::cout << karma::format(
karma::omit[ karma::bool_(true) ] << " const" | "",
const_qualifier);
It feels a bit... clumsy. I'll have a look later what I'm forgetting :)
UPDATE Here's a slightly more elegant take using karma::symbols<>:
#include <boost/spirit/include/karma.hpp>
namespace karma = boost::spirit::karma;
int main()
{
karma::symbols<bool, const char*> const_;
const_.add(true, "const")(false, "");
for (bool const_qualifier : { true, false })
{
std::cout << karma::format_delimited("void foo()" << const_, ' ', const_qualifier) << "\n";
}
}
Prints:
void foo() const
void foo()
I have a simple struct
// in namespace client
struct UnaryExpression
{
std::string key;
SomeEnums::CompareType op;
};
SomeEnums::CompareType is an enum where I define a symbol table as such:
struct UnaryOps : bsq::symbols<char, SomeEnums::CompareType>
{
UnaryOps() : bsq::symbols<char, SomeEnums::CompareType>(std::string("UnaryOps"))
{
add("exists", SomeEnums::Exists)
("nexists", SomeEnums::NotExists);
}
};
I have two different ways I want to parse the struct, which I asked about in another thread and got to work (mostly).
My grammar looks as follows:
template<typename Iterator>
struct test_parser : bsq::grammar<Iterator, client::UnaryExpression(), bsq::ascii::space_type>
{
test_parser()
: test_parser::base_type(unaryExp, std::string("Test"))
{
using bsq::no_case;
key %= bsq::lexeme[bsq::alnum >> +(bsq::alnum | bsq::char_('.'))];
unaryExp %= unaryE | unaryF;
unaryE %= key >> no_case[unaryOps];
unaryF %= no_case[unaryOps] >> '(' >> key >> ')';
};
UnaryOps unaryOps;
bsq::rule<Iterator, std::string(), bsq::ascii::space_type> key;
bsq::rule<Iterator, client::UnaryExpression(), bsq::ascii::space_type> unaryExp;
bsq::rule<Iterator, client::UnaryExpression(), bsq::ascii::space_type> unaryE;
bsq::rule<Iterator, client::UnaryFunction(), bsq::ascii::space_type> unaryF;
};
And I'm parsing the code using the following logic:
bool r = phrase_parse(iter, end, parser, bsq::ascii::space, exp);
if (r && iter == end)
{
std::cout << "-------------------------\n";
std::cout << "Parsing succeeded\n";
std::cout << "key: " << exp.key << "\n";
std::cout << "op : " << exp.op << "\n";
std::cout << "-------------------------\n";
}
This all works fine if I do the input like foo exists and exp.key equals "foo" and exp.op equals the corresponding enum value (in this case 0). Something like foo1 nexists also works.
However, that second rule doesn't work like I expect. If I give it input of nexists(foo) then I get the following output:
-------------------------
Parsing succeeded
key: nexistsfoo
op : 1
-------------------------
It seems that the enum value is getting set appropriately but I can't figure out why the "nexsts" is getting prepended to the key string. Can someone please tell me how I can fix my rule so that the key would equal just 'foo' with the second rule?
I have posted a copy of the stripped down code that illustrates my problem here: http://pastebin.com/402M9iTS