Related
I tried two ways for the same issue, one works and the other doesn't. The following version works, returning a value of 4.
int gather_list()
{
mpi::environment env;
mpi::communicator world;
std::srand(time(0) + world.rank());
int my_number = std::rand();
std::vector<int> tmp_vec;
tmp_vec.push_back(my_number);
tmp_vec.push_back(my_number + 1);
if (world.rank() == 0)
{
std::vector<int> all_numbers;
gather(world, &tmp_vec[0], tmp_vec.size(), all_numbers, 0);
std::cout << all_numbers.size() << std::endl;
} else
{
gather(world, &tmp_vec[0], tmp_vec.size(), 0);
}
return 0;
}
The following version doesn't work, returning a value of 2.
class ArchivableVecInt : public std::vector<int>
{
public:
ArchivableVecInt() = default;
explicit ArchivableVecInt(const std::vector<int> &vec)
{
auto base_ptr = static_cast<std::vector<int> *>(this);
*base_ptr = vec;
}
template<class Archiver>
void serialize(Archiver &ar, unsigned int)
{
for (auto i: *this)
{
ar & i;
}
}
protected:
};
int gather_list()
{
mpi::environment env;
mpi::communicator world;
std::srand(time(0) + world.rank());
int my_number = std::rand();
std::vector<int> tmp_vec;
tmp_vec.push_back(my_number);
tmp_vec.push_back(my_number + 1);
ArchivableVecInt my_vec(tmp_vec);
if (world.rank() == 0)
{
std::vector<ArchivableVecInt> all_numbers;
gather(world, my_vec, all_numbers, 0);
for (int proc = 0; proc < world.size(); ++proc)
std::cout << "Process #" << proc << " thought of "
<< all_numbers[proc].size() << std::endl;
} else
{
gather(world, my_vec, 0);
}
return 0;
}
I tried 2 processes, while rank 0 returns a value of 2, rank 1 returns zero. Seems the gather(world, my_vec, 0) didn't work, why?
result as below.
Thanks in advance.
In your second example your receive buffer is a vector<vector<int>> meaning that the data is not contiguous. MPI needs contiguous buffers.
I need to load these values from INI file and print them in the application using C++ Boost Library. The sections have duplicate names. I have been restricted to using C++ Boost Library only.
numColors = 4
boardSize = 11
numSnails = 2
[initialization]
id = 0
row = 3
col = 4
orientation = 0
[initialization]
id = 1
row = 5
col = 0
orientation = 1
[color]
id = 0
nextColor = 1
deltaOrientation = +2
[color]
id = 1
nextColor = 2
deltaOrientation = +1
[color]
id = 2
nextColor = 3
deltaOrientation = -2
[color]
id = 3
nextColor = 0
deltaOrientation = -1
I figured that for a random passer-by the Boost Spirit answer might seem complex/overkill.
I wanted to try again, since it's 2021, whether with C++17, we can do a reasonable job with just the standard library.
Turns out it's a lot more work. The Qi implementation takes 86 lines of code, but the standard library implementation takes 136 lines. Also, it took me a lot longer (several hours) to debug/write. In particular it was hard to get '=', '[', ']' as token boundaries with std::istream&. I used the ctype facet approach from this answer: How do I iterate over cin line by line in C++?
I did leave in the DebugPeeker (20 lines) so you can perhaps understand it yourself.
Short Expo
The top level parse function looks sane and shows what I wanted to achieve: natural std::istream extraction:
static Ast::File std_parse_game(std::string_view input) {
std::istringstream iss{std::string(input)};
using namespace Helpers;
if (Ast::File parsed; iss >> parsed)
return parsed;
throw std::runtime_error("Unable to parse game");
}
All the rest lives in namespace Helpers:
static inline std::istream& operator>>(std::istream& is, Ast::File& v) {
for (section s; is >> s;) {
if (s.name == "parameters")
is >> v.parameters;
else if (s.name == "initialization")
is >> v.initializations.emplace_back();
else if (s.name == "color")
is >> v.colors.emplace_back();
else
is.setstate(std::ios::failbit);
}
if (is.eof())
is.clear();
return is;
}
So far, this is paying out well. The different section types are similar:
static inline std::istream& operator>>(std::istream& is, Ast::Parameters& v) {
return is
>> entry{"numColors", v.numColors}
>> entry{"boardSize", v.boardSize}
>> entry{"numSnails", v.numSnails};
}
static inline std::istream& operator>>(std::istream& is, Ast::Initialization& v) {
return is
>> entry{"id", v.id}
>> entry{"row", v.row}
>> entry{"col", v.col}
>> entry{"orientation", v.orientation};
}
static inline std::istream& operator>>(std::istream& is, Ast::Color& v) {
return is
>> entry{"id", v.id}
>> entry{"nextColor", v.nextColor}
>> entry{"deltaOrientation", v.deltaOrientation};
}
Now, if all was plain sailing like this, I would not be recommending Spirit. Now we get into conditional parsings.
The entry{"name", value} formulations use a "manipulator type":
template <typename T> struct entry {
entry(std::string name, T& into) : _name(name), _into(into) {}
std::string _name;
T& _into;
friend std::istream& operator>>(std::istream& is, entry e) {
return is >> expect{e._name} >> expect{'='} >> e._into;
}
};
Similarly, sections are using expect and token:
struct section {
std::string name;
friend std::istream& operator>>(std::istream& is, section& s) {
if (is >> expect('['))
return is >> token{s.name} >> expect{']'};
return is;
}
};
The conditional is important to be able to detect EOF without putting the stream into a hard failed mode (is.bad() != is.fail()).
expect is built on top of token:
template <typename T> struct expect {
expect(T expected) : _expected(expected) {}
T _expected;
friend std::istream& operator>>(std::istream& is, expect const& e) {
if (T actual; is >> token{actual})
if (actual != e._expected)
is.setstate(std::ios::failbit);
return is;
}
};
You will notice that there's much less error information. We just make the
stream fail() in case an expected token is not found.
Here's where the real complexity comes. I didn't want to parse character by
character. But reading std::string using operator>> would only stop on
whitespace, meaning that a section name would "eat" the bracket: parameters]
instead of parameters, and keys might eat the = character if there was no
separating space.
In the answer linked above we learn how to build our own character
classification locale facet:
// make sure =,[,] break tokens
struct mytoken_ctype : std::ctype<char> {
static auto const* get_table() {
static std::vector rc(table_size, std::ctype_base::mask());
rc[' '] = rc['\f'] = rc['\v'] = rc['\t'] = rc['\r'] = rc['\n'] =
std::ctype_base::space;
// crucial for us:
rc['='] = rc['['] = rc[']'] = std::ctype_base::space;
return rc.data();
}
mytoken_ctype() : std::ctype<char>(get_table()) {}
};
And then we need to use it but only if we parse a std::string token. That
way, if we expect('=') it will not skip over '=' because our facet calls it whitespace...
template <typename T> struct token {
token(T& into) : _into(into) {}
T& _into;
friend std::istream& operator>>(std::istream& is, token const& t) {
std::locale loc = is.getloc();
if constexpr (std::is_same_v<std::decay_t<T>, std::string>) {
loc = is.imbue(std::locale(std::locale(), new mytoken_ctype()));
}
try { is >> t._into; is.imbue(loc); }
catch (...) { is.imbue(loc); throw; }
return is;
}
};
I tried to keep it pretty condensed. Had I used proper formatting, we would
have had more lines of code still :)
DEMO AND TEST
I used the same Ast types, so it made sense to test both implementations and
compare the results for equality.
NOTES:
On Compiler Explorer so we can enjoy libfmt for easy output
For comparison I used one C++20 feature to get compiler generated operator==
Live On Compiler Explorer
#include <boost/spirit/home/qi.hpp>
#include <boost/fusion/include/io.hpp>
#include <fstream>
#include <sstream>
#include <iomanip>
#include <fmt/ranges.h>
#include <fmt/ostream.h>
namespace qi = boost::spirit::qi;
namespace Ast {
using Id = unsigned;
using Size = uint16_t; // avoiding char types for easy debug/output
using Coord = Size;
using ColorNumber = Size;
using Orientation = Size;
using Delta = signed;
struct Parameters {
Size numColors{}, boardSize{}, numSnails{};
bool operator==(Parameters const&) const = default;
};
struct Initialization {
Id id;
Coord row;
Coord col;
Orientation orientation;
bool operator==(Initialization const&) const = default;
};
struct Color {
Id id;
ColorNumber nextColor;
Delta deltaOrientation;
bool operator==(Color const&) const = default;
};
struct File {
Parameters parameters;
std::vector<Initialization> initializations;
std::vector<Color> colors;
bool operator==(File const&) const = default;
};
using boost::fusion::operator<<;
template <typename T>
static inline std::ostream& operator<<(std::ostream& os, std::vector<T> const& v) {
return os << fmt::format("vector<{}>{}",
boost::core::demangle(typeid(T).name()), v);
}
} // namespace Ast
BOOST_FUSION_ADAPT_STRUCT(Ast::Parameters, numColors, boardSize, numSnails)
BOOST_FUSION_ADAPT_STRUCT(Ast::Initialization, id, row, col, orientation)
BOOST_FUSION_ADAPT_STRUCT(Ast::Color, id, nextColor, deltaOrientation)
BOOST_FUSION_ADAPT_STRUCT(Ast::File, parameters, initializations, colors)
template <typename It>
struct GameParser : qi::grammar<It, Ast::File()> {
GameParser() : GameParser::base_type(start) {
using namespace qi;
start = skip(blank)[file];
auto section = [](const std::string& name) {
return copy('[' >> lexeme[lit(name)] >> ']' >> (+eol | eoi));
};
auto required = [](const std::string& name, auto value) {
return copy(lexeme[eps > lit(name)] > '=' > value >
(+eol | eoi));
};
file = parameters >
*initialization >
*color >
eoi; // must reach end of input
parameters = section("parameters") >
required("numColors", _size) >
required("boardSize", _size) >
required("numSnails", _size);
initialization = section("initialization") >
required("id", _id) >
required("row", _coord) >
required("col", _coord) >
required("orientation", _orientation);
color = section("color") >
required("id", _id) >
required("nextColor", _colorNumber) >
required("deltaOrientation", _delta);
BOOST_SPIRIT_DEBUG_NODES((file)(parameters)(initialization)(color))
}
private:
using Skipper = qi::blank_type;
qi::rule<It, Ast::File()> start;
qi::rule<It, Ast::File(), Skipper> file;
// sections
qi::rule<It, Ast::Parameters(), Skipper> parameters;
qi::rule<It, Ast::Initialization(), Skipper> initialization;
qi::rule<It, Ast::Color(), Skipper> color;
// value types
qi::uint_parser<Ast::Id> _id;
qi::uint_parser<Ast::Size> _size;
qi::uint_parser<Ast::Coord> _coord;
qi::uint_parser<Ast::ColorNumber> _colorNumber;
qi::uint_parser<Ast::Orientation> _orientation;
qi::int_parser<Ast::Delta> _delta;
};
static Ast::File qi_parse_game(std::string_view input) {
using SVI = std::string_view::const_iterator;
static const GameParser<SVI> parser{};
try {
Ast::File parsed;
if (qi::parse(input.begin(), input.end(), parser, parsed)) {
return parsed;
}
throw std::runtime_error("Unable to parse game");
} catch (qi::expectation_failure<SVI> const& ef) {
std::ostringstream oss;
auto where = ef.first - input.begin();
auto sol = 1 + input.find_last_of("\r\n", where);
auto lineno = 1 + std::count(input.begin(), input.begin() + sol, '\n');
auto col = 1 + where - sol;
auto llen = input.substr(sol).find_first_of("\r\n");
oss << "input.txt:" << lineno << ":" << col << " Expected: " << ef.what_ << "\n"
<< " note: " << input.substr(sol, llen) << "\n"
<< " note:" << std::setw(col) << "" << "^--- here";
throw std::runtime_error(oss.str());
}
}
namespace Helpers {
struct DebugPeeker {
DebugPeeker(std::istream& is, int line) : is(is), line(line) { dopeek(); }
~DebugPeeker() { dopeek(); }
private:
std::istream& is;
int line;
void dopeek() const {
std::char_traits<char> t;
auto ch = is.peek();
std::cerr << "DEBUG " << line << " Peek: ";
if (std::isgraph(ch))
std::cerr << "'" << t.to_char_type(ch) << "'";
else
std::cerr << "<" << ch << ">";
std::cerr << " " << std::boolalpha << is.good() << "\n";
}
};
#define DEBUG_PEEK(is) // Peeker _peek##__LINE__(is, __LINE__);
// make sure =,[,] break tokens
struct mytoken_ctype : std::ctype<char> {
static auto const* get_table() {
static std::vector rc(table_size, std::ctype_base::mask());
rc[' '] = rc['\f'] = rc['\v'] = rc['\t'] = rc['\r'] = rc['\n'] =
std::ctype_base::space;
// crucial for us:
rc['='] = rc['['] = rc[']'] = std::ctype_base::space;
return rc.data();
}
mytoken_ctype() : std::ctype<char>(get_table()) {}
};
template <typename T> struct token {
token(T& into) : _into(into) {}
T& _into;
friend std::istream& operator>>(std::istream& is, token const& t) {
DEBUG_PEEK(is);
std::locale loc = is.getloc();
if constexpr (std::is_same_v<std::decay_t<T>, std::string>) {
loc = is.imbue(std::locale(std::locale(), new mytoken_ctype()));
}
try { is >> t._into; is.imbue(loc); }
catch (...) { is.imbue(loc); throw; }
return is;
}
};
template <typename T> struct expect {
expect(T expected) : _expected(expected) {}
T _expected;
friend std::istream& operator>>(std::istream& is, expect const& e) {
DEBUG_PEEK(is);
if (T actual; is >> token{actual})
if (actual != e._expected)
is.setstate(std::ios::failbit);
return is;
}
};
template <typename T> struct entry {
entry(std::string name, T& into) : _name(name), _into(into) {}
std::string _name;
T& _into;
friend std::istream& operator>>(std::istream& is, entry e) {
DEBUG_PEEK(is);
return is >> expect{e._name} >> expect{'='} >> e._into;
}
};
struct section {
std::string name;
friend std::istream& operator>>(std::istream& is, section& s) {
DEBUG_PEEK(is);
if (is >> expect('['))
return is >> token{s.name} >> expect{']'};
return is;
}
};
static inline std::istream& operator>>(std::istream& is, Ast::Parameters& v) {
DEBUG_PEEK(is);
return is
>> entry{"numColors", v.numColors}
>> entry{"boardSize", v.boardSize}
>> entry{"numSnails", v.numSnails};
}
static inline std::istream& operator>>(std::istream& is, Ast::Initialization& v) {
DEBUG_PEEK(is);
return is
>> entry{"id", v.id}
>> entry{"row", v.row}
>> entry{"col", v.col}
>> entry{"orientation", v.orientation};
}
static inline std::istream& operator>>(std::istream& is, Ast::Color& v) {
DEBUG_PEEK(is);
return is
>> entry{"id", v.id}
>> entry{"nextColor", v.nextColor}
>> entry{"deltaOrientation", v.deltaOrientation};
}
static inline std::istream& operator>>(std::istream& is, Ast::File& v) {
DEBUG_PEEK(is);
for (section s; is >> s;) {
if (s.name == "parameters")
is >> v.parameters;
else if (s.name == "initialization")
is >> v.initializations.emplace_back();
else if (s.name == "color")
is >> v.colors.emplace_back();
else
is.setstate(std::ios::failbit);
}
if (is.eof())
is.clear();
return is;
}
}
static Ast::File std_parse_game(std::string_view input) {
std::istringstream iss{std::string(input)};
using namespace Helpers;
if (Ast::File parsed; iss >> parsed)
return parsed;
throw std::runtime_error("Unable to parse game");
}
std::string read_file(const std::string& name) {
std::ifstream ifs(name);
return std::string(std::istreambuf_iterator<char>(ifs), {});
}
int main() {
std::string const game_save = read_file("input.txt");
Ast::File g1, g2;
try {
std::cout << "Qi: " << (g1 = qi_parse_game(game_save)) << "\n";
} catch (std::exception const& e) { std::cerr << e.what() << "\n"; }
try {
std::cout << "std: " << (g2 = std_parse_game(game_save)) << "\n";
} catch (std::exception const& e) { std::cerr << e.what() << "\n"; }
std::cout << "Equal: " << std::boolalpha << (g1 == g2) << "\n";
}
To my great relief, the parsers agree on the data:
Qi: ((4 11 2)
vector<Ast::Initialization>{(0 3 4 0), (1 5 0 1)}
vector<Ast::Color>{(0 1 2), (1 2 1), (2 3 -2), (3 0 -1)})
std: ((4 11 2)
vector<Ast::Initialization>{(0 3 4 0), (1 5 0 1)}
vector<Ast::Color>{(0 1 2), (1 2 1), (2 3 -2), (3 0 -1)})
Equal: true
Summary/Conclusion
Although this answer is "standard" and "portable" it has some drawbacks.
For example it is certainly not easier to get right, it has virtually no debug options or error reporting, it doesn't validate the input format as much. E.g. it will still read this unholy mess and accept it:
[parameters] numColors=999 boardSize=999 numSnails=999
[color] id=0 nextColor=1 deltaOrientation=+2 [color] id=1 nextColor=2
deltaOrientation=+1 [
initialization] id=1 row=5 col=0 orientation=1
[color] id=2 nextColor=3 deltaOrientation=-2
[parameters] numColors=4 boardSize=11 numSnails=2
[color] id=3 nextColor=0 deltaOrientation=-1
[initialization] id=0 row=3 col=4 orientation=0
If your input format is not stable and computer-written, I would highly recommend against the standard-library approach because it will lead to hard-to-diagnose problems and just horrible UX (don't make your users want to throw their computer out of the window because of unhelpful error messages like "Unable to parse game data").
Otherwise, you might. For one thing, it'll be faster to compile.
What It Isn't
In short, this is not INI format at all. It just very loosely resembles it. Which is nice.
What Is It Instead?
You don't specify a lot, so I'm going to make assumptions.
I'm going to, for simplicity, assume that
initialization sections precede color sections
keys in like sections have the same order always
all keys shown are mandatory in like sections
the deltas are signed integral values (positive sign being optional)
all other values are non-negative integral numbers
whitespace is not significant
case is significant
all numbers are in decimal form (regardless of leading zeros)
Non-essential deductions (could be used to add more validation):
the number of of initializations = numSnails
the board size dictates row and col are in [0, boardSize)
Data Structures
To represent the file, I'd make:
namespace Ast {
struct Initialization {
unsigned id, row, col, orientation;
};
struct Color {
unsigned id, nextColor;
int deltaOrientation;
};
struct File {
unsigned numColors, boardSize, numSnails;
std::vector<Initialization> initializations;
std::vector<Color> colors;
};
}
That's the simplest I can think of.
Parsing It
Is a nice job for Boost Spirit. If we adapt the data structures as Fusion Sequences:
BOOST_FUSION_ADAPT_STRUCT(Ast::Initialization, id, row, col, orientation)
BOOST_FUSION_ADAPT_STRUCT(Ast::Color, id, nextColor, deltaOrientation)
BOOST_FUSION_ADAPT_STRUCT(Ast::File, numColors, boardSize, numSnails,
initializations, colors)
We can basically let the parser "write itself":
template <typename It>
struct GameParser : qi::grammar<It, Ast::File()> {
GameParser() : GameParser::base_type(start) {
using namespace qi;
start = skip(blank)[file];
auto section = [](std::string name) {
return copy('[' >> lexeme[lit(name)] >> ']' >> (+eol | eoi));
};
auto required = [](std::string name) {
return copy(lexeme[eps > lit(name)] > '=' > auto_ >
(+eol | eoi));
};
file =
required("numColors") >
required("boardSize") >
required("numSnails") >
*initialization >
*color >
eoi; // must reach end of input
initialization = section("initialization") >
required("id") >
required("row") >
required("col") >
required("orientation");
color = section("color") >
required("id") >
required("nextColor") >
required("deltaOrientation");
BOOST_SPIRIT_DEBUG_NODES((file)(initialization)(color))
}
private:
using Skipper = qi::blank_type;
qi::rule<It, Ast::File()> start;
qi::rule<It, Ast::File(), Skipper> file;
qi::rule<It, Ast::Initialization(), Skipper> initialization;
qi::rule<It, Ast::Color(), Skipper> color;
};
Because of the many assumptions we've made we littered the place with expectation points (operator> sequences, instead of operator>>). This means we get "helpful" error messages on invalid input, like
Expected: nextColor
Expected: =
Expected: <eoi>
See also BONUS section below that improves this a lot
Testing/Live Demo
Testing it, we will read the file first and then parse it using that parser:
std::string read_file(std::string name) {
std::ifstream ifs(name);
return std::string(std::istreambuf_iterator<char>(ifs), {});
}
static Ast::File parse_game(std::string_view input) {
using SVI = std::string_view::const_iterator;
static const GameParser<SVI> parser{};
try {
Ast::File parsed;
if (qi::parse(input.begin(), input.end(), parser, parsed)) {
return parsed;
}
throw std::runtime_error("Unable to parse game");
} catch (qi::expectation_failure<SVI> const& ef) {
std::ostringstream oss;
oss << "Expected: " << ef.what_;
throw std::runtime_error(oss.str());
}
}
A lot could be improved, but for now it works and parses your input:
Live On Coliru
int main() {
std::string game_save = read_file("input.txt");
Ast::File data = parse_game(game_save);
}
The absense of output means success.
BONUS
Some improvements, instead of using auto_ to generate the right parser for the type, we can make that explicit:
namespace Ast {
using Id = unsigned;
using Size = uint8_t;
using Coord = Size;
using ColorNumber = Size;
using Orientation = Size;
using Delta = signed;
struct Initialization {
Id id;
Coord row;
Coord col;
Orientation orientation;
};
struct Color {
Id id;
ColorNumber nextColor;
Delta deltaOrientation;
};
struct File {
Size numColors{}, boardSize{}, numSnails{};
std::vector<Initialization> initializations;
std::vector<Color> colors;
};
} // namespace Ast
And then in the parser define the analogous:
qi::uint_parser<Ast::Id> _id;
qi::uint_parser<Ast::Size> _size;
qi::uint_parser<Ast::Coord> _coord;
qi::uint_parser<Ast::ColorNumber> _colorNumber;
qi::uint_parser<Ast::Orientation> _orientation;
qi::int_parser<Ast::Delta> _delta;
Which we then use e.g.:
initialization = section("initialization") >
required("id", _id) >
required("row", _coord) >
required("col", _coord) >
required("orientation", _orientation);
Now we can improve the error messages to be e.g.:
input.txt:2:13 Expected: <unsigned-integer>
note: boardSize = (11)
note: ^--- here
Or
input.txt:16:19 Expected: <alternative><eol><eoi>
note: nextColor = 1 deltaOrientation = +2
note: ^--- here
Full Code, Live On Coliru
//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/home/qi.hpp>
#include <fstream>
#include <sstream>
#include <iomanip>
namespace qi = boost::spirit::qi;
namespace Ast {
using Id = unsigned;
using Size = uint8_t;
using Coord = Size;
using ColorNumber = Size;
using Orientation = Size;
using Delta = signed;
struct Initialization {
Id id;
Coord row;
Coord col;
Orientation orientation;
};
struct Color {
Id id;
ColorNumber nextColor;
Delta deltaOrientation;
};
struct File {
Size numColors{}, boardSize{}, numSnails{};
std::vector<Initialization> initializations;
std::vector<Color> colors;
};
} // namespace Ast
BOOST_FUSION_ADAPT_STRUCT(Ast::Initialization, id, row, col, orientation)
BOOST_FUSION_ADAPT_STRUCT(Ast::Color, id, nextColor, deltaOrientation)
BOOST_FUSION_ADAPT_STRUCT(Ast::File, numColors, boardSize, numSnails,
initializations, colors)
template <typename It>
struct GameParser : qi::grammar<It, Ast::File()> {
GameParser() : GameParser::base_type(start) {
using namespace qi;
start = skip(blank)[file];
auto section = [](const std::string& name) {
return copy('[' >> lexeme[lit(name)] >> ']' >> (+eol | eoi));
};
auto required = [](const std::string& name, auto value) {
return copy(lexeme[eps > lit(name)] > '=' > value >
(+eol | eoi));
};
file =
required("numColors", _size) >
required("boardSize", _size) >
required("numSnails", _size) >
*initialization >
*color >
eoi; // must reach end of input
initialization = section("initialization") >
required("id", _id) >
required("row", _coord) >
required("col", _coord) >
required("orientation", _orientation);
color = section("color") >
required("id", _id) >
required("nextColor", _colorNumber) >
required("deltaOrientation", _delta);
BOOST_SPIRIT_DEBUG_NODES((file)(initialization)(color))
}
private:
using Skipper = qi::blank_type;
qi::rule<It, Ast::File()> start;
qi::rule<It, Ast::File(), Skipper> file;
qi::rule<It, Ast::Initialization(), Skipper> initialization;
qi::rule<It, Ast::Color(), Skipper> color;
qi::uint_parser<Ast::Id> _id;
qi::uint_parser<Ast::Size> _size;
qi::uint_parser<Ast::Coord> _coord;
qi::uint_parser<Ast::ColorNumber> _colorNumber;
qi::uint_parser<Ast::Orientation> _orientation;
qi::int_parser<Ast::Delta> _delta;
};
std::string read_file(const std::string& name) {
std::ifstream ifs(name);
return std::string(std::istreambuf_iterator<char>(ifs), {});
}
static Ast::File parse_game(std::string_view input) {
using SVI = std::string_view::const_iterator;
static const GameParser<SVI> parser{};
try {
Ast::File parsed;
if (qi::parse(input.begin(), input.end(), parser, parsed)) {
return parsed;
}
throw std::runtime_error("Unable to parse game");
} catch (qi::expectation_failure<SVI> const& ef) {
std::ostringstream oss;
auto where = ef.first - input.begin();
auto sol = 1 + input.find_last_of("\r\n", where);
auto lineno = 1 + std::count(input.begin(), input.begin() + sol, '\n');
auto col = 1 + where - sol;
auto llen = input.substr(sol).find_first_of("\r\n");
oss << "input.txt:" << lineno << ":" << col << " Expected: " << ef.what_ << "\n"
<< " note: " << input.substr(sol, llen) << "\n"
<< " note:" << std::setw(col) << "" << "^--- here";
throw std::runtime_error(oss.str());
}
}
int main() {
std::string game_save = read_file("input.txt");
try {
Ast::File data = parse_game(game_save);
} catch (std::exception const& e) {
std::cerr << e.what() << "\n";
}
}
Look here for various failure modes and BOOST_SPIRIT_DEBUG ouput:
A parser application where I’m working on calls for recursive rules. Besides looking into the Recursive AST tutorial examples of Boost Spirit X3 which can be found here:
https://www.boost.org/doc/libs/develop/libs/spirit/doc/x3/html/index.html, I was looking for a solution with a std::variant of some types as well as a std::vector of that same
variant type.
In the StackOverflow post titled: Recursive rule in Spirit.X3, I found the code from the answer from sehe a decent starting point for my parser.
I have repeated the code here but I have limited the input strings to be tested. Because the full list from the original is not relevant for this question here.
//#define BOOST_SPIRIT_X3_DEBUG
#include <iostream>
#include <boost/fusion/adapted.hpp>
#include <boost/spirit/home/x3.hpp>
#include <string>
#include <vector>
#include <variant>
struct value: std::variant<int,float,std::vector<value>>
{
using base_type = std::variant<int,float,std::vector<value>>;
using base_type::variant;
friend std::ostream& operator<<(std::ostream& os, base_type const& v) {
struct {
std::ostream& operator()(float const& f) const { return _os << "float:" << f; }
std::ostream& operator()(int const& i) const { return _os << "int:" << i; }
std::ostream& operator()(std::vector<value> const& v) const {
_os << "tuple: [";
for (auto& el : v) _os << el << ",";
return _os << ']';
}
std::ostream& _os;
} vis { os };
return std::visit(vis, v);
}
};
namespace parser {
namespace x3 = boost::spirit::x3;
x3::rule<struct value_class, value> const value_ = "value";
x3::rule<struct o_tuple_class, std::vector<value> > o_tuple_ = "tuple";
x3::real_parser<float, x3::strict_real_policies<float> > float_;
const auto o_tuple__def = "tuple" >> x3::lit(':') >> ("[" >> value_ % "," >> "]");
const auto value__def
= "float" >> (':' >> float_)
| "int" >> (':' >> x3::int_)
| o_tuple_
;
BOOST_SPIRIT_DEFINE(value_, o_tuple_)
const auto entry_point = x3::skip(x3::space) [ value_ ];
}
int main()
{
for (std::string const str : {
"float: 3.14",
"int: 3",
"tuple: [float: 3.14,int: 3]",
"tuple: [float: 3.14,int: 3,tuple: [float: 4.14,int: 4]]"
}) {
std::cout << "============ '" << str << "'\n";
//using boost::spirit::x3::parse;
auto first = str.begin(), last = str.end();
value val;
if (parse(first, last, parser::entry_point, val))
std::cout << "Parsed '" << val << "'\n";
else
std::cout << "Parse failed\n";
if (first != last)
std::cout << "Remaining input: '" << std::string(first, last) << "'\n";
}
}
However I would like to use a traditional visitor class rather than making ostream a friend in the variant class. You know just a struct/class with a bunch of function objects for each type you encounter in the variant and a "for loop" for the vector that calls std::visit for each
element.
My goal for the traditional visitor class is to be able to maintain a state machine during printing.
My own attempts to write this visitor class did fail because I ran into an issue with my GCC 8.1 compiler. With GCC during compilation std::variant happens to be std::variant_size somehow and I got the following error:
error: incomplete type 'std::variant_size' used in nested name specifier
More about this here:
Using std::visit on a class inheriting from std::variant - libstdc++ vs libc++
Is it possible giving this constraint on GCC to write a visitor class for the code example I included, so that the ostream stuff can be removed?
Is it possible giving this constraint on GCC to write a visitor class for the code example I included, so that the ostream stuff can be removed?
Sure. Basically, I see three approaches:
1. Add the template machinery
You can specialize the implementation details accidentally required by GCC:
struct value: std::variant<int,float,std::vector<value>> {
using base_type = std::variant<int,float,std::vector<value>>;
using base_type::variant;
};
namespace std {
template <> struct variant_size<value> :
std::variant_size<value::base_type> {};
template <size_t I> struct variant_alternative<I, value> :
std::variant_alternative<I, value::base_type> {};
}
See it live on Wandbox (GCC 8.1)
2. Don't (again live)
Extending the std namespace is fraught (though I think it's legal for
user-defined types). So, you can employ my favorite pattern and hide th
estd::visit dispatch in the function object itself:
template <typename... El>
void operator()(std::variant<El...> const& v) const { std::visit(*this, v); }
Now you can simply call the functor and it will automatically dispatch
on your own variant-derived type because that operator() overload does
NOT have the problems that GCC stdlib has:
if (parse(first, last, parser::entry_point, val))
{
display_visitor display { std::cout };
std::cout << "Parsed '";
display(val);
std::cout << "'\n";
}
3. Make things explicit
I like this the least, but it does have merit: there's no magic and no
tricks:
struct value: std::variant<int,float,std::vector<value>> {
using base_type = std::variant<int,float,std::vector<value>>;
using base_type::variant;
base_type const& as_variant() const { return *this; }
base_type& as_variant() { return *this; }
};
struct display_visitor {
void operator()(value const& v) const { std::visit(*this, v.as_variant()); }
// ...
Again, live
SUMMARY
After thinking a bit more, I'd recommend the last approach, due to the relative simplicity. Clever is often a code-smell :)
Full listing for future visitors:
//#define BOOST_SPIRIT_X3_DEBUG
#include <iostream>
#include <boost/fusion/adapted.hpp>
#include <boost/spirit/home/x3.hpp>
#include <string>
#include <vector>
#include <variant>
struct value: std::variant<int,float,std::vector<value>> {
using base_type = std::variant<int,float,std::vector<value>>;
using base_type::variant;
base_type const& as_variant() const { return *this; }
base_type& as_variant() { return *this; }
};
struct display_visitor {
std::ostream& _os;
void operator()(value const& v) const { std::visit(*this, v.as_variant()); }
void operator()(float const& f) const { _os << "float:" << f; }
void operator()(int const& i) const { _os << "int:" << i; }
void operator()(std::vector<value> const& v) const {
_os << "tuple: [";
for (auto& el : v) {
operator()(el);
_os << ",";
}
_os << ']';
}
};
namespace parser {
namespace x3 = boost::spirit::x3;
x3::rule<struct value_class, value> const value_ = "value";
x3::rule<struct o_tuple_class, std::vector<value> > o_tuple_ = "tuple";
x3::real_parser<float, x3::strict_real_policies<float> > float_;
const auto o_tuple__def = "tuple" >> x3::lit(':') >> ("[" >> value_ % "," >> "]");
const auto value__def
= "float" >> (':' >> float_)
| "int" >> (':' >> x3::int_)
| o_tuple_
;
BOOST_SPIRIT_DEFINE(value_, o_tuple_)
const auto entry_point = x3::skip(x3::space) [ value_ ];
}
int main()
{
for (std::string const str : {
"float: 3.14",
"int: 3",
"tuple: [float: 3.14,int: 3]",
"tuple: [float: 3.14,int: 3,tuple: [float: 4.14,int: 4]]"
}) {
std::cout << "============ '" << str << "'\n";
//using boost::spirit::x3::parse;
auto first = str.begin(), last = str.end();
value val;
if (parse(first, last, parser::entry_point, val))
{
display_visitor display { std::cout };
std::cout << "Parsed '";
display(val);
std::cout << "'\n";
}
else
std::cout << "Parse failed\n";
if (first != last)
std::cout << "Remaining input: '" << std::string(first, last) << "'\n";
}
}
I'm new to both C++ and Boost spirit.
I'm stuck on this for a day now.
I want to parse two strings separated by a dot.
basically, I need following strings to be parsed to an integer.
eg: [field] --> integer // working
eg2: [instance.field] --> integer // not working
For the 2nd one, I need to take two strings as parameters and evaluate them and return the relevant integer value.
I must have missed a basic thing, but I can't figure it out.
Please let me know the error in my code or a better way to do it.
calling a method and getting the value is needed. I can't change that.
this is the code peace.
#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/spirit/include/support_utree.hpp>
#include <iostream>
#include <string>
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
namespace spirit = boost::spirit;
namespace phx = boost::phoenix;
int fieldVal(std::vector<char> fieldName) {
std::string fieldValue(fieldName.begin(), fieldName.end());
std::cout << "Field Name Recieved.::: " << fieldValue << std::endl;
int i = 50; //just for test
return i;
}
int instanceFieldVal(std::string instance, std::string fieldName) {
std::cout << "Recieved ::: " << instance <<" : "<< fieldName << std::endl;
int i = 60; //just for test
return i;
}
namespace client
{
template <typename Iterator>
struct calculator : qi::grammar<Iterator, ascii::space_type, int()>
{
calculator() : calculator::base_type(instanceFieldValue)
/*change base type to "field" and comment everything relevant to
"instanceFieldValue", then it's working */
{
using qi::int_;
using qi::_val;
using qi::_1;
using qi::char_;
using qi::lexeme;
field = lexeme[
'['
>> +(~char_(".]["))
>> ']'
][qi::_val = phx::bind(&fieldVal, qi::_1)]; // this is working
instanceField = '['
>> +(~char_(".]["))
>> '.'
>> +(~char_(".]["))
>> ']';
instanceFieldValue
= instanceField[qi::_val = phx::bind(&instanceFieldVal, qi::_1)];
// how ^this line should be changed??
}
qi::rule<Iterator, ascii::space_type, int()> field, instanceFieldValue;
qi::rule<Iterator, ascii::space_type, std::string(), std::string()>instanceField;
};
}
int main()
{
std::cout << "Type an expression...or [q or Q] to quit\n\n";
using boost::spirit::ascii::space;
using boost::spirit::utree;
typedef std::string::const_iterator iterator_type;
typedef client::calculator<iterator_type> calculator;
calculator calc; // Our grammar
std::string str;
while (std::getline(std::cin, str))
{
if (str.empty() || str[0] == 'q' || str[0] == 'Q')
break;
std::string::const_iterator iter = str.begin();
std::string::const_iterator end = str.end();
int val;
bool r = phrase_parse(iter, end, calc, space, val);
if (r && iter == end)
{
std::cout << "-------------------------\n";
std::cout << "Parsing succeeded: " << val << "\n";
std::cout << "-------------------------\n";
}
else
{
std::string rest(iter, end);
std::cout << "-------------------------\n";
std::cout << "Parsing failed\n";
std::cout << "stopped at: \": " << rest << "\"\n";
std::cout << "-------------------------\n";
}
}
std::cout << "Bye... :-) \n\n";
return 0;
}
Well, you say it's a function with multiple arguments, but then you only pass one arg: _1.
How can that work? It can't. It's also completely what you even want to pass, because the second parameter is likely from the instanceField expression, but the first is ... magical context.
Like always, I'd try to minimize state and semantic actions. In fact, I'd suggest separation of concerns:
parse to std:vector<std::string> first
on successful parse, evaluate it
That would lead to a grammar like
template <typename Iterator>
struct path : qi::grammar<Iterator, std::deque<std::string>()> {
path() : path::base_type(start) {
using namespace qi;
name = +(graph - char_(".][")); // not matching spaces please
qualifiedName = name % '.';
start = skip(ascii::space) ['[' >> qualifiedName >> ']'];
BOOST_SPIRIT_DEBUG_NODES((start)(qualifiedName)(name))
}
private:
qi::rule<Iterator, std::deque<std::string>(), qi::ascii::space_type> qualifiedName;
qi::rule<Iterator, std::string()> name;
qi::rule<Iterator, std::deque<std::string>()> start;
};
You can of course combine it using a semantic action if you insist:
template <typename Iterator>
struct property : qi::grammar<Iterator, int()> {
property() : property::base_type(start) {
using namespace qi;
name = +(graph - char_(".][")); // not matching spaces please
qualifiedName = name % '.';
start = skip(ascii::space) ['[' >> qualifiedName >> ']']
[_val = phx::bind(lookup, phx::cref(sample), _1) ]
;
BOOST_SPIRIT_DEBUG_NODES((start)(qualifiedName)(name))
}
private:
qi::rule<Iterator, std::deque<std::string>(), qi::ascii::space_type> qualifiedName;
qi::rule<Iterator, std::string()> name;
qi::rule<Iterator, int()> start;
};
Now, just implement a function
int lookup(Node const& context, std::deque<std::string> path);
As an example, let's imagine a sample context:
using Leaf = int;
using Node = boost::make_recursive_variant< Leaf, std::map<std::string, boost::recursive_variant_> >::type;
using SubTree = std::map<std::string, Node>;
static Node sample = SubTree {
{ "simple", 100 },
{ "compound", SubTree {
{ "first", 200 },
{ "second", 300 },
}, },
{ "deep", SubTree {
{ "nested", SubTree {
{ "compound", SubTree {
{ "buried", 400 },
{ "secrets", 500 },
}, }, }, }, }, }
};
Now, an implementation of lookup could be as simple as:
int lookup(Node const& context, std::deque<std::string> path) {
if (path.empty())
return boost::get<Leaf>(context);
auto& sub = boost::get<SubTree>(context);
auto element = path.front();
path.pop_front();
try {
return lookup(sub.at(element), std::move(path));
} catch(std::out_of_range const& e) {
throw std::runtime_error("'" + element + "' not found");
}
}
Full Demo
Live On Coliru
//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <iostream>
#include <map>
namespace qi = boost::spirit::qi;
namespace phx = boost::phoenix;
using Leaf = int;
using Node = boost::make_recursive_variant< Leaf, std::map<std::string, boost::recursive_variant_> >::type;
using SubTree = std::map<std::string, Node>;
static Node sample = SubTree {
{ "simple", 100 },
{ "compound", SubTree {
{ "first", 200 },
{ "second", 300 },
}, },
{ "deep", SubTree {
{ "nested", SubTree {
{ "compound", SubTree {
{ "buried", 400 },
{ "secrets", 500 },
},
},
},
},
},
}
};
int lookup(Node const& context, std::deque<std::string> path) {
if (path.empty())
return boost::get<Leaf>(context);
auto& sub = boost::get<SubTree>(context);
auto element = path.front();
path.pop_front();
try {
return lookup(sub.at(element), std::move(path));
} catch(std::out_of_range const& e) {
throw std::runtime_error("'" + element + "' not found");
}
}
namespace parser {
template <typename Iterator>
struct property : qi::grammar<Iterator, int()> {
property() : property::base_type(start) {
using namespace qi;
name = +(graph - char_(".][")); // not matching spaces please
qualifiedName = name % '.';
start = skip(ascii::space) ['[' >> qualifiedName >> ']']
[_val = phx::bind(lookup, phx::cref(sample), _1) ]
;
BOOST_SPIRIT_DEBUG_NODES((start)(qualifiedName)(name))
}
private:
qi::rule<Iterator, std::deque<std::string>(), qi::ascii::space_type> qualifiedName;
qi::rule<Iterator, std::string()> name;
qi::rule<Iterator, int()> start;
};
}
int main() {
using It = std::string::const_iterator;
parser::property<It> calc;
for (std::string const str : {
"[simple]",
"[compound.first]",
"[compound.second]",
"[deep.nested.compound.buried]",
"[deep.nested.compound.secrets]",
// whitespace is ok
" [ compound.\tfirst ]",
// failing:
"[]",
"[missing]",
"[deep.missing.compound.buried]",
// whitespace not ok inside names
" [ compound.\tfi rst ]",
})
try {
std::cout << " ===== Input: '" << str << "'\n";
It iter = str.begin(), end = str.end();
int val;
bool r = parse(iter, end, calc, val);
if (r) {
std::cout << "Parsing succeeded: " << val << "\n";
} else {
std::cout << "Parsing failed\n";
}
if (iter != end) {
std::cout << " - Remaining unparsed input: '" << std::string(iter, end) << "'\n";
}
} catch(std::exception const& e) {
std::cout << "Exception: " << e.what() << "\n";
}
}
Prints:
===== Input: '[simple]'
Parsing succeeded: 100
===== Input: '[compound.first]'
Parsing succeeded: 200
===== Input: '[compound.second]'
Parsing succeeded: 300
===== Input: '[deep.nested.compound.buried]'
Parsing succeeded: 400
===== Input: '[deep.nested.compound.secrets]'
Parsing succeeded: 500
===== Input: ' [ compound. first ]'
Parsing succeeded: 200
===== Input: '[]'
Parsing failed
- Remaining unparsed input: '[]'
===== Input: '[missing]'
Exception: 'missing' not found
===== Input: '[deep.missing.compound.buried]'
Exception: 'missing' not found
===== Input: ' [ compound. fi rst ]'
Parsing failed
- Remaining unparsed input: ' [ compound. fi rst ]'
I am trying to build up a client to get data via a specific protocol from a server.
I know that my code is not the best - but at the moment I am still experimenting with the basic functions of Boost ASIO.
I want to implement an read from TCP-Function which blocks until a specific amount of bytes have been received.
My Problem:
When I call boost::asio::read or boost::asio::write i geht following error:
error C2039: 'read_some' : is not a member of boost::shared_ptr'
I am working with VS2013 Professional, Boost 1.55.00 (precompiled).
Here is my Code: ( You can find the line by the comment "//HEEERE"
boost::mutex cout_lock;
int main()
{
// creating io_service
boost::shared_ptr<boost::asio::io_service> io_service(new boost::asio::io_service);
// creating work and assigning it to io_service
boost::shared_ptr<boost::asio::io_service::work> work(new boost::asio::io_service::work(*io_service));
// creating strand and assigning it to io_service
boost::shared_ptr<boost::asio::io_service::strand> strand(new boost::asio::io_service::strand(*io_service));
// creating socket
boost::shared_ptr<boost::asio::ip::tcp::socket> socket(new boost::asio::ip::tcp::socket(*io_service));
try {
// creating resolver
boost::asio::ip::tcp::resolver resolver(*io_service);
// creating query
boost::asio::ip::tcp::resolver::query query(IPConfig_str, boost::lexical_cast<std::string>(IPConfig_PortNr));
// creating iterator
boost::asio::ip::tcp::resolver::iterator iterator = resolver.resolve(query);
// creating endpoint
boost::asio::ip::tcp::endpoint endpoint = *iterator;
// connecting synchronously
socket->connect(endpoint);
}
catch(std::exception &ex) {
cout_lock.lock();
std::cout << "[main]:\t" << "Exception:" << ex.what() << std::endl;
cout_lock.unlock();
}
// Create Query
CommandCreator CMDCreator;
Command sendCommand;
CMDCreator.Create_fpga_GetSwVers(&sendCommand);
std::cout << std::endl;
std::cout << "SENT:" << std::endl;
for (int i = 0; i < sendCommand.length; i++)
{
std::cout << std::hex << std::setw(2) << std::setfill('0') << int(sendCommand.buffer[i]) << ", ";
}
std::cout << std::endl;
// Send Query
boost::system::error_code ec;
socket->async_send(boost::asio::buffer(sendCommand.buffer, sendCommand.length), boost::asio::transfer_all());
Sleep(300); // sleep 100 ms (at least 85 <- not stable!)
// Receive Answer - Header
Command receiveCommandHeader;
receiveCommandHeader.InitBuffer(4);
// Async
// socket->async_receive(boost::asio::buffer(receiveCommandHeader.buffer, receiveCommandHeader.length), 0, boost::bind(HandleRead, ec));
//HEEERE
boost::asio::read(socket, boost::asio::buffer(receiveCommandHeader.buffer, receiveCommandHeader.length), boost::asio::transfer_all(), ec);
//shutting down
socket->shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
socket->close(ec);
io_service->stop();
return 0;
}
class Command
{
friend class CommandCreator; // TODO: is there a better and as simple method as a friend class?
public:
Command() : buffer(0)
{}
virtual ~Command()
{
delete[] buffer;
buffer = 0;
}
void InitBuffer(int const len)
{
this->length = len;
this->buffer = new uint8_t[len];
}
uint8_t* buffer;
int length;
};
Actually the problem is located at this part of boost in the file read.hpp, where async_read_some is called from 'stream_'.
void operator()(const boost::system::error_code& ec,
std::size_t bytes_transferred, int start = 0)
{
std::size_t n = 0;
switch (start_ = start)
{
case 1:
n = this->check_for_completion(ec, total_transferred_);
for (;;)
{
stream_.async_read_some(
boost::asio::buffer(buffer_ + total_transferred_, n),
BOOST_ASIO_MOVE_CAST(read_op)(*this));
return; default:
total_transferred_ += bytes_transferred;
if ((!ec && bytes_transferred == 0)
|| (n = this->check_for_completion(ec, total_transferred_)) == 0
|| total_transferred_ == boost::asio::buffer_size(buffer_))
break;
}
handler_(ec, static_cast<const std::size_t&>(total_transferred_));
}
}
Okey, I've just found the problem.
// creating socket
boost::shared_ptr<boost::asio::ip::tcp::socket> socket(new boost::asio::ip::tcp::socket(*io_service));
I created the socket as a pointer but all the interfaces of read, read_some and other boost-library functions require the object. Therefore adding the dereferencing operator did it:
boost::asio::async_read(*socket, boost::asio::buffer(receiveCommandHeader.buffer, receiveCommandHeader.length),
boost::asio::transfer_all(), boost::bind(HandleRead, ec));