How to implement Boost::Multi-index on a list of lists
I have a hierarchical tree as follows:
typedef std::list<struct obj> objList // the object list
typedef std::list<objList> topLevelList // the list of top-level object lists
struct obj
{
int Id; // globally unique Id
std::string objType;
std::string objAttributes;
....
topLevelList childObjectlist;
}
At the top-level, I have a std::list of struct obj
Then, each of these top-level obj can have any number of child objects,
which are contained in a topLevelList list for that object. This can continue, with a child in the nested list also having its own children.
Some objects can only be children, while others are containers and can have children of their own. Container objects have X number of sub-containers, each sub-container having its own list of child objects and that is why I have topLevelList in each obj struct, rather than simply objList.
I want to index this list of lists with boost::Multi-index to obtain random access to any of the objects in either the top-level list or the descendant list by its globally unique Id.
Can this be accomplished? I have searched for examples with no success.
I think the only way to have a flattened master search index by object Ids is to make the lists above to be lists of pointers to the objects, then traverse the completed hierarchical list, and log into the master search index the pointer where each object is physically allocated in memory. Then any object can be located via the master search index.
With Boost::Multi-index, I'd still have to traverse the hierarchy, though hopefully with the ability to use random instead of sequential access in each list encountered, in order to find a desired object.
Using nested vectors instead of lists is a problem - as additions and deletions occur in the vectors, there is a performance penalty as well as the prospect of pointers to objects becoming invalidated as the vectors are reallocated.
I'm almost talking myself into implementing the flattened master objId search index of pointers, unless someone has a better solution that can leverage Boost::Multi-index.
Edit on 1/31/2020:
I'm having trouble with the implementation of nested lists below. I have cases where the code does not properly place top-level parent objects into the top level, and thus in the "bracketed" printout, we don't see the hierarchy for that parent. However, in the "Children of xxx" printout, the children of that parent do display correctly. Here is a section of main.cpp which demonstrates the problem:
auto it=c.insert({170}).first;
it=c.insert({171}).first;
it=c.insert({172}).first;
it=c.insert({173}).first;
auto it141=c.insert({141}).first;
auto it137=insert_under(c,it141,{137}).first;
insert_under(c,it137,{8});
insert_under(c,it137,{138});
auto it9=insert_under(c,it137,{9}).first;
auto it5=insert_under(c,it9,{5}).first;
insert_under(c,it5,{6});
insert_under(c,it5,{7});
insert_under(c,it137,{142});
auto it143=insert_under(c,it137,{143}).first;
insert_under(c,it143,{144});
If you place this code in Main.cpp instead of the demo code and run it you will see the problem. Object 141 is a parent object and is placed at the top level. But it does not print in the "Bracketed" hierarchy printout. Why is this?
Edit on 2/2/2020:
Boost::Serialize often delivers an exception on oarchive, complaining that re-creating a particular object would result in duplicate objects. Some archives save and re-load successfully, but many result in the error above. I have not been able yet to determine the exact conditions under which the error occurs, but I have proven that none of the content used to populate the nested_container and the flat object list contains duplicate object IDs. I am using text archive, not binary. Here is how I have modified the code for nested_container and also for another, separate flat object list in order to do Boost::Serialize:
struct obj
{
int id;
const obj * parent = nullptr;
obj()
:id(-1)
{ }
obj(int object)
:id(object)
{ }
int getObjId() const
{
return id;
}
bool operator==(obj obj2)
{
if (this->getObjId() == obj2.getObjId())
return true;
else
return false;
}
#if 1
private:
friend class boost::serialization::access;
friend std::ostream & operator<<(std::ostream &os, const obj &obj);
template<class Archive>
void serialize(Archive &ar, const unsigned int file_version)
{
ar & id & parent;
}
#endif
};
struct subtree_obj
{
const obj & obj_;
subtree_obj(const obj & ob)
:obj_(ob)
{ }
#if 1
private:
friend class boost::serialization::access;
friend std::ostream & operator<<(std::ostream &os, const subtree_obj &obj);
template<class Archive>
void serialize(Archive &ar, const unsigned int file_version)
{
ar & obj_;
}
#endif
};
struct path
{
int id;
const path *next = nullptr;
path(int ID, const path *nex)
:id(ID), next(nex)
{ }
path(int ID)
:id(ID)
{ }
#if 1
private:
friend class boost::serialization::access;
friend std::ostream & operator<<(std::ostream &os, const path &pathe);
template<class Archive>
void serialize(Archive &ar, const unsigned int file_version)
{
ar & id & next;
}
#endif
};
struct subtree_path
{
const path & path_;
subtree_path(const path & path)
:path_(path)
{ }
#if 1
private:
friend class boost::serialization::access;
friend std::ostream & operator<<(std::ostream &os, const subtree_path &pathe);
template<class Archive>
void serialize(Archive &ar, const unsigned int file_version)
{
ar & path_;
}
#endif
};
//
// My flattened object list
//
struct HMIObj
{
int objId;
std::string objType;
HMIObj()
:objId(-1), objType("")
{ }
bool operator==(HMIObj obj2)
{
if (this->getObjId() == obj2.getObjId())
&& this->getObjType() == obj2.getObjType())
return true;
else
return false;
}
int getObjId() const
{
return objId;
}
std::string getObjType() const
{
return objType;
}
#if 1
private:
friend class boost::serialization::access;
friend std::ostream & operator<<(std::ostream &os, const HMIObj &obj);
template<class Archive>
void serialize(Archive &ar, const unsigned int file_version)
{
ar & objId & objType;
}
#endif
};
In case it helps, you can use Boost.MultiIndex to implement a sort of hierarchical container using the notion of path ordering.
Suppose we have the following hierarchy of objects, identified by their IDs:
|-------
| |
0 4
|---- |----
| | | | | |
1 2 3 5 8 9
|--
| |
6 7
We define the path of each object as the sequence of IDs from the root down to the object:
0 --> 0
1 --> 0, 1
2 --> 0, 2
3 --> 0, 3
4 --> 4
5 --> 4, 5
6 --> 4, 5, 6
7 --> 4, 5, 7
8 --> 4, 8
9 --> 4, 9
These paths can be ordered lexicographically so that a sequence of objects sorted by path is actually a representation of the underlying hierarchy. If we add a parent pointer to objects to model parent-child relationships:
struct obj
{
int id;
const obj* parent=nullptr;
};
then we can define a multi_index_container with both O(1) access by ID and hierarchy-based indexing:
using nested_container=multi_index_container<
obj,
indexed_by<
hashed_unique<member<obj,int,&obj::id>>,
ordered_unique<identity<obj>,obj_less>
>
>;
where obj_less compares objects according to path ordering. All types of tree manipulations and visitations are possible as exemplified below (code is not entirely trivial, feel free to ask).
Live On Coliru
#include <boost/multi_index_container.hpp>
#include <boost/multi_index/hashed_index.hpp>
#include <boost/multi_index/ordered_index.hpp>
#include <boost/multi_index/identity.hpp>
#include <boost/multi_index/member.hpp>
#include <iterator>
struct obj
{
int id;
const obj* parent=nullptr;
};
struct subtree_obj
{
const obj& obj_;
};
struct path
{
int id;
const path* next=nullptr;
};
struct subtree_path
{
const path& path_;
};
inline bool operator<(const path& x,const path& y)
{
if(x.id<y.id)return true;
else if(y.id<x.id)return false;
else if(!x.next) return y.next;
else if(!y.next) return false;
else return *(x.next)<*(y.next);
}
inline bool operator<(const subtree_path& sx,const path& y)
{
const path& x=sx.path_;
if(x.id<y.id)return true;
else if(y.id<x.id)return false;
else if(!x.next) return false;
else if(!y.next) return false;
else return subtree_path{*(x.next)}<*(y.next);
}
inline bool operator<(const path& x,const subtree_path& sy)
{
return x<sy.path_;
}
struct obj_less
{
private:
template<typename F>
static auto apply_to_path(const obj& x,F f)
{
return apply_to_path(x.parent,path{x.id},f);
}
template<typename F>
static auto apply_to_path(const obj* px,const path& x,F f)
->decltype(f(x))
{
return !px?f(x):apply_to_path(px->parent,{px->id,&x},f);
}
public:
bool operator()(const obj& x,const obj& y)const
{
return apply_to_path(x,[&](const path& x){
return apply_to_path(y,[&](const path& y){
return x<y;
});
});
}
bool operator()(const subtree_obj& x,const obj& y)const
{
return apply_to_path(x.obj_,[&](const path& x){
return apply_to_path(y,[&](const path& y){
return subtree_path{x}<y;
});
});
}
bool operator()(const obj& x,const subtree_obj& y)const
{
return apply_to_path(x,[&](const path& x){
return apply_to_path(y.obj_,[&](const path& y){
return x<subtree_path{y};
});
});
}
};
using namespace boost::multi_index;
using nested_container=multi_index_container<
obj,
indexed_by<
hashed_unique<member<obj,int,&obj::id>>,
ordered_unique<identity<obj>,obj_less>
>
>;
template<typename Iterator>
inline auto insert_under(nested_container& c,Iterator it,obj x)
{
x.parent=&*it;
return c.insert(std::move(x));
}
template<typename Iterator,typename F>
void for_each_in_level(
nested_container& c,Iterator first,Iterator last, F f)
{
if(first==last)return;
const obj* parent=first->parent;
auto first_=c.project<1>(first),
last_=c.project<1>(last);
do{
f(*first_);
auto next=std::next(first_);
if(next->parent!=parent){
next=c.get<1>().upper_bound(subtree_obj{*first_});
}
first_=next;
}while(first_!=last_);
}
template<typename ObjPointer,typename F>
void for_each_child(nested_container& c,ObjPointer p,F f)
{
auto [first,last]=c.get<1>().equal_range(subtree_obj{*p});
for_each_in_level(c,std::next(first),last,f);
}
#include <iostream>
auto print=[](const obj& x){std::cout<<x.id<<" ";};
void print_subtree(nested_container& c,const obj& x)
{
std::cout<<x.id<<" ";
bool visited=false;
for_each_child(c,&x,[&](const obj& x){
if(!visited){
std::cout<<"[ ";
visited=true;
}
print_subtree(c,x);
});
if(visited)std::cout<<"] ";
}
int main()
{
nested_container c;
auto it=c.insert({0}).first;
insert_under(c,it,{1});
insert_under(c,it,{2});
insert_under(c,it,{3});
it=c.insert({4}).first;
auto it2=insert_under(c,it,{5}).first;
insert_under(c,it2,{6});
insert_under(c,it2,{7});
insert_under(c,it,{8});
insert_under(c,it,{9});
std::cout<<"preorder:\t";
std::for_each(c.get<1>().begin(),c.get<1>().end(),print);
std::cout<<"\n";
std::cout<<"top level:\t";
for_each_in_level(c,c.get<1>().begin(),c.get<1>().end(),print);
std::cout<<"\n";
std::cout<<"children of 0:\t";
for_each_child(c,c.find(0),print);
std::cout<<"\n";
std::cout<<"children of 4:\t";
for_each_child(c,c.find(4),print);
std::cout<<"\n";
std::cout<<"children of 5:\t";
for_each_child(c,c.find(5),print);
std::cout<<"\n";
std::cout<<"bracketed:\t";
for_each_in_level(c,c.get<1>().begin(),c.get<1>().end(),[&](const obj& x){
print_subtree(c,x);
});
std::cout<<"\n";
}
Output
preorder: 0 1 2 3 4 5 6 7 8 9
top level: 0 4
children of 0: 1 2 3
children of 4: 5 8 9
children of 5: 6 7
bracketed: 0 [ 1 2 3 ] 4 [ 5 [ 6 7 ] 8 9 ]
Update 2020/02/02:
When accessing top-level elements, I've changed the code from:
std::for_each(c.begin(),c.end(),...;
for_each_in_level(c,c.begin(),c.end(),...);
to
std::for_each(c.get<1>().begin(),c.get<1>().end(),...;
for_each_in_level(c,c.get<1>().begin(),c.get<1>().end(),...);
This is because index #0 is hashed and does not necessarily show elements sorted by ID.
For instance, if elements with IDs (170,171,173,173,141) are inserted in this order, index #0 lists them as
170,171,173,173,141 (coincidentally, same order as inserted),
while index #1 lists them as
141,170,171,173,173 (sorted by ID).
The way the code is implemented, for_each_in_level(c,c.begin(),c.end(),...); gets internally mapped to index #1 range [170,...,173], leaving out 141. The way to make sure all top elements are included is then to write for_each_in_level(c,c.get<1>().begin(),c.get<1>().end(),...);.
Good morning everyone.
I am new to C++ 11 multithreading theme and trying to write down a code that add two vectors of equal sizes in asynchronous way.
This means, if I have two vectors:
vector<int> fisrt = {1, 2, 3};
vector<int> second = {3, 2, 1};
first += second; // first = {4, 4, 4}
I wrote Paginator class which splits vector on "pages" with appropriate page_size.
My idea of vectors addition in asynchronous way is next: split vectors on pages with choosen page_size, and add pages of first and second vectors in asynchronous way.
Paginator class implementation
template<class Iter>
class IterRange {
public:
explicit IterRange(Iter first, Iter last) : first_(first), last_(last) {}
Iter begin() { return first_; }
const Iter begin() const { return first_; }
Iter end() { return last_; }
const Iter end() const { return last_; }
private:
Iter first_;
Iter last_;
};
template<class Iter>
IterRange<Iter> MakeIterRange(Iter first, Iter last) {
return IterRange<Iter> {first, last };
}
template<class Iter>
class Paginator {
public:
Paginator(Iter first, Iter last, size_t page_size) : page_size_(page_size) {
size_t pages_count = static_cast<size_t> (floor((double)distance(first, last) / page_size_));
pages_.reserve(pages_count);
size_t page_id = 0u;
Iter begin_page = first;
for (page_id, begin_page; page_id < pages_count; ++page_id, begin_page += page_size_) {
pages_.push_back(MakeIterRange( begin_page, begin_page + page_size ));
}
// If some elements less than page_size_ is left
if (begin_page != last) {
pages_.push_back(MakeIterRange(begin_page, begin_page + distance(begin_page, last)));
}
}
auto begin() { return pages_.begin(); }
auto begin() const { return pages_.begin(); }
auto end() { return pages_.end(); }
auto end() const { return pages_.end(); }
private:
size_t page_size_;
vector<IterRange<Iter>> pages_;
};
template<class Iter>
Paginator<Iter> MakePaginator(Iter first, Iter last, size_t page_size) {
return{ first, last, page_size };
}
template<class Container> // And the same for non constant Container
auto Paginate(const Container & c, size_t page_size) {
return MakePaginator(begin(c), end(c), page_size);
}
This Paginate procedure is used in operator+= in my Matrix class.
Matrix class fields are:
Matrix sizes along Ox and Oy direction respectively: size_t nx_, size_t ny_.
Vector of (nx_ * ny_) size, which stores all elements in matrix: vector body_.
Operator += for Matrix
template
inline Matrix & Matrix::operator+=(const Matrix & other) {
size_t threads_numb = thread::hardware_concurrency();
size_t page_size = static_cast<size_t> (ceil((double)body_.size() / threads_numb));
vector<future<void>> futures;
auto page_1 = page::Paginate(body_, page_size);
auto page_2 = page::Paginate(other.body_, page_size);
auto it_2 = page_2.begin();
for (auto it = page_1.begin(); it != page_1.end(); ++it, ++it_2) {
futures.push_back(
async([it, it_2] { transform(it->begin(), it->end(), it_2->begin(), it->begin(), plus<T>()); })
);
}
return *this;
}
But as a result I get iterator out of range error! How could I fix this?
P.S. Sorry for bad representation of first string in operator +=. Could not fix this problem :(
I want to use expression templates to create a tree of objects that persists across statement. Building the tree initially involves some computations with the Eigen linear algebra library. The persistent expression template will have additional methods to compute other quantities by traversing the tree in different ways (but I'm not there yet).
To avoid problems with temporaries going out of scope, subexpression objects are managed through std::unique_ptr. As the expression tree is built, the pointers should be propagated upwards so that holding the pointer for the root object ensures all objects are kept alive. The situation is complicated by the fact that Eigen creates expression templates holding references to temporaries that go out of scope at the end of the statement, so all Eigen expressions must be evaluated while the tree is being constructed.
Below is a scaled-down implementation that seems to work when the val type is an object holding an integer, but with the Matrix type it crashes while constructing the output_xpr object. The reason for the crash seems to be that Eigen's matrix product expression template (Eigen::GeneralProduct) gets corrupted before it is used. However, none of the destructors either of my own expression objects or of GeneralProduct seems to get called before the crash happens, and valgrind doesn't detect any invalid memory accesses.
Any help will be much appreciated! I'd also appreciate comments on my use of move constructors together with static inheritance, maybe the problem is there somewhere.
#include <iostream>
#include <memory>
#include <Eigen/Core>
typedef Eigen::MatrixXi val;
// expression_ptr and derived_ptr: contain unique pointers
// to the actual expression objects
template<class Derived>
struct expression_ptr {
Derived &&transfer_cast() && {
return std::move(static_cast<Derived &&>(*this));
}
};
template<class A>
struct derived_ptr : public expression_ptr<derived_ptr<A>> {
derived_ptr(std::unique_ptr<A> &&p) : ptr_(std::move(p)) {}
derived_ptr(derived_ptr<A> &&o) : ptr_(std::move(o.ptr_)) {}
auto operator()() const {
return (*ptr_)();
}
private:
std::unique_ptr<A> ptr_;
};
// value_xpr, product_xpr and output_xpr: expression templates
// doing the actual work
template<class A>
struct value_xpr {
value_xpr(const A &v) : value_(v) {}
const A &operator()() const {
return value_;
}
private:
const A &value_;
};
template<class A,class B>
struct product_xpr {
product_xpr(expression_ptr<derived_ptr<A>> &&a, expression_ptr<derived_ptr<B>> &&b) :
a_(std::move(a).transfer_cast()), b_(std::move(b).transfer_cast()) {
}
auto operator()() const {
return a_() * b_();
}
private:
derived_ptr<A> a_;
derived_ptr<B> b_;
};
// Top-level expression with a matrix to hold the completely
// evaluated output of the Eigen calculations
template<class A>
struct output_xpr {
output_xpr(expression_ptr<derived_ptr<A>> &&a) :
a_(std::move(a).transfer_cast()), result_(a_()) {}
const val &operator()() const {
return result_;
}
private:
derived_ptr<A> a_;
val result_;
};
// helper functions to create the expressions
template<class A>
derived_ptr<value_xpr<A>> input(const A &a) {
return derived_ptr<value_xpr<A>>(std::make_unique<value_xpr<A>>(a));
}
template<class A,class B>
derived_ptr<product_xpr<A,B>> operator*(expression_ptr<derived_ptr<A>> &&a, expression_ptr<derived_ptr<B>> &&b) {
return derived_ptr<product_xpr<A,B>>(std::make_unique<product_xpr<A,B>>(std::move(a).transfer_cast(), std::move(b).transfer_cast()));
}
template<class A>
derived_ptr<output_xpr<A>> eval(expression_ptr<derived_ptr<A>> &&a) {
return derived_ptr<output_xpr<A>>(std::make_unique<output_xpr<A>>(std::move(a).transfer_cast()));
}
int main() {
Eigen::MatrixXi mat(2, 2);
mat << 1, 1, 0, 1;
val one(mat), two(mat);
auto xpr = eval(input(one) * input(two));
std::cout << xpr() << std::endl;
return 0;
}
Your problem appears to be that you are using someone else's expression templates, and storing the result in an auto.
(This happens in product_xpr<A>::operator(), where you call *, which if I read it right, is an Eigen multiplication that uses expression templates).
Expression templates are often designed to presume the entire expression will occur on a single line, and it will end with a sink type (like a matrix) that causes the expression template to be evaluated.
In your case, you have a*b expression template, which is then used to construct an expression template return value, which you later evaluate. The lifetime of temporaries passed to * in a*b are going to be over by the time you reach the sink type (matrix), which violates what the expression templates expect.
I am struggling to come up with a solution to ensure that all temporary objects have their lifetime extended. One thought I had was some kind of continuation passing style, where instead of calling:
Matrix m = (a*b);
you do
auto x = { do (a*b) pass that to (cast to matrix) }
replace
auto operator()() const {
return a_() * b_();
}
with
template<class F>
auto operator()(F&& f) const {
return std::forward<F>(f)(a_() * b_());
}
where the "next step' is passed to each sub-expression. This gets trickier with binary expressions, in that you have to ensure that the evaluation of the first expression calls code that causes the second sub expression to be evaluated, and then the two expressions are combined, all in the same long recursive call stack.
I am not proficient enough in continuation passing style to untangle this knot completely, but it is somewhat popular in the functional programming world.
Another approach would be to flatten your tree into a tuple of optionals, then construct each optional in the tree using a fancy operator(), and manually hook up the arguments that way. Basically do manual memory management of the intermediate values. This will work if the Eigen expression templates are either move-aware or do not have any self-pointers, so that moving at the point of construction doesn't break things. Writing that would be challenging.
Continuation passing style, suggested by Yakk, solves the problem and isn't too insane (not more insane than template metaprogramming in general anyhow). The double lambda evaluation for the arguments of binary expressions can be tucked away in a helper function, see binary_cont in the code below. For reference, and since it's not entirely trivial, I'm posting the fixed code here.
If somebody understands why I had to put a const qualifier on the F type in binary_cont, please let me know.
#include <iostream>
#include <memory>
#include <Eigen/Core>
typedef Eigen::MatrixXi val;
// expression_ptr and derived_ptr: contain unique pointers
// to the actual expression objects
template<class Derived>
struct expression_ptr {
Derived &&transfer_cast() && {
return std::move(static_cast<Derived &&>(*this));
}
};
template<class A>
struct derived_ptr : public expression_ptr<derived_ptr<A>> {
derived_ptr(std::unique_ptr<A> &&p) : ptr_(std::move(p)) {}
derived_ptr(derived_ptr<A> &&o) = default;
auto operator()() const {
return (*ptr_)();
}
template<class F>
auto operator()(F &&f) const {
return (*ptr_)(std::forward<F>(f));
}
private:
std::unique_ptr<A> ptr_;
};
template<class A,class B,class F>
auto binary_cont(const derived_ptr<A> &a_, const derived_ptr<B> &b_, const F &&f) {
return a_([&b_, f = std::forward<const F>(f)] (auto &&a) {
return b_([a = std::forward<decltype(a)>(a), f = std::forward<const F>(f)] (auto &&b) {
return std::forward<const F>(f)(std::forward<decltype(a)>(a), std::forward<decltype(b)>(b));
});
});
}
// value_xpr, product_xpr and output_xpr: expression templates
// doing the actual work
template<class A>
struct value_xpr {
value_xpr(const A &v) : value_(v) {}
template<class F>
auto operator()(F &&f) const {
return std::forward<F>(f)(value_);
}
private:
const A &value_;
};
template<class A,class B>
struct product_xpr {
product_xpr(expression_ptr<derived_ptr<A>> &&a, expression_ptr<derived_ptr<B>> &&b) :
a_(std::move(a).transfer_cast()), b_(std::move(b).transfer_cast()) {
}
template<class F>
auto operator()(F &&f) const {
return binary_cont(a_, b_,
[f = std::forward<F>(f)] (auto &&a, auto &&b) {
return f(std::forward<decltype(a)>(a) * std::forward<decltype(b)>(b));
});
}
private:
derived_ptr<A> a_;
derived_ptr<B> b_;
};
template<class A>
struct output_xpr {
output_xpr(expression_ptr<derived_ptr<A>> &&a) :
a_(std::move(a).transfer_cast()) {
a_([this] (auto &&x) { this->result_ = x; });
}
const val &operator()() const {
return result_;
}
private:
derived_ptr<A> a_;
val result_;
};
// helper functions to create the expressions
template<class A>
derived_ptr<value_xpr<A>> input(const A &a) {
return derived_ptr<value_xpr<A>>(std::make_unique<value_xpr<A>>(a));
}
template<class A,class B>
derived_ptr<product_xpr<A,B>> operator*(expression_ptr<derived_ptr<A>> &&a, expression_ptr<derived_ptr<B>> &&b) {
return derived_ptr<product_xpr<A,B>>(std::make_unique<product_xpr<A,B>>(std::move(a).transfer_cast(), std::move(b).transfer_cast()));
}
template<class A>
derived_ptr<output_xpr<A>> eval(expression_ptr<derived_ptr<A>> &&a) {
return derived_ptr<output_xpr<A>>(std::make_unique<output_xpr<A>>(std::move(a).transfer_cast()));
}
int main() {
Eigen::MatrixXi mat(2, 2);
mat << 1, 1, 0, 1;
val one(mat), two(mat), three(mat);
auto xpr = eval(input(one) * input(two) * input(one) * input(two));
std::cout << xpr() << std::endl;
return 0;
}