I am trying to send a user-defined structure named ABC using boost::mpi::send () call.
The given struct contains a vector "data" whose size is determined at runtime. Objects of struct ABC are sent by master to slaves.
But the slaves need to know the size of vector "data" so that the sufficient buffer is available on the slave to receive this data.
I can work around it by sending the size first and initialize sufficient buffer on the slave before receiving the objects of struct ABC. But that defeats the whole purpose of using STL containers.
Does anyone know of a better way to do handle this ? Any suggestions are greatly appreciated.
Here is a sample code that describes the intent of my program. This code fails at runtime due to above mentioned reason.
struct ABC
{
double cur_stock_price;
double strike_price;
double risk_free_rate;
double option_price;
std::vector <char> data;
};
namespace boost
{
namespace serialization
{
template<class Archive>
void serialize (Archive &ar,
struct ABC &abc,
unsigned int version)
{
ar & abc.cur_stock_price;
ar & abc.strike_price;
ar & abc.risk_free_rate;
ar & abc.option_price;
ar & bopr.data;
}
}
}
BOOST_IS_MPI_DATATYPE (ABC);
int main(int argc, char* argv[])
{
mpi::environment env (argc, argv);
mpi::communicator world;
if (world.rank () == 0)
{
ABC abc_obj;
abc.cur_stock_price = 1.0;
abc.strike_price = 5.0;
abc.risk_free_rate = 2.5;
abc.option_price = 3.0;
abc_obj.data.push_back ('a');
abc_obj.data.push_back ('b');
world.send ( 1, ANY_TAG, abc_obj;);
std::cout << "Rank 0 OK!" << std::endl;
}
else if (world.rank () == 1)
{
ABC abc_obj;
// Fails here because abc_obj is not big enough
world.recv (0,ANY_TAG, abc_obj;);
std::cout << "Rank 1 OK!" << std::endl;
for (int i = 0; i < abc_obj;.data.size(); i++)
std::cout << i << "=" << abc_obj.data[i] << std::endl;
}
MPI_Finalize();
return 0;
}
You shouldn't send the vector object itself in the message, as the receiver only needs its contents, and the internal state of the vector would probably be all messed up after receiving anyway (memory addresses that were valid on the sending end probably won't be valid on the receiving end).
Instead, here's what you need to do:
Define a separate struct with four doubles and a simple char array.
Whenever you need to send, create a temporary variable of this new struct, and fill the char array with the contents of the vector you want to send.
Define a temporary MPI datatype that matches the size of this specific struct object.
Send the temporary struct as an instance of this temporary datatype.
Receive it in a suitably large receive buffer (or send the size in advance)
Reconstruct the ABC struct based on the contents of the char array.
At least, that's what you need to do with vanilla C++ and MPI. I'm not familiar with boost, so I don't know if it makes any of these steps easier.
BOOST_IS_MPI_DATATYPE is only for fixed length data. Your array is not fixed length and this is why it fails.
Related
in C++, if a method is accepting left reference + pointer only,
it seems it suffices if we only have a template method with T& as its parameter, why we usually overload with test(T* ) as well ?
proof of concept: left reference method can take pointer argument.
#include <iostream>
using namespace std;
template<class T>
void test(T& arg) {
T value = arg;
cout << *value << endl;
}
int main() {
int b = 4;
int* a = &b;
test(a); // compiles and runs without issue.
return 0;
}
Why [do] we usually overload with test(T* ) as well?
I am not sure that we usually do anything of the sort, but if one were to overload for a pointer, it would be because pointers behave differently than object types. Remember, a pointer in fact is not an object but an address to an object.
The reason that test(a) compiles and runs without issue is because it is accepting a reference to a pointer to an object as its parameter. Thus, when the line cout << *value << endl; executes, the pointer is dereferenced back to an object and we see 4 printed to standard out.
As #HolyBlackCat mentioned, we usually want do different things for T& and T*.
As indicated in the example, for test(T&) we usually need to manually do dereference, this would result in the difference in the behavior, so it makes sense to have a overload like this.
If I have a code for example like this:
#include <iostream>
using namespace std;
void swap(void** a) {
int tmp = 5;
void* b = &tmp;
a = &b;
}
int main()
{
int x=11;
void* y=&x;
void** z=&y;
swap(z);
void* a = *z;
cout << *(int*)a << endl;
return 0;
}
The code above prints 11, but I want to update the value of z (its address) to point to a place so I can print 5 (I mean update it). What should I do so that when I send z to the function and get back to main I can receive 5 instead of 11.
I'm just not that good with pointers.
EDIT: I must send to swap an argument with void**
You can't update the value of a void** (i.e. what it points to) by passing it to a function that takes a void**. That only allows to modify the pointed-to memory, not what address the pointer you pass to the function points to.
To update what it points to, the parameter should be a void**& or a void***.
Regardless of what solution you choose, the code you posted is extremely error prone and a hell to maintain. You should totally avoid it.
Also, note that &tmp becomes invalid as long as you exit the function, because the local variable tmp gets destroyed.
It seems that if I write
#include <random>
std::minstd_rand engine(1);
std::cout << engine;
then this prints out the internal state of the engine (which is a linear congruential generator). Right now the state equals the seed (1), but if I call a random number and print out engine, it returns some large number, which is probably the state.
How do I actually get the state, in a variable?
Use a string stream instead of stdout. Example:
#include <sstream>
...
std::ostringstream os;
os << engine;
string mystate = os.str();
The o in ostringstream is for output.
The state should be last random number generated, which is why there is not an easier way to do this. It's not as ideal as something like int a; a << engine, but it'll have to do. If you need it that often, make the stringstream operation a function (Including perhaps a conversion from string to integer). You can also typedef a pair of engine/integer with the integer being the state, and make a couple of methods so it's autoset every generation call if you need the performance.
If you don't care about the state, and just want it for the future, do
int engineState = engine();
Now you have the state. Though it's not the same as what it was before, it might not matter depending on your use case.
Output from linear congruential RNG is the state. Or, as alreadynoted, use operator<< to output and convert state
Code
#include <random>
#include <iostream>
#include <sstream>
int main() {
auto engine = std::minstd_rand{ 1 };
auto q = engine();
auto os = std::ostringstream{};
os << engine;
auto r = std::stoul(os.str()); // use ul to fit output
std::cout << q << " " << os.str() << " " << r << '\n';
return 0;
}
prints
48271 48271 48271
Alternative might be if particular implementation implements discard properly in O(log2(N)) time, according to paper by F.Brown https://laws.lanl.gov/vhosts/mcnp.lanl.gov/pdf_files/anl-rn-arb-stride.pdf. In such case you could move one position back, call RNG again and get your state as output.
Compiler and library I use - Visual C++ 2017 15.7 - has not implemented discard in such way, and useless for moving back.
LCGs consist of a simple state that is represented by a single integer.
This means you can treat this pointer as a pointer to an integer.
Below, I have provided an example of a template function that gets
the state (seed) of an engine and even works for classes deriving LCGs.
#include <random>
template <class T, T... v>
T getSeed(std::linear_congruential_engine<T, v...>& rand) {
static_assert(sizeof(rand) == sizeof(T));
return *reinterpret_cast<T*>(&rand);
}
#include <iostream>
int main() {
std::minstd_rand engine(19937);
auto seed = getSeed(engine);
std::cout << sizeof(engine);
std::cout << '\t' << seed;
}
^ This method is way more efficient (x320 times) than serializing through a stream,
or by creating a dummy ostream and specializing std::operator<< for every case.
template<class T, T... v>
using LCG = std::linear_congruential_engine<T, v...>;
#define DummyRandSpec32 uint_fast32_t, 0xDEADBEEF, 0xCAFE, 0xFFFFFFFF
typedef LCG<DummyRandSpec32> DummyRand32; // the same engine type
template<class T, class R>
T* getSeed(R& rand) // getSeed 70:1 nextInt
{ // creating stream is heavy operation
// return rand._M_x; // cannot access private
__dummy_ostream<T> dumdum; // workaround
auto& didey = *reinterpret_cast<DummyRand32*>(&rand);
std::operator<<(dumdum, didey); // specialized
return dumdum.retrieve(); // pointer to state
}
int main() {
std::minstd_rand engine(19937);
std::cout << *getSeed<uint_fast32_t>(engine);
std::cout << std::endl << engine << std::endl;
}
^ Here is ill-coded my first attempt at a solution, if you want to compare.
It is worth mentioning that a field name of the state is implementation-specific.
Purposefully left out std::operator<< and __dummy_ostream.
I have a std::vector<double> that I have to move to a boost::container::flat_set<double>.
Both containers are contiguous, so after sorting the vector in principle I could move the data from one to the other.
Is there a way to move the whole data between these two different containers?
Please, take into account that I want to move the whole data, not element by element.
I can move data between containers of the same type, but not between different containers.
std::vector<double> v1 = ...
std::sort(v1.begin(), v1.end());
std::vector<double> v2(std::move(v1)); // ok
boost::flat_set<double> f2(v1.begin(), v1.end()); // doesn't move, it copies
boost::flat_set<double> f3(std::move(v1)); // doesn't compile
It seems that for this to work flat_set should have a move constructor from containers with .data(), where the pointer is stolen from the argument.
I believe there is some way to verify whenever data alignment in both containers match and memcpy could be used (and source cleared without destructing) exists and maybe someone will share it with us, but as long as we want to use STL there is a way: the std::move_iterator. It makes your container constructor move elements instead of copying. It does not remove elements out of source container though, but leaves them stateless (e.g. empty strings as in example).
#include <iostream>
#include <vector>
#include <string>
#include <algorithm>
#include <boost/container/flat_set.hpp>
int main()
{
std::vector<std::string> v1 = {"a","v","d"};
std::sort(v1.begin(), v1.end());
std::vector<std::string> v2(std::move(v1)); // ok
boost::container::flat_set<std::string> f1(std::make_move_iterator(v2.begin()), std::make_move_iterator(v2.end())); // moves, but does not remove elements from of source container
for(auto& s : v1)
std::cout << "'" << s << "'" << ' ';
std::cout << " <- v1 \n";
for(auto& s : v2)
std::cout << "'" << s << "'" << ' ';
std::cout << " <- v2 \n";
for(auto& s : f1)
std::cout << "'" << s << "'" << ' ';
std::cout << " <- f1 \n";
}
Output
<- v1
'' '' '' <- v2
'a' 'd' 'v' <- f1
Online code: https://wandbox.org/permlink/ZLbocXKdqYHT0zYi
It looks like it is not possible without modifying the constructor boost::container::flat.
Without modifying either class it seems that the only a hack would do it, for example using reinterpret_cast.
The solution I found is either to use an alternative implementation of vector or very ugly code.
Before going into my solution, I must that say that this is probably a
defect of both classes. These clases should have a set of
release()/aquire(start, end) functions that respectively
returns the pointer range to the data releasing the ownership and
gets the pointer range owning it from then on. An alternative could be to
have a constructor that moves from any other container that has a the
data member function.
Solution using reinterpret_cast and a different implementation of vector
It turns out that reinterpret_casting from std::vector to boost::container::flat_set is not possible, because the layout is not compatible.
However it is possible to reinterpret_cast from boost::container::vector to boost::container::flat_set out of the box (that is because they have a common implementation).
#include<cassert>
#include<boost/container/flat_set.hpp>
int main(){
boost::container::vector<double> v = {1.,2.,3.};
boost::container::flat_set<double> fs = std::move(reinterpret_cast<boost::container::flat_set<double>&>(v));
assert(v.size() == 0);
assert(*fs.find(2.) == 2.);s
assert(fs.find(4.) == fs.end());
}
So, I can replace std::vector by boost::container::vector and I can move data to a flat_set.
Non-portable solution using std::vector and ugly code
The reason the layout of std::vector and boost::container::vector are different is that boost::container::vector stores metadata in this way:
class boost::container::vector{
pointer m_start;
size_type m_size;
size_type m_capacity;
}
while std::vector (in GCC) is basically pure pointers,
class std::vector{
pointer _M_start;
pointer _M_finish;
pointer _M_end_of_storage;
}
So, my conclusion is that moving is possible only through a hack given that the implementation I use of std::vector is not compatible with boost::container::flat_set.
In an extreme case, one can do this (sorry if this code offends someone, the code is not portable):
template<class T>
boost::container::flat_set<T> to_flat_set(std::vector<T>&& from){
// struct dummy_vector{T* start; T* finish; T* end_storarge;}&
// dfrom = reinterpret_cast<dummy_vector&>(from);
boost::container::flat_set<T> ret;
struct dummy_flat_set{T* start; std::size_t size; std::size_t capacity;}&
dret = reinterpret_cast<dummy_flat_set&>(ret);
dret = {from.data(), from.size(), from.capacity()};
// dfrom.start = dfrom.finish = dfrom.end_storarge = nullptr;
new (&from) std::vector<T>();
return ret;
};
int main(){
std::vector<double> v = {1.,2.,3.};
boost::container::flat_set<double> fs = to_flat_set(std::move(v));
assert(v.size() == 0);
assert(*fs.find(2.) == 2.);
assert(fs.find(4.) == fs.end());
}
Note that I am not taking into account allocator issues at all. I am not sure how to handle allocators here.
In retrospect I don't mind using a form of cast for this specific problem, because somehow I have to tell that the vector is sorted before moving to flat_set. (The problem is that this goes to extreme because it is a reinterpret_cast.)
However this is a secondary issue, there should be legal way to move from std::vector to boost::container::vector.
I coded in Borland C++ ages ago, and now I'm trying to understand the "new"(to me) C+11 (I know, we're in 2015, there's a c+14 ... but I'm working on an C++11 project)
Now I have several ways to assign a value to a string.
#include <iostream>
#include <string>
int main ()
{
std::string test1;
std::string test2;
test1 = "Hello World";
test2.assign("Hello again");
std::cout << test1 << std::endl << test2;
return 0;
}
They both work. I learned from http://www.cplusplus.com/reference/string/string/assign/ that there are another ways to use assign . But for simple string assignment, which one is better? I have to fill 100+ structs with 8 std:string each, and I'm looking for the fastest mechanism (I don't care about memory, unless there's a big difference)
Both are equally fast, but = "..." is clearer.
If you really want fast though, use assign and specify the size:
test2.assign("Hello again", sizeof("Hello again") - 1); // don't copy the null terminator!
// or
test2.assign("Hello again", 11);
That way, only one allocation is needed. (You could also .reserve() enough memory beforehand to get the same effect.)
I tried benchmarking both the ways.
static void string_assign_method(benchmark::State& state) {
std::string str;
std::string base="123456789";
// Code inside this loop is measured repeatedly
for (auto _ : state) {
str.assign(base, 9);
}
}
// Register the function as a benchmark
BENCHMARK(string_assign_method);
static void string_assign_operator(benchmark::State& state) {
std::string str;
std::string base="123456789";
// Code before the loop is not measured
for (auto _ : state) {
str = base;
}
}
BENCHMARK(string_assign_operator);
Here is the graphical comparitive solution. It seems like both the methods are equally faster. The assignment operator has better results.
Use string::assign only if a specific position from the base string has to be assigned.