I want to read a binary file in 16 bit words. Right now, I'm using an std::ifstream to read into a 2 character array c.
#include <iostream>
#include <fstream>
#include <stdint.h>
int main() {
std::ifstream file("./tetris.rom", std::ios::in | std::ios::binary);
char c[2];
while (file.read(c, 2)) {
uint16_t word = (static_cast<uint8_t>(c[0]) << 8) | static_cast<uint8_t>(c[1]);
std::cout << "word\t" << std::hex << word << std::endl;
}
}
This works for me, but is there a better (either safer or faster) way of doing this in C++11?
There are no new APIs of reading files in C++11.
If the file fits into your RAM, the most optimal way is to map it into memory and access it as a byte array. However, the C++ standard library does not provide an API for that. You can do that with Boost though, see Boost.Interprocess Memory Mapped Files.
The usual advice stands though: start with your simple and correctly working code, benchmark and see if file reading is the bottleneck.
Related
I'm using eigen to do some matrix operation, but the compile time for the src files using eigen is very slow, by slow I mean it take about 40s when the file is only 300 lines. I only use Matrix smaller than Matrix4f and not even use dynamic size matrix, for only some matrix multiplication and matrix decomposition(SVD and FullPivLU).
In my other project where a cpp file is 1000 lines, it take minutes to compile, and the output .so file is really big, about 100M. I have to open the -bigobj choice.
This happens both in debug(where I set the opimization to -O0) and release(-O3) mode. I've tried add #define NDEBUG and EIGEN_NO_DEBUG in the header, not help.
I write this piece of very small code test.cpp below:
#include </home/user/mywork/software/eigen-3.3.9/Eigen/Dense>
#include "iostream"
using namespace std;
using namespace Eigen;
void test()
{
Matrix3f A;
A << 1,2,3,4,2,8,5,4,9;
BDCSVD<Matrix3f> svd(A, ComputeFullU | ComputeFullV);
Matrix3f U = svd.matrixU();
Matrix3f V = svd.matrixV();
cout << "------Eigen------" << endl;
cout << "A" << A << endl;
cout << "U" << U << endl;
cout << "V" << V << endl;
}
int main()
{
test();
return 0;
}
compile command
g++ test.cpp -o test_eigen
it take 20s to compile, and the output 'test_eigen' is 5.6M!!!
my os is ubuntu 16.04 and I use cmake, in the CMakelist, only thing related to eigen is include its directory. eigen version is 3.3.9.
does anyone have any clue? Thanks a lot.
Eigen consists of 7.8MB of pure header files. So any .cpp including it will have a hard life compiling (uses giga-bytes of memory and extremely slow). In addition, due to extensive use of templates and inline functions, the DEBUG version will be so slow and unusable for debug purpose. (For example, google's Cartographer will warn about debug version) and I also made some tests.
So I think Eigen should be used as follow:
Encapsulate Eigen. Create simple interface according to your own needs. (For example, create matrix_4x4.h, vector4.h, standard_deviation_calculaor.h, etc.) Include Eigen only in .cpp files. The exposed .h files will be so simple and will not propagate to slow down compiling of other files.
Proper Modularization. Precompile the library into static/shared libraries. And use the simple headers in other parts of your program.
Restrain Use eigen only in performance-critical parts of my program. For example, the following function will not use eigen and should be inlined(exposed in header):
// vector3.h
inline Vector3 operator + (const Vector3& l, const Vector3& r) {
return Vector3(l.x + r.x, l.y + r.y, l.z + r.z);
}
But Matrix4x4::multiplyInplace(const Matrix4x4& r) will be
implemented in .cpp and use Eigen.
I'm using boost-1.68 and gcc-7.3.0 with -std=c++11. The following code as written throws a SIGSEGV for me.
The problem is caused by the combination of using std::unordered_map along with the boost interprocess shrink_to_fit() method. Changing the shared::unordered_map type from std::unordered_map to boost::unordered_map and/or commenting out the call to shrink_to_fit() will fix the code.
I ultimately don't intend to use std::unordered_map, but the fact that this minimal example is failing makes me concerned that I'm not properly setting up and/or not properly using the allocators.
EDIT: Should have mentioned that using shrink_to_fit() is necessary for my purposes.
Any insight would be appreciated.
#include <cstdio>
#include <iostream>
#include <unordered_map>
#include <scoped_allocator>
#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/container_hash/hash.hpp>
#include <boost/unordered_map.hpp>
using std::cout;
using std::endl;
namespace shared {
using segment_type = boost::interprocess::managed_mapped_file;
using segment_manager = segment_type::segment_manager;
template <typename T>
using Alloc = boost::interprocess::allocator<T, segment_manager>;
template <typename T>
using ScopedAlloc = std::scoped_allocator_adaptor<Alloc<T>>;
using string = boost::interprocess::basic_string<char, std::char_traits<char>, Alloc<char>>;
template <typename Key, typename Val, typename KHash = boost::hash<Key>, typename KEqual = std::equal_to<Key>>
using unordered_map =
std::unordered_map<Key, Val, KHash, KEqual, ScopedAlloc<std::pair<const Key, Val>>>;
}
struct MyStruct {
shared::unordered_map<shared::string, int> some_map;
MyStruct (shared::segment_manager *smgr)
: some_map{smgr}
{ }
friend std::ostream& operator<< (std::ostream& os, const MyStruct& m) {
os << "{";
for (auto& e : m.some_map) {
os << "{\"" << e.first << "\"," << e.second << "}";
}
os << "}";
return os;
}
};
int main () {
const char *filename = "content.bin";
std::remove(filename);
{
shared::segment_type segment(boost::interprocess::create_only, filename, (1<<20));
MyStruct *o1 = segment.construct<MyStruct>("MyStruct01")(segment.get_segment_manager());
o1->some_map.emplace("entry_1", 1);
o1->some_map.emplace("entry_2", 2);
o1->some_map.emplace("entry_3", 3);
cout << "constructed MyStruct01: " << *o1 << endl;
}
shared::segment_type::shrink_to_fit(filename);
{
shared::segment_type segment(boost::interprocess::open_read_only, filename);
MyStruct *o1 = (segment.find<MyStruct>("MyStruct01")).first;
cout << " found MyStruct01: " << *o1 << endl;
}
cout << "completed" << endl;
return 0;
}
EDIT: I put some instrumentation in the code to do some sleuthing. I won't include it here, but I used std::system to run ls against the the mapped file, and to run pmap against the process at different points. In particular, I wanted to see where the mapped file is in the process address space. I also dumped out the addresses of the entries returned by the range-for loop in the operator<< method to see where they were in the address space.
When using std::unordered_map, I got the following information:
Before shrink_to_fit():
Mapped file size: 1048576
File mapping address range: 0x7f4f912d5000 - 0x7f4f913d5000
Addresses of map entries: 0x7f4f912d5238, 0x7f4f912d5188, 0x7f4f912d51f8
So, everything looks good there.
After shrink_to_fit():
Mapped file size: 688
File mapping address range: 0x7f4f919c6000 - 0x7f4f919c62b0
Addresses of map entry: 0x7f4f912d5238 (SEGV)
That map entry is completely outside the file mapping address range.
Compare to using boost::unordered_map:
Before shrink_to_fit():
Mapped file size: 1048576
File mapping address range: 0x7fe06957f000 - 0x7fe06967f000
Addresses of map entries: 0x7fe06957f2d0, 0x7fe06957f290, 0x7fe06957f180
All good.
After shrink_to_fit():
Mapped file size: 784
File mapping address range: 0x7fe069c70000 - 0x7fe069c70310
Addresses of map entries: 0x7fe069c702d0, 0x7fe069c70290, 0x7fe069c70180
All within the mapped address space.
Switching back to std::unordered_map, if I omit the call to shrink_to_fit(), then the file is mapped back into the same address space, and everything "works". If had been mapped into a different address space, then it probably would have crashed similarly.
If I had to guess, I would say that std::unordered_map is using regular pointers, rather than the boost interprocess offset_ptr that it should be using.
"Handling map of files in c++" says no, one shall use std::map<std::string, std::ofstream*>, but this leads to the new and delete actions, which is not so neat.
Since "Is std::ofstream movable? Yes!" and it's possible to "std::map<>::insert using non-copyable objects and uniform initialization", is it possible to handle a collection of ofstream using std::map? so that one won't worry about closing filestreams and delete to release memory.
I can compromise that during using std::map<std::string, std::ofstream>, only create, use (it to write) and close, not to copy it.
Yes it is possible. See sample code below.
I can compromise that during using std::map<std::string, std::ofstream>, only create, use (it to write) and close, not to copy it.
They are not copyable, so in your final comment, you are correct, you will be unable to copy it. You can move assign though, if that's what you want to do.
#include <iostream>
#include <fstream>
#include <map>
int main()
{
std::map<std::string, std::ofstream> map;
map.emplace("foo", std::ofstream("/tmp/foo"));
map.emplace("bar", std::ofstream("/tmp/bar"));
map["foo"] << "test";
map["foo"].flush();
std::ifstream ifs("/tmp/foo");
std::string data;
ifs >> data;
std::cout << data << '\n';
return 0;
}
Output:
test
I would like to use gzip from C++ (or C) to gzip a string. If possible, I would like to use zlib.
When I learned that I would have to use zlib to compress and uncompress, I Googled it for a few minutes and then quickly wrote a program to gzip a file and then ungzip it. However, I don't actually have any need to do that. I need to use gzip to compress and uncompress a string, not a file. I couldn't find much good documentation for using gzip on strings. Every example I find works with files.
Could someone show me a simple example?
Thanks in advance.
It's built into Poco (C++ library/framework, lots of utilities, networking, what you have). Here's a sample program:
#include <iostream>
#include <sstream>
#include <Poco/InflatingStream.h>
#include <Poco/DeflatingStream.h>
#include <Poco/StreamCopier.h>
int main() {
std::ostringstream stream1;
Poco::DeflatingOutputStream
gzipper(stream1, Poco::DeflatingStreamBuf::STREAM_GZIP);
gzipper << "Hello World!";
gzipper.close();
std::string zipped_string = stream1.str();
std::cout << "zipped_string: [" << zipped_string << "]\n";
std::ostringstream stream2;
Poco::InflatingOutputStream
gunzipper(stream2, Poco::InflatingStreamBuf::STREAM_GZIP);
gunzipper << zipped_string;
gunzipper.close();
std::string unzipped_string = stream2.str();
std::cout << "unzipped_string back: [" << unzipped_string << "]\n";
return 0;
}
The nice thing is that instead of the ostringstreams above, you could hook up the Poco gzipping streams to files etc..
How can one convert a shared_ptr that points to a const object to a shared_ptr that points to a non-const object.
I am trying to do the following :
boost::shared_ptr<const A> Ckk(new A(4));
boost::shared_ptr<A> kk=const_cast< boost::shared_ptr<A> > Ckk;
But it does not work.
'boost::const_pointer_cast' will do what you're asking for, but the obligatory second half of the answer is that you probably shouldn't use it. 99% of the time when it seems like you need to cast away the const property of a variable, it means that you have a design flaw. Const is sometimes more than just window dressing and casting it away may lead to unexpected bugs.
Without knowing more details of your situation one can't say for certain. But no discussion of const-cast is complete without mentioning this fact.
use boost::const_pointer_cast, documentation.
the proper way should be this
boost::shared_ptr<A> kk (boost::const_pointer_cast<A>(Ckk));
std::const_cast_pointer makes a second managed pointer. After the cast you have a writable pointer and the original const-pointer. The pointee remains the same. The reference count has been increased by 1.
Note that const_cast is a builtin keyword, but const_pointer_cast is a template function in namespace std.
The writable pointer can then be used to change the value from under the shared_ptr<const T>. IMHO the writable pointer should only persist temporarily on the stack; otherwise there must be a design flaw.
I once wrote a small test program to make this clear to myself which I adapted for this thread:
#include <memory>
#include <iostream>
#include <cassert>
using namespace std;
typedef shared_ptr<int> int_ptr;
typedef shared_ptr<const int> const_int_ptr;
int main(void)
{
const_int_ptr Ckk(new int(1));
assert(Ckk.use_count() == 1);
cout << "Ckk = " << *Ckk << endl;
int_ptr kk = const_pointer_cast<int>(Ckk); // obtain a 2nd reference
*kk = 2; // change value under the const pointer
assert(Ckk.use_count() == 2);
cout << "Ckk = " << *Ckk << endl; // prints 3
}
Under UNIX or Windows/Cygwin, compile with
g++ -std=c++0x -lm const_pointer_cast.cpp