HKEY hkey;
LPCSTR data = "HelloWrold\0";
if (ERROR_SUCCESS == RegOpenKeyExA(HKEY_LOCAL_MACHINE, "SOFTWARE\\Sample", 0,KEY_ALL_ACCESS, &hkey))
{
cout << "Registry Open SUccess" << endl;
if (ERROR_SUCCESS == RegSetValueEx(hkey, L"NAME", 0, REG_SZ, (LPBYTE)(data), strlen(data)+1))
cout << "Value Set Success" << endl;
else
cout << "Value Set Failed" << endl;
}
else
cout << "Registry Open Failed " << GetLastError() << endl;
when i use this code the code works fine, but in registry I'm not getting Helloworld but getting some chinese characters, any suggestion what to do?
Why does misdetected Unicode text tend to show up as Chinese characters?
If you take an ASCII string and cast it to Unicode, the results are usually nonsense Chinese.
Your project setting are probably defining UNICODE and your call to RegSetValueEx ends up calling RegSetValueExW but you are passing it a narrow C-style string.
Change it to RegSetValueExA(hkey, "NAME", 0, REG_SZ, (LPBYTE)(data), strlen(data)+1) or change data to be a Unicode string (WCHAR/LPWSTR).
You can catch this sort of error in C++ by changing (LPBYTE)(data) to (LPBYTE)const_cast<LPTSTR>(data).
Intro: I am trying to write a program which connects to a FLIR AX5(GigE Vision) camera and then save images after regular intervals to a pre-specified location on my PC. These images must be 14bit which contains the temperature information. Later I need to process these images using openCV to get some meaningful results from obtained temperature data.
Current Position: I can save image at regular interval but the image which I am getting doesn't contain 14 bit data but 8 bit data instead. This even after I change the PixelFormat to 14 bit, CMOS and LVDT bit depths to 14 bit. I checked the resulting .bin file in matlab and found that the max pixel value is 255 which means image is being stored in 8 bit format. I am using the sample code provided by eBus SDK to do this job. In this code I have made some changes as per my requirement.
Please help in saving the image in the raw format from which I can read the temperature data.
P.S. Relevant code is here.
// If the buffer contains an image, display width and height.
uint32_t lWidth = 0, lHeight = 0;
lType = lBuffer->GetPayloadType();
cout << fixed << setprecision( 1 );
cout << lDoodle[ lDoodleIndex ];
cout << " BlockID: " << uppercase << hex << setfill( '0' ) << setw( 16 ) << lBuffer->GetBlockID();
if (lType == PvPayloadTypeImage)
{
// Get image specific buffer interface.
PvImage *lImage = lBuffer->GetImage();
// Read width, height.
lWidth = lImage->GetWidth();
lHeight = lImage->GetHeight();
cout << " W: " << dec << lWidth << " H: " << lHeight;
lBuffer->GetImage()->Alloc(lWidth, lHeight, lBuffer->GetImage()->GetPixelType());
if (lBuffer->GetBlockID()%50==0) {
char filename[]= IMAGE_SAVE_LOC;
std::string s=std::to_string(lBuffer->GetBlockID());
char const *schar=s.c_str();
strcat(filename, schar);
strcat(filename,".bin");
lBufferWriter.Store(lBuffer,filename);
}
Be sure that the streaming is configured for 14 bits stream.
Before create PvStream you have to set PixelFormat to 14 bits. If you PvDevice object it's called _pvDevice:
_pvDevice->GetParameters()->SetEnumValue("PixelFormat", PvPixelMono14);
_pvDevice->GetParameters()->SetEnumValue("DigitalOutput", 3);
UPDATE 2 / TL;DR
Is there some way to prevent dirty pages of a windows FILE_FLAG_DELETE_ON_CLOSE temporary file from being flushed as a result of closing memory maps opened on these files?
Yes. If you do not need to do anything with the files themselves after their initial creation and you implement some naming conventions, this is possible through the strategy explained in this answer.
Note: I am still quite interested in finding out the reasons for why there is so much difference in the behavior depending on how maps are created and the order of disposal/unmapping.
I have been looking into some strategies for an inter-process shared memory data structure that allows growing and shrinking its committed capacity on windows by using a chain of "memory chunks."
One possible way is to use pagefile backed named memory maps as the chunk memory. An advantage of this strategy is the possibility to use SEC_RESERVE to reserve a big chunk of memory address space and incrementally allocate it using VirtualAlloc with MEM_COMMIT. Disadvantages appear to be (a) the requirement to have SeCreateGlobalPrivilege permissions to allow using a shareable name in the Global\ namespace and (b) the fact that all committed memory contributes to the system commit charge.
To circumvent these disadvantages, I started investigating the use of temporary file backed memory maps. I.e. memory maps over files created using the FILE_FLAG_DELETE_ON_CLOSE | FILE_ATTRIBUTE_TEMPORARY flags combination. This appears to be a recommended strategy that according to e.g. this blog post should prevent flushing the mapped memory to disk (unless memory pressure causes dirty mapped pages to be paged out).
I am however observing that closing the map/file handle before the owning process exits, causes dirty pages to be flushed to disk. This occurs even if the view/file handle is not the one through which the dirty pages were created and when these views/file handles were opened after the pages were 'dirtied' in a different view.
It appears that changing the order of disposal (i.e. unmapping the view first or closing the file handle first) has some impact on when the disk flush is initiated, but not on the fact that flushing takes place.
So my questions are:
Is there some way to use temporary file backed memory maps and prevent them from flushing dirty pages when the map/file is closed, taking into account that multiple threads within a process/multiple processes may have open handles/views to such a file?
If not, what is/could be the reason for the observed behavior?
Is there an alternative strategy that I may have overlooked?
UPDATE
Some additional info: When running the "arena1" and "arena2" parts of the sample code below in two separate (independent) processes, with "arena1" being the process that creates the shared memory regions and "arena2" the one that opens them, the following behavior is observed for maps/chunks that have dirty pages:
If closing the view before the file handle in the "arena1" process, it flushes each of these chunks to disk in what seems a (partially) synchronous process (i.e. it blocks the disposing thread for several seconds), independent of whether or not the "arena2" process was started.
If closing the file handle before the view, disk flushes only occur for those maps/chunks that are closed in the "arena1" process while the "arena2" process still has an open handle to those chunks, and they appear to be 'asynchronous', i.e. not blocking the application thread.
Refer to the (c++) sample code below that allows reproducing the problem on my system (x64, Win7):
static uint64_t start_ts;
static uint64_t elapsed() {
return ::GetTickCount64() - start_ts;
}
class PageArena {
public:
typedef uint8_t* pointer;
PageArena(int id, const char* base_name, size_t page_sz, size_t chunk_sz, size_t n_chunks, bool dispose_handle_first) :
id_(id), base_name_(base_name), pg_sz_(page_sz), dispose_handle_first_(dispose_handle_first) {
for (size_t i = 0; i < n_chunks; i++)
chunks_.push_back(new Chunk(i, base_name_, chunk_sz, dispose_handle_first_));
}
~PageArena() {
for (auto i = 0; i < chunks_.size(); ++i) {
if (chunks_[i])
release_chunk(i);
}
std::cout << "[" << ::elapsed() << "] arena " << id_ << " destructed" << std::endl;
}
pointer alloc() {
auto ptr = chunks_.back()->alloc(pg_sz_);
if (!ptr) {
chunks_.push_back(new Chunk(chunks_.size(), base_name_, chunks_.back()->capacity(), dispose_handle_first_));
ptr = chunks_.back()->alloc(pg_sz_);
}
return ptr;
}
size_t num_chunks() {
return chunks_.size();
}
void release_chunk(size_t ndx) {
delete chunks_[ndx];
chunks_[ndx] = nullptr;
std::cout << "[" << ::elapsed() << "] chunk " << ndx << " released from arena " << id_ << std::endl;
}
private:
struct Chunk {
public:
Chunk(size_t ndx, const std::string& base_name, size_t size, bool dispose_handle_first) :
map_ptr_(nullptr), tail_(nullptr),
handle_(INVALID_HANDLE_VALUE), size_(0),
dispose_handle_first_(dispose_handle_first) {
name_ = name_for(base_name, ndx);
if ((handle_ = create_temp_file(name_, size)) == INVALID_HANDLE_VALUE)
handle_ = open_temp_file(name_, size);
if (handle_ != INVALID_HANDLE_VALUE) {
size_ = size;
auto map_handle = ::CreateFileMappingA(handle_, nullptr, PAGE_READWRITE, 0, 0, nullptr);
tail_ = map_ptr_ = (pointer)::MapViewOfFile(map_handle, FILE_MAP_ALL_ACCESS, 0, 0, size);
::CloseHandle(map_handle); // no longer needed.
}
}
~Chunk() {
if (dispose_handle_first_) {
close_file();
unmap_view();
} else {
unmap_view();
close_file();
}
}
size_t capacity() const {
return size_;
}
pointer alloc(size_t sz) {
pointer result = nullptr;
if (tail_ + sz <= map_ptr_ + size_) {
result = tail_;
tail_ += sz;
}
return result;
}
private:
static const DWORD kReadWrite = GENERIC_READ | GENERIC_WRITE;
static const DWORD kFileSharing = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE;
static const DWORD kTempFlags = FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_FLAG_DELETE_ON_CLOSE | FILE_ATTRIBUTE_TEMPORARY;
static std::string name_for(const std::string& base_file_path, size_t ndx) {
std::stringstream ss;
ss << base_file_path << "." << ndx << ".chunk";
return ss.str();
}
static HANDLE create_temp_file(const std::string& name, size_t& size) {
auto h = CreateFileA(name.c_str(), kReadWrite, kFileSharing, nullptr, CREATE_NEW, kTempFlags, 0);
if (h != INVALID_HANDLE_VALUE) {
LARGE_INTEGER newpos;
newpos.QuadPart = size;
::SetFilePointerEx(h, newpos, 0, FILE_BEGIN);
::SetEndOfFile(h);
}
return h;
}
static HANDLE open_temp_file(const std::string& name, size_t& size) {
auto h = CreateFileA(name.c_str(), kReadWrite, kFileSharing, nullptr, OPEN_EXISTING, kTempFlags, 0);
if (h != INVALID_HANDLE_VALUE) {
LARGE_INTEGER sz;
::GetFileSizeEx(h, &sz);
size = sz.QuadPart;
}
return h;
}
void close_file() {
if (handle_ != INVALID_HANDLE_VALUE) {
std::cout << "[" << ::elapsed() << "] " << name_ << " file handle closing" << std::endl;
::CloseHandle(handle_);
std::cout << "[" << ::elapsed() << "] " << name_ << " file handle closed" << std::endl;
}
}
void unmap_view() {
if (map_ptr_) {
std::cout << "[" << ::elapsed() << "] " << name_ << " view closing" << std::endl;
::UnmapViewOfFile(map_ptr_);
std::cout << "[" << ::elapsed() << "] " << name_ << " view closed" << std::endl;
}
}
HANDLE handle_;
std::string name_;
pointer map_ptr_;
size_t size_;
pointer tail_;
bool dispose_handle_first_;
};
int id_;
size_t pg_sz_;
std::string base_name_;
std::vector<Chunk*> chunks_;
bool dispose_handle_first_;
};
static void TempFileMapping(bool dispose_handle_first) {
const size_t chunk_size = 256 * 1024 * 1024;
const size_t pg_size = 8192;
const size_t n_pages = 100 * 1000;
const char* base_path = "data/page_pool";
start_ts = ::GetTickCount64();
if (dispose_handle_first)
std::cout << "Mapping with 2 arenas and closing file handles before unmapping views." << std::endl;
else
std::cout << "Mapping with 2 arenas and unmapping views before closing file handles." << std::endl;
{
std::cout << "[" << ::elapsed() << "] " << "allocating " << n_pages << " pages through arena 1." << std::endl;
PageArena arena1(1, base_path, pg_size, chunk_size, 1, dispose_handle_first);
for (size_t i = 0; i < n_pages; i++) {
auto ptr = arena1.alloc();
memset(ptr, (i + 1) % 256, pg_size); // ensure pages are dirty.
}
std::cout << "[" << elapsed() << "] " << arena1.num_chunks() << " chunks created." << std::endl;
{
PageArena arena2(2, base_path, pg_size, chunk_size, arena1.num_chunks(), dispose_handle_first);
std::cout << "[" << ::elapsed() << "] arena 2 loaded, going to release chunks 1 and 2 from arena 1" << std::endl;
arena1.release_chunk(1);
arena1.release_chunk(2);
}
}
}
Please refer to this gist that contains the output of running the above code and links to screen captures of system free memory and disk activity when running TempFileMapping(false) and TempFileMapping(true) respectively.
After the bounty period expired without any answers that provided more insight or solved the mentioned problem, I decided to dig a little deeper and experiment some more with several combinations and sequences of operations.
As a result, I believe I have found a way to achieve memory maps shared between processes over temporary, delete-on-close files, that are not being flushed to disk when they are closed.
The basic idea involves creating the memory map when a temp file is newly created with a map name that can be used in a call to OpenFileMapping:
// build a unique map name from the file name.
auto map_name = make_map_name(file_name);
// Open or create the mapped file.
auto mh = ::OpenFileMappingA(FILE_MAP_ALL_ACCESS, false, map_name.c_str());
if (mh == 0 || mh == INVALID_HANDLE_VALUE) {
// existing map could not be opened, create the file.
auto fh = ::CreateFileA(name.c_str(), kReadWrite, kFileSharing, nullptr, CREATE_NEW, kTempFlags, 0);
if (fh != INVALID_HANDLE_VALUE) {
// set its size.
LARGE_INTEGER newpos;
newpos.QuadPart = desired_size;
::SetFilePointerEx(fh, newpos, 0, FILE_BEGIN);
::SetEndOfFile(fh);
// create the map
mh = ::CreateFileMappingA(mh, nullptr, PAGE_READWRITE, 0, 0, map_name.c_str());
// close the file handle
// from now on there will be no accesses using file handles.
::CloseHandle(fh);
}
}
Thus, the file handle is only used when the file is newly created, and closed immediately after the map is created, while the map handle itself remains open, to allow opening the mapping without requiring access to a file handle. Note that a race condition exists here, that we would need to deal with in any "real code" (as well as adding decent error checking and handling).
So if we got a valid map handle, we can create the view:
auto map_ptr = MapViewOfFile(mh, FILE_MAP_ALL_ACCESS, 0, 0, 0);
if (map_ptr) {
// determine its size.
MEMORY_BASIC_INFORMATION mbi;
if (::VirtualQuery(map_ptr, &mbi, sizeof(MEMORY_BASIC_INFORMATION)) > 0)
map_size = mbi.RegionSize;
}
When, some time later closing a mapped file: close the map handle before unmapping the view:
if (mh == 0 || mh == INVALID_HANDLE_VALUE) {
::CloseHandle(mh);
mh = INVALID_HANDLE_VALUE;
}
if (map_ptr) {
::UnmapViewOfFile(map_ptr);
map_ptr = 0;
map_size = 0;
}
And, according to the test I have performed so far, this does not cause flushing dirty pages to disk on close, problem solved. Well partially anyway, there may still be a cross-session map name sharing issue.
If I take it correctly, commenting out Arena2 part of code shall reproduce the issue without the need for second process. I have tried this:
I edited base_path as follows for convenience:
char base_path[MAX_PATH];
GetTempPathA(MAX_PATH, base_path);
strcat_s(base_path, MAX_PATH, "page_pool");
I edited n_pages = 1536 * 128 to bring the used memory to 1.5GB, compared to your ~800mb.
I have tested TempFileMapping(false) and TempFileMapping(true), one at a time, for the same results.
I have tested with Arena2 commented out and intact, for the same results.
I have tested on Win8.1 x64 and Win7 x64, for ±10% same results.
In my tests, code runs in 2400ms ±10%, only 500ms ±10% spent on deallocating. That's clearly not enough for a flush of 1.5GB on a low-spinning silent HDDs I have there.
So, the question is, what are you observing? I'd suggest that you:
Provide your times for comparison
Use a different computer for tests, paying attention to excluding software issues such as "same antivirus"
Verify that you're not experiencing a RAM shortage.
Use xperf to see what's happening during the freeze.
Update
I have tested on yet another Win7 x64, and times are 890ms full, 430ms spent on dealloc. I have looked into your results, and what is VERY suspicious is that almost exactly 4000ms is spent in freeze each time on your machine. That can't be a mere coincidence, I believe. Also, it's rather obvious now the the problem is somehow bound to a specific machine you're using. So my suggestions are:
As stated above, test on another computer yourself
As stated above, Use XPerf, it will allow you to see what exactly happens in user mode and kernel mode during the freeze (I really suspect some non-standard driver in the middle)
Play with number of pages and see how it affects the freeze length.
Try to store files on a different disk drive on the same computer where you have tested initially.
I have the following function
std::tuple<int,val*>Socket::recv(val* values ) // const
{
char buf [ MAXRECV + 1 ];
memset ( buf, 0, MAXRECV + 1 );
int status = ::recv ( m_sock, buf, MAXRECV, 0 );
if ( status == -1 )
{
std::cout << "status == -1 errno == " << errno << " in Socket::recv\n";
// return std::make_tuple(0,NULL);//this is not working
}
else if ( status == 0 )
{
//return std::make_tuple(0,NULL); //this is not working
}
else
{
struct val* values=(struct val*) buf;
if(!std::isnan(values->val1) &&
!std::isnan(values->val2) &&
!std::isnan(values->val3) &&
!std::isnan(values->val4),
!std::isnan(values->val5),
!std::isnan(values->val6))
printf("received:%f %f %f %f %f %f\n", values->val1, values->val2,
values->val3, values->val4, values->val5, values->val6);
return std::make_tuple(status,values);
}
}
The received values are printed out in to standard output correctly within the function.
But when I try to access these received values out of the function by calling as follows what I get is all 0's.[after creating Socket rcvd object]
Would you tell me how to access these values outside the function?
1.
std::cout << std::get<1>(rcvd.recv(&values)->val1)
<< std::get<1>(rcvd.recv(&values)->val2)
<< std::get<1>(rcvd.recv(&values)->val3)
<< std::get<1>(rcvd.recv(&values)->val4)
<< std::get<1>(rcvd.recv(&values)->val5)
<< std::get<1>(rcvd.recv(&values)->val6)
<< std::endl;
2.
std::cout << std::get<1>(rcvd.recv(&values).val1)
<< std::get<1>(rcvd.recv(&values).val2)
<< std::get<1>(rcvd.recv(&values).val3)
<< std::get<1>(rcvd.recv(&values).val4)
<< std::get<1>(rcvd.recv(&values).val5)
<< std::get<1>(rcvd.recv(&values).val6)
<< std::endl;
3.
std::cout << std::get<1>(rcvd.recv(&values)[0])
<< std::get<1>(rcvd.recv(&values)[1])
<< std::get<1>(rcvd.recv(&values)[2])
<< std::get<1>(rcvd.recv(&values)[3])
<< std::get<1>(rcvd.recv(&values)[4])
<< std::get<1>(rcvd.recv(&values)[5])
<< std::endl;
where "values" comes from
struct val {
val1;
val2;
val3;
val4;
val5;
val6;} values;
All the three options of calling the function or access the struct val could not work for me.
Would you tell me
how to access these received values externally from any function?
how to return zero to struct pointer [NULL is not working ] when status is 0 or -1
Try
return std::make_tuple<int, val*>(0, nullptr);
The type of tuple is deduced from arguments, so by using 0,NULL you are actually using the null constant wich is evaluted to 0 and hence deduced type is <int,int>.
By the way, I see no reason for using NULL in C++11, if you need that really for some reason then cast NULL to val*
static_cast<val*>(NULL);
EDIT:
Other viable alternatives are
val* nullval = nullptr;
return std::make_tuple(0, nullval);
Or
return std::make_tuple(0, static_cast<val*>(nullptr));
Or (as comment suggest)
return {0, nullptr};
Choose the one that seems more clear to you.
You are lucky that the outside function is printing zeroes. It might have as well just dumped the core on you :)
What you are doing is accessing a buffer, that was created on a stack, after that stack was released (once the function's execution finished). That is HIGHLY UNSAFE and, pretty much, illegal.
Instead what you should do is allocate your data buffer in a 'free memory", using functions like malloc (in C) or operator new/new[] (in C++).
The quick fix is to replace the line
char buf [ MAXRECV + 1 ];
with
char * buf = new char [ MAXRECV + 1 ];
And when you do a type casting on line
struct val* values=(struct val*) buf;
you really ought to be sure that what you do is correct. If the sizeof() of you struct val is more than the sizeof(char[MAXRECV + 1]) you'll get in memory access troubles.
After you are done using the returned data buffer don't forget to release it with a call to free (in C) or delete/delete[] (in C++). Otherwise you'd have what is called a memory leak.
If I have multiple files in a large project, all of which share a large number of included header files, is there any way to share the work of parsing the header files? I had hoped that creating one Index and then adding multiple translationUnits to it could cause some work to be shared - however even code along the lines of (pseudocode)
index = clang_createIndex();
clang_parseTranslationUnit(index, "myfile");
clang_parseTranslationUnit(index, "myfile");
seems to take the full amount of time for each call to parseTranslationUnit, performing no better than
index1 = clang_createIndex();
clang_parseTranslationUnit(index1, "myfile");
index2 = clang_createIndex();
clang_parseTranslationUnit(index2, "myfile");
I am aware that there are specialized functions for reparsing the exact same file; however what I really want is that parsing "myfile1" and "myfile2" can share the work of parsing "myheader.h", and reparsing-specific functions won't help there.
As a sub-question, is there any meaningful difference between reusing an index and creating a new index for each translation unit?
One way of doing this consists in creating Precompiled Headers (PCH file) from the shared header in your project.
Something along these lines seems to work (you can see the whole example here):
auto Idx = clang_createIndex (0, 0);
CXTranslationUnit TU;
Timer t;
{
char const *args[] = { "-xc++", "foo.hxx" };
int nargs = 2;
t.reset();
TU = clang_parseTranslationUnit(Idx, 0, args, nargs, 0, 0, CXTranslationUnit_ForSerialization);
std::cerr << "PCH parse time: " << t.get() << std::endl;
displayDiagnostics (TU);
clang_saveTranslationUnit (TU, "foo.pch", clang_defaultSaveOptions(TU));
clang_disposeTranslationUnit (TU);
}
{
char const *args[] = { "-include-pch", "foo.pch", "foo.cxx" };
int nargs = 3;
t.reset();
TU = clang_createTranslationUnitFromSourceFile(Idx, 0, nargs, args, 0, 0);
std::cerr << "foo.cxx parse time: " << t.get() << std::endl;
displayDiagnostics (TU);
clang_disposeTranslationUnit (TU);
}
{
char const *args[] = { "-include-pch", "foo.pch", "foo2.cxx" };
int nargs = 3;
t.reset();
TU = clang_createTranslationUnitFromSourceFile(Idx, 0, nargs, args, 0, 0);
std::cerr << "foo2.cxx parse time: " << t.get() << std::endl;
displayDiagnostics (TU);
clang_disposeTranslationUnit (TU);
}
yielding the following output:
PCH parse time: 5.35074
0 diagnostics
foo1.cxx parse time: 0.158232
0 diagnostics
foo2.cxx parse time: 0.143654
0 diagnostics
I did not find much information about libclang and precompiled headers in the API documentation, but here are a few pages where the keyword appears: CINDEX and TRANSLATION_UNIT
Please note that this solution is not optimal by any ways. I'm looking forward to seeing better answers. In particular:
each source file can have at most one precompiled header
nothing here is libclang-specific ; this is the exact same strategy that is used for build time optimization using the standard clang command lines.
it is not really automated, in that you have to explicitly create the precompiled header (and must thus know the name of the shared header file)
I don't think using different CXIndex objects would have made any difference here