node.js c++ addon - afraid of memory leak - v8

first of all I admit I'm a newbie in C++ addons for node.js.
I'm writing my first addon and I reached a good result: the addon does what I want. I copied from various examples I found in internet to exchange complex data between the two languages, but I understood almost nothing of what I wrote.
The first thing scaring me is that I wrote nothing that seems to free some memory; another thing which is seriously worrying me is that I don't know if what I wrote may helps or creating confusion for the V8 garbage collector; by the way I don't know if there are better ways to do what I did (iterating over js Object keys in C++, creating js Objects in C++, creating Strings in C++ to be used as properties of js Objects and what else wrong you can find in my code).
So, before going on with my job writing the real math of my addon, I would like to share with the community the nan and V8 part of it to ask if you see something wrong or that can be done in a better way.
Thank you everybody for your help,
iCC
#include <map>
#include <nan.h>
using v8::Array;
using v8::Function;
using v8::FunctionTemplate;
using v8::Local;
using v8::Number;
using v8::Object;
using v8::Value;
using v8::String;
using Nan::AsyncQueueWorker;
using Nan::AsyncWorker;
using Nan::Callback;
using Nan::GetFunction;
using Nan::HandleScope;
using Nan::New;
using Nan::Null;
using Nan::Set;
using Nan::To;
using namespace std;
class Data {
public:
int dt1;
int dt2;
int dt3;
int dt4;
};
class Result {
public:
int x1;
int x2;
};
class Stats {
public:
int stat1;
int stat2;
};
typedef map<int, Data> DataSet;
typedef map<int, DataSet> DataMap;
typedef map<float, Result> ResultSet;
typedef map<int, ResultSet> ResultMap;
class MyAddOn: public AsyncWorker {
private:
DataMap *datas;
ResultMap results;
Stats stats;
public:
MyAddOn(Callback *callback, DataMap *set): AsyncWorker(callback), datas(set) {}
~MyAddOn() { delete datas; }
void Execute () {
for(DataMap::iterator i = datas->begin(); i != datas->end(); ++i) {
int res = i->first;
DataSet *datas = &i->second;
for(DataSet::iterator l = datas->begin(); l != datas->end(); ++l) {
int dt4 = l->first;
Data *data = &l->second;
// TODO: real population of stats and result
}
// test result population
results[res][res].x1 = res;
results[res][res].x2 = res;
}
// test stats population
stats.stat1 = 23;
stats.stat2 = 42;
}
void HandleOKCallback () {
Local<Object> obj;
Local<Object> res = New<Object>();
Local<Array> rslt = New<Array>();
Local<Object> sts = New<Object>();
Local<String> x1K = New<String>("x1").ToLocalChecked();
Local<String> x2K = New<String>("x2").ToLocalChecked();
uint32_t idx = 0;
for(ResultMap::iterator i = results.begin(); i != results.end(); ++i) {
ResultSet *set = &i->second;
for(ResultSet::iterator l = set->begin(); l != set->end(); ++l) {
Result *result = &l->second;
// is it ok to declare obj just once outside the cycles?
obj = New<Object>();
// is it ok to use same x1K and x2K instances for all objects?
Set(obj, x1K, New<Number>(result->x1));
Set(obj, x2K, New<Number>(result->x2));
Set(rslt, idx++, obj);
}
}
Set(sts, New<String>("stat1").ToLocalChecked(), New<Number>(stats.stat1));
Set(sts, New<String>("stat2").ToLocalChecked(), New<Number>(stats.stat2));
Set(res, New<String>("result").ToLocalChecked(), rslt);
Set(res, New<String>("stats" ).ToLocalChecked(), sts);
Local<Value> argv[] = { Null(), res };
callback->Call(2, argv);
}
};
NAN_METHOD(AddOn) {
Local<Object> datas = info[0].As<Object>();
Callback *callback = new Callback(info[1].As<Function>());
Local<Array> props = datas->GetOwnPropertyNames();
Local<String> dt1K = Nan::New("dt1").ToLocalChecked();
Local<String> dt2K = Nan::New("dt2").ToLocalChecked();
Local<String> dt3K = Nan::New("dt3").ToLocalChecked();
Local<Array> props2;
Local<Value> key;
Local<Object> value;
Local<Object> data;
DataMap *set = new DataMap();
int res;
int dt4;
DataSet *dts;
Data *dt;
for(uint32_t i = 0; i < props->Length(); i++) {
// is it ok to declare key, value, props2 and res just once outside the cycle?
key = props->Get(i);
value = datas->Get(key)->ToObject();
props2 = value->GetOwnPropertyNames();
res = To<int>(key).FromJust();
dts = &((*set)[res]);
for(uint32_t l = 0; l < props2->Length(); l++) {
// is it ok to declare key, data and dt4 just once outside the cycles?
key = props2->Get(l);
data = value->Get(key)->ToObject();
dt4 = To<int>(key).FromJust();
dt = &((*dts)[dt4]);
int dt1 = To<int>(data->Get(dt1K)).FromJust();
int dt2 = To<int>(data->Get(dt2K)).FromJust();
int dt3 = To<int>(data->Get(dt3K)).FromJust();
dt->dt1 = dt1;
dt->dt2 = dt2;
dt->dt3 = dt3;
dt->dt4 = dt4;
}
}
AsyncQueueWorker(new MyAddOn(callback, set));
}
NAN_MODULE_INIT(Init) {
Set(target, New<String>("myaddon").ToLocalChecked(), GetFunction(New<FunctionTemplate>(AddOn)).ToLocalChecked());
}
NODE_MODULE(myaddon, Init)
One year and half later...
If somebody is interested, my server is up and running since my question and the amount of memory it requires is stable.
I can't say if the code I wrote really does not has some memory leak or if lost memory is freed at each thread execution end, but if you are afraid as I was, I can say that using same structure and calls does not cause any real problem.

You do actually free up some of the memory you use, with the line of code:
~MyAddOn() { delete datas; }
In essence, C++ memory management boils down to always calling delete for every object created by new. There are also many additional architecture-specific and legacy 'C' memory management functions, but it is not strictly necessary to use these when you do not require the performance benefits.
As an example of what could potentially be a memory leak: You're passing the object held in the *callback pointer to the function AsyncQueueWorker. Yet nowhere in your code is this pointer freed, so unless the Queue worker frees it for you, there is a memory leak here.
You can use a memory tool such as valgrind to test your program further. It will spot most memory problems for you and comes highly recommended.
One thing I've observed is that you often ask (paraphrased):
Is it okay to declare X outside my loop?
To which the answer actually is that declaring variables inside of your loops is better, whenever you can do it. Declare variables as deep inside as you can, unless you have to re-use them. Variables are restricted in scope to the outermost set of {} brackets. You can read more about this in this question.
is it ok to use same x1K and x2K instances for all objects?
In essence, when you do this, if one of these objects modifies its 'x1K' string, then it will change for all of them. The advantage is that you free up memory. If the string is the same for all these objects anyway, instead of having to store say 1,000,000 copies of it, your computer will only keep a single one in memory and have 1,000,000 pointers to it instead. If the string is 9 ASCII characters long or longer under amd64, then that amounts to significant memory savings.
By the way, if you don't intend to modify a variable after its declaration, you can declare it as const, a keyword short for constant which forces the compiler to check that your variable is not modified after declaration. You may have to deal with quite a few compiler errors about functions accepting only non-const versions of things they don't modify, some of which may not be your own code, in which case you're out of luck. Being as conservative as possible with non-const variables can help spot problems.

Related

ESP32 heap corruption error when releasing allocated memory

I am currently programming the ESP32 board in C++ and I am having trouble with my dataContainer class and releasing/allocating memory.
I do use the following DataContainer class (simplyfied):
template <typename Elementtype>
class DataContainer
{
private:
Elementtype **datalist;
int maxsize;
std::size_t currentsize; // How much data is saved in datalist
public:
DataContainer(int maxcapacity);
~DataContainer();
...some methods...
void reset_all_data();
};
And here is the reset_all_data() definition:
/* Deletes all Data of Datacontainer and allocates new memory*/
template <typename Elementtype>
void DataContainer<Elementtype>::reset_all_data()
{
for (int i = 0; i < currentsize; i++)
{
if (datalist[i])
Serial.println(heap_caps_check_integrity_all(true));
delete datalist[i]; <-- Error is triggered here!!!
Serial.println(heap_caps_check_integrity_all(true));
}
delete datalist;
datalist = new Elementtype *[maxsize];
for (int i = 0; i < maxsize; i++) // Declare a memory block of size maxsize (maxsize = 50)
{
datalist[i] = new Elementtype[5];
}
currentsize = 0;
}
As you can see, I have added some integrity checks, but the one before delete datalist (this seems to trigger the error). When I call reset_all_data() from my main.cpp at a certain point in my program the following error is triggered:
CORRUPT HEAP: Bad head at 0x3ffbb0f0. Expected 0xabba1234 got 0x3ffb9a34
assert failed: multi_heap_free multi_heap_poisoning.c:253 (head != NULL)
Backtrace:0x40083881:0x3ffb25400x4008e7e5:0x3ffb2560 0x40093d55:0x3ffb2580 0x4009399b:0x3ffb26b0 0x40083d41:0x3ffb26d0
0x40093d85:0x3ffb26f0 0x4014e3f5:0x3ffb2710 0x400d2dc6:0x3ffb2730 0x400d31e3:0x3ffb2750 0x400d9b02:0x3ffb2820
One more thing, the error is only triggered when a certain function is called right before it, even when the whole code inside this function is commented. This is the function's head: void write_data_container_to_file(fs::FS &fs, const char *path, DataContainer<uint16_t> data, const char *RTC_timestamp)thus, the mere call of the function plays an import role here.
Right now I am completely lost - any suggestion/idea is welcome on how to proceed.
EDIT: The dataContainer holds a 2D array of uint16_t.
I finally tracked down the, rather obvious, reason for the HEAP CORRUPTION. In the end I only called delete datalist but it would have been correct to call delete[] datalist after the for loop. The reason is, that within the for loop I delete the pointers pointing to arrays, which represent the "rows" of my allcoated 2D memory. In the end, I also have to delete the pointer, which points to the array holding the pointers I deleted within the for loop.
So I was not paying attention and one should watch out that when it comes to releasing the previously allocated memory, care should be taken if delete or delete[]should be called.

Is there an easier way to set/get values in a gSOAP request/response?

I am using gSOAP to configure an ONVIF compatible camera.
Currently, I am manually setting all the parameters in the request by doing something like this. This is for the SetVideEncoderConfiguration
MediaBindingProxy mediaDevice (uri);
AUTHENTICATE (mediaDevice);
_trt__SetVideoEncoderConfiguration req;
_trt__SetVideoEncoderConfigurationResponse resp;
struct tt__VideoEncoderConfiguration encoderConfig;
struct tt__VideoResolution resolutionConfig;
encoderConfig.Name = strdup (name);
encoderConfig.UseCount = 1;
encoderConfig.Quality = 50;
if (strcmp (encoding, "H264") == 0)
encoderConfig.Encoding = tt__VideoEncoding__H264;
else if (strcmp (encoding, "JPEG") == 0)
encoderConfig.Encoding = tt__VideoEncoding__JPEG;
encoderConfig.token = strdup (profileToken);
encoderConfig.SessionTimeout = (LONG64)"PT0S";
resolutionConfig.Width=1280;
resolutionConfig.Height=720;
encoderConfig.Resolution = &resolutionConfig;
tt__VideoRateControl rateControl;
rateControl.FrameRateLimit = 15;
rateControl.EncodingInterval = 1;
rateControl.BitrateLimit = 4500;
encoderConfig.RateControl = &rateControl;
struct tt__H264Configuration h264;
h264.GovLength = 30;
h264.H264Profile = tt__H264Profile__Baseline;
encoderConfig.H264 = &h264;
struct tt__MulticastConfiguration multicast;
struct tt__IPAddress address;
address.IPv4Address = strdup ("0.0.0.0");
multicast.Address = &address;
encoderConfig.Multicast = &multicast;
req.Configuration = &encoderConfig;
req.ForcePersistence = true;
int ret = mediaDevice.SetVideoEncoderConfiguration (&req, resp);
qDebug () << "Set Encoder: " << ret;
Is there an easier way to do this? May be some function calls that set the request parameters? Another way I found with GetMediaUri was to use something like
soap_new_req__trt__GetStreamUri (mediaDevice.soap,soap_new_req_tt__StreamSetup (mediaDevice.soap, (enum tt__StreamType)0, soap_new_tt__Transport(mediaDevice.soap), 1, NULL), "profile1");
Are these the only two ways for client side code with gSOAP?
-Mandar Joshi
There are four variations of soap_new_T() to allocate data of type T in C++ with gSOAP:
T * soap_new_T(struct soap*) returns a new instance of T that is default
initialized and allocated on the heap managed by the soap context.
T * soap_new_T(struct soap*, int n) returns an array of n new instances of
T on the managed heap. The instances in the array are default initialized as described above.
T * soap_new_req_T(struct soap*, ...) (structs and classes only) returns a
new instance of T allocated on the managed heap and sets the required data members to the values specified in the other arguments ....
T * soap_new_set_T(struct soap*, ...) (structs and classes only) returns a
new instance of T on the managed heap and sets the public/serializable data members to the values specified in the other arguments ....
Use soap_strdup(struct soap*, const char*) instead of strdup to dup strings onto the managed heap.
All data on the managed heap is mass-deleted with soap_destroy(soap) and
soap_end(soap) (call these in that order) which must be called before soap_done(soap) or soap_free(soap).
To allocate pointers to data, use templates:
template<class T>
T * soap_make(struct soap *soap, T val)
{
T *p = (T*)soap_malloc(soap, sizeof(T));
if (p)
*p = val;
return p;
}
template<class T>
T **soap_make_array(struct soap *soap, T* array, int n)
{
T **p = (T**)soap_malloc(soap, n * sizeof(T*));
for (int i = 0; i < n; ++i)
p[i] = &array[i];
return p;
}
Then use soap_make<int>(soap, 123) to create a pointer to the value 123 on the managed heap and soap_make_array(soap, soap_new_CLASSNAME(soap, 100), 100) to create 100 pointers to 100 instances of CLASSNAME.
The gSOAP tools also generate deep copy operations for you: CLASSNAME::soap_dup(struct soap*) creates a deep copy of the object and allocates it in a another soap context that you provide as argument. Use NULL as this argument to allocate unmanaged deep copies (but these cannot have pointer cycles!). Then delete unmanaged copies with CLASSNAME::soap_del() for deep deletion of all members and then delete the object itself.
See Memory management in C++ for more details. Use gSOAP 2.8.39 and greater.

how do I allocate memory for some of the structure elements

I want to allocate memory for some elements of a structure, which are pointers to other small structs.How do I allocate and de-allocate memory in best way?
Ex:
typedef struct _SOME_STRUCT {
PDATATYPE1 PDatatype1;
PDATATYPE2 PDatatype2;
PDATATYPE3 PDatatype3;
.......
PDATATYPE12 PDatatype12;
} SOME_STRUCT, *PSOME_STRUCT;
I want to allocate memory for PDatatype1,3,4,6,7,9,11.Can I allocate memory with single malloc? or what is the best way to allocate memory for only these elements and how to free the whole memory allocated?
There is a trick that allows a single malloc, but that also has to weighed against doing a more standard multiple malloc approach.
If [and only if], once the DatatypeN elements of SOME_STRUCT are allocated, they do not need to be reallocated in any way, nor does any other code do a free on any of them, you can do the following [the assumption that PDATATYPEn points to DATATYPEn]:
PSOME_STRUCT
alloc_some_struct(void)
{
size_t siz;
void *vptr;
PSOME_STRUCT sptr;
// NOTE: this optimizes down to a single assignment
siz = 0;
siz += sizeof(DATATYPE1);
siz += sizeof(DATATYPE2);
siz += sizeof(DATATYPE3);
...
siz += sizeof(DATATYPE12);
sptr = malloc(sizeof(SOME_STRUCT) + siz);
vptr = sptr;
vptr += sizeof(SOME_STRUCT);
sptr->Pdatatype1 = vptr;
// either initialize the struct pointed to by sptr->Pdatatype1 here or
// caller should do it -- likewise for the others ...
vptr += sizeof(DATATYPE1);
sptr->Pdatatype2 = vptr;
vptr += sizeof(DATATYPE2);
sptr->Pdatatype3 = vptr;
vptr += sizeof(DATATYPE3);
...
sptr->Pdatatype12 = vptr;
vptr += sizeof(DATATYPE12);
return sptr;
}
Then, the when you're done, just do free(sptr).
The sizeof above should be sufficient to provide proper alignment for the sub-structs. If not, you'll have to replace them with a macro (e.g. SIZEOF) that provides the necessary alignment. (e.g.) for 8 byte alignment, something like:
#define SIZEOF(_siz) (((_siz) + 7) & ~0x07)
Note: While it is possible to do all this, and it is more common for things like variable length string structs like:
struct mystring {
int my_strlen;
char my_strbuf[0];
};
struct mystring {
int my_strlen;
char *my_strbuf;
};
It is debatable whether it's worth the [potential] fragility (i.e. somebody forgets and does the realloc/free on the individual elements). The cleaner way would be to embed the actual structs rather than the pointers to them if the single malloc is a high priority for you.
Otherwise, just do the the [more] standard way and do the 12 individual malloc calls and, later, the 12 free calls.
Still, it is a viable technique, particularly on small memory constrained systems.
Here is the [more] usual way involving per-element allocations:
PSOME_STRUCT
alloc_some_struct(void)
{
void *vptr;
PSOME_STRUCT sptr;
sptr = malloc(sizeof(SOME_STRUCT));
// either initialize the struct pointed to by sptr->Pdatatype1 here or
// caller should do it -- likewise for the others ...
sptr->Pdatatype1 = malloc(sizeof(DATATYPE1));
sptr->Pdatatype2 = malloc(sizeof(DATATYPE2));
sptr->Pdatatype3 = malloc(sizeof(DATATYPE3));
...
sptr->Pdatatype12 = malloc(sizeof(DATATYPE12));
return sptr;
}
void
free_some_struct(PSOME_STRUCT sptr)
{
free(sptr->Pdatatype1);
free(sptr->Pdatatype2);
free(sptr->Pdatatype3);
...
free(sptr->Pdatatype12);
free(sptr);
}
If your structure contains the others structures as elements instead of pointers, you can allocate memory for the combined structure in one shot:
typedef struct _SOME_STRUCT {
DATATYPE1 Datatype1;
DATATYPE2 Datatype2;
DATATYPE3 Datatype3;
.......
DATATYPE12 Datatype12;
} SOME_STRUCT, *PSOME_STRUCT;
PSOME_STRUCT p = (PSOME_STRUCT)malloc(sizeof(SOME_STRUCT));
// Or without malloc:
PSOME_STRUCT p = new SOME_STRUCT();

atomics/mutex hybrid counter

I know that using atomics is dangerous (I watched Herb Sutter's 3hr lecture a few days ago), but the following use case seems reasonable to me, in terms of being simple and well contained.
Questions: (a) Is there something wrong with this? (Surely, there must be.) (b) Is there a name for this kind of hybrid atomic/mutex based approach? (c) Is there a simpler way of achieving the same thing?
The goal is to have a thread-safe counter class which we can call attempt_invalidation() on, knowing that it will only set its invalid flag to true if the count is at zero. There will be no other public methods on the class, but we will have a friend class specially designed to do RAII incrementing/decrementing of the counter.
class hybrid_counter{
friend class hybrid_counter_user;
bool invalidated = false;
int counter_a = 0;
std::atomic_int counter_b;
std::mutex mu;
bool increment_safely(){
std::lock_guard<std::mutex> gaurd(mu);
if ( !invalidated )
counter_a++;
return invalidated;
};
void increment_dangerously(){
counter_b++;
};
void decrement(){
counter_b--;
};
public:
bool attempt_invalidation(){
if(counter_a + counter_b == 0){
std::lock_guard<std::mutex> gaurd(mu);
if(counter_a + counter_b == 0)
invalidated = true;
}
return invalidated;
};
};
This is the friend class that knows how to use the counter correctly:
class hybrid_counter_user{
public:
hybrid_counter_user(hybrid_counter& hc){
if(hc.increment_safely() == false) // is not yet invalidated
c = &hc;
else
c = nullptr;
};
~hybrid_counter_user(){
if(c)
c->decrement();
};
hybrid_counter_user(hybrid_counter_user&& old){
c = old.c;
old.c = nullptr;
}
hybrid_counter_user(hybrid_counter_user& other){
c = other.c;
if(c)
c->increment_dangerously();
}
private:
hybrid_counter* c;
};
Note that the copy constructor uses the fact that hybrid_counter remains valid while other is in scope and other's destructor cannot be reordered with increment_dangerously because both involve the same atomic var.
The move constructor is simply transferring responsibility for decrementing.

Trying to use lambda functions as predicate for condition_variable wait method

I am trying to make the producer-consumer method using c++11 concurrency. The wait method for the condition_variable class has a predicate as second argument, so I thought of using a lambda function:
struct LimitedBuffer {
int* buffer, size, front, back, count;
std::mutex lock;
std::condition_variable not_full;
std::condition_variable not_empty;
LimitedBuffer(int size) : size(size), front(0), back(0), count(0) {
buffer = new int[size];
}
~LimitedBuffer() {
delete[] buffer;
}
void add(int data) {
std::unique_lock<std::mutex> l(lock);
not_full.wait(l, [&count, &size]() {
return count != size;
});
buffer[back] = data;
back = (back+1)%size;
++count;
not_empty.notify_one();
}
int extract() {
std::unique_lock<std::mutex> l(lock);
not_empty.wait(l, [&count]() {
return count != 0;
});
int result = buffer[front];
front = (front+1)%size;
--count;
not_full.notify_one();
return result;
}
};
But I am getting this error:
[Error] capture of non-variable 'LimitedBuffer::count'
I don't really know much about c++11 and lambda functions so I found out that class members can't be captured by value. By value though, I am capturing them by reference, but it seems like it's the same thing.
In a display of brilliance I stored the struct members values in local variables and used them in the lambda function, and it worked! ... or not:
int ct = count, sz = size;
not_full.wait(l, [&ct, &sz]() {
return ct != sz;
});
Obviously I was destroying the whole point of the wait function by using local variables since the value is assigned once and the fun part is checking the member variables which may, should and will change. Silly me.
So, what are my choices? Is there any way I can make the wait method do what it has to do, using the member variables? Or I am forced to not use lambda functions so I'd have to declare auxiliary functions to do the work?
I don't really get why I can't use members variables in lambda functions, but since the masters of the universe dessigned lamba functions for c++11 this way, there must be some good reason.
count is a member variable. Member variables can not be captured directly. Instead, you can capture this to achieve the same effect:
not_full.wait(l, [this] { return count != size; });

Resources