I am having trouble with the C routines in libb64, here is my code:
base64_encodestate state;
int outBufLen = 2 * nInBuf;
*outBuf = new char[outBufLen];
base64_init_encodestate(&state);
int r1 = base64_encode_block(inBuf, nInBuf, *outBuf, &state);
int r2 = base64_encode_blockend(*outBuf, &state);
base64_init_encodestate(&state);
This puts the = at the beginning, not at the end.
So I tried this:
base64_encodestate state;
int outBufLen = 2 * nInBuf;
*outBuf = new char[outBufLen];
base64_init_encodestate(&state);
int r1 = base64_encode_block(inBuf, nInBuf, *outBuf, &state);
int r2 = base64_encode_blockend(*outBuf+ r1, &state);
base64_init_encodestate(&state);
This works, but not for "large" (~800KB text) files, then it skips the end = entirely. In that case base64_encode_blockend(code_out,state), enters case step_C where state->result = 0. I tried writing the b64 data to a file using the size reported by the libb64 functions, but it misses the end or is impartial. Im not sure.
I'm pretty much fed up with this. I based my code on the struct encode and decode.
Also do anyone know if there is a Windows API for base64 encode/decode? I am not using any C++ standard stuff, thats why I don't use the structs.
Related
Hi I am utilizing Protobuf for my personal project about neural networks.
Here is my Protobuf definitions:
syntax = "proto3";
package NGNET;
message InputLayer {
string name = 1;
uint32 size = 2;
}
message ComputeLayer {
string name = 1;
uint32 size = 2;
repeated LayerLink inputs = 3;
}
message LayerLink {
InputLayer il_input = 1;
ComputeLayer cl_input = 2;
uint32 output_size = 3;
repeated float weights = 4;
}
message NNET {
string name = 1;
repeated ComputeLayer outputs = 3;
}
The network is created like this:
ComputeLayer output1 = ComputeLayer(10, "output1");
ComputeLayer output2 = ComputeLayer(10, "output2");
ComputeLayer hidden = ComputeLayer(100, "hidden");
InputLayer input1 = InputLayer(784, "input1");
InputLayer input2 = InputLayer(784, "input2");
output1.link(&hidden);
output2.link(&hidden);
hidden.link(&input1);
hidden.link(&input2);
hidden.link(&extra);
The link functions are defined as:
void ComputeLayer::link(ComputeLayer* to_link) {
NGNET::LayerLink* link = new NGNET::LayerLink();
link->set_output_size(internal->size());
link->set_allocated_cl_input(to_link->getInternal());
internal->mutable_inputs()->AddAllocated(link);
}
void ComputeLayer::link(InputLayer* to_link) {
NGNET::LayerLink* link = new NGNET::LayerLink();
link->set_output_size(internal->size());
link->set_allocated_il_input(to_link->getInternal());
internal->mutable_inputs()->AddAllocated(link);
}
Note: The getInternal() function returns a NGNET::ComputeLayer or NGNET::InputLayer
Then the outputs are liked to a NNET with:
nnet->mutable_outputs()->AddAllocated(output1->getInternal());
nnet->mutable_outputs()->AddAllocated(output2->getInternal());
When nnet is deleted the program crashes with a segment fault.
I believe this is due to the hidden layer gets deleted twice. Is there any way I can safely free the memory that was allocated ?
Thanks.
The add_allocated_*() and set_allocated_*() methods take ownership of the pointer they are given. This means that you have to make sure that no other code will delete those pointers later, because the Protobuf implementation will delete them when the message is destroyed.
If you don't want Protobuf to take ownership of these objects, you should make copies instead:
link->mutable_il_input()->CopyFrom(*to_link->getInternal());
nnet->mutable_outputs()->Add()->CopyFrom(*output2->getInternal());
Generally, unless you are doing intense memory allocation optimizations, you probably never want to call the "allocated" protobuf accessors.
first of all I admit I'm a newbie in C++ addons for node.js.
I'm writing my first addon and I reached a good result: the addon does what I want. I copied from various examples I found in internet to exchange complex data between the two languages, but I understood almost nothing of what I wrote.
The first thing scaring me is that I wrote nothing that seems to free some memory; another thing which is seriously worrying me is that I don't know if what I wrote may helps or creating confusion for the V8 garbage collector; by the way I don't know if there are better ways to do what I did (iterating over js Object keys in C++, creating js Objects in C++, creating Strings in C++ to be used as properties of js Objects and what else wrong you can find in my code).
So, before going on with my job writing the real math of my addon, I would like to share with the community the nan and V8 part of it to ask if you see something wrong or that can be done in a better way.
Thank you everybody for your help,
iCC
#include <map>
#include <nan.h>
using v8::Array;
using v8::Function;
using v8::FunctionTemplate;
using v8::Local;
using v8::Number;
using v8::Object;
using v8::Value;
using v8::String;
using Nan::AsyncQueueWorker;
using Nan::AsyncWorker;
using Nan::Callback;
using Nan::GetFunction;
using Nan::HandleScope;
using Nan::New;
using Nan::Null;
using Nan::Set;
using Nan::To;
using namespace std;
class Data {
public:
int dt1;
int dt2;
int dt3;
int dt4;
};
class Result {
public:
int x1;
int x2;
};
class Stats {
public:
int stat1;
int stat2;
};
typedef map<int, Data> DataSet;
typedef map<int, DataSet> DataMap;
typedef map<float, Result> ResultSet;
typedef map<int, ResultSet> ResultMap;
class MyAddOn: public AsyncWorker {
private:
DataMap *datas;
ResultMap results;
Stats stats;
public:
MyAddOn(Callback *callback, DataMap *set): AsyncWorker(callback), datas(set) {}
~MyAddOn() { delete datas; }
void Execute () {
for(DataMap::iterator i = datas->begin(); i != datas->end(); ++i) {
int res = i->first;
DataSet *datas = &i->second;
for(DataSet::iterator l = datas->begin(); l != datas->end(); ++l) {
int dt4 = l->first;
Data *data = &l->second;
// TODO: real population of stats and result
}
// test result population
results[res][res].x1 = res;
results[res][res].x2 = res;
}
// test stats population
stats.stat1 = 23;
stats.stat2 = 42;
}
void HandleOKCallback () {
Local<Object> obj;
Local<Object> res = New<Object>();
Local<Array> rslt = New<Array>();
Local<Object> sts = New<Object>();
Local<String> x1K = New<String>("x1").ToLocalChecked();
Local<String> x2K = New<String>("x2").ToLocalChecked();
uint32_t idx = 0;
for(ResultMap::iterator i = results.begin(); i != results.end(); ++i) {
ResultSet *set = &i->second;
for(ResultSet::iterator l = set->begin(); l != set->end(); ++l) {
Result *result = &l->second;
// is it ok to declare obj just once outside the cycles?
obj = New<Object>();
// is it ok to use same x1K and x2K instances for all objects?
Set(obj, x1K, New<Number>(result->x1));
Set(obj, x2K, New<Number>(result->x2));
Set(rslt, idx++, obj);
}
}
Set(sts, New<String>("stat1").ToLocalChecked(), New<Number>(stats.stat1));
Set(sts, New<String>("stat2").ToLocalChecked(), New<Number>(stats.stat2));
Set(res, New<String>("result").ToLocalChecked(), rslt);
Set(res, New<String>("stats" ).ToLocalChecked(), sts);
Local<Value> argv[] = { Null(), res };
callback->Call(2, argv);
}
};
NAN_METHOD(AddOn) {
Local<Object> datas = info[0].As<Object>();
Callback *callback = new Callback(info[1].As<Function>());
Local<Array> props = datas->GetOwnPropertyNames();
Local<String> dt1K = Nan::New("dt1").ToLocalChecked();
Local<String> dt2K = Nan::New("dt2").ToLocalChecked();
Local<String> dt3K = Nan::New("dt3").ToLocalChecked();
Local<Array> props2;
Local<Value> key;
Local<Object> value;
Local<Object> data;
DataMap *set = new DataMap();
int res;
int dt4;
DataSet *dts;
Data *dt;
for(uint32_t i = 0; i < props->Length(); i++) {
// is it ok to declare key, value, props2 and res just once outside the cycle?
key = props->Get(i);
value = datas->Get(key)->ToObject();
props2 = value->GetOwnPropertyNames();
res = To<int>(key).FromJust();
dts = &((*set)[res]);
for(uint32_t l = 0; l < props2->Length(); l++) {
// is it ok to declare key, data and dt4 just once outside the cycles?
key = props2->Get(l);
data = value->Get(key)->ToObject();
dt4 = To<int>(key).FromJust();
dt = &((*dts)[dt4]);
int dt1 = To<int>(data->Get(dt1K)).FromJust();
int dt2 = To<int>(data->Get(dt2K)).FromJust();
int dt3 = To<int>(data->Get(dt3K)).FromJust();
dt->dt1 = dt1;
dt->dt2 = dt2;
dt->dt3 = dt3;
dt->dt4 = dt4;
}
}
AsyncQueueWorker(new MyAddOn(callback, set));
}
NAN_MODULE_INIT(Init) {
Set(target, New<String>("myaddon").ToLocalChecked(), GetFunction(New<FunctionTemplate>(AddOn)).ToLocalChecked());
}
NODE_MODULE(myaddon, Init)
One year and half later...
If somebody is interested, my server is up and running since my question and the amount of memory it requires is stable.
I can't say if the code I wrote really does not has some memory leak or if lost memory is freed at each thread execution end, but if you are afraid as I was, I can say that using same structure and calls does not cause any real problem.
You do actually free up some of the memory you use, with the line of code:
~MyAddOn() { delete datas; }
In essence, C++ memory management boils down to always calling delete for every object created by new. There are also many additional architecture-specific and legacy 'C' memory management functions, but it is not strictly necessary to use these when you do not require the performance benefits.
As an example of what could potentially be a memory leak: You're passing the object held in the *callback pointer to the function AsyncQueueWorker. Yet nowhere in your code is this pointer freed, so unless the Queue worker frees it for you, there is a memory leak here.
You can use a memory tool such as valgrind to test your program further. It will spot most memory problems for you and comes highly recommended.
One thing I've observed is that you often ask (paraphrased):
Is it okay to declare X outside my loop?
To which the answer actually is that declaring variables inside of your loops is better, whenever you can do it. Declare variables as deep inside as you can, unless you have to re-use them. Variables are restricted in scope to the outermost set of {} brackets. You can read more about this in this question.
is it ok to use same x1K and x2K instances for all objects?
In essence, when you do this, if one of these objects modifies its 'x1K' string, then it will change for all of them. The advantage is that you free up memory. If the string is the same for all these objects anyway, instead of having to store say 1,000,000 copies of it, your computer will only keep a single one in memory and have 1,000,000 pointers to it instead. If the string is 9 ASCII characters long or longer under amd64, then that amounts to significant memory savings.
By the way, if you don't intend to modify a variable after its declaration, you can declare it as const, a keyword short for constant which forces the compiler to check that your variable is not modified after declaration. You may have to deal with quite a few compiler errors about functions accepting only non-const versions of things they don't modify, some of which may not be your own code, in which case you're out of luck. Being as conservative as possible with non-const variables can help spot problems.
How to get post script name,full path of type 1 font on 64 bit Windows7? APIs for these are provided by Adobe ATM library for 32 bit win OS only like- ATMGetPostScriptName, ATMGetFontPaths, etc.
As per my knowledge type 1 fonts are supported by OS now. I am able to get all these font information through GetFontData, RegQueryMultipleValues, etc for TTF and OTF fonts but these APIs are failing for type1 fonts (.pfm, .pfb fonts).
First time I am using this forum and hoping someone can help me out quickly.
Thanks in Advance,
Vijendra
I had a similar problem, I wanted the postscript name and to convert the font to postscript. The solution sits in a little library called ttf2pt1, found at http://ttf2pt1.sourceforge.net/README.html
The routine you need is below. The routine digs out all the various internal names for the font using OutlineTextMetrics. The nameFace value is the one you want.
short i;
short val;
long offset;
OUTLINETEXTMETRIC *otm;
UINT nSize;
UINT retVal;
char *sptr;
char *nameFamily;
char *nameFace;
char *nameStyle;
char *nameFull;
int fontType;
char otmBuffer[4096];
SelectObject(hdc, theFont);
nSize = GetOutlineTextMetrics(hdc, 0, NULL);
if(nSize){
otm = (OUTLINETEXTMETRIC *) otmBuffer;
retVal = GetOutlineTextMetrics(hdc, nSize, otm);
val = otm->otmTextMetrics.tmPitchAndFamily;
offset = (long) otm->otmpFamilyName;
sptr = &otmBuffer[offset];
nameFamily = sptr;
offset = (long) otm->otmpFaceName;
sptr = &otmBuffer[offset];
nameFace = sptr;
offset = (long) otm->otmpStyleName;
sptr = &otmBuffer[offset];
nameStyle = sptr;
offset = (long) otm->otmpFullName;
sptr = &otmBuffer[offset];
nameFull = sptr;
if(val & TMPF_TRUETYPE){
fontType = kFontTypeTrueType;
}
else{
if(val & TMPF_VECTOR){
fontType = kFontTypePostscript;
}
}
}
else{
fontType = kFontTypeUnknown;
}
I have a C++ code that I am trying to interface with Matlab. My mex file runs fine in the first run but crashes in the second run. However, if I clear all the variables in the Matlab before execution (using clear all) program never crashes. So I have a question in this:
1. Can mex function takes variables from the Matlab workspace without using some special functions? Am I doing it somehow in my code, unintentionally?
I have a posted the mex function that I wrote. It has a one dimensional vector called "block" that is read inside the C++ function called sphere_detector. For the present problem the block size is 1x1920 and it is read in the chunk of 16 elements inside the sphere_detector. Program crashed when I read the SECOND chunk of 16 elements. The first element that I read in the chunk will throw this error:
First-chance exception at 0x000007fefac7206f (sphere_decoder.mexw64) in MATLAB.exe: 0xC0000005: Access violation reading location 0xffffffffffffffff.
MATLAB.exe has triggered a breakpoint
I checked my block vector, it should have all the values initialized and it has that. So, I am little confused as to why I am facing this problem.
I am using Matlab 2010a and Visual Studio 2010 Professional.
Here is the mex function:
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
double *mod_scheme, *Mt, *Mr, *block_length, *SNR;
mod_scheme = mxGetPr(prhs[0]);
Mt = mxGetPr(prhs[1]);
Mr = mxGetPr(prhs[2]);
block_length = mxGetPr(prhs[3]);
SNR = mxGetPr(prhs[4]);
/* Now take the input block. This is an encoded block and sphere detector will do the transmission too -- I can change it later */
double *block = mxGetPr(prhs[5]);
double *LIST_SIZE = mxGetPr(prhs[6]);
double **cand_sym;
int a = *mod_scheme;
int b = *Mt;
int c = *Mr;
int d = *block_length;
int e = *SNR;
int f = *LIST_SIZE;
int bitSize = (int)(log10(1.0*a)/log10(2.0));
for(int i=0; i<(int)*block_length; ++i)
{
printf("%d\n", (int)block[i]);
}
printf("Hello world %d %d %d %d %d!\n", (int)*mod_scheme, (int)*Mt, (int)*Mr, (int)*block_length, (int)*SNR);
/* Inputs are read correctly now set the outputs */
double *llr, *cand_dist;
/* for llrs */
plhs[0] = mxCreateDoubleMatrix(1, d, mxREAL);
llr = mxGetPr(plhs[0]);
/* for cand_dist */
int no_mimo_sym = d/(b*bitSize);
plhs[1] = mxCreateDoubleMatrix(1, f*no_mimo_sym, mxREAL);
cand_dist = mxGetPr(plhs[1]);
/* for cand_syms */
plhs[2] = mxCreateDoubleMatrix(b*bitSize*no_mimo_sym, f,mxREAL); //transposed version
double *candi;
candi = mxGetPr(plhs[2]);
cand_sym = (double**)mxMalloc(f*sizeof(double*));
if(cand_sym != NULL)
{
for(int i=0;i<f; ++i)
{
cand_sym[i] = candi + i*b*bitSize*no_mimo_sym;
}
}
sphere_decoder(a,b,c,d,e,block,f,llr,cand_dist,cand_sym);
// mxFree(cand_sym);
}
The portion inside the sphere decoder code where I get read exception looks like this:
for(int _block_length=0;_block_length<block_length; _block_length+=Mt*bitSize)
{
printf("Transmitting MIMO Symbol: %d\n", _block_length/(Mt*bitSize));
for(int _antenna = 0; _antenna < Mt; ++_antenna)
for(int _tx_part=0;_tx_part<bitSize; _tx_part++)
{
// PROGRAM CRASHES EXECUTING THIS LINE
bitstream[_antenna][_tx_part] = (int)block_data[_block_length + _antenna*bitSize + _tx_part];
}
............................REST OF THE CODE..................
}
Any help would be appreciated.
With regards,
Newbie
Well I finally managed to solve the problem. It was a very stupid mistake that I made. I had a pointer to a pointer(double *a;) of data type double and by mistake I assigned it memory of integer (I ran a find and replace command where I changed lots of int to double but this one left). Hence heap was getting corrupted. Also I changed my Mex function where I created dynamic variables using calloc and passed them to the C++ function. Once C++ function returned I copied there values to matlab variables and freed them usind free().
I have the following code, I use to Open a File Open Dialog using Win32 API. It works fine in 32bit, but fails when I use in a 64bit (In a DLL). What am I doing wrong?
char Filestring[256];
Filter = "OBJ files\0*.obj\0\0";
char* returnstring = NULL;
OPENFILENAME opf;
opf.hwndOwner = mainHWND;
opf.lpstrFilter = Filter;
opf.lpstrCustomFilter = 0;
opf.nMaxCustFilter = 0L;
opf.nFilterIndex = 1L;
opf.lpstrFile = Filestring;
opf.lpstrFile[0] = '\0';
opf.nMaxFile = 256;
opf.lpstrFileTitle = 0;
opf.nMaxFileTitle=50;
opf.lpstrInitialDir = Path;
opf.lpstrTitle = "Open Obj File";
opf.nFileOffset = 0;
opf.nFileExtension = 0;
opf.lpstrDefExt = "*.*";
opf.lpfnHook = NULL;
opf.lCustData = 0;
opf.Flags = (OFN_PATHMUSTEXIST | OFN_OVERWRITEPROMPT) & ~OFN_ALLOWMULTISELECT;
opf.lStructSize = sizeof(OPENFILENAME);
if(GetOpenFileName(&opf))
{
returnstring = opf.lpstrFile;
if (returnstring) {
result = returnstring;
}
}
EDIT: By failing, I meant that the Open File Dialog doesn't show up. The code still returns zero without any errors.
EDIT 2: I have called CommDlgExtendedError() and it returned 1. From the MSDN reference, does it mean the dialog has invalid lStructSize? I have checked the sizeof(OPENFILENAME) and it returned 140 bytes.
UPDATE: In my Project Settings, Under Code Generation the "Struct Member Alignment" is set to 4 Bytes(/Zp4). I changed this to default and it magically worked. Look for the answers and their comments below for more information.
You aren't initialising lpTemplateName and so it contains random stack noise. This in turn will lead to 'hInstance` being references which also contains stack noise.
When calling a function like this you should first of all zero out the struct and only fill in the fields that are non-zero. Something like this:
OPENFILENAME opf={0};
opf.lStructSize = sizeof(OPENFILENAME);
opf.hwndOwner = mainHWND;
opf.lpstrFilter = Filter;
opf.nFilterIndex = 1L;
opf.lpstrFile = Filestring;
opf.lpstrFile[0] = '\0';
opf.nMaxFile = 256;
opf.lpstrInitialDir = Path;
opf.lpstrTitle = "Open Obj File";
opf.lpstrDefExt = "*.*";
opf.Flags = OFN_PATHMUSTEXIST | OFN_OVERWRITEPROMPT;
There was no need to exclude OFN_ALLOWMULTISELECT explicitly since you were not including it in the first place!
EDIT
You state in a comment that this doesn't work. Calling CommDlgExtendedError is a good idea and should tell you why it fails.
You could also try to run the minimal possible GetOpenFileName which is this:
char Filestring[MAX_PATH] = "\0";
OPENFILENAME opf={0};
opf.lStructSize = sizeof(OPENFILENAME);
opf.lpstrFile = Filestring;
opf.nMaxFile = MAX_PATH;
GetOpenFileName(&opf);
I have the very same problem and a partial solution :
+ the simple following simple example (proposed abobe) was not working in x64 mode.
+ I changed the complie option "struct Member Alignment" from 1byte /Zp1 to default which solved this problem (by introducing others !!!)
char Filestring[MAX_PATH] = "\0";
OPENFILENAME opf={0};
opf.lStructSize = sizeof(OPENFILENAME);
opf.lpstrFile = Filestring;
opf.nMaxFile = MAX_PATH;
GetOpenFileName(&opf);
To find out more you should call CommDlgExtendedError to get the error code what went wrong. Besides this I would initialize all member of the struct to 0 with
ZeroMemory(&opf, sizeof(opf));
Since the file open dialog is in reality a COM component it could be worth to check out if your thread apartment state is different under 64 bit.
if( RPC_E_CHANGED_MODE == CoInitialize(NULL) )
ASSERT(FALSE); // MTA Apartment found
CoUnitialize()
Yours,
Alois Kraus
As a note in Microsoft Office 2010 64-bit we gave up and used the internal wrappers as the structure turned into 140 bytes and we were not sure how to change alignment.
Application.GetOpenFilename(FileFilter, FilterIndex, Title, ButtonText, MultiSelect)
and Application.GetSaveAsFilename(InitialFilename, FileFilter, FilterIndex, Title, ButtonText)
http://msdn.microsoft.com/en-us/library/ff834966.aspx
http://msdn.microsoft.com/en-us/library/microsoft.office.interop.excel._application.getopenfilename.aspx
Needless to say we think all individuals with fairly heavy applications in Excel should start considering other options as maintaining future versions across multiple clients and platforms may just be... insane!
I managed to get around this problem by setting the packing appropriately before including the header file. That way, for the purpose of this one function, we were using the 'default' 16 byte alignment, but did not have to change the packing alignment for the rest of our program:
#ifdef _WIN64
#pragma pack( push )
#pragma pack( 16 )
#include "Commdlg.h"
#pragma pack( pop )
#else
#include "Commdlg.h"
#endif // _WIN64