I am currently setting feature testing for the API of a project I am currently working on. However, I seem to be running into issues with memory issues when running the tests in bitbucket pipelines. This seems to be caused by some memory leaks.
I have done the following research but can not seem to find a solution.
I used roave/no-leaks to find which tests where causing memory leaks. This returned that all my tests where causing memory leaks.
This lead me to believe the memory leak is present in my application, not in the actual test.
I found an article "Laravel: Fixing memory leaks on tests" addressing the issue.
Using the information in the article I set up the following test in order to test my memory leakage:
public function test_memory_leak()
{
$this->app->flush();
$this->app = null;
for($i = 1; $i < 250; ++$i) {
$this->createApplication()->flush();
echo('Using ' . ((int) (memory_get_usage(true) / (1024 * 1024))) . 'MB as ' . $i . ' iterations.');
echo("\n");
}
$this->app = $this->createApplication();
}
This test creates and flushes an app instance 250 times and echoing the memory usage after each iteration. When starting the test I use around 28MB or memory. After 250 app instances this is 74MB. I am not using any setUp or tearDown methods in combination with this test.
After some investigation I found that not using Route::group() in my routing files decreased the total memory usage to 56MB. Does anybody have experience or tips for finding out why Route::group() causes such a memory leak and how to find the remaining memory leaks?
The project is set up in Laravel 8 and the testing is set up using PHPUnit 9.5.20. I am not using RefreshDatabase, DatabaseTransanactions or similar traits. I am developing on Windows but have access to Linux if needed for debugging software.
Thanks in advance for any help!
Kind Regards,
Simon
Related
I am having some issues with my virtualHBA driver on Windows Server 2016. A ran the HLK crashdump support test. 3 times out of 10 the test passed. In those 3 failing tests, the crashdump hangs at 0% while taking Complete dump, or Kernel dump or minidump.
By kernel debugging my code, I found that the call to ExAllocatePoolWithTag() for buffer allocation never actually returns.
Below is the statement which never returns.
pDeviceExtension->pcmdbuf=(struct mycmdrsp *)ExAllocatePoolWithTag(NonPagedPoolCacheAligned,pcmdqSignalSize,((ULONG)'TA1'));
I searched on the web regarding this. However, all of the found pages are focusing on this function returning NULL which in my case never returns.
Any help on how to move forward would be highly appreciated.
Thanks in advance.
You can't allocate memory in crash dump mode. You're running at HIGH_LEVEL with interrupts disabled and so you're calling this API at the wrong IRQL.
The typical solution for a hardware adapter is to set the RequestedDumpBufferSize in the PORT_CONFIGURATION_INFORMATION structure during the normal HwFindAdapter call. Then when you're called again in crash dump mode you use the CrashDumpRegion field to get your dump buffer allocation. You then need to write your own "crash dump mode only" allocator to allocate buffers out of this memory region.
It's a huge pain, especially given that it's difficult/impossible to know how much memory you're ultimately going to need. I usually calculate some minimal configuration overhead (i.e. 1 channel, 8 I/O requests at a time, etc.) and then add in a registry configurable slush. The only benefit is that the environment is stripped down so you don't need to be in your all singing, all dancing configuration.
I have a service written in go that takes 6-7G memory at runtime (RES in top). So I used the pprof tool trying to figure out where the problem is.
go tool pprof --pdf http://<service>/debug/pprof/heap > heap_prof.pdf
But there are only about 1-2G memory in result ('Total MB' in pdf). Where's the rest ?
And I've tried profile my service with GOGC=off, as a result the 'Total MB' is exactly the same as 'RES' in top. It seems that memory is GCed but haven't been return to kernel won't be profiled.
Any idea?
P.S, I've tested in both 1.0.3 and 1.1rc3.
This is because Go currently does not give memory of GC-ed objects back to the operating system, to be precise, only for objects smaller then predefined limit (32KB). Instead memory is cached to speed up future allocations Go:malloc. Also, it seems that this is going to be fixed in the future TODO.
Edit:
New GC behavior: If the memory is not used for a while (about 5 min), runtime will advise the kernel to remove the physical mappings from the unused virtual ranges. This process can be forced by calling runtime.FreeOSMemory()
My Malloc is failing in my project.
Malloc runs several times via a one of the functions but fails due to lack of memory.
I am trying to increase the heap size in my VC++ but it gives me the error as above in the subject.
Can someone please tell me what is wrong in this ?
Windows server 2003 R2 Enterprise edition
And i am using VC++ 98 edition.
I tried some search but could not get anything conclusive on how to use /HEAP OPTION.
should the numbers be in MB ?
message_t* Allocate_momory(MsgType_t msgType, UInt16 dataLength)
{
// TO DO: Allocate memenory and return the pointer
message_t* mes_t;
mes_t = (message_t*) malloc(sizeof (message_t));
mes_t->msgType = msgType;
mes_t->dataLength = 0;
mes_t->clientID = 0;
mes_t->usageCount = 0;
mes_t->dataBuf = malloc(sizeof (dataLength));
return mes_t;
}
Yes it worked... But it unfortunatly did not solve my problem with malloc :( !!
This is a huge project with too many files.
I can't post the code but can someone guide me how should i try to debug a problem where malloc is failing ?
/HEAP sets the heap size in bytes. Also the square brackets in the documentation denote an optional parameter - you don't actually type these in. So it would be e.g.
/HEAP:1073741824
for a 1 GB heap, or
/HEAP:1073741824,16777216
if you really do want to specify the "commit" parameter in addition to the heap size (you probably don't).
Unfortunately I don't think this will solve your real problem, which is that you are running out of memory. You may have memory leaks, which you can track down with a tool such as valgrind. If that's not the case then you have a bad design, which will be a lot harder to fix than memory leaks.
I'm hosting IronPython in a c#-based WebService to be able to provide custom extension scripts. However, I'm finding that memory usage sharply increases when I do simple load testing by executing the webservice repeatedly in a loop.
IronPython-1.1 implemented IDisposable on its objects so that you can dispose of them when they are done. The new IronPython-2 engine based on the DLR has no such concept.
From what I understood, everytime you execute a script in the ScriptEngine a new assembly is injected in the appdomain and can't be unloaded.
Is there any way around this?
You could try creating a new AppDomain every time you run one of your IronPython scripts. Although assebmlies cannot be unloaded from memory you can unload an AppDomain and this will allow you to get the injected assembly out of memory.
You need to disable the optimized code generation:
var runtime = Python.CreateRuntime();
var engine = runtime.GetEngine("py");
PythonCompilerOptions pco = (PythonCompilerOptions)engine.GetCompilerOptions();
pco.Module &= ~ModuleOptions.Optimized;
// this shouldn't leak now
while(true) {
var code = engine.CreateScriptSourceFromString("1.0+2.0").Compile(pco);
code.Execute();
}
Turns out, after aspnet_wp goes to about 500mb, the garbage collector kicks in and cleans out the mess. The memory usage then drops to about 20mb and steadily starts increasing again during load testing.
So there's no memory 'leak' as such.
I'm writing the memory manager for an application, as part of a team of twenty-odd coders. We're running out of memory quota and we need to be able to see what's going on, since we only appear to be using about 700Mb. I need to be able to report where it's all going - fragmentation etc. Any ideas?
You can use existing memory debugging tools for this, I found Memory Validator 1 quite useful, it is able to track both API level (heap, new...) and OS level (Virtual Memory) allocations and show virtual memory maps.
The other option which I also found very usefull is to be able to dump a map of the whole virtual space based on VirtualQuery function. My code for this looks like this:
void PrintVMMap()
{
size_t start = 0;
// TODO: make portable - not compatible with /3GB, 64b OS or 64b app
size_t end = 1U<<31; // map 32b user space only - kernel space not accessible
SYSTEM_INFO si;
GetSystemInfo(&si);
size_t pageSize = si.dwPageSize;
size_t longestFreeApp = 0;
int index=0;
for (size_t addr = start; addr<end; )
{
MEMORY_BASIC_INFORMATION buffer;
SIZE_T retSize = VirtualQuery((void *)addr,&buffer,sizeof(buffer));
if (retSize==sizeof(buffer) && buffer.RegionSize>0)
{
// dump information about this region
printf(.... some buffer information here ....);
// track longest feee region - usefull fragmentation indicator
if (buffer.State&MEM_FREE)
{
if (buffer.RegionSize>longestFreeApp) longestFreeApp = buffer.RegionSize;
}
addr += buffer.RegionSize;
index+= buffer.RegionSize/pageSize;
}
else
{
// always proceed
addr += pageSize;
index++;
}
}
printf("Longest free VM region: %d",longestFreeApp);
}
You can also find out information about the heaps in a process with Heap32ListFirst/Heap32ListNext, and about loaded modules with Module32First/Module32Next, from the Tool Help API.
'Tool Help' originated on Windows 9x. The original process information API on Windows NT was PSAPI, which offers functions which partially (but not completely) overlap with Tool Help.
Our (huge) application (a Win32 game) started throwing "Not enough quota" exceptions recently, and I was charged with finding out where all the memory was going. It is not a trivial job - this question and this one were my first attempts at finding out. Heap behaviour is unexpected, and accurately tracking how much quota you've used and how much is available has so far proved impossible. In fact, it's not particularly useful information anyway - "quota" and "somewhere to put things" are subtly and annoyingly different concepts. The accepted answer is as good as it gets, although enumerating heaps and modules is also handy. I used DebugDiag from MS to view the true horror of the situation, and understand how hard it is to actually thoroughly track everything.