I'm having a problem while running a Scilab code.
As title suggests, I get the error numeric factorization: not enough memory, related to the umfpack function.
In task manager I see a memory usage around 3GB (my system has 16GB).
Can anyone help me with this issue?
My guess is that you are attempting to use umfpack with matrices which are too big, and then it is unable to allocate the memory required (probably more than the 13GB you still have available).
See also https://scicomp.stackexchange.com/a/8972/21926
Related
I downloaded the stanford segmentator and I am following the instructions but I am getting a memory error, the full message is here:
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.regex.Pattern.matcher(Pattern.java:1093)
at edu.stanford.nlp.wordseg.Sighan2005DocumentReaderAndWriter.shapeOf(Sighan2005DocumentReaderAndWriter.java:230)
at edu.stanford.nlp.wordseg.Sighan2005DocumentReaderAndWriter.access$300(Sighan2005DocumentReaderAndWriter.java:49)
at edu.stanford.nlp.wordseg.Sighan2005DocumentReaderAndWriter$CTBDocumentParser.apply(Sighan2005DocumentReaderAndWriter.java:169)
at edu.stanford.nlp.wordseg.Sighan2005DocumentReaderAndWriter$CTBDocumentParser.apply(Sighan2005DocumentReaderAndWriter.java:114)
at edu.stanford.nlp.objectbank.LineIterator.setNext(LineIterator.java:42)
at edu.stanford.nlp.objectbank.LineIterator.<init>(LineIterator.java:31)
at edu.stanford.nlp.objectbank.LineIterator$LineIteratorFactory.getIterator(LineIterator.java:108)
at edu.stanford.nlp.wordseg.Sighan2005DocumentReaderAndWriter.getIterator(Sighan2005DocumentReaderAndWriter.java:86)
at edu.stanford.nlp.objectbank.ObjectBank$OBIterator.setNextObjectHelper(ObjectBank.java:435)
at edu.stanford.nlp.objectbank.ObjectBank$OBIterator.setNextObject(ObjectBank.java:419)
at edu.stanford.nlp.objectbank.ObjectBank$OBIterator.<init>(ObjectBank.java:412)
at edu.stanford.nlp.objectbank.ObjectBank.iterator(ObjectBank.java:250)
at edu.stanford.nlp.sequences.ObjectBankWrapper.iterator(ObjectBankWrapper.java:45)
at edu.stanford.nlp.ie.AbstractSequenceClassifier.classifyAndWriteAnswers(AbstractSequenceClassifier.java:1193)
at edu.stanford.nlp.ie.AbstractSequenceClassifier.classifyAndWriteAnswers(AbstractSequenceClassifier.java:1137)
at edu.stanford.nlp.ie.AbstractSequenceClassifier.classifyAndWriteAnswers(AbstractSequenceClassifier.java:1091)
at edu.stanford.nlp.ie.crf.CRFClassifier.main(CRFClassifier.java:3023)
Before executing the file I tried increasing the heap space by doing export JAVA_OPTS=-Xmx4000m. I also tried splitting the file but still had the same error - I split the file to 8 chunks, so each had around 15MB each. What should I do to adjust the memory problem?
The segment.sh script that ships with the segmenter limits the memory to 2G, which is probably the cause of the error. Editing that file will hopefully fix the issue for you.
I'm using Julia 1.1 with JLD and HDF5 to save a file onto the disk, where I met a couple of question about the memory usage.
Issue 1:
First, I defined a 4 GB matrix A.
A = zeros(ComplexF64,(243,243,4000));
When I type the command and look at windows task manager:
A=nothing
It took several minutes for Julia to release those memory back to me. Most of the time, (In Task manager) Julia just doesn't release the memory usage at all, even though the command returned results saying that A occupied 0 bytes instantly.
varinfo()
name size summary
–––––––––––––––– ––––––––––– –––––––
A 0 bytes Nothing
Base Module
Core Module
InteractiveUtils 162.930 KiB Module
Main Module
ans 0 bytes Nothing
Issue 2:
Further, when I tried to use JLD and HDF5 to save file onto the disk. This time, the task manager told me that, when using the save("test.jld", "A", A) command, an extra 4GB memory was used.
using JLD,HDF5
A = zeros(ComplexF64,(243,243,4000));
save("test.jld", "A", A)
Further, after I typed
A=nothing
Julia won't release the 8 GB memory back to me.
Finding 3:
An interesting thing I found was that, if I retype the command
A = zeros(ComplexF64,(243,243,4000));
The task manager would told me the cashed memory was released, and the total memory usage was again only 4GB.
Question 1:
What's going on with memory management in Julia? Was it just a mistake by Windows, or some command in Julia? How to check the Julia memory usage instantly?
Question 2:
How to tell the Julia to instantly release the memory usage?
Question 3:
Is there a way to tell JLD package not use those extra 4GB meomory?
(Better, could someone tell me how to create A directly on the disk without even creating it in the memory? I knew there's memory mapped I/O in JLD package. I have tried it, but it seemed to require me to create matrix A in the memory and save A onto the disk first, before I could recall the memory mapped A again. )
This is a long question, so thanks ahead!
Julia uses garbage collector to de-alocate the memory. Usually a garbage collector does not run after every line of code but only when needed.
Try to force garbage collection by running the command:
GC.gc()
This releases memory space for unreferenced Julia objects. In this way you can check whether the memory actually has been released.
Side note: JLD used to be somewhat not-always-working (I do not know the current status). Hence you first consideration for non-cross-platform object persistence always should be the serialize function from the in-built Serialization package - check the documentation at https://docs.julialang.org/en/v1/stdlib/Serialization/index.html#Serialization.serialize
I am having some issues with my virtualHBA driver on Windows Server 2016. A ran the HLK crashdump support test. 3 times out of 10 the test passed. In those 3 failing tests, the crashdump hangs at 0% while taking Complete dump, or Kernel dump or minidump.
By kernel debugging my code, I found that the call to ExAllocatePoolWithTag() for buffer allocation never actually returns.
Below is the statement which never returns.
pDeviceExtension->pcmdbuf=(struct mycmdrsp *)ExAllocatePoolWithTag(NonPagedPoolCacheAligned,pcmdqSignalSize,((ULONG)'TA1'));
I searched on the web regarding this. However, all of the found pages are focusing on this function returning NULL which in my case never returns.
Any help on how to move forward would be highly appreciated.
Thanks in advance.
You can't allocate memory in crash dump mode. You're running at HIGH_LEVEL with interrupts disabled and so you're calling this API at the wrong IRQL.
The typical solution for a hardware adapter is to set the RequestedDumpBufferSize in the PORT_CONFIGURATION_INFORMATION structure during the normal HwFindAdapter call. Then when you're called again in crash dump mode you use the CrashDumpRegion field to get your dump buffer allocation. You then need to write your own "crash dump mode only" allocator to allocate buffers out of this memory region.
It's a huge pain, especially given that it's difficult/impossible to know how much memory you're ultimately going to need. I usually calculate some minimal configuration overhead (i.e. 1 channel, 8 I/O requests at a time, etc.) and then add in a registry configurable slush. The only benefit is that the environment is stripped down so you don't need to be in your all singing, all dancing configuration.
I'm trying to find current flag count in KMines by using gdb. I know that I should look for memory mappings first to avoid non-existent memory locations. So I ran info proc mappings command to see the memory segments. I picked up a random memory gap (0xd27000-0x168b000) from the result and executed the find command like this: find 0x00d27000, 0x0168b000, 10
But I got the warning: Unable to access 1458 bytes of target memory at 0x168aa4f, halting search. error. Although the address 0x168aa4f is between 0xd27000 and 0x168b000, gdb says that it can't access to it. Why does this happen? What can I do to avoid this situation? Or is there a way to ignore unmapped/unaccessible memory locations?
Edit: I tried to set the value of the address 0x168aa4f to 1 and it works, so gdb can actually access that address but gives error when used with the find command. But why?
I guess I have solved my own problem, I can't believe how simple the solution was. The only thing I did was to decrease the 2nd parameter's value by one. So the code should be find 0x00d27000, 0x0168afff, 10 because linux allocates the memory by using maps in [x,y) format, so if the line in root/proc/pid/maps says something like this;
01a03000-0222a000 rw-p
The memory allocated includes 0x01a03000 but not 0x0222a000. Hope this silly mistake of mine helps someone :D
Edit: The root of the problem is the algorithm implemented in target.c(gdb's source code I mean) the algorithm reads and searches the memory as chunks at the size of 16000 bytes. So even if the last byte of the chunk is invalid, gdb will throw the entire chunk into the trash and won't even give any proper information about the invalid byte, it only reports the beginning of the current chunk.
Hi I know that this error which I'm going to show can't be fixed through code. I just want to know why and how is it caused and I also know its due to JVM trying to access address space of another program.
A fatal error has been detected by the Java Runtime Environment:
EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6dcd422a, pid=4024, tid=3900
JRE version: 6.0_14-b08
Java VM: Java HotSpot(TM) Server VM (14.0-b16 mixed mode windows-x86 )
Problematic frame:
V [jvm.dll+0x17422a]
An error report file with more information is saved as:
C:\PServer\server\bin\hs_err_pid4024.log
If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
The book "modern operating systems" from Tanenbaum, which is available online here:
http://lovingod.host.sk/tanenbaum/Unix-Linux-Windows.html
Covers the topic in depth. (Chapter 4 is in Memory Management and Chapter 4.8 is on Memory segmentation). The short version:
It would be very bad if several programs on your PC could access each other's memory. Actually even within one program, even in one thread you have multiple areas of memory that must not influence one other. Usually a process has at least one memory area called the "stack" and one area called the "heap" (commonly every process has one heap + one stack per thread. There MAY be more segments, but this is implementation dependent and it does not matter for the explanation here). On the stack things like you function's arguments and your local variables are saved. On the heap are variables saved that's size and lifetime cannot be determined by the compiler at compile time (that would be in Java everything that you use the "new"-Operator on. Example:
public void bar(String hi, int myInt)
{
String foo = new String("foobar");
}
in this example are two String objects: (referenced by "foo" and "hi"). Both these objects are on the heap (you know this, because at some point both Strings were allocated using "new". And in this example 3 values are on the stack. This would be the value of "myInt", "hi" and "foo". It is important to realize that "hi" and "foo" don't really contain Strings directly, but instead they contain some id that tells them were on the heap they can find the String. (This is not as easy to explain using java because java abstracts a lot. In C "hi" and "foo" would be a Pointer which is actually just an integer which represents the address in the heap where the actual value is stored).
You might ask yourself why there is a stack and a heap anyway. Why not put everything in the same place. Explaining that unfortunately exceeds the scope of this answer. Read the book I linked ;-). The short version is that stack and heap are differently managed and the separation is done for reasons of optimization.
The size of stack and heap are limited. (On Linux execute ulimit -a and you'll get a list including "data seg size" (heap) and "stack size" (yeah... stack :-)).).
The stack is something that just grows. Like an array that gets bigger and bigger if you append more and more data. Eventually you run out of space. In this case you may end up writing in the memory area that does not belong to you anymore. And that would be extremely bad. So the operating systems notices that and stops the program if that happens. On Linux you get a "Segmenation fault" and on Windows you get an "Access violation".
In other languages like in C, you need to manage your memory manually. A tiny error can easily cause you to accidentally write into some space that does not belong to you. In Java you have "automatic memory management" which means that the JVM does all this for you. You don't need to care and that takes loads from your shoulders as a developer (it usually does. I bet there are people out there who would disagree about the "loads" part ;-)). This means that it /should/ be impossible to produce segmentation faults with java. Unfortunatelly the JVM is not perfect. Sometimes it has bugs and screws up. And then you get what you got.