I'm facing a problem I can't seems to fix and I need your help.
I'm generating a list of PDF that I write to the hard drive and everything works fine for a small amount of files, but when I start to generate more files (via a for loop), the creations stops and the others PDF files arent created.
I'm using Play Framework with the PDF module, that rely on ITextRenderer to generate the PDF.
I localized the problem (well, I believe it's here) by adding outputs to see where it stops, and the problem is when I call .createPDF(os);.
At first, I was able to only create 16 files and after that, it would stops, but I created a Singleton that creates the renderer in the Class instance and re-use the same instance (in order to avoid adding the fonts and settings everytime) and I went to 61 files created, but no more.
I though about a memory leak that blocks the process, but can't see where nor how to find it correctly.
Here's my part of the code :
List models; // I got a list of MyModel from a db query, this MyModel contains a path to a file
List<InputStream> files = new ArrayList<InputStream>();
for (MyModel model : models) {
if (!model.getFile().exists()) {
model.generatePdf();
}
files.add(new FileInputStream(model.getFile()));
}
// The generatePDF :
public void generatePdf() {
byte[] bytes = PDF.toBytes(views.html.invoices.pdf.invoice.render(this, due));
FileOutputStream output;
try {
File file = getFile();
if (!file.getParentFile().exists()) {
file.getParentFile().mkdirs();
}
if (file.exists()) {
file.delete();
}
output = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(output);
bos.write(bytes);
bos.flush();
bos.close();
output.flush();
output.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
As you can see, I do my best to avoid memory leaks but this isn't enough.
In order to locate the problem, I replaced PDF.toBytes and all subsequent calls from that class to a copy/paste version inside my class, and added outputs. That's how I found that the thread hangs at createPDF, line
Update 1:
I have two (indentical) PlayFramework applications running with those parameters :
-Xms1024m -Xmx1024m -XX:PermSize=512m -XX:MaxPermSize=512m
I tried to stop one instance and re-execute the PDF generation, but it didn't impact the number of file generated, it stops at the same amount of files.
I also tried to update the allocated memories :
-Xms1536m -Xmx1536m -XX:PermSize=1024m -XX:MaxPermSize=1024m
No changes at all neither.
For information, the server has 16 Gb of RAM.
cat /proc/cpuinfo :
model name : Intel(R) Core(TM) i5-2400 CPU # 3.10GHz
cpu MHz : 3101.000
cpu cores : 4
cache size : 6144 KB
Hope it'll helps.
Well I'm really surprised the bug has absolutely nothing related to memory, memory leaks or available memory left.
I'm astonished.
It's related to an image that was loaded via an url, in the same server (local), that was taking to long to load. Removing that image fixed the issue.
I will make a base64 encoded image and it should fix the issue.
I still can't believe it!
The module is developed by Jörg Viola, I think it's safe to assume everything is fine on this side. From the IText library, I also believe it's safe to assume that everything is safe.
The bottleneck, as you guessed, was from your code. The interesting part was that it's wasn't some memory not properly managed, but from a network request that was making the PDF rendering slower and slower everytime, until it would ultimately fails.
It's nice you finally make it work.
Related
Every time I upload a dicom file, the memory grow up. How can I clean the memory of the old ones
You can see various examples of how to free the memory on the examples code, example on the loader:
let loader = new LoadersVolume();
loader.free(); // Free memory
loader = null;
Another one:
let stackHelper = new HelpersStack();
stackHelper.dispose(); // Free memory
stackHelper = null;
I suggest to read the following document to know how garbage collection works on most browsers.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_Management
The garbage collector deletes from the memory everything that is not referenced somewhere.
Having a high use of memory even if you don't you the object anymore means there still a reference to it somewhere. Look for a variable that can still access to your old data, including the 3D scene, the AMI stackHelper, the AMI loader...
Before going frustrated about the title, I would like to clear out that I am a fresher on JavaFX UI. I have been a developer for 9 years, using Swing and currently I decided to give a try to the JavaFX. Examples on the net shows that JavaFX really can create beautiful GUIs compared to Swing. Maybe I am trying to create and deploy GUIs the wrong way, but one thing is for sure. JavaFX panes load slower than Swing and consumes a lot more memory. The same GUI was redesigned with JAVAFX and it takes almost 200Mb while the Swing GUI take only 50Mb.
Here I give an example of the code of how I create the GUIs programmatically using FXML.
public class PanelCreator {
private FXMLPane<LoginPaneController> loginFXML;
private FXMLPane<RegistrationPaneController> registerFXML;
private FXMLPane<EmailValidationPaneController> emailValidationFXML;
public PanelCreator() {
this.rootPane = rootPane;
try {
loginFXML = new FXMLPane<LoginPaneController>("Login.fxml");
registerFXML = new FXMLPane<RegistrationPaneController>("Register.fxml");
emailValidationFXML = new FXMLPane<EmailValidationPaneController>("EmailValidation.fxml");
} catch (IOException e) {e.printStackTrace();} // catch
} // Constructor Method
public Pane getLoginPane() {
return loginFXML.getPane();
} // getLoginPane()
public Pane getRegisterPane() {
return registerFXML.getPane();
} // getRegisterPane
public Pane getEmailValidationPane() {
return emailValidationFXML.getPane();
} // getEmailValidationPane
public LoginPaneController getLoginPaneController() {
return loginFXML.getController();
} // getLoginPaneController()
public RegistrationPaneController getRegistrationPaneController() {
return registerFXML.getController();
} // getRegistrationPaneController()
} // class PanelCreator
The constructor method of PanelCreator creates 3 FXMLPane classes, a class that combines both the FXML Pane and its Controller. An code of FXMLPane class is shown on the following code.
public class FXMLPane<T> {
private Pane pane;
private T paneController;
public FXMLPane(String url) throws IOException {
URL location = getClass().getResource(url);
FXMLLoader fxmlLoader = new FXMLLoader();
fxmlLoader.setLocation(location);
fxmlLoader.setBuilderFactory(new JavaFXBuilderFactory());
pane = fxmlLoader.load(location.openStream());
paneController = fxmlLoader.<T>getController();
} // Constructor Method
public Pane getPane() {
return pane;
} // getPane()
public T getController() {
return paneController;
} // getController()
}
Through PanelCreator now I can use the get methods to get each JavaFX Panel and its controller and I do not have to run the FXML load method every time to get the panel. Currently, what bothers me is not that the creation of FXML GUIs is slower than Swing but more that the RAM is 3x and 4x times more than the correspoing Swing version.
Can someone explain to me what I am doing wrong? The FXML files have just basic components on a Grid Pane, components like buttons, layers and textfields.
The code for the above example can be found here
Summarizing the answers from the comment section:
JavaFX needs more memory in general. E.g. JavaFX uses double precision for all properties along the UI-components, while Swing uses integer values most of the time. But the difference should not be noticeable.
Java consumes more memory as it needs to. As a default Java does not return memory back to your system even if you trigger the garbage collection. Thus if a JavaFX program needs a lot of memory on the initialization process but frees it afterwards, the JRE continues to hold the maximum level of memory for ever (see picture 1). As a side effect the GC will be triggered less often, because there is so much free unused memory (see picture 2). You can change the default by using the JVM option -XX:+UseG1GC. This changes the behavior of how memory is allocated, how it's freed and when the GC is triggered. With this option the allocated memory should better fit in with the used memory. If you want more tuning see Java Heap Tuning
JavaFX is a new framework compared to Swing. It will be improved over time in performance and resources consumption. As you can see in picture 1 and 3 it has already been improved. It now uses 8 to 9MB of memory on a 64Bit Linux machine. This is even less memory than the Swing version. I used Oracle Java
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
Picture 1: Memory consumption over time for the JavaFX example program. It shows a huge amount of free memory compared to the used memory. The GC was triggered manually multiple times to show used memory part without garbage.
Picture 2: Memory consumption over time for the JavaFX example program, but without manually triggering the GC. The used memory grows and grows because the GC isn't triggered.
Picture 3: Memory consumption over time for the JavaFX example program using the GC option -XX:+UseG1GC. After the first GC cycle the memory size was reduced to fit the real size of used memory.
Dear programmers, i wrote a program wich target a Windows Mobile platform (NetCF 3.5).
My programm has a method of answers check and this method show dynamically created pictureboxes, textboxes and images in new form. Here is a method logic:
private void ShowAnswer()
{
PictureBox = new PictureBox();
PictureBox.BackColor = Color.Red;
PictureBox.Location = new Point(x,y);
PictureBox.Name = "Name";
PictureBox.Size = Size(w,h);
PictureBox.Image = new Bitmap(\\Image01.jpg);
}
My problem is in memory leaks or something. If the user work with a programm aproximately 30 minutes and run the ShowAnswer() method several times, Out of memry exception appears. I know that the reason may be in memory allocation of bitmaps, but i even handle the ShowAnswers form closing event and manually trying to release all controls resources and force a garbage collector:
foreach(Control cntrl in this.Controls)
{
cntrl.Dispose();
GC.Collect();
}
It seems like everything collects and disposes well, every time i check the taskmanager on my windows mobile device during the programm tests and see that memory were released and child form was closed properly, but in every ShowAnswer() method call and close i see a different memory amount in device taskmanager (somtimes it usues 7.5 Mb, sometimes 11.5, sometimes 9.5) any time its different, but it seems like sometimes when the method start to run as usual memory is not allocated and Out of memory exception appears.. Please advice me how to solve my problem.. Maybe i should use another Dispose methods, or i should set bitmap another way.. thank you in advance!!!
Depending on how you're handling the form generation, you might need to dispose of the old Image before loading a new one.
private void ShowAnswer()
{
PictureBox = new PictureBox();
PictureBox.BackColor = Color.Red;
PictureBox.Location = new Point(x,y);
PictureBox.Name = "Name";
PictureBox.Size = Size(w,h);
if(PictureBox.Image != null) //depending on how you construct the form
PictureBox.Image.Dispose();
PictureBox.Image = new Bitmap(\\Image01.jpg);
}
However, you should also check before you load the image that it's not so obscenely large that it munches up all of your device's memory.
Edit: I don't just mean the size of the compressed image in memory - I also mean the physical size of the image (height & width). The Bitmap will create an uncompressed image that will take up much, much more memory than is resident on storage memory (height*width*4). For a more in-depth explanation, check out the following SO question:
OutOfMemoryException loading big image to Bitmap object with the Compact Framework
My Win32 application performs numerous disk operations in a designated temporary folder while functioning, and seriously redesigning it is out of the question.
Some clients have antivirus software that scans the same temporary directory (it simply scans everything). We tried to talk them into disabling it - it doesn't work, so it's out of the question either.
Every once in a while (something like once for every one thousand file operations) my application tries to perform an operation on a file which is at that very time opened by the antivirus and is therefore locked by the operating system. A sharing violation occurs and causes an error in my application. This happens about once in three minutes on average.
The temporary folder can contain up to 100k files in most typical scenarios, so I don't like the idea of having them open at all times because this could cause running out of resources on some edge conditions.
Is there some reasonable strategy for my application to react to situations when a needed file is locked? Maybe something like this?
for( int i = 0; i < ReasonableNumber; i++ ) {
try {
performOperation(); // do useful stuff here
break;
} catch( ... ) {
if( i == ReasonableNumber - 1 ) {
throw; //not to hide errors if unlock never happens
}
}
Sleep( ReasonableInterval );
}
Is this a viable strategy? If so, how many times and how often should my application retry? What are better ideas if any?
A virusscanner that locks files while it's scanning them is quite bad. Clients who have virusscanners this bad need to have their brains replaced... ;-)
Okay, enough ranting. If a file is locked by some other process then you can use a "try again" strategy like you suggest. OTOH, do you really need to close and then re-open those files? Can't you keep them open until your process is done?
One tip: Add a delay (sleep) when you try to re-open the file again. About 100 ms should be enough. If the virusscanner keeps the file open that long then it's a real bad scanner. Clients with scanners that bad deserve the exception message that they'll see.
Typically, try up to three times... -> Open, on failure try again, on second failure try again, on third failure just crash.
Remember to crash in a user-friendly way.
I've had experience with antivirus software made by both Symantec and AVG which resulted in files being unavailable for open.
A common problem we experienced back in the 2002 time frame with Symantec was with MSDev6 when a file was updated in this sequence:
a file is opened
contents are modified in memory
application needs to commit changes
application creates new tmp file with new copy of file + changes
application deletes old file
application copies tmp file to old file name
application deletes the tmp file
The problem would occur between step 5 and step 6. Symantec would do something to slowdown the delete preventing the creation of a file with the same name (CreateFile returned ERROR_DELETE_PENDING). MSDev6 would fail to notice that - meaning step 6 failed. Step 7 still happened though. The delete of the original would eventually finish. So the file no longer existed on disk!
With AVG, we've been experiencing intermittent problems being able to open files that have just been modified.
Our resolution was a try/catch in a reasonable loop as in the question. Our loop count is 5.
If there is the possibility that some other process - be it the antivirus software, a backup utility or even the user themselves - can open the file, then you must code for that possibility.
Your solution, while perhaps not the most elegant, will certainly work as long as ReasonableNumber is sufficiently large - in the past I've used 10 as the reasonable number. I certainly wouldn't go any higher and you could get away with a lower value such as 5.
The value of sleep? 100ms or 200ms at most
Bear in mind that most of the time your application will get the file first time anyway.
Depends on how big your files are, but for 10s to 100s of Kb I find that 5 trys with 100ms (0.1 seconds) to be sufficient. If you still hit the error once in a while, double the wait, but YMMV.
If you have a few places in the code which needs to do this, may I suggest taking a functional approach:
using System;
namespace Retry
{
class Program
{
static void Main(string[] args)
{
int i = 0;
Utils.Retry(() =>
{
i = i + 1;
if (i < 3)
throw new ArgumentOutOfRangeException();
});
Console.WriteLine(i);
Console.Write("Press any key...");
Console.ReadKey();
}
}
class Utils
{
public delegate void Retryable();
static int RETRIES = 5;
static int WAIT = 100; /*ms*/
static public void Retry( Retryable retryable )
{
int retrys = RETRIES;
int wait = WAIT;
Exception err;
do
{
try
{
err = null;
retryable();
}
catch (Exception e)
{
err = e;
if (retrys != 1)
{
System.Threading.Thread.Sleep(wait);
wait *= 2;
}
}
} while( --retrys > 0 && err != null );
if (err != null)
throw err;
}
}
}
Could you change your application so you don't release the file handle? If you hold a lock on the file yourself the antivir application will not be able to scan it.
Otherwise a strategy such as yours will help, a bit, because it only reduces the probability but it doesn't solve the problem.
Tough problem. Most ideas that I have go into a direction that you don't want (e.g. redesign).
I don't know how many files you have in your directory, but if it's not that much you may be able to work around your problem by keeping all files open and locked while your program runs.
That way the virus scanner will have no chance to interrupt your file-accesses anymore.
I'm hosting IronPython in a c#-based WebService to be able to provide custom extension scripts. However, I'm finding that memory usage sharply increases when I do simple load testing by executing the webservice repeatedly in a loop.
IronPython-1.1 implemented IDisposable on its objects so that you can dispose of them when they are done. The new IronPython-2 engine based on the DLR has no such concept.
From what I understood, everytime you execute a script in the ScriptEngine a new assembly is injected in the appdomain and can't be unloaded.
Is there any way around this?
You could try creating a new AppDomain every time you run one of your IronPython scripts. Although assebmlies cannot be unloaded from memory you can unload an AppDomain and this will allow you to get the injected assembly out of memory.
You need to disable the optimized code generation:
var runtime = Python.CreateRuntime();
var engine = runtime.GetEngine("py");
PythonCompilerOptions pco = (PythonCompilerOptions)engine.GetCompilerOptions();
pco.Module &= ~ModuleOptions.Optimized;
// this shouldn't leak now
while(true) {
var code = engine.CreateScriptSourceFromString("1.0+2.0").Compile(pco);
code.Execute();
}
Turns out, after aspnet_wp goes to about 500mb, the garbage collector kicks in and cleans out the mess. The memory usage then drops to about 20mb and steadily starts increasing again during load testing.
So there's no memory 'leak' as such.