Not quite sure if this is a Jackson question or a Springboot question, or Jetty:
My microservice became unresponsive in production apparently due to excessive memory usage (telling from OS RSS stat) but no OOM.
I obtained a heap dump via jcmd [pid] GC.heap_dump and later opened it in Eclipse Memory Analyzer Tool (MAT) installed via Eclipse Marketplace.
I'm greeted by this finding:
I think this says Jackson ObjectMapper ate 80% of my heap (395M out of the default 512M size).
What would cause this and how can I prevent it?
UPDATES
I started digging into Jackson's SeralizerCache.
There was indeed a reproducible memory leak but it was fixed in 2.7.x: https://github.com/FasterXML/jackson-databind/issues/1049
This SO question also applied to pre-2.7 versions: Too Many objects in single instance of ObjectMapper SerializerCache causing memory leak
My version is 2.13.1, so the above shouldn't matter.
Found the culprit:
#PostMapping(value = "/headers", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
#ResponseBody
public ListingHeader[] head(#RequestBody ListingDetailPK[] parms) {
final ListingInfo[] all = readerDao.getAll(Arrays.asList(parms));
ObjectMapper mapper = JSON.mapper().copy();
mapper.registerModule(new MrBeanModule());
try {
return mapper.readValue(mapper.writerFor(ListingHeader[].class)
.writeValueAsString(all), ListingHeader[].class);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
}
A clue was provided in a comment by #Pawel Zieminski:
Is there a large number of (possibly generated) classes in your
system? Some proxy classes maybe?
I suspect that the dynamic proxies generated by MrBeanModule are causing the problem.
Related
I am working on a project to read from our existing ElasticSearch instance and produce messages in Pulsar. If I do this in a highly multithreaded way without any explicit synchronization, I get many occurances of the following log line:
Message with sequence id X might be a duplicate but cannot be determined at this time.
That is produced from this line of code in the Pulsar Java client:
https://github.com/apache/pulsar/blob/a4c3034f52f857ae0f4daf5d366ea9e578133bc2/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L653
When I add a synchronized block to my method, synchronizing on the pulsar template, the error disappears, but my publish rate drops substantially.
Here is the current working implementation of my method that sends Protobuf messages to Pulsar:
public <T extends GeneratedMessageV3> CompletableFuture<MessageId> persist(T o) {
var descriptor = o.getDescriptorForType();
PulsarPersistTopicSettings settings = pulsarPersistConfig.getSettings(descriptor);
MessageBuilder<T> messageBuilder = Optional.ofNullable(pulsarPersistConfig.getMessageBuilder(descriptor))
.orElse(DefaultMessageBuilder.DEFAULT_MESSAGE_BUILDER);
Optional<ProducerBuilderCustomizer<T>> producerBuilderCustomizerOpt =
Optional.ofNullable(pulsarPersistConfig.getProducerBuilder(descriptor));
PulsarOperations.SendMessageBuilder<T> sendMessageBuilder;
sendMessageBuilder = pulsarTemplate.newMessage(o)
.withSchema(Schema.PROTOBUF_NATIVE(o.getClass()))
.withTopic(settings.getTopic());
producerBuilderCustomizerOpt.ifPresent(sendMessageBuilder::withProducerCustomizer);
sendMessageBuilder.withMessageCustomizer(mb -> messageBuilder.applyMessageBuilderKeys(o, mb));
synchronized (pulsarTemplate) {
try {
return sendMessageBuilder.sendAsync();
} catch (PulsarClientException re) {
throw new PulsarPersistException(re);
}
}
}
The original version of the above method did not have the synchronized(pulsarTemplate) { ... } block. It performed faster, but generated a lot of logs about duplicate messages, which I knew to be incorrect. Adding the synchronized block got rid of the log messages, but slowed down publishing.
What are the best practices for multithreaded access to the PulsarTemplate? Is there a better way to achieve very high throughput message publishing?
Should I look at using the reactive client instead?
EDIT: I've updated the code block to show the minimum synchronization necessary to avoid the log lines, which is just synchronizing during the .sendAsync(...) call.
Your usage w/o the synchronized should work. I will look into that though to see if I see anything else going on. In the meantime, it would be great to give the Reactive client a try.
This issue was initially tracked here, and the final resolution was that it was an issue that has been resolved in Pulsar 2.11.
Please try updating the Pulsar 2.11.
We are using OptaPlanner(8.2.0) library in Spring Boot to solve knapsack problem using construction heuristic algorithm.
While running the application we observed that threads created by SolverManager are not getting released even after solving the problem. Because of that, performance of the application starts degrading after some time. Also, solver manager starts responding slowly of the increased thread count.
We also tried with latest version(8.17.0) but issue still persist.
Termination conditions:
<termination>
<millisecondsSpentLimit>200</millisecondsSpentLimit>
</termination>
optaplanner:
solver:
termination:
best-score-limit: 0hard/*soft
Code:
#Component
#Slf4j
public class SolutionManager {
private final SolverManager<Solution, String> solutionManager;
public SolutionManager(SolverManager<Solution, String> solutionManager) {
this.solutionManager = solutionManager;
}
public Solution getSolutionResponse(String solutionId, Solution unsolvedProblem)
throws InterruptedException, ExecutionException {
SolverJob<Solution, String> solve = solutionManager.solve(solutionId, unsolvedProblem);
Solution finalBestSolution = solve.getFinalBestSolution();
return finalBestSolution;
}
}
Thread metrics:
I wasn't able to reproduce the problem; after a load represented by solving several datasets in parallel, the number of threads drops back to the same value as before the load started.
The chart you shared doesn't clearly suggest there is a thread leak either; if you take a look at ~12:40 PM and compare it with ~2:00 PM, the number of threads actually did decrease.
Let me also add that the getFinalBestSolution() method actually blocks the calling thread until the solver finishes. If you instead use solve(ProblemId_ problemId, Solution_ problem, Consumer<? super Solution_> finalBestSolutionConsumer), this method returns immediately and the Consumer you provide is called when the solver finishes.
It looks like you might not be using OptaPlanner Spring Boot Starter.
If that's the case, upgrade to a recent version of OptaPlanner and add a dependency to optaplanner-spring-boot-starter. See the docs spring quickstart and the optaplanner-quickstarts repository (in the directory technology) for an example how to use it.
I have a spring batch massive loading that reading from a huge xml file (2GB), processing and write into DB oracle with hibernate persist query. I used chunk of 100 elements for this.
The problem is that when I running this batch on server the memory allocation increase until the process is killed for 'out of memory' (i used top command on server and the process comes to use 20Gb memory!)..i think that, for some reason, spring not deallocate memory after ending the chunk elements.
Can you help me to understand what happened?
Are you using JAXB/JAXB2 to unmarshal the xml data, by any chance? If so, the problem could be related to the initialization of the JAXBContext in your method, instead of initializing it once in your application. Initializing the JAXBContext is an expensive operation and a frequent cause of memory leaks. More info related to this issue can be found here.
Are you using JAXB/JAXB2 to unmarshal the xml data, by any chance? If so, the problem could be related to the initialization of the JAXBContext in your method, instead of initializing it once in your application. Initializing the JAXBContext is an expensive operation and a frequent cause of memory leaks. More info related to this issue can be found here.
I'm using Stax as follow:
public class ClassReader<T> extends StaxEventItemReader<T> {
public MyClassReader(Class<T> t) {
super();
XStreamMarshaller unmarshaller = new XStreamMarshaller();
HashMap<String, Object> aliases = new HashMap<String, Object>();
aliases.put("RECORD", t);
unmarshaller.setAliases(aliases);
this.setFragmentRootElementName("RECORD");
this.setUnmarshaller(unmarshaller);
}
}
I don't think this is the problem..
I get a different output from the same code snippet under different minor version of java. I was not able to find the related ticket on the open jdk bug tracker.
CompletableFuture<String> completableFuture = new CompletableFuture<>();
completableFuture.complete("xxx");
completableFuture.thenCompose(str -> {
CompletableFuture<String> completableFuture1 = new CompletableFuture<>();
completableFuture1.completeExceptionally(new Exception("hello"));
return completableFuture1;
}).exceptionally(ex -> {
System.out.println(ex.getMessage());
return null;
}).get();
Output under JDK 1.8.0_25:
hello
Output under JDK 1.8.0_102:
java.lang.Exception: hello
Is the newer one a fix or a regression? What is the related ticket?
There's a bug report here discussing this change. The key is in the Javadoc of CompletableFuture#thenCompose
Returns a new CompletionStage that, when this stage completes
normally, is executed with this stage as the argument to the supplied
function. See the CompletionStage documentation for rules covering
exceptional completion.
and and class documentation of CompletionStage
In all other cases, if a stage's computation terminates abruptly with
an (unchecked) exception or error, then all dependent stages requiring
its completion complete exceptionally as well, with a
CompletionException holding the exception as its cause.
That's what you see here. The Function you pass to exceptionally now receives a CompletionException holding the Exception that completed the triggering CompletableFuture.
The behavior you see now is the expected behavior.
You'll have to unwrap to get the cause of the exceptional completion
Throwable cause = ex.getCause(); // this is your Exception("Hello")
This is an issue and is been already fixed in 9 and backported to 8u60. You will see java.lang.Exception: hello from 8u60 onwards.
We have a simple microservice setup, based on Spring Boot and Java 8 on Windows servers.
Many of the services have a low load, since they serve as integrations to all kinds of external partners. So they are idle a lot of the time.
The problem is that the JVM only releases memory back to the OS, when a garbage collection is triggered. So a service might start using 32mb, then serve a single request and allocate 2GB of memory. If there is no other activity on that service, it will not GC and other services on the server will suffer.
Triggering a GC externally or internally with a System.gc works just fine and I have figured out how to use -XX:MaxHeapFreeRatio and -XX:MinHeapFreeRatio with -XX:+UseG1GC to control when the heap should expand and release memory to the OS.
My question is: What is the best way to ensure that memory is relased back to the OS when the JVM is idle?
One idea would be to have the service monitor itself and trigger a System.gc efter a period of idleness, but that might be tricky and errorprone. So hoping for better suggestions.
You can reproduce by running X instances of this program. About 10 made my Windows machine with 8GB give up.
import java.util.*;
public class Load {
public static void main(String[] args) throws Exception {
alloc();
Scanner s = new Scanner(System.in);
System.out.println("enter to gc ");
s.nextLine();
System.gc();
System.out.println("enter to exit");
s.nextLine();
}
private static void alloc() {
ArrayList<String[]> strings = new ArrayList<>();
int max = 1000000;
for (int i = 0; i < max; i++) {
strings.add(new String[500]);
}
}
}
c:\> java -server -XX:+UseG1GC -Xms32m -Xmx2048m Load
Edit: This was marked as a duplicate two times, but it is not a duplicate of the linked questions. The first question is a 2010 version of the same question, but that question is on why the GC does not release memory back to the OS (which was not possible at that time). The other question is about basic GC settings, that I already wrote that I understand. I wish a discussion of how to trigger the garbage collector when the system is idle. So running System.gc every five seconds is not acceptable, because that would have a high risk of colliding with valid requests and ruin the response times.
If calling System.gc() fulfills your needs, I would recomend to use spring scheduler to run a periodic task every x sedonds.
This is quite easy to implement, some annotations
#EnableAsync
#EnableScheduling
#Scheduled(cron = "...")
is all you need.
See spring scheduling for details.
Edit
Calling System.gc() gives only suggests to start the garbage collection, its still up to the JVM to decide when to do it or not.
To find out, if your system is idle or not, you could use the spring metrics.
There are some sub classes of
org.springframework.boot.actuate.endpoint.PublicMetrics
like TomcatPublicMetrics or SystemPublicMetrics that give you information about the system.
You can get them injected using #Autowire and call mertics() to get single values. Based on that you might be able to decide, if your system is idle or not,