sftp large file(>1GB)failed with only specific data bytes transferred.Why? - amazon-ec2

Currently our system (Client) on AWS EC2 instance encountered a sftp problem when trying to send a large file to a customer remote sftp server (“160.xxx.xxx.35.bc.googleusercontent.com). For the small sized files, the sftp transfer works fine but when the file size is about or larger than 1GB, we found that only 1068392448 bytes were transferred to Server sftp site. However, when we send the same large file with the same code and same environments to our own non-googleusercontent remote sftp server (with only the URL/UserName/Password differ), it was successful and all data transferred correctly.
This problem happened after the customer server side made some changes by adding the Load Balancer process. The customer server side did some investigation, adjusted the load balancer timeout but it did not help to resolve this issue. It is said the Client side stopped the data transfer after 1068392448 bytes and Server side waited and dropped the connection after the allowed idle time (~50second).
Our investigation noticed that the large source file was read from AWS S3 and saved to local correctly. When the large file data is written to Server sftp site reach 1068392448 bytes (consistently for all test results), the TCP socket connection status changed from ESTABLISHED to CLOSE_WAIT after Server allowed idle time (about ~50s). The process stayed in this status forever until it is manually stopped/killed. When the TCP socket connection is in CLOSE_WAIT status, the data transfer process showed in dump is in a wait loop at awaitSpace() method (in PipedInputStream class of java.io package). The buffer is indicated to be full and waiting to be written to the Server side. Below is the codes of the wait loop:
private void checkStateForReceive() throws IOException {
if (!connected) {
throw new IOException("Pipe not connected");
} else if (closedByWriter || closedByReader) {
throw new IOException("Pipe closed");
} else if (readSide != null && !readSide.isAlive()) {
throw new IOException("Read end dead");
}
}
private void awaitSpace() throws IOException {
while (in == out) {
checkStateForReceive();
/* full: kick any waiting readers */
notifyAll();
try {
wait(1000);
} catch (InterruptedException ex) {
throw new java.io.InterruptedIOException();
}
}
}
Below is the dump:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode):
"Connect thread files.liveramp.com session" #13 daemon prio=5 os_prio=0 tid=0x000000001e711800 nid=0x63c0 in Object.wait() [0x000000001f81f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273)
at java.io.PipedInputStream.receive(PipedInputStream.java:231)
- locked <0x00000006c1c944c8> (a com.jcraft.jsch.Channel$MyPipedInputStream)
at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
at com.jcraft.jsch.IO.put(IO.java:64)
at com.jcraft.jsch.Channel.write(Channel.java:438)
at com.jcraft.jsch.Session.run(Session.java:1459)
at java.lang.Thread.run(Thread.java:745)
"org.apache.commons.vfs2.cache.SoftRefFilesCache$SoftRefReleaseThread" #11 daemon prio=5 os_prio=0 tid=0x000000001ea2d000 nid=0x3078 in Object.wait() [0x000000001efcf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c1c24710> (a java.lang.ref.ReferenceQueue$Lock)
at org.apache.commons.vfs2.cache.SoftRefFilesCache$SoftRefReleaseThread.run(SoftRefFilesCache.java:74)
"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000000001dfc5000 nid=0x1944 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #9 daemon prio=9 os_prio=2 tid=0x000000001df12800 nid=0x6848 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #8 daemon prio=9 os_prio=2 tid=0x000000001df07800 nid=0x5720 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #7 daemon prio=9 os_prio=2 tid=0x000000001df04800 nid=0x6358 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Monitor Ctrl-Break" #6 daemon prio=5 os_prio=0 tid=0x000000001df56000 nid=0x6910 runnable [0x000000001e1cf000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x00000006c1c24f88> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x00000006c1c24f88> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001c69e800 nid=0x4880 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001c69d800 nid=0x2534 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000001c67e800 nid=0x4908 in Object.wait() [0x000000001d9df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c1c26660> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c1c26660> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000030b5000 nid=0x6340 in Object.wait() [0x000000001d8df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c1c26818> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c1c26818> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x0000000002fc4000 nid=0x56dc in Object.wait() [0x0000000002e0f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.jcraft.jsch.Session.write(Session.java:1269)
- locked <0x00000006c1c943d0> (a com.jcraft.jsch.ChannelSftp)
at com.jcraft.jsch.ChannelSftp.sendWRITE(ChannelSftp.java:2646)
at com.jcraft.jsch.ChannelSftp.access$100(ChannelSftp.java:36)
at com.jcraft.jsch.ChannelSftp$1.write(ChannelSftp.java:806)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
- eliminated <0x00000006c1c27a88> (a org.apache.commons.vfs2.provider.sftp.SftpFileObject$SftpOutputStream)
at org.apache.commons.vfs2.util.MonitorOutputStream.write(MonitorOutputStream.java:123)
- locked <0x00000006c1c27a88> (a org.apache.commons.vfs2.provider.sftp.SftpFileObject$SftpOutputStream)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
- eliminated <0x00000006c1c26a50> (a org.apache.commons.vfs2.provider.DefaultFileContent$FileContentOutputStream)
at org.apache.commons.vfs2.util.MonitorOutputStream.write(MonitorOutputStream.java:123)
- locked <0x00000006c1c26a50> (a org.apache.commons.vfs2.provider.DefaultFileContent$FileContentOutputStream)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:805)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:784)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:755)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:771)
at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:37)
at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(AbstractFileObject.java:295)
at com.merkleinc.dat.sftptran.cli.SftpVfs2App.main(SftpVfs2App.java:88)
"VM Thread" os_prio=2 tid=0x000000001c657000 nid=0x28b4 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000002fda800 nid=0x2b54 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000002fdc000 nid=0x4528 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002fdd800 nid=0x4850 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002fdf800 nid=0x7f0 runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000001e023000 nid=0x5d58 waiting on condition
JNI global references: 225
Heap
PSYoungGen total 49664K, used 14491K [0x000000076b400000, 0x000000076ed00000, 0x00000007c0000000)
eden space 49152K, 29% used [0x000000076b400000,0x000000076c21efd0,0x000000076e400000)
from space 512K, 6% used [0x000000076e500000,0x000000076e508000,0x000000076e580000)
to space 4608K, 0% used [0x000000076e880000,0x000000076e880000,0x000000076ed00000)
ParOldGen total 175104K, used 2879K [0x00000006c1c00000, 0x00000006cc700000, 0x000000076b400000)
object space 175104K, 1% used [0x00000006c1c00000,0x00000006c1ecfc70,0x00000006cc700000)
Metaspace used 11230K, capacity 11448K, committed 11648K, reserved 1058816K
class space used 1265K, capacity 1318K, committed 1408K, reserved 1048576K
Currently, we are using com.github’s vfs-s3 Version 2.4.2 with related apache.commons vfs2 version 2.1. and com.jcraft’s jsch version 0.1.54 for sftp data transfer. Our basic codes are as below:
import org.apache.commons.vfs2.*;
Public void upload(String sourceUri, String targetUri) throws IOException, URISyntaxException {
StandardFileSystemManager fsManager = new StandardFileSystemManager();
fsManager.init();
try (FileObject sourceFile = getFileObject(sourceUri, fsManager);
FileObject remoteFile = fsManager.resolveFile(createConnectionString(targetUri), createOptions())) {
// Copy s3 file to sftp server
remoteFile.copyFrom(sourceFile, Selectors.SELECT_SELF);
} finally {
fsManager.close();
}
};
We tried to test the same code in the local Window environments. Occasionally we could see the same problem symptoms as in ec2 instance but most time the problem is not there (inconsistent). We tested on other linux system to send the large file to the customer server sftp site and there was no problem.
We have tried to upgrade to com.github.abashev’s vfs-s3 Version 3.0.0 and 4.0.0 with corresponding versions of vfs2 on AWS EC2 instance. However, it reproduced the same problem results for the same large files. Tried to set loadOpenSSHConfig to be true with proper config file values to keep the connection alive, but it did not help for this situation.
We tried to test sftp directly by sftp "put" command with large file to the remote server from the EC2 instance (and other Windows/Linux platforms) and the large file data transfer is always successful.
The question is where is the potential root cause of the problem? Why the server side stopped receiving the data after 1068392448 bytes or why the rest of the data could not be sent to the server side? Any hard limits on our EC2 environments blocked the data transfer operation (we tried to check a few limits but not still not clear)? Or the Server site incorrectly sent the connection close “FIN” request to Client site (how to prove it)? Thanks for any suggestions for the potential solution.

Related

How to turn off Logging to a File in Spring Boot/JPA/Hibernate Show_sql is true

I have show_sql to true in my properties because I want it to log the queries into my console when I am running our Spring Boot server app in IntelliJ. We do not have any config to write those to a file, but for some reason it is trying to write to a file. Here is what we see in a thread dump
"http-nio-8080-exec-2" #28 daemon prio=5 os_prio=31 tid=0x00007fa208deb000 nid=0x9403 runnable [0x000070000bef8000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
- locked <0x00000006c00430b8> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:480)
- locked <0x00000006c0043098> (a java.io.PrintStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
- locked <0x00000006c00431d8> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
at java.io.PrintStream.write(PrintStream.java:527)
- locked <0x00000006c0043098> (a java.io.PrintStream)
at java.io.PrintStream.print(PrintStream.java:669)
at java.io.PrintStream.println(PrintStream.java:806)
- locked <0x00000006c0043098> (a java.io.PrintStream)
at org.hibernate.engine.jdbc.spi.SqlStatementLogger.logStatement(SqlStatementLogger.java:111)
All the other logging of SQL is going to the console.
"http-nio-8080-exec-1" #29 daemon prio=5 os_prio=31 tid=0x00007fa200fef000 nid=0x110b waiting for monitor entry [0x0000700009b8c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.io.PrintStream.println(PrintStream.java:805)
- waiting to lock <0x00000006c0043098> (a java.io.PrintStream)
at org.hibernate.engine.jdbc.spi.SqlStatementLogger.logStatement(SqlStatementLogger.java:111)
at org.hibernate.engine.jdbc.spi.SqlStatementLogger.logStatement(SqlStatementLogger.java:94)
I have searched everywhere in our code base to try and find anything that might make it want to log to a file, even in the run configuration of IntelliJ, but I have not had any luck.
I want it to stop logging to a file, because it ends up hanging my IDE.

Maven can create folders and lock file, but hangs while downloading

I have a maven build that I'm running in a docker container using the official maven image from the Docker hub. The .m2 directory is mounted to an NFS share.
This works in one environment, but in another identical environment it will always hang after writing the lock file. It never completes the download, but just hangs there forever. Since maven debug didn't give me any details after the point where it hung, I decided to watch the .m2 directory to see what happened.
ubuntu#kubernetes-dev-nfs-pv:/nfs-shares/jenkins/.m2$ inotifywait -m -r .
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
./ CREATE,ISDIR repository
./ OPEN,ISDIR repository
./ CLOSE_NOWRITE,CLOSE,ISDIR repository
./repository/ CREATE,ISDIR org
./repository/ OPEN,ISDIR org
./repository/ CLOSE_NOWRITE,CLOSE,ISDIR org
./repository/org/ CREATE,ISDIR springframework
./repository/org/ OPEN,ISDIR springframework
./repository/org/ CLOSE_NOWRITE,CLOSE,ISDIR springframework
./repository/org/springframework/ CREATE,ISDIR boot
./repository/org/springframework/ OPEN,ISDIR boot
./repository/org/springframework/ CLOSE_NOWRITE,CLOSE,ISDIR boot
./repository/org/springframework/boot/ CREATE,ISDIR spring-boot-starter-parent
./repository/org/springframework/boot/ OPEN,ISDIR spring-boot-starter-parent
./repository/org/springframework/boot/ CLOSE_NOWRITE,CLOSE,ISDIR spring-boot-starter-parent
./repository/org/springframework/boot/spring-boot-starter-parent/ CREATE,ISDIR 1.3.7.RELEASE
./repository/org/springframework/boot/spring-boot-starter-parent/ OPEN,ISDIR 1.3.7.RELEASE
./repository/org/springframework/boot/spring-boot-starter-parent/ CLOSE_NOWRITE,CLOSE,ISDIR 1.3.7.RELEASE
./repository/org/springframework/boot/spring-boot-starter-parent/1.3.7.RELEASE/ CREATE spring-boot-starter-parent-1.3.7.RELEASE.pom.part.lock
Maven seems to be working, it creates a number of folders and even the lock file, but then it hangs. How can I get maven to complete or find some additional information to help me solve this.
By the way, if I use ephemeral storage inside the container, it downloads the package as expected.
UDPATE: one of the comments suggested a thread dump. Below you can see I attached to the running container. I confirm that the container can modify files in the .m2 directory and then I use jstack to get a thread dump of the process.
ec2-user#murano-necrhj0ld3vkx4-kube-3-gm5dmsfnftxn:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c7d1f4c91559 maven:alpine "cat" About an hour ago Up About an hour agitated_cori
ec2-user#murano-necrhj0ld3vkx4-kube-3-gm5dmsfnftxn:~$ sudo docker exec -ti c7d1f4c91559 /bin/bash
bash-4.3$ ps
PID USER TIME COMMAND
1 1000 0:00 cat
6 1000 0:00 sh -c echo $$ > '/var/jenkins_home/workspace/api-product#tmp/durable-ca9825bd/pid'; jsc=durable-04ba6b757bca34373f180bd01ef64ca1; JENKINS_SERVER_COOKIE=$jsc '/var/jenkins_home/workspace/api-product#tmp/durable-ca
12 1000 0:00 {script.sh} /bin/sh -xe /var/jenkins_home/workspace/api-product#tmp/durable-ca9825bd/script.sh
13 1000 0:07 /usr/lib/jvm/java-1.8-openjdk/bin/java -classpath /usr/share/maven/boot/plexus-classworlds-2.5.2.jar -Dclassworlds.conf=/usr/share/maven/bin/m2.conf -Dmaven.home=/usr/share/maven -Dmaven.multiModuleProjectDirecto
1584 1000 0:00 /bin/bash
1589 1000 0:00 ps
bash-4.3$ cat /var/jenkins_home/workspace/api-product#tmp/durable-ca9825bd/script.sh
#!/bin/sh -xe
mvn -Dmaven.repo.local="$PWD"/../../.m2/repository clean compile
bash-4.3$ ls -la /var/jenkins_home/.m2/
total 16
drwxr-xr-x 3 1000 1000 4096 May 2 21:14 .
drwxrwxr-x 23 1000 1000 4096 May 3 11:55 ..
-rw-r--r-- 1 1000 1000 6 May 2 21:14 file.txt
drwxr-xr-x 3 1000 1000 4096 May 2 20:50 repository
bash-4.3$ cat /var/jenkins_home/.m2/file.txt
hello
bash-4.3$ vi /var/jenkins_home/.m2/file.txt
bash-4.3$ cat /var/jenkins_home/.m2/file.txt
hello
another
bash-4.3$ jstack 13
2017-05-03 13:04:37
Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode):
"Attach Listener" #11 daemon prio=9 os_prio=0 tid=0x00007fc4a4956800 nid=0x6a7 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fc4a4343000 nid=0x2c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fc4a4311800 nid=0x2b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fc4a4302000 nid=0x2a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fc4a42ff000 nid=0x29 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fc4a42fc800 nid=0x28 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fc4a42d5000 nid=0x27 in Object.wait() [0x00007fc48ba4b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dab108d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000dab108d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fc4a42ca800 nid=0x26 in Object.wait() [0x00007fc48bb4c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dab18178> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000dab18178> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007fc4a4179800 nid=0x20 runnable [0x00007fc4a3426000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115)
at org.eclipse.aether.connector.basic.PartialFile$LockFile.tryLock(PartialFile.java:135)
at org.eclipse.aether.connector.basic.PartialFile$LockFile.lock(PartialFile.java:80)
at org.eclipse.aether.connector.basic.PartialFile$LockFile.<init>(PartialFile.java:67)
at org.eclipse.aether.connector.basic.PartialFile$Factory.newInstance(PartialFile.java:219)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:441)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:359)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:76)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:590)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:258)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:529)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:430)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:255)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:232)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:303)
at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:193)
at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:243)
at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:1051)
at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:829)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:331)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:429)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:398)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:361)
at org.apache.maven.graph.DefaultGraphBuilder.collectProjects(DefaultGraphBuilder.java:400)
at org.apache.maven.graph.DefaultGraphBuilder.getProjectsForMavenReactor(DefaultGraphBuilder.java:391)
at org.apache.maven.graph.DefaultGraphBuilder.build(DefaultGraphBuilder.java:78)
at org.apache.maven.DefaultMaven.buildGraph(DefaultMaven.java:511)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:221)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
"VM Thread" os_prio=0 tid=0x00007fc4a42c0000 nid=0x25 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fc4a4190800 nid=0x21 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fc4a4192000 nid=0x22 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fc4a4194000 nid=0x23 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fc4a4195800 nid=0x24 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fc4a4382800 nid=0x2d waiting on condition
JNI global references: 235
bash-4.3$
Inside the container I have just confirmed that I can access the pom and I show the debug output, which just hangs at downloading.
https://gist.github.com/dwatrous/34e1edc1db5e4756d4b33c83a9c2ccd0
It is highly likely related to NFS, and file locking bugs and/or semantics.
Others have reported similar problems with FileChannel#tryLock over NFS; see e.g. JDK-8156026 and JDK-8065927.
The contract of that method says that tryLock does not block, so any blocking which occurs is due to a native system call not returning when it should. It is possible Maven could try to work around such bugs somehow, but I think any attempt to do so would be quite hacky and maybe introduce more bugs than it avoids.
You could try different versions of Java, including both Oracle and OpenJDK, at different release versions...
In our case the problem was definitely related to out NetApp ONTAP NFS and was caused by not having port 4045 for nlockmgr open in the firewall. Writing files worked fine, locking files before writing, like maven does for its cache did not.
Hard to debug because nothing timed out, the process just hung until the port was opened. Really curious why it does not time out if the port is closed.
Which Network File System (NFS) TCP and NFS UDP ports are used on the storage system?

How to work out how to resolve a performance issue using the threading tools in Mule?

I am using Anypoint Studio 6.1 and Mule 3.8.1 and have a workflow that is not performing as the thread monitoring looks like alot of the 250 max active threads are not being used even though the flow is being called by 250 concurrent users in an infinite loop from JMeter 3.1.
I have used tools like VisualVM and YourKit to investigate the problem and try to identify a solution. It looks like the majority of threads are parked or waiting but I'm not sure how you go about finding the optimal performance solution from the thread dumps and the analysis tools?
TEST XML
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns:metadata="http://www.mulesoft.org/schema/mule/metadata" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:json="http://www.mulesoft.org/schema/mule/json" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:mongo="http://www.mulesoft.org/schema/mule/mongo" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/json http://www.mulesoft.org/schema/mule/json/current/mule-json.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/mongo http://www.mulesoft.org/schema/mule/mongo/current/mule-mongo.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd">
<ee:object-store-caching-strategy name="Caching_Strategy" doc:name="Caching Strategy">
<managed-store storeName="cacheManagedObjectStore" maxEntries="1000" entryTTL="846000" expirationInterval="846000"/>
</ee:object-store-caching-strategy>
<http:listener-config name="HTTP_Listener_Config" host="0.0.0.0" port="8081" basePath="/api" doc:name="HTTP Listener Configuration">
<http:worker-threading-profile maxThreadsActive="250" maxThreadsIdle="2" threadTTL="10000" poolExhaustedAction="WAIT" threadWaitTimeout="10000" maxBufferSize="0"/>
</http:listener-config>
<flow name="testFlow" initialState="started">
<http:listener config-ref="HTTP_Listener_Config" path="/perf-test" allowedMethods="GET" doc:name="HTTP"/>
<set-payload value="#['{"EffectiveEndDateTime" : null}']" doc:name="Set Payload"/>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
<ee:cache cachingStrategy-ref="Caching_Strategy" doc:name="Cache">
<mongo:find-documents config-ref="Mongo_DB__Configuration" collection="${test.collection}" doc:name="Mongo DB"/>
<json:object-to-json-transformer encoding="UTF-8" mimeType="application/json" doc:name="Object to JSON"/>
</ee:cache>
<logger level="INFO" doc:name="Logger"/>
</flow>
</mule>
MONITORING GRAPHS
GARBAGE COLLECTION
THREADs
THREAD DUMP SNAPSHOT (FULL LOG TOO BIG)
2017-02-16 21:59:58
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):
"[test-project].HTTP_Listener_Config.worker.147" #263 daemon prio=5 os_prio=0 tid=0x000000002636e800 nid=0x31a0 waiting on condition [0x000000003efce000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da7ec940> (a java.util.concurrent.SynchronousQueue$TransferStack)
Locked ownable synchronizers:
- None
"[test-project].HTTP_Listener_Config.worker.91" #207 daemon prio=5 os_prio=0 tid=0x0000000027dfc000 nid=0x1988 waiting on condition [0x000000003b7ce000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da7ec940> (a java.util.concurrent.SynchronousQueue$TransferStack)
Locked ownable synchronizers:
- None
"RMI TCP Connection(3)-192.168.0.16" #200 daemon prio=5 os_prio=0 tid=0x0000000027df9000 nid=0x4668 runnable [0x000000003afcd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000006d885d660> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
Locked ownable synchronizers:
- <0x00000006d885e008> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-1-thread-1" #153 daemon prio=5 os_prio=0 tid=0x0000000027bb6000 nid=0x5524 waiting on condition [0x0000000035b2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da8578e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- None
"[test-project].HTTP_Listener_Config.worker.38" #152 daemon prio=5 os_prio=0 tid=0x0000000027bb8000 nid=0x2988 waiting on condition [0x0000000035a2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da7ec940> (a java.util.concurrent.SynchronousQueue$TransferStack)
Locked ownable synchronizers:
- None
"cluster-ClusterId{value='58a620afa3cb7258cc6cd828', description='null'}-test-project-shard-00-02.mongodb.net:27017" #151 daemon prio=5 os_prio=0 tid=0x0000000027bb9800 nid=0x33c0 waiting on condition [0x000000003495e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006daa03680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
- locked <0x00000006daa03650> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"cluster-ClusterId{value='58a620afa3cb7258cc6cd828', description='null'}-test-project-shard-00-01.mongodb.net:27017" #150 daemon prio=5 os_prio=0 tid=0x0000000027bbc800 nid=0x208c waiting on condition [0x000000003485f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da856910> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
- locked <0x00000006da8568e0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"[test-project].HTTP_Listener_Config.worker.37" #149 daemon prio=5 os_prio=0 tid=0x0000000027bbb000 nid=0x3668 waiting on condition [0x000000003475f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da7ec940> (a java.util.concurrent.SynchronousQueue$TransferStack)
Locked ownable synchronizers:
- None
"cluster-ClusterId{value='58a620afa3cb7258cc6cd828', description='null'}-test-project-shard-00-00.mongodb.net:27017" #148 daemon prio=5 os_prio=0 tid=0x0000000027bbc000 nid=0x2844 waiting on condition [0x000000003465e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006daacb940> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
- locked <0x00000006daacb910> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"[test-project].HTTP_Listener_Config.worker.34" #145 daemon prio=5 os_prio=0 tid=0x0000000027bb9000 nid=0x274 waiting on condition [0x000000003435f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da7ec940> (a java.util.concurrent.SynchronousQueue$TransferStack)
Locked ownable synchronizers:
- None
"[test-project].HTTP_Listener_Configuration.worker.02" #111 daemon prio=5 os_prio=0 tid=0x000000001d6f3000 nid=0x24a8 waiting on condition [0x0000000030d8e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da86d580> (a java.util.concurrent.SynchronousQueue$TransferStack)
Locked ownable synchronizers:
- None
"[test-project].HTTP_Listener_Configuration.worker.01" #110 daemon prio=5 os_prio=0 tid=0x00000000261e7000 nid=0x4ee0 waiting on condition [0x0000000030c8e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006da86d580> (a java.util.concurrent.SynchronousQueue$TransferStack)
Locked ownable synchronizers:
- None
"DestroyJavaVM" #109 prio=5 os_prio=0 tid=0x00000000261e5800 nid=0x7330 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Mule.app.deployer.monitor.1.thread.1" #108 prio=1 os_prio=-2 tid=0x00000000261e5000 nid=0x3d24 waiting on condition [0x000000003098f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006daacc138> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- None
"[test-project].processing.time.monitor" #107 daemon prio=5 os_prio=0 tid=0x00000000261e6800 nid=0x6268 in Object.wait() [0x000000002f6bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006d7a6bc10> (a java.lang.ref.ReferenceQueue$Lock)
Locked ownable synchronizers:
- None
"[test-project].testFlow.stage1.01" #104 prio=5 os_prio=0 tid=0x00000000261dc800 nid=0x2fb4 in Object.wait() [0x000000002f3be000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mule.util.queue.AbstractQueueStoreDelegate.poll(AbstractQueueStoreDelegate.java:81)
- locked <0x00000006daa523b8> (a org.mule.util.queue.DefaultQueueStoreDelegate)
Locked ownable synchronizers:
- <0x00000006da6616e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Thread-8" #57 daemon prio=5 os_prio=0 tid=0x0000000023cec800 nid=0x30c0 waiting on condition [0x000000002c4be000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d9b219f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- None
"pool-7-thread-6" #56 daemon prio=5 os_prio=0 tid=0x000000001eb5f800 nid=0x2914 waiting on condition [0x000000002bcbf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d88e7270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- None
"[test-project].http.listener.HttpIdleConnectionCloser.01" #50 prio=5 os_prio=0 tid=0x00000000241cd000 nid=0x2b50 in Object.wait() [0x000000002aa2f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:172)
- locked <0x00000006d8bdac18> (a java.lang.Object)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Locked ownable synchronizers:
- <0x00000006d8bbf6c0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"[test-project].http.listener.worker(8)" #49 daemon prio=5 os_prio=0 tid=0x00000000241d3800 nid=0xb34 waiting on condition [0x000000002a92f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d8bcf9a8> (a java.util.concurrent.LinkedTransferQueue)
Locked ownable synchronizers:
- None
"[test-project].http.listener.worker(7)" #48 daemon prio=5 os_prio=0 tid=0x00000000241d3000 nid=0x67d4 waiting on condition [0x000000002a82e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d8bcf9a8> (a java.util.concurrent.LinkedTransferQueue)
Locked ownable synchronizers:
- None
"[test-project].http.listener(5) SelectorRunner" #41 daemon prio=5 os_prio=0 tid=0x00000000241cf000 nid=0x4288 runnable [0x000000002a12e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.$$YJP$$poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(WindowsSelectorImpl.java)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006da0f16c0> (a sun.nio.ch.Util$3)
- locked <0x00000006da0f16d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006d8a843c0> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:115)
Locked ownable synchronizers:
- None
"[test-project].http.listener(4) SelectorRunner" #40 daemon prio=5 os_prio=0 tid=0x000000001efaa000 nid=0x2bb0 runnable [0x000000002a02f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.$$YJP$$poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(WindowsSelectorImpl.java)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006d8bdac48> (a sun.nio.ch.Util$3)
- locked <0x00000006d8bdac60> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006d8770f40> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
Locked ownable synchronizers:
- None
"Log4j2-AsyncLoggerConfig-4" #36 daemon prio=5 os_prio=0 tid=0x000000001daee000 nid=0x6afc waiting on condition [0x0000000026f1e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d9a27c00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- <0x00000006d6fa29f8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"[test-project].Mule.01" #35 prio=5 os_prio=0 tid=0x000000001daec800 nid=0x114c waiting on condition [0x000000002544e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d808bcc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- <0x00000006d80a0ae8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Log4j2-AsyncLoggerConfig-3" #34 daemon prio=5 os_prio=0 tid=0x000000001daeb800 nid=0x6274 runnable [0x0000000024b4e000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
- locked <0x00000006d697bce0> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:480)
- locked <0x00000006d68e2f20> (a java.io.PrintStream)
at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.java:50)
at org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:123)
- locked <0x00000006d6c8f7f0> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
at org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:136)
Locked ownable synchronizers:
- <0x00000006d80a0b18> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Log4j2-AsyncLoggerConfig-2" #28 daemon prio=5 os_prio=0 tid=0x000000001daef800 nid=0x6eec waiting on condition [0x000000002327f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d7a78c00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- <0x00000006d6e6d4a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"agw-policy-watcher" #27 prio=5 os_prio=0 tid=0x000000001daf1000 nid=0x1870 waiting on condition [0x000000002317f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.commons.io.monitor.FileAlterationMonitor.run(FileAlterationMonitor.java:188)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"MapDB writer #2" #24 daemon prio=5 os_prio=0 tid=0x000000001daf0000 nid=0x6984 waiting on condition [0x000000002307f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at org.mapdb.AsyncWriteEngine$WriterRunnable.run(AsyncWriteEngine.java:165)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"MapDB writer #1" #22 daemon prio=5 os_prio=0 tid=0x000000001daee800 nid=0x6098 waiting on condition [0x0000000022f7f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at org.mapdb.AsyncWriteEngine$WriterRunnable.run(AsyncWriteEngine.java:165)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"JMX server connection timeout 21" #21 daemon prio=5 os_prio=0 tid=0x000000001e58f800 nid=0x3d50 in Object.wait() [0x0000000022e7f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
- locked <0x00000006d6e7e948> (a [I)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" #20 daemon prio=5 os_prio=0 tid=0x000000001dba8800 nid=0x3590 waiting on condition [0x0000000022d7e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d68e1878> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- None
"Log4j2-AsyncLoggerConfig-1" #19 daemon prio=5 os_prio=0 tid=0x000000001d920000 nid=0x334 waiting on condition [0x0000000022c7f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d68e99c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Locked ownable synchronizers:
- <0x00000006d68f47e0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Connection(2)-192.168.0.16" #18 daemon prio=5 os_prio=0 tid=0x000000001da4a000 nid=0x4eb8 runnable [0x0000000022b7d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
- locked <0x00000006daa99310> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
Locked ownable synchronizers:
- <0x00000006d68e9ba0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Accept-0" #17 daemon prio=5 os_prio=0 tid=0x000000001d793000 nid=0x273c runnable [0x000000002297e000]
java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x00000006d68eded0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
Locked ownable synchronizers:
- None
"Service Thread" #13 daemon prio=9 os_prio=0 tid=0x000000001b40e800 nid=0x1478 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread2" #12 daemon prio=9 os_prio=2 tid=0x000000001bdcc800 nid=0x6d1c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #11 daemon prio=9 os_prio=2 tid=0x000000001bdcb800 nid=0x1448 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #10 daemon prio=9 os_prio=2 tid=0x000000001b40a000 nid=0x4144 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"VM JFR Buffer Thread" #9 daemon prio=5 os_prio=0 tid=0x000000001b3fc000 nid=0x5e30 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JFR request timer" #7 daemon prio=5 os_prio=0 tid=0x0000000021824000 nid=0x135c in Object.wait() [0x000000002207e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006d68e18c0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000006d68e18c0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Attach Listener" #6 daemon prio=5 os_prio=2 tid=0x000000001b2b6800 nid=0x1008 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #5 daemon prio=9 os_prio=2 tid=0x000000001b2b5000 nid=0x2da8 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x000000001b2be000 nid=0xdd8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000000021474800 nid=0x6fa0 in Object.wait() [0x0000000021c7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006d68e18d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:193)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x000000002144d800 nid=0x6b30 in Object.wait() [0x0000000021b7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006d68e99f0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=2 tid=0x0000000021441000 nid=0x5ad8 runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=2 tid=0x0000000001875800 nid=0x401c runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=2 tid=0x0000000001876000 nid=0x6060 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=2 tid=0x0000000001877800 nid=0x1fc runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=2 tid=0x0000000001879000 nid=0x4a5c runnable
"Concurrent Mark-Sweep GC Thread" os_prio=2 tid=0x00000000018bd800 nid=0x17b8 runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000001be76800 nid=0x4dd0 waiting on condition
JNI global references: 17658
The problem description doesn't seem to be correct. You mentioned "looks like alot of the 250 max active threads are not being used". Is that a problem? If the processing is efficient enough to answer all requests with less resource usage than the maximum allowed, then why would that be an issue? You'll need to clarify if you are seeing some actual problem related to that.
Thread dumps are mostly used to find thread locking issues, or threads delayed significantly in some task. For performance issues it would be better to use a profiler, like YourKit or the more limited one in VisualVM. Again, you need to define the problem more clearly to see what is the correct tool to use.

IntelliJ Idea 2016.2 starting very slow

My IntelliJ Idea 2016.2 is starting very slow, it takes about 5 minutes to display "Choose your project". It used to load quickly, but suddenly in a day, it started to load slowly. It's not just IntelliJ Idea that is being affected, but also Android Studio is starting slow and gradle run on a gradle project starts slowly.
But after the application has loaded, it runs well, except that it freezes from time to time for a few seconds.
According to an IntelliJ help website, I tried adding
-agentlib:yjpagent=delay=10000
at the end of the idea.vmoptions file under ~/Library/Preferences/<product>, but this doesn't work at all, in fact, IntelliJ doesn't even start this way.
Device: This is on a Mac with OS X El Capitan 10.11.4.
Thread dump:
2016-11-22 10:01:37.716 idea[1275:29621] allVms required 1.8*,1.8+
2016-11-22 10:01:37.723 idea[1275:29627] fullFileName is: /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2016-11-22 10:01:37.723 idea[1275:29627] fullFileName exists: /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2016-11-22 10:01:37.723 idea[1275:29627] Value of IDEA_VM_OPTIONS is (null)
2016-11-22 10:01:37.724 idea[1275:29627] Processing VMOptions file at /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2016-11-22 10:01:37.724 idea[1275:29627] Done
^\2016-11-22 10:01:40
Full thread dump OpenJDK 64-Bit Server VM (25.112-b2 mixed mode):
"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007fc0b912f800 nid=0x5203 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #8 daemon prio=9 os_prio=31 tid=0x00007fc0ba08d800 nid=0x5003 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=31 tid=0x00007fc0ba085000 nid=0x4e03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=31 tid=0x00007fc0ba014800 nid=0x4c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #5 daemon prio=9 os_prio=31 tid=0x00007fc0ba000000 nid=0x4a03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=31 tid=0x00007fc0b90f1800 nid=0x4607 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fc0b90da800 nid=0x4403 in Object.wait() [0x00007000011b4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000791208e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000791208e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fc0b90d7800 nid=0x4203 in Object.wait() [0x00007000010b1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000791206b40> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000791206b40> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=31 tid=0x00007fc0b9808000 nid=0x2703 runnable [0x0000700000999000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:219)
at java.util.zip.ZipFile.<init>(ZipFile.java:149)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:103)
at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:893)
at sun.misc.URLClassPath$JarLoader.access$700(URLClassPath.java:756)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:838)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:831)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:830)
at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:803)
at sun.misc.URLClassPath$3.run(URLClassPath.java:530)
at sun.misc.URLClassPath$3.run(URLClassPath.java:520)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:492)
- locked <0x000000079124ed78> (a sun.misc.URLClassPath)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
- locked <0x000000079124ed78> (a sun.misc.URLClassPath)
at sun.misc.URLClassPath.getResource(URLClassPath.java:211)
at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked <0x0000000791256058> (a java.lang.Object)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x0000000791255fd0> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
"VM Thread" os_prio=31 tid=0x00007fc0b9016000 nid=0x4003 runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0ba010800 nid=0x3603 runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0ba011800 nid=0x3803 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0b9012000 nid=0x3a03 runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0b9013000 nid=0x3c03 runnable
"Concurrent Mark-Sweep GC Thread" os_prio=31 tid=0x00007fc0b9014000 nid=0x3e03 runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007fc0ba084000 nid=0x5403 waiting on condition
JNI global references: 3
Heap
par new generation total 39296K, used 1397K [0x0000000791200000, 0x0000000793ca0000, 0x00000007a0c00000)
eden space 34944K, 4% used [0x0000000791200000, 0x000000079135d780, 0x0000000793420000)
from space 4352K, 0% used [0x0000000793420000, 0x0000000793420000, 0x0000000793860000)
to space 4352K, 0% used [0x0000000793860000, 0x0000000793860000, 0x0000000793ca0000)
concurrent mark-sweep generation total 87424K, used 0K [0x00000007a0c00000, 0x00000007a6160000, 0x00000007c0000000)
Metaspace used 2297K, capacity 4480K, committed 4480K, reserved 1056768K
class space used 247K, capacity 384K, committed 384K, reserved 1048576K
Update 1
It turns out, that running a Processing sketch takes longer than it used to. It takes some time for the sketch to run after I click on Run Sketch.
All I'm looking for is a solution to this problem, on how to fix my IntelliJ Idea, how to make it run normally (quickly) and without any freezes. Any help is appreciated.

Does the OpenJDK JVM parallelize bytecode?

I've implemented an algorithm using single-threaded Java code. When I run my program using JIT compilation enabled it saturates all 8 cores on my machine. When I run the same program using the -Xint JVM option to disable JIT compilation it runs on a single core as expected.
This is my Java version info:
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.10.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
Why does it seem like my code gets parallelized and where can I find more information on when HotSpot can parallelize code?
Java doesn't auto parallelize code, my guess is that the core saturation you are seeing is the JIT compiling your code. Make your test program input larger so it runs longer(maybe a 2-3 min.) and see if it tails off after a while.
It doesn't automatically parallize your code directly, but it uses much more of the machines resources to make the code run more quickly. It is profiling, compiling, garbage collecting and constantly recompiling based on runtime data. Whenever possible these operations will be done on other CPUs.
It might decide you call a method enough with the same parameter that it completley inlines the result for that parameter, or it might optimize out quite a few if statements in a given method if they are never taken, resulting to the original if you have different parameters. It doesn't want these operations to slow down/block your program, so it does them on different threads.
I'd guess if you ran long enough though you'd see it go back to filling a single cpu.
The answers given above are by and large correct. I'm just answering to complete the picture. Obviously JVM does not auto prallelize the user code and it has it's own threads going on. These threads vary in number, but generally We have following threads. This snapshot of thread stack is taken at the start of the live phase of the hotspot (OpenJDK) JVM. We can see 11 threads here which can easily occupy all the cores of an octacore or a quadcore machine. The thread names also explain their purposes.
Full thread dump OpenJDK 64-Bit Server VM (25.71-b00-debug mixed mode):
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007ffff0160800 nid=0x2f91 runnable [0x0000000000000000]
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ffff0158800 nid=0x2f90 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ffff0156000 nid=0x2f8f waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ffff0153000 nid=0x2f8e waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ffff0150800 nid=0x2f8d waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ffff014e800 nid=0x2f8c runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ffff0103800 nid=0x2f8b in Object.wait() [0x00007ffff412f000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ffff00fa000 nid=0x2f8a in Object.wait() [0x00007ffff4230000]
"main" #1 prio=5 os_prio=0 tid=0x00007ffff000d000 nid=0x2f86 runnable [0x00007ffff7fca000]
"VM Thread" os_prio=0 tid=0x00007ffff00ef800 nid=0x2f89 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007ffff015f000 nid=0x2f92 waiting on condition
Similarly, following snapshot of the thread stack shows Garbage Collector in action. This snap was taken after GC invocation. There are 8 threads alone for the GC since I'm running a parallel implementation of GC (I guess GC threads are equal to the number of cores because I'm testing on an octacore machine).
Full thread dump OpenJDK 64-Bit Server VM (25.71-b00-debug mixed mode):
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007ffff017e800 nid=0xaa1 runnable (no locks) [0x0000000000000000]
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ffff016e800 nid=0xaa0 waiting on condition (no locks) [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ffff016b800 nid=0xa9f waiting on condition (no locks) [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ffff0169800 nid=0xa9e waiting on condition (no locks) [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ffff0166000 nid=0xa9d waiting on condition (no locks) [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ffff0164800 nid=0xa9c runnable (no locks) [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ffff0119000 nid=0xa9b in Object.wait() (no locks) [0x00007fffba33d000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ffff0110000 nid=0xa9a in Object.wait() (no locks) [0x00007fffba43e000]
"main" #1 prio=5 os_prio=0 tid=0x00007ffff000d000 nid=0xa8b runnable (no locks) [0x00007ffff7fc9000]
"VM Thread" os_prio=0 tid=0x00007ffff0105000 nid=0xa99 runnable (no locks)
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ffff0026800 nid=0xa91 runnable (no locks)
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ffff0028800 nid=0xa92 runnable (no locks)
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007ffff002a800 nid=0xa93 runnable (no locks)
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007ffff002c800 nid=0xa94 runnable (no locks)
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007ffff002e800 nid=0xa95 runnable (no locks)
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007ffff0030800 nid=0xa96 runnable (no locks)
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007ffff0032800 nid=0xa97 runnable (no locks)
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007ffff0034800 nid=0xa98 runnable (no locks)
"VM Periodic Task Thread" os_prio=0 tid=0x00007ffff017f800 nid=0xaa2 waiting on condition (no locks)
Conclusion
Since number of threads vary during runtime and many threads are spawned and killed on the fly, you would almost never see a single core in use by a java program no matter how long it runs.
The above output was generated using GDB and an internal debug API of openJDK. If anybody is interested in knowing more about these threads and their purpose, you can refer to: Thread Management in openJDK

Resources