I am trying to use LibreOffice as a frontend on the Drill DB engine, through JDBC interface. Currently I am working in embedded mode, then I will switch to distributed mode.
Drill is working fine in CLI or GUI interface.
I am using Oracle JRE 1.7
I have registered the Java class org.apache.drill.jdbc.Driver, and it is found correctly.
But as soon as I try the connection the answer is always
"java.lang.NullPointerException".
"Reference Handler" daemon prio=10 tid=0x00007fc64c140800 nid=0x916 in Object.wait() [0x00007fc63f2f1000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000d7704470> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000000d7704470> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00007fc64c13c000 nid=0x915 runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fc64c02b000 nid=0x911 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fc64c02d000 nid=0x912 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fc64c02f000 nid=0x913 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fc64c030800 nid=0x914 runnable "VM Periodic Task Thread" prio=10 tid=0x00007fc64c17c800 nid=0x91d waiting on condition JNI global references: 145
Same JDBC interface seems to work in Squirrel, and it opens correctly.
Has anybody managed to work it out?
An help would be highly appreciated.
Let me add some clue (thanks Mark, thanks Jim).
I am just using LibreOffice Base as an application, as it comes out from Ubuntu Desktop.
Here's some detail :
LibreOffice : Versione: 4.2.8.2
Build ID: 420m0(Build:2)
riccardo#coccodrill:~$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
riccardo#coccodrill:~$ uname -a
Linux coccodrill 3.19.0-30-generic #34~14.04.1-Ubuntu SMP Fri Oct 2 22:09:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Mark, sorry, I don't know how to retrieve the stacktrace of the NullPointer Exception, since I am not writing an app, so I have no development environment around me. Sort of working blind, I suppose.
I think it's not compatible. I tried to do the same, but with newer versions (Apache Drill 1.7 and LibreOffice 5.1), and it showed this error: Running Drill in embedded mode using Drill's jdbc-all JDBC driver Jar file alone is not supported.
But I did get it working with ODBC driver instead.
Similar issue here: Apache Drill - connection to Drill in Embedded Mode [java]
Related
Currently our system (Client) on AWS EC2 instance encountered a sftp problem when trying to send a large file to a customer remote sftp server (“160.xxx.xxx.35.bc.googleusercontent.com). For the small sized files, the sftp transfer works fine but when the file size is about or larger than 1GB, we found that only 1068392448 bytes were transferred to Server sftp site. However, when we send the same large file with the same code and same environments to our own non-googleusercontent remote sftp server (with only the URL/UserName/Password differ), it was successful and all data transferred correctly.
This problem happened after the customer server side made some changes by adding the Load Balancer process. The customer server side did some investigation, adjusted the load balancer timeout but it did not help to resolve this issue. It is said the Client side stopped the data transfer after 1068392448 bytes and Server side waited and dropped the connection after the allowed idle time (~50second).
Our investigation noticed that the large source file was read from AWS S3 and saved to local correctly. When the large file data is written to Server sftp site reach 1068392448 bytes (consistently for all test results), the TCP socket connection status changed from ESTABLISHED to CLOSE_WAIT after Server allowed idle time (about ~50s). The process stayed in this status forever until it is manually stopped/killed. When the TCP socket connection is in CLOSE_WAIT status, the data transfer process showed in dump is in a wait loop at awaitSpace() method (in PipedInputStream class of java.io package). The buffer is indicated to be full and waiting to be written to the Server side. Below is the codes of the wait loop:
private void checkStateForReceive() throws IOException {
if (!connected) {
throw new IOException("Pipe not connected");
} else if (closedByWriter || closedByReader) {
throw new IOException("Pipe closed");
} else if (readSide != null && !readSide.isAlive()) {
throw new IOException("Read end dead");
}
}
private void awaitSpace() throws IOException {
while (in == out) {
checkStateForReceive();
/* full: kick any waiting readers */
notifyAll();
try {
wait(1000);
} catch (InterruptedException ex) {
throw new java.io.InterruptedIOException();
}
}
}
Below is the dump:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode):
"Connect thread files.liveramp.com session" #13 daemon prio=5 os_prio=0 tid=0x000000001e711800 nid=0x63c0 in Object.wait() [0x000000001f81f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273)
at java.io.PipedInputStream.receive(PipedInputStream.java:231)
- locked <0x00000006c1c944c8> (a com.jcraft.jsch.Channel$MyPipedInputStream)
at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
at com.jcraft.jsch.IO.put(IO.java:64)
at com.jcraft.jsch.Channel.write(Channel.java:438)
at com.jcraft.jsch.Session.run(Session.java:1459)
at java.lang.Thread.run(Thread.java:745)
"org.apache.commons.vfs2.cache.SoftRefFilesCache$SoftRefReleaseThread" #11 daemon prio=5 os_prio=0 tid=0x000000001ea2d000 nid=0x3078 in Object.wait() [0x000000001efcf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c1c24710> (a java.lang.ref.ReferenceQueue$Lock)
at org.apache.commons.vfs2.cache.SoftRefFilesCache$SoftRefReleaseThread.run(SoftRefFilesCache.java:74)
"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000000001dfc5000 nid=0x1944 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #9 daemon prio=9 os_prio=2 tid=0x000000001df12800 nid=0x6848 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #8 daemon prio=9 os_prio=2 tid=0x000000001df07800 nid=0x5720 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #7 daemon prio=9 os_prio=2 tid=0x000000001df04800 nid=0x6358 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Monitor Ctrl-Break" #6 daemon prio=5 os_prio=0 tid=0x000000001df56000 nid=0x6910 runnable [0x000000001e1cf000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x00000006c1c24f88> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x00000006c1c24f88> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001c69e800 nid=0x4880 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001c69d800 nid=0x2534 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000001c67e800 nid=0x4908 in Object.wait() [0x000000001d9df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c1c26660> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c1c26660> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000030b5000 nid=0x6340 in Object.wait() [0x000000001d8df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c1c26818> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c1c26818> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x0000000002fc4000 nid=0x56dc in Object.wait() [0x0000000002e0f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.jcraft.jsch.Session.write(Session.java:1269)
- locked <0x00000006c1c943d0> (a com.jcraft.jsch.ChannelSftp)
at com.jcraft.jsch.ChannelSftp.sendWRITE(ChannelSftp.java:2646)
at com.jcraft.jsch.ChannelSftp.access$100(ChannelSftp.java:36)
at com.jcraft.jsch.ChannelSftp$1.write(ChannelSftp.java:806)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
- eliminated <0x00000006c1c27a88> (a org.apache.commons.vfs2.provider.sftp.SftpFileObject$SftpOutputStream)
at org.apache.commons.vfs2.util.MonitorOutputStream.write(MonitorOutputStream.java:123)
- locked <0x00000006c1c27a88> (a org.apache.commons.vfs2.provider.sftp.SftpFileObject$SftpOutputStream)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
- eliminated <0x00000006c1c26a50> (a org.apache.commons.vfs2.provider.DefaultFileContent$FileContentOutputStream)
at org.apache.commons.vfs2.util.MonitorOutputStream.write(MonitorOutputStream.java:123)
- locked <0x00000006c1c26a50> (a org.apache.commons.vfs2.provider.DefaultFileContent$FileContentOutputStream)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:805)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:784)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:755)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:771)
at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:37)
at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(AbstractFileObject.java:295)
at com.merkleinc.dat.sftptran.cli.SftpVfs2App.main(SftpVfs2App.java:88)
"VM Thread" os_prio=2 tid=0x000000001c657000 nid=0x28b4 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000002fda800 nid=0x2b54 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000002fdc000 nid=0x4528 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002fdd800 nid=0x4850 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002fdf800 nid=0x7f0 runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000001e023000 nid=0x5d58 waiting on condition
JNI global references: 225
Heap
PSYoungGen total 49664K, used 14491K [0x000000076b400000, 0x000000076ed00000, 0x00000007c0000000)
eden space 49152K, 29% used [0x000000076b400000,0x000000076c21efd0,0x000000076e400000)
from space 512K, 6% used [0x000000076e500000,0x000000076e508000,0x000000076e580000)
to space 4608K, 0% used [0x000000076e880000,0x000000076e880000,0x000000076ed00000)
ParOldGen total 175104K, used 2879K [0x00000006c1c00000, 0x00000006cc700000, 0x000000076b400000)
object space 175104K, 1% used [0x00000006c1c00000,0x00000006c1ecfc70,0x00000006cc700000)
Metaspace used 11230K, capacity 11448K, committed 11648K, reserved 1058816K
class space used 1265K, capacity 1318K, committed 1408K, reserved 1048576K
Currently, we are using com.github’s vfs-s3 Version 2.4.2 with related apache.commons vfs2 version 2.1. and com.jcraft’s jsch version 0.1.54 for sftp data transfer. Our basic codes are as below:
import org.apache.commons.vfs2.*;
Public void upload(String sourceUri, String targetUri) throws IOException, URISyntaxException {
StandardFileSystemManager fsManager = new StandardFileSystemManager();
fsManager.init();
try (FileObject sourceFile = getFileObject(sourceUri, fsManager);
FileObject remoteFile = fsManager.resolveFile(createConnectionString(targetUri), createOptions())) {
// Copy s3 file to sftp server
remoteFile.copyFrom(sourceFile, Selectors.SELECT_SELF);
} finally {
fsManager.close();
}
};
We tried to test the same code in the local Window environments. Occasionally we could see the same problem symptoms as in ec2 instance but most time the problem is not there (inconsistent). We tested on other linux system to send the large file to the customer server sftp site and there was no problem.
We have tried to upgrade to com.github.abashev’s vfs-s3 Version 3.0.0 and 4.0.0 with corresponding versions of vfs2 on AWS EC2 instance. However, it reproduced the same problem results for the same large files. Tried to set loadOpenSSHConfig to be true with proper config file values to keep the connection alive, but it did not help for this situation.
We tried to test sftp directly by sftp "put" command with large file to the remote server from the EC2 instance (and other Windows/Linux platforms) and the large file data transfer is always successful.
The question is where is the potential root cause of the problem? Why the server side stopped receiving the data after 1068392448 bytes or why the rest of the data could not be sent to the server side? Any hard limits on our EC2 environments blocked the data transfer operation (we tried to check a few limits but not still not clear)? Or the Server site incorrectly sent the connection close “FIN” request to Client site (how to prove it)? Thanks for any suggestions for the potential solution.
I have show_sql to true in my properties because I want it to log the queries into my console when I am running our Spring Boot server app in IntelliJ. We do not have any config to write those to a file, but for some reason it is trying to write to a file. Here is what we see in a thread dump
"http-nio-8080-exec-2" #28 daemon prio=5 os_prio=31 tid=0x00007fa208deb000 nid=0x9403 runnable [0x000070000bef8000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
- locked <0x00000006c00430b8> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:480)
- locked <0x00000006c0043098> (a java.io.PrintStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
- locked <0x00000006c00431d8> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
at java.io.PrintStream.write(PrintStream.java:527)
- locked <0x00000006c0043098> (a java.io.PrintStream)
at java.io.PrintStream.print(PrintStream.java:669)
at java.io.PrintStream.println(PrintStream.java:806)
- locked <0x00000006c0043098> (a java.io.PrintStream)
at org.hibernate.engine.jdbc.spi.SqlStatementLogger.logStatement(SqlStatementLogger.java:111)
All the other logging of SQL is going to the console.
"http-nio-8080-exec-1" #29 daemon prio=5 os_prio=31 tid=0x00007fa200fef000 nid=0x110b waiting for monitor entry [0x0000700009b8c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.io.PrintStream.println(PrintStream.java:805)
- waiting to lock <0x00000006c0043098> (a java.io.PrintStream)
at org.hibernate.engine.jdbc.spi.SqlStatementLogger.logStatement(SqlStatementLogger.java:111)
at org.hibernate.engine.jdbc.spi.SqlStatementLogger.logStatement(SqlStatementLogger.java:94)
I have searched everywhere in our code base to try and find anything that might make it want to log to a file, even in the run configuration of IntelliJ, but I have not had any luck.
I want it to stop logging to a file, because it ends up hanging my IDE.
I have a maven build that I'm running in a docker container using the official maven image from the Docker hub. The .m2 directory is mounted to an NFS share.
This works in one environment, but in another identical environment it will always hang after writing the lock file. It never completes the download, but just hangs there forever. Since maven debug didn't give me any details after the point where it hung, I decided to watch the .m2 directory to see what happened.
ubuntu#kubernetes-dev-nfs-pv:/nfs-shares/jenkins/.m2$ inotifywait -m -r .
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
./ CREATE,ISDIR repository
./ OPEN,ISDIR repository
./ CLOSE_NOWRITE,CLOSE,ISDIR repository
./repository/ CREATE,ISDIR org
./repository/ OPEN,ISDIR org
./repository/ CLOSE_NOWRITE,CLOSE,ISDIR org
./repository/org/ CREATE,ISDIR springframework
./repository/org/ OPEN,ISDIR springframework
./repository/org/ CLOSE_NOWRITE,CLOSE,ISDIR springframework
./repository/org/springframework/ CREATE,ISDIR boot
./repository/org/springframework/ OPEN,ISDIR boot
./repository/org/springframework/ CLOSE_NOWRITE,CLOSE,ISDIR boot
./repository/org/springframework/boot/ CREATE,ISDIR spring-boot-starter-parent
./repository/org/springframework/boot/ OPEN,ISDIR spring-boot-starter-parent
./repository/org/springframework/boot/ CLOSE_NOWRITE,CLOSE,ISDIR spring-boot-starter-parent
./repository/org/springframework/boot/spring-boot-starter-parent/ CREATE,ISDIR 1.3.7.RELEASE
./repository/org/springframework/boot/spring-boot-starter-parent/ OPEN,ISDIR 1.3.7.RELEASE
./repository/org/springframework/boot/spring-boot-starter-parent/ CLOSE_NOWRITE,CLOSE,ISDIR 1.3.7.RELEASE
./repository/org/springframework/boot/spring-boot-starter-parent/1.3.7.RELEASE/ CREATE spring-boot-starter-parent-1.3.7.RELEASE.pom.part.lock
Maven seems to be working, it creates a number of folders and even the lock file, but then it hangs. How can I get maven to complete or find some additional information to help me solve this.
By the way, if I use ephemeral storage inside the container, it downloads the package as expected.
UDPATE: one of the comments suggested a thread dump. Below you can see I attached to the running container. I confirm that the container can modify files in the .m2 directory and then I use jstack to get a thread dump of the process.
ec2-user#murano-necrhj0ld3vkx4-kube-3-gm5dmsfnftxn:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c7d1f4c91559 maven:alpine "cat" About an hour ago Up About an hour agitated_cori
ec2-user#murano-necrhj0ld3vkx4-kube-3-gm5dmsfnftxn:~$ sudo docker exec -ti c7d1f4c91559 /bin/bash
bash-4.3$ ps
PID USER TIME COMMAND
1 1000 0:00 cat
6 1000 0:00 sh -c echo $$ > '/var/jenkins_home/workspace/api-product#tmp/durable-ca9825bd/pid'; jsc=durable-04ba6b757bca34373f180bd01ef64ca1; JENKINS_SERVER_COOKIE=$jsc '/var/jenkins_home/workspace/api-product#tmp/durable-ca
12 1000 0:00 {script.sh} /bin/sh -xe /var/jenkins_home/workspace/api-product#tmp/durable-ca9825bd/script.sh
13 1000 0:07 /usr/lib/jvm/java-1.8-openjdk/bin/java -classpath /usr/share/maven/boot/plexus-classworlds-2.5.2.jar -Dclassworlds.conf=/usr/share/maven/bin/m2.conf -Dmaven.home=/usr/share/maven -Dmaven.multiModuleProjectDirecto
1584 1000 0:00 /bin/bash
1589 1000 0:00 ps
bash-4.3$ cat /var/jenkins_home/workspace/api-product#tmp/durable-ca9825bd/script.sh
#!/bin/sh -xe
mvn -Dmaven.repo.local="$PWD"/../../.m2/repository clean compile
bash-4.3$ ls -la /var/jenkins_home/.m2/
total 16
drwxr-xr-x 3 1000 1000 4096 May 2 21:14 .
drwxrwxr-x 23 1000 1000 4096 May 3 11:55 ..
-rw-r--r-- 1 1000 1000 6 May 2 21:14 file.txt
drwxr-xr-x 3 1000 1000 4096 May 2 20:50 repository
bash-4.3$ cat /var/jenkins_home/.m2/file.txt
hello
bash-4.3$ vi /var/jenkins_home/.m2/file.txt
bash-4.3$ cat /var/jenkins_home/.m2/file.txt
hello
another
bash-4.3$ jstack 13
2017-05-03 13:04:37
Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode):
"Attach Listener" #11 daemon prio=9 os_prio=0 tid=0x00007fc4a4956800 nid=0x6a7 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fc4a4343000 nid=0x2c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fc4a4311800 nid=0x2b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fc4a4302000 nid=0x2a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fc4a42ff000 nid=0x29 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fc4a42fc800 nid=0x28 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fc4a42d5000 nid=0x27 in Object.wait() [0x00007fc48ba4b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dab108d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000dab108d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fc4a42ca800 nid=0x26 in Object.wait() [0x00007fc48bb4c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dab18178> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000dab18178> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007fc4a4179800 nid=0x20 runnable [0x00007fc4a3426000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115)
at org.eclipse.aether.connector.basic.PartialFile$LockFile.tryLock(PartialFile.java:135)
at org.eclipse.aether.connector.basic.PartialFile$LockFile.lock(PartialFile.java:80)
at org.eclipse.aether.connector.basic.PartialFile$LockFile.<init>(PartialFile.java:67)
at org.eclipse.aether.connector.basic.PartialFile$Factory.newInstance(PartialFile.java:219)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:441)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:359)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:76)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:590)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:258)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:529)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:430)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:255)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:232)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:303)
at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:193)
at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:243)
at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:1051)
at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:829)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:331)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:429)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:398)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:361)
at org.apache.maven.graph.DefaultGraphBuilder.collectProjects(DefaultGraphBuilder.java:400)
at org.apache.maven.graph.DefaultGraphBuilder.getProjectsForMavenReactor(DefaultGraphBuilder.java:391)
at org.apache.maven.graph.DefaultGraphBuilder.build(DefaultGraphBuilder.java:78)
at org.apache.maven.DefaultMaven.buildGraph(DefaultMaven.java:511)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:221)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
"VM Thread" os_prio=0 tid=0x00007fc4a42c0000 nid=0x25 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fc4a4190800 nid=0x21 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fc4a4192000 nid=0x22 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fc4a4194000 nid=0x23 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fc4a4195800 nid=0x24 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fc4a4382800 nid=0x2d waiting on condition
JNI global references: 235
bash-4.3$
Inside the container I have just confirmed that I can access the pom and I show the debug output, which just hangs at downloading.
https://gist.github.com/dwatrous/34e1edc1db5e4756d4b33c83a9c2ccd0
It is highly likely related to NFS, and file locking bugs and/or semantics.
Others have reported similar problems with FileChannel#tryLock over NFS; see e.g. JDK-8156026 and JDK-8065927.
The contract of that method says that tryLock does not block, so any blocking which occurs is due to a native system call not returning when it should. It is possible Maven could try to work around such bugs somehow, but I think any attempt to do so would be quite hacky and maybe introduce more bugs than it avoids.
You could try different versions of Java, including both Oracle and OpenJDK, at different release versions...
In our case the problem was definitely related to out NetApp ONTAP NFS and was caused by not having port 4045 for nlockmgr open in the firewall. Writing files worked fine, locking files before writing, like maven does for its cache did not.
Hard to debug because nothing timed out, the process just hung until the port was opened. Really curious why it does not time out if the port is closed.
Which Network File System (NFS) TCP and NFS UDP ports are used on the storage system?
My IntelliJ Idea 2016.2 is starting very slow, it takes about 5 minutes to display "Choose your project". It used to load quickly, but suddenly in a day, it started to load slowly. It's not just IntelliJ Idea that is being affected, but also Android Studio is starting slow and gradle run on a gradle project starts slowly.
But after the application has loaded, it runs well, except that it freezes from time to time for a few seconds.
According to an IntelliJ help website, I tried adding
-agentlib:yjpagent=delay=10000
at the end of the idea.vmoptions file under ~/Library/Preferences/<product>, but this doesn't work at all, in fact, IntelliJ doesn't even start this way.
Device: This is on a Mac with OS X El Capitan 10.11.4.
Thread dump:
2016-11-22 10:01:37.716 idea[1275:29621] allVms required 1.8*,1.8+
2016-11-22 10:01:37.723 idea[1275:29627] fullFileName is: /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2016-11-22 10:01:37.723 idea[1275:29627] fullFileName exists: /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2016-11-22 10:01:37.723 idea[1275:29627] Value of IDEA_VM_OPTIONS is (null)
2016-11-22 10:01:37.724 idea[1275:29627] Processing VMOptions file at /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2016-11-22 10:01:37.724 idea[1275:29627] Done
^\2016-11-22 10:01:40
Full thread dump OpenJDK 64-Bit Server VM (25.112-b2 mixed mode):
"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007fc0b912f800 nid=0x5203 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #8 daemon prio=9 os_prio=31 tid=0x00007fc0ba08d800 nid=0x5003 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=31 tid=0x00007fc0ba085000 nid=0x4e03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=31 tid=0x00007fc0ba014800 nid=0x4c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #5 daemon prio=9 os_prio=31 tid=0x00007fc0ba000000 nid=0x4a03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=31 tid=0x00007fc0b90f1800 nid=0x4607 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fc0b90da800 nid=0x4403 in Object.wait() [0x00007000011b4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000791208e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000791208e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fc0b90d7800 nid=0x4203 in Object.wait() [0x00007000010b1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000791206b40> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000791206b40> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=31 tid=0x00007fc0b9808000 nid=0x2703 runnable [0x0000700000999000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:219)
at java.util.zip.ZipFile.<init>(ZipFile.java:149)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:103)
at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:893)
at sun.misc.URLClassPath$JarLoader.access$700(URLClassPath.java:756)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:838)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:831)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:830)
at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:803)
at sun.misc.URLClassPath$3.run(URLClassPath.java:530)
at sun.misc.URLClassPath$3.run(URLClassPath.java:520)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:492)
- locked <0x000000079124ed78> (a sun.misc.URLClassPath)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
- locked <0x000000079124ed78> (a sun.misc.URLClassPath)
at sun.misc.URLClassPath.getResource(URLClassPath.java:211)
at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked <0x0000000791256058> (a java.lang.Object)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x0000000791255fd0> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
"VM Thread" os_prio=31 tid=0x00007fc0b9016000 nid=0x4003 runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0ba010800 nid=0x3603 runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0ba011800 nid=0x3803 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0b9012000 nid=0x3a03 runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=31 tid=0x00007fc0b9013000 nid=0x3c03 runnable
"Concurrent Mark-Sweep GC Thread" os_prio=31 tid=0x00007fc0b9014000 nid=0x3e03 runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007fc0ba084000 nid=0x5403 waiting on condition
JNI global references: 3
Heap
par new generation total 39296K, used 1397K [0x0000000791200000, 0x0000000793ca0000, 0x00000007a0c00000)
eden space 34944K, 4% used [0x0000000791200000, 0x000000079135d780, 0x0000000793420000)
from space 4352K, 0% used [0x0000000793420000, 0x0000000793420000, 0x0000000793860000)
to space 4352K, 0% used [0x0000000793860000, 0x0000000793860000, 0x0000000793ca0000)
concurrent mark-sweep generation total 87424K, used 0K [0x00000007a0c00000, 0x00000007a6160000, 0x00000007c0000000)
Metaspace used 2297K, capacity 4480K, committed 4480K, reserved 1056768K
class space used 247K, capacity 384K, committed 384K, reserved 1048576K
Update 1
It turns out, that running a Processing sketch takes longer than it used to. It takes some time for the sketch to run after I click on Run Sketch.
All I'm looking for is a solution to this problem, on how to fix my IntelliJ Idea, how to make it run normally (quickly) and without any freezes. Any help is appreciated.
I've implemented an algorithm using single-threaded Java code. When I run my program using JIT compilation enabled it saturates all 8 cores on my machine. When I run the same program using the -Xint JVM option to disable JIT compilation it runs on a single core as expected.
This is my Java version info:
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.10.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
Why does it seem like my code gets parallelized and where can I find more information on when HotSpot can parallelize code?
Java doesn't auto parallelize code, my guess is that the core saturation you are seeing is the JIT compiling your code. Make your test program input larger so it runs longer(maybe a 2-3 min.) and see if it tails off after a while.
It doesn't automatically parallize your code directly, but it uses much more of the machines resources to make the code run more quickly. It is profiling, compiling, garbage collecting and constantly recompiling based on runtime data. Whenever possible these operations will be done on other CPUs.
It might decide you call a method enough with the same parameter that it completley inlines the result for that parameter, or it might optimize out quite a few if statements in a given method if they are never taken, resulting to the original if you have different parameters. It doesn't want these operations to slow down/block your program, so it does them on different threads.
I'd guess if you ran long enough though you'd see it go back to filling a single cpu.
The answers given above are by and large correct. I'm just answering to complete the picture. Obviously JVM does not auto prallelize the user code and it has it's own threads going on. These threads vary in number, but generally We have following threads. This snapshot of thread stack is taken at the start of the live phase of the hotspot (OpenJDK) JVM. We can see 11 threads here which can easily occupy all the cores of an octacore or a quadcore machine. The thread names also explain their purposes.
Full thread dump OpenJDK 64-Bit Server VM (25.71-b00-debug mixed mode):
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007ffff0160800 nid=0x2f91 runnable [0x0000000000000000]
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ffff0158800 nid=0x2f90 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ffff0156000 nid=0x2f8f waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ffff0153000 nid=0x2f8e waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ffff0150800 nid=0x2f8d waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ffff014e800 nid=0x2f8c runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ffff0103800 nid=0x2f8b in Object.wait() [0x00007ffff412f000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ffff00fa000 nid=0x2f8a in Object.wait() [0x00007ffff4230000]
"main" #1 prio=5 os_prio=0 tid=0x00007ffff000d000 nid=0x2f86 runnable [0x00007ffff7fca000]
"VM Thread" os_prio=0 tid=0x00007ffff00ef800 nid=0x2f89 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007ffff015f000 nid=0x2f92 waiting on condition
Similarly, following snapshot of the thread stack shows Garbage Collector in action. This snap was taken after GC invocation. There are 8 threads alone for the GC since I'm running a parallel implementation of GC (I guess GC threads are equal to the number of cores because I'm testing on an octacore machine).
Full thread dump OpenJDK 64-Bit Server VM (25.71-b00-debug mixed mode):
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007ffff017e800 nid=0xaa1 runnable (no locks) [0x0000000000000000]
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ffff016e800 nid=0xaa0 waiting on condition (no locks) [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ffff016b800 nid=0xa9f waiting on condition (no locks) [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ffff0169800 nid=0xa9e waiting on condition (no locks) [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ffff0166000 nid=0xa9d waiting on condition (no locks) [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ffff0164800 nid=0xa9c runnable (no locks) [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ffff0119000 nid=0xa9b in Object.wait() (no locks) [0x00007fffba33d000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ffff0110000 nid=0xa9a in Object.wait() (no locks) [0x00007fffba43e000]
"main" #1 prio=5 os_prio=0 tid=0x00007ffff000d000 nid=0xa8b runnable (no locks) [0x00007ffff7fc9000]
"VM Thread" os_prio=0 tid=0x00007ffff0105000 nid=0xa99 runnable (no locks)
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ffff0026800 nid=0xa91 runnable (no locks)
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ffff0028800 nid=0xa92 runnable (no locks)
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007ffff002a800 nid=0xa93 runnable (no locks)
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007ffff002c800 nid=0xa94 runnable (no locks)
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007ffff002e800 nid=0xa95 runnable (no locks)
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007ffff0030800 nid=0xa96 runnable (no locks)
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007ffff0032800 nid=0xa97 runnable (no locks)
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007ffff0034800 nid=0xa98 runnable (no locks)
"VM Periodic Task Thread" os_prio=0 tid=0x00007ffff017f800 nid=0xaa2 waiting on condition (no locks)
Conclusion
Since number of threads vary during runtime and many threads are spawned and killed on the fly, you would almost never see a single core in use by a java program no matter how long it runs.
The above output was generated using GDB and an internal debug API of openJDK. If anybody is interested in knowing more about these threads and their purpose, you can refer to: Thread Management in openJDK