Spring boot maven plugin hangs - maven

Question

We use spring-boot 2.1.5 with spring-boot-maven-plugin to start the package for integration tests. Important parts of our pom look like this:
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.30.0</version>
<configuration>
<images>
<image>
<name>postgres:9.6.13</name>
<alias>app-it</alias>
<run>
<ports>
<port>app.datasource.port:5432</port>
</ports>
<wait>
<log>(?s)database system is ready to accept connections.</log>
<time>20000</time>
</wait>
</run>
</image>
</images
</configuration>
<executions>
<execution>
<id>prepare-it-database</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>remove-it-database</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<executable>true</executable>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
<execution>
<id>pre-integration-test</id>
<goals>
<goal>start</goal>
</goals>
<configuration>
<workingDirectory>${project.basedir}</workingDirectory>
<environmentVariables>
<app.datasource.port>${app.datasource.port}</app.datasource.port>
</environmentVariables>
<profiles>
<profile>integration</profile>
</profiles>
</configuration>
</execution>
<execution>
<id>post-integration-test</id>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>${frontend-maven-plugin.version}</version>
<executions>
<execution>
<id>run e2e tests</id>
<phase>integration-test</phase>
<goals>
<goal>npm</goal>
</goals>
<configuration>
<workingDirectory>../frontend</workingDirectory>
<arguments>run verify</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
Sometimes everything works fine. But sometimes the process just hangs. So I started it with
mvn -X clean verify
The result is rather strange as the spring-boot-maven plugin hangs and is checking the ready state of the spring application only once. All other attemps only follow after hitting Ctrl-C.
...
[DEBUG] Active profile(s): integration
[DEBUG] Application argument(s):
[DEBUG] Environment variable(s): app.datasource.port=16032
[DEBUG] Connecting to local MBeanServer at port 9001
[DEBUG] Waiting for spring application to start...
[DEBUG] MBean server at port 9001 is not up yet...
[DEBUG] Spring application is not ready yet, waiting 500ms (attempt 1)
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.5.RELEASE)
2019-06-12 15:34:06.482 INFO 113957 --- [ main] ...: Starting Application on kt101 with PID 113957 ...
...
2019-06-12 15:34:30.801 INFO 113957 --- [ main] ...: Started Application in 24.862 seconds (JVM running for 25.284)
The process hangs. Nothing happens. No more messages checking the application state (like attempt 2)
After hitting [Ctrl-C]
2019-06-12 16:05:02.258 INFO 113957 --- [ Thread-25] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService
[DEBUG] DOCKER> Closing LogWaitChecker callback
[DEBUG] MBean server at port 9001 is not up yet...
[DEBUG] Spring application is not ready yet, waiting 500ms (attempt 2)
[DEBUG] MBean server at port 9001 is not up yet...
[DEBUG] Spring application is not ready yet, waiting 500ms (attempt 3)
...
[DEBUG] MBean server at port 9001 is not up yet...
[DEBUG] Spring application is not ready yet, waiting 500ms (attempt 60)
...
[INFO] backend ................................... FAILURE [33:37 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:47 min
[INFO] Finished at: 2019-06-12T16:06:05+02:00
[INFO] Final Memory: 238M/6637M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.1.5.RELEASE:start (pre-integration-test) on project backend: Could not figure out if the application has started: Failed to connect to MBean server at port 9001: Spring application did not start before the configured timeout (60000ms -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.1.5.RELEASE:start (pre-integration-test) on project backend: Could not figure out if the application has started
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Could not figure out if the application has started
at org.springframework.boot.maven.StartMojo.waitForSpringApplication (StartMojo.java:198)
at org.springframework.boot.maven.StartMojo.runWithForkedJvm (StartMojo.java:97)
at org.springframework.boot.maven.AbstractRunMojo.doRunWithForkedJvm (AbstractRunMojo.java:284)
at org.springframework.boot.maven.AbstractRunMojo.run (AbstractRunMojo.java:245)
at org.springframework.boot.maven.AbstractRunMojo.execute (AbstractRunMojo.java:198)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to connect to MBean server at port 9001
at org.springframework.boot.maven.StartMojo.waitForForkedSpringApplication (StartMojo.java:224)
at org.springframework.boot.maven.StartMojo.waitForSpringApplication (StartMojo.java:187)
at org.springframework.boot.maven.StartMojo.runWithForkedJvm (StartMojo.java:97)
at org.springframework.boot.maven.AbstractRunMojo.doRunWithForkedJvm (AbstractRunMojo.java:284)
at org.springframework.boot.maven.AbstractRunMojo.run (AbstractRunMojo.java:245)
at org.springframework.boot.maven.AbstractRunMojo.execute (AbstractRunMojo.java:198)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Spring application did not start before the configured timeout (60000ms
at org.springframework.boot.maven.StartMojo.execute (StartMojo.java:277)
at org.springframework.boot.maven.StartMojo.waitForForkedSpringApplication (StartMojo.java:207)
at org.springframework.boot.maven.StartMojo.waitForSpringApplication (StartMojo.java:187)
at org.springframework.boot.maven.StartMojo.runWithForkedJvm (StartMojo.java:97)
at org.springframework.boot.maven.AbstractRunMojo.doRunWithForkedJvm (AbstractRunMojo.java:284)
at org.springframework.boot.maven.AbstractRunMojo.run (AbstractRunMojo.java:245)
at org.springframework.boot.maven.AbstractRunMojo.execute (AbstractRunMojo.java:198)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :backend
This is rather strange as the spring-boot-maven-plugin seems to hang in the line synchronized(this.lock) and can not enter the synchronized block.
https://github.com/spring-projects/spring-boot/blob/9b5cb4f9b86f258403419a92124a70eb3c099667/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin/src/main/java/org/springframework/boot/maven/StartMojo.java#L246
String message = "Spring application is not ready yet, waiting " + wait + "ms (attempt " + (i + 1) + ")";
getLog().debug(message);
synchronized (this.lock) {
try {
this.lock.wait(wait);
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while waiting for Spring Boot app to start.");
}
}
And even more strange: it happens one out of four times I execute this command.
I have two Java Processes running. The original maven process and my Spring Application. I took ThreadDumps of both.
Maven process:
https://pastebin.com/KCfZxDEa
Forked Process with Spring Application
https://pastebin.com/fn0SHjVW
The Spring Application runs fine by the way if I open localhost:8080
The maven plugin seems to hang in the JMXConnection:
"main" #1 prio=5 os_prio=0 tid=0x00007f28dc00b800 nid=0x1e2f runnable [0x00007f28e6865000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000006b0963158> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:903)
at org.springframework.boot.maven.SpringApplicationAdminClient.isReady(SpringApplicationAdminClient.java:63)
How can I fix this? Why does it happen sometimes and not always?