File.exists() sometimes wrong on Windows 10 with Java 8.191 - java-8

Question

I have a bunch of unit tests which contain code like:
File file = new File("src/main/java/com/pany/Foo.java");
assertTrue("Missing file: " + file.getAbsolutePath(), file.exists());
This test is suddenly failing when running it with Maven Surefire and -DforkCount=0. With -DforkCount=1, it works.
Things I tried so far:
The file does exist. Windows Explorer, command line (copy & paste), text editors, Cygwin can all find it and show the contents. That's why I think it's not a permission problem.
It's not modified by the unit tests or anything else. Git shows no modifications for the last two months.
I've checked the file system, it's clean.
I've tried other versions of Java 8, namely 8u171 and 8u181. Same problem.
I've run Maven from within Cygwin and the command prompt. Same result.
Reboot :-) No effect :-(
More details:
When I see this problem, I start to see the "The forked VM terminated without properly saying goodbye. VM crash or System.exit called?" in other projects. That's why I tried forkCount=0 which often helps in this case to find out why the forked VM crashed.
This has started recently, maybe around the October 2018 update of Windows 10. Before that, the builds were rock solid for about three years. My machine was switched to Windows 10 late 2017, I think.
I'm using Maven 3.6 and can't easily try an older version because of an important bug that was fixed with it. I did see the VM crash above with Maven 3.5.2 as well.
It's always the same files which fail (so it's stable).
ulimit (from Cygwin) says:
$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 2032
cpu time (seconds, -t) unlimited
max user processes (-u) 256
virtual memory (kbytes, -v) unlimited
I'm wondering if the "open files" limit of 256 only applied to Cygwin processes or whether that's something which Cygwin reads from Windows.
Let me know if you need anything else. I'm running out of ideas what I could try.
Update 1
Bernhard asked me to print absolute names. My answer was that I was already using absolute names but I was wrong. The actual code was:
File file = new File("src/main/java/com/pany/Foo.java");
if (!file.exists()) {
log.debug("Missing file {}", file.getAbsolutePath());
... fail ...
}
... do something with file...
I have now changed this to:
File file = new File("src/main/java/com/pany/Foo.java").getAbsoluteFile();
if (!file.exists()) {
log.debug("Missing file {}", file);
}
and that fixed the problem. I just can't understand why.
When Maven creates a forked VM to run the tests with Surefire, then it can change the current directory. So in this case, it would make sense that the tests work when forked but fail when running in the same VM (since the VM was created in the root folder of the multi-module build). But why is making the path absolute before the call to exists() fixing the issue?

Answer 1

Some background. Each process has a notion of "current directory". When started from the command line, then it's the directory in which the command was executed. When started from the UI, it's usually the folder in which the program (the .exe file) is.
In the command prompt or BASH, you can change this folder with cd for the process which runs the command prompt.
When Maven builds a multi-module project, it has to change this for each module (so that the relative path src/main/java/ always points to the right place). Unfortunately, Java doesn't have a "set current directory" method anywhere. You can only specify one when creating a new process and you can modify the system property user.dir.
That's why new File("a").exists() and new File("a").getAbsoluteFile().exists() work differently.
The latter will use new File(System.getProperty("user.dir"), "a") to determine the path and the former will use the Windows API function _wgetdcwd (docs) which in turn uses a field of the Windows process to get the current directory - in our case, that's always the folder in which Maven was originally started because Java doesn't update the field in the process when someone changes user.dir and Maven can only change this property to "simulate" changing folders.
WinNTFileSystem_md.c calls fileToNTPath(). That's defined in io_util_md.c and calls pathToNTPath(). For relative paths, it will call currentDirLength() which calls currentDir() which calls _wgetdcwd().
See also:
https://github.com/openjdk-mirror/jdk7u-jdk/blob/jdk7u6-b08/src/windows/native/java/io/WinNTFileSystem_md.c
https://github.com/openjdk-mirror/jdk7u-jdk/blob/jdk7u6-b08/src/windows/native/java/io/io_util_md.c
and here is the place where the Surefire plugin modifies the Property user.dir: https://github.com/apache/maven-surefire/blob/56d41b4c903b6c134c5e1a2891f9f08be7e5039f/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/AbstractSurefireMojo.java#L1060
When not forking, it's copied into the current VM's System properties: https://github.com/apache/maven-surefire/blob/56d41b4c903b6c134c5e1a2891f9f08be7e5039f/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/AbstractSurefireMojo.java#L1133

Answer 2

So I have checked via printing out the system properties with some simple tests.
During the tests via maven-surefire-plugin the user.dir will be changed to the root of the appropriate module in a multi module build.
But as I mentioned already there is a system property available basedir which can be used to correctly handle the location for tests which needs to access them via File...The basedir is pointed to the location of the pom.xml of the appropriate module.
But unfortunately the basedir property is not set by IDEA IntelliJ during the test run.
But this can be solved by a setup like this:
private String basedir;
#Before
public void before() {
this.basedir = System.getProperty("basedir", System.getProperty("user.dir", "Need to think about the default value"));
}
#Test
public void testX() {
File file = new File(this.basedir, "src/main/java/com/pany/Foo.java");
assertTrue("Missing file: " + file.getAbsolutePath(), file.exists());
}
This will work in Maven Surefire with -DforkCount=0 as well as -DforkCount=1 and in IDE's (checked only IDEA IntelliJ).
And yes that is an issue in Maven Surefire plugin changing the user.dir.
We might convince IDE's to support the basedir property as well ?

Answer 3

Aaron, we develop the Surefire. We can help you if you provide the path for this:
assertTrue("Missing file: " + file.getAbsolutePath(), file.exists());
Pls post the actual path, expected path and basedir where your POM resides.
The theory would not help here. We are testing all the spectrum of JDKs 7-12 but we do not have the combination Cygwin+Windows which must be considered.
The code setting user.dir in Surefire you mentioned exists a decade.