Differing behavior in maven multi-module projects when run in TeamCity vs locally - maven

Question

I have a maven multi-module project on TeamCity. I'm using TeamCity's built in maven 3.5 tooling.
In one of the child projects, in the section of its pom.xml I set "<target.env>dev</target.env>".
Later in the pom I use the properties-maven-plugin to load a file with the name "${target.env}.env.properties"
Locally if I run "mvn package -Dtarget.env=prod" in the parent project, the child project loads prod.env.properties as expected.
If I configure my teamcity build with param("system.target.env", "prod"), I can see "-Dtarget.env=prod" passed to the maven execution in the build log (where teamcity invokes the plexus-classworlds launcher to do so), the child project loads dev.env.properties, breaking the build.
Here's my questions:
Why does the behavior differ?
How do I reconcile this?
Update including some of the information #khmarbaise asked for:
The properties-maven-plugin is being used to load an environment specific set of properties based on which environment the application will run in.
It is set up to choose which file to load based on a system property, and a default value is set in the properties block to avoid having to constantly
add -Dtarget.env=dev during development. The properties-maven-plugin configuration for the child project is as follows:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<executions>
<execution>
<id>load-environment-properties</id>
<phase>validate</phase> <!-- Bound to validate phase to ensure it comes before loading of local.build.properties --> # No local.build.properties on TeamCity, so nothing clobbers this in practice
<goals>
<goal>read-project-properties</goal>
</goals>
<configuration>
<files>
<file>src/main/targetEnvironment/default.env.properties</file>
<file>src/main/targetEnvironment/${target.env}.env.properties</file>
</files>
</configuration>
</execution>
<execution>
<id>write-properties</id>
<phase>generate-resources</phase>
<goals>
<goal>write-project-properties</goal>
</goals>
<configuration>
<outputFile>${project.build.directory}/effective.build.properties</outputFile>
</configuration>
</execution>
</executions>
</plugin>
Version info:
Teamcity version: 2020.2.1 (build 85633)
Maven version: 3.3.9
Java version:
openjdk version "1.8.0_272"
OpenJDK Runtime Environment Corretto-8.272.10.3 (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM Corretto-8.272.10.3 (build 25.272-b10, mixed mode)
This is the line TeamCity is using to invoke maven, I've only included property definitions that seemed relevant since there were so many that are definitely irrelevant (build numbers, names, timestamps, and other TC specifc, maven agnostic, config)
/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/bin/java
-Dclassworlds.conf=/home/ec2-user/BuildAgent/temp/buildTmp/teamcity.m2.conf
-Dmaven.home=/home/ec2-user/BuildAgent/tools/maven3_3
-DskipTests=true
-Dteamcity.build.properties.file=/home/ec2-user/BuildAgent/temp/buildTmp/teamcity.build4380238471360533686.properties
-Dtarget.env=prod
-classpath /home/ec2-user/BuildAgent/tools/maven3_3/boot/plexus-classworlds-2.5.2.jar: org.codehaus.plexus.classworlds.launcher.Launcher -f /home/ec2-user/BuildAgent/work/4508a7116faa21f3/pom.xml -B clean package
The contents of teamcity.m2.conf is as follows:
main is org.apache.maven.cli.MavenCli from plexus.core
set maven.home default ${user.home}/m2
[plexus.core]
load ${teamcity.maven.watcher.home}/*.jar
optionally ${maven.home}/lib/ext/*.jar
load ${maven.home}/lib/*.jar
load ${maven.home}/conf/logging
teamcity.build4380238471360533686.properties contains many properties, the value of target.env within that file is 'prod' as expected

Answer 1

This ended up being a known bug in TeamCity
The underlying issue seems to be that TeamCity uses the MAVEN_OPTS environment variable to pass system properties into maven by default, but properties in MAVEN_OPTS are treated differently from properties passed as arguments to the maven command itself.
The workaround is that for any property "foo" that gets set in a POM section, that you want to override in a TeamCity build, you have to specify it in the "Additional Maven command line parameters" with -Dfoo=value, or, if you're setting the value in a system or build property within the TeamCity build -Dfoo=%system.foo%.