Dependency errors when running Apache Beam with Kafka - maven

Question

When I run my Apache Beam code using the Direct Runner I get the following error:
Caused by: java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.writeLog (LogContext.java:428)
at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.info (LogContext.java:382)
at org.apache.kafka.clients.consumer.KafkaConsumer.assign (KafkaConsumer.java:1123)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
Running it as follows works:
mvn package
java -cp target/myjar.jar \
com.CLASSNAME \
--runner=DirectRunner \
...
Running it as follows does not work:
mvn compile exec:java \
-Dexec.mainClass="com.CLASSNAME" \
-Dexec.args="..."
It seems this is related to slf4j dependency conflicts.
I've been looking at this for hours and do not seem to make any progress.
Things I've tried:
changing versions of libraries
putting the slf4j-api dependency on top
excluding dependencies
changing scopes
inspecting dependencies using mvn dependency:tree, but all looks ok
changing jdk versions
I have noted the following conflict between identical versions:
Any additional ideas or input are appreciated greatly.
Cleaned version of pom file for completeness:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>...</groupId>
<artifactId>...</artifactId>
<version>...</version>
<packaging>jar</packaging>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<beam.version>2.19.0</beam.version>
<google.cloud.core.version>1.108.1</google.cloud.core.version>
<maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
<maven-surefire-plugin.version>2.21.0</maven-surefire-plugin.version>
<maven-jar-plugin.version>3.0.2</maven-jar-plugin.version>
<maven-shade-plugin.version>3.1.0</maven-shade-plugin.version>
<maven-exec-plugin.version>1.6.0</maven-exec-plugin.version>
<slf4j.version>1.7.30</slf4j.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
...
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-kafka</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-direct-java</artifactId>
<version>${beam.version}</version>
</dependency>
</dependencies>
</project>

Answer 1

I've managed to resolve the issue. Apparently this was not caused by the dependencies themselves, but rather by a plugin. I omitted the pom.xml in my question as I did not expect this part to have a specific influence.
Cause of error:
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${maven-exec-plugin.version}</version>
<configuration>
<!-- the following is needed for logging to pick up the log4j.properties file -->
<includePluginDependencies>true</includePluginDependencies>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
</plugins>
</pluginManagement>
We had to remove the includePluginDependencies subtag and now it works:
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${maven-exec-plugin.version}</version>
<configuration>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
</plugins>
</pluginManagement>
Note that that line was apparently there in order to pick op the log4j property file (see the comment), which we might have to solve otherwise now. We still have to look into whether this gives any issues.