How to utilize/import Flink's TestHarness class with bazel? - maven

Question

My intuition might have been right to use the packaging and/or Classifier. For finding the correct import Tag for the Build File I recommending visting the .cache/bazel/bazel_user/.../external/maven/BUILD file
EDITED the Post into working setup
I want to test my own async flink Stream Operator RichAsyncFunction using a Bazel Build Tool.
This is basicly the same questions as How to utilize Flink's TestHarness class?
The main problem is that i can not find the import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness
And all the other Code needed for testing Steaming Operators.
I tried to follow the provided answer, but I struggle with the bazel syntax, asking myself if it is even possible to use those imports with bazel.
I am using newest Bazel Version and IntelliJ 2019.3.4 with the bazel plugin.
The linked Answer supposes this Maven Dependencies:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils-junit</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime_2.11</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
My WORKSPACE looks like this
...
http_archive(
name = "rules_jvm_external",
sha256 = RULES_JVM_EXTERNAL_SHA,
strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)
load("#rules_jvm_external//:defs.bzl", "maven_install")
load("#rules_jvm_external//:defs.bzl", "artifact")
load("#rules_jvm_external//:specs.bzl", "maven")
load("#io_grpc_grpc_java//:repositories.bzl", "IO_GRPC_GRPC_JAVA_ARTIFACTS")
load("#io_grpc_grpc_java//:repositories.bzl", "IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS")
maven_install(
artifacts = [
...
"org.apache.commons:commons-lang3:3.9",
"org.javatuples:javatuples:1.2",
"junit:junit:4.13",
"org.testcontainers:testcontainers:1.14.1",
"org.testcontainers:kafka:1.14.1",
"org.testcontainers:postgresql:1.14.1",
"commons-io:commons-io:2.6",
"com.google.code.findbugs:jsr305:1.3.9",
"com.google.errorprone:error_prone_annotations:2.0.18",
"com.google.j2objc:j2objc-annotations:1.1",
"com.google.protobuf:protobuf-java:3.11.1",
"com.google.protobuf:protobuf-java-util:3.6.1",
"info.picocli:picocli:4.1.0",
"org.slf4j:slf4j-log4j12:1.7.5",
"org.slf4j:slf4j-api:1.7.28",
"com.github.jasync-sql:jasync-postgresql:1.0.11",
"com.github.jasync-sql:jasync-common:1.0.11",
"org.postgresql:postgresql:42.2.5",
"org.mongodb:mongodb-driver-reactivestreams:4.0.2",
"org.mongodb:mongodb-driver-core:4.0.2",
"org.mongodb:bson:4.0.2",
"org.reactivestreams:reactive-streams:1.0.3",
"joda-time:joda-time:2.9.7",
"org.apache.kafka:kafka-clients:2.4.0",
# "io.grpc:grpc-netty-shaded:%s" % GRPC_JAVA_VERSION,
# "io.grpc:grpc-protobuf:%s" % GRPC_JAVA_VERSION,
# "io.grpc:grpc-stub:%s" % GRPC_JAVA_VERSION,
"org.apache.flink:flink-core:%s" % FLINK_VERSION,
"org.apache.flink:flink-java:%s" % FLINK_VERSION,
"org.apache.flink:flink-streaming-java_%s:%s" % (SCALA_VERSION, FLINK_VERSION),
"org.apache.flink:flink-connector-kafka-0.11_%s:%s" % (SCALA_VERSION, FLINK_VERSION),
"org.apache.flink:flink-cep_2.11:%s" % FLINK_VERSION,
] + IO_GRPC_GRPC_JAVA_ARTIFACTS,
generate_compat_repositories = True,
override_targets = IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS,
repositories = [
"https://jcenter.bintray.com/",
"https://maven.google.com",
"https://repo1.maven.org/maven2",
],
)
maven_install(
name = "testing",
artifacts = [
maven.artifact(
group = "org.apache.flink",
artifact = "flink-runtime_2.11",
version = FLINK_VERSION,
classifier = "tests",
packaging = "test-jar",
),
maven.artifact(
group = "org.apache.flink",
artifact = "flink-streaming-java_2.11",
version = FLINK_VERSION,
classifier = "tests",
packaging = "test-jar",
),
maven.artifact(
group = "org.apache.flink",
artifact = "flink-test-utils-junit",
version = FLINK_VERSION,
),
"org.apache.flink:flink-tests:%s" % FLINK_VERSION,
],
repositories = [
"https://jcenter.bintray.com/",
"https://maven.google.com",
"https://repo1.maven.org/maven2",
],
)
load("#maven//:compat.bzl", "compat_repositories")
compat_repositories()
...
The testing maven install "group" is my try on figuring out how to get the needed Dependencies imported, since there is no explicit "type" and "scope" in Bazels maven syntax as far as i know.
The test BUILD files looks like this
load("#rules_java//java:defs.bzl", "java_test")
load("#rules_jvm_external//:specs.bzl", "maven")
load("#rules_jvm_external//:defs.bzl", "artifact", "maven_install")
java_test(
name = "some_module",
size = "medium",
srcs = ["DataOperatorTests.java"],
tags = [
"docker",
"integration",
],
test_class = "org.some.project.DataOperatorTests",
deps = [
"#maven//:com_google_protobuf_protobuf_java",
"#maven//:junit_junit_4_13",
"#maven//:org_apache_flink_flink_connector_kafka_0_11_2_11",
"#maven//:org_apache_flink_flink_core",
"#maven//:org_apache_flink_flink_java",
"#maven//:org_javatuples_javatuples",
"#maven//:org_mongodb_bson",
"#maven//:org_mongodb_mongodb_driver_core",
"#maven//:org_mongodb_mongodb_driver_reactivestreams",
"#maven//:org_testcontainers_kafka",
"#maven//:org_testcontainers_testcontainers",
"#maven//:org_apache_flink_flink_streaming_java_2_11",
"#testing//:org_apache_flink_flink_runtime_2_11_tests",
"#testing//:org_apache_flink_flink_streaming_java_2_11_tests",
"#testing//:org_apache_flink_flink_test_utils_junit",
"#testing//:org_apache_flink_flink_tests",
],
)
test_suite(name = "smoke",tags = ["-docker","-internal",],)
test_suite(name = "unit",tags = ["-internal","unit",],)
test_suite(name = "integration",tags = ["-internal","integration",],)
test_suite(name = "internal")
With the above setup the following Error does no longer occur - leaving it for reference
The Packages seem to exist but in the actual test BUILD file i get this error on the sync process.
Error:(5, 1) no such target '#testing//:org_apache_flink_flink_streaming_java_2_11': target 'org_apache_flink_flink_streaming_java_2_11' not declared in package '' defined by /home/user/.cache/bazel/_bazel_user/7b62e4e31c70ee640c6d33972433da28/external/testing/BUILD and referenced by '//core/src/test/java/org/some/project/some_module:some_module'

Answer 1

I solved the issue by myself. Adjusted the above Code to be a suitable solution.
To sum it up.
When a maven Repository is give like this:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils-junit</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime_2.11</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
The corresponding Bazel WORKSPACE file has to have entries like this:
load("#rules_jvm_external//:defs.bzl", "maven_install")
load("#rules_jvm_external//:defs.bzl", "artifact")
load("#rules_jvm_external//:specs.bzl", "maven")
maven_install(
...
artifacts = [
maven.artifact(
group = "org.apache.flink",
artifact = "flink-runtime_2.11",
version = FLINK_VERSION,
classifier = "tests",
packaging = "test-jar",
),
maven.artifact(
group = "org.apache.flink",
artifact = "flink-streaming-java_2.11",
version = FLINK_VERSION,
classifier = "tests",
packaging = "test-jar",
),
maven.artifact(
group = "org.apache.flink",
artifact = "flink-test-utils-junit",
version = FLINK_VERSION,
),
"org.apache.flink:flink-tests:%s" % FLINK_VERSION,
],
repositories = [
"https://jcenter.bintray.com/",
"https://maven.google.com",
"https://repo1.maven.org/maven2",
],
)
The Detailed Dependencies explanation can be found here:
https://github.com/bazelbuild/rules_jvm_external#detailed-dependency-information-specifications
and here:
https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/api.md#mavenartifact
To find the resulting targets for import in the Build files use like mentioned by #Jin
bazel query #maven//:all | grep flink_flink_streaming
In most cases the target might have just the suffix _tests
Here the necessary imports for the BUILD File:
load("#rules_jvm_external//:specs.bzl", "maven")
load("#rules_jvm_external//:defs.bzl", "artifact", "maven_install")
java_test(
name = "some_module",
size = "medium",
srcs = ["DataOperatorTests.java"],
tags = [
"docker",
"integration",
],
test_class = "org.some.project.DataOperatorTests",
deps = [
...
"#maven//:org_apache_flink_flink_runtime_2_11_tests",
"#maven//:org_apache_flink_flink_streaming_java_2_11_tests",
"#maven//:org_apache_flink_flink_test_utils_junit",
"#maven//:org_apache_flink_flink_tests",
],
)

Answer 2

The Harness files are part of the "flink-streaming-java_2.12" artifact with type test-jar.
So if you project code is using both the test and source library, then better to use a classifier for your test dependency.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.12</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<classifier>tests</classifier>
<type>test-jar</type>
</dependency>
Check the External Libraries in intellij to see if you have 2 jars now, one for source and other for tests.