Retrieving external .proto files for protoc with protobuf-maven-plugin - maven

Question

Context: My team is working on making our suite of Java services containerized and dynamically scalable. To accomplish this, our plan is to use Envoy backed by etcd, with a custom-build Endpoint Discovery Service as described in the Envoy documentation, using the v2 gRPC-based API. We'll then generate Docker images for each service and deploy/manage them with Kubernetes.
We use Maven as a build system. I'm pretty well versed in Maven, but this is my first time with gRPC or protocol buffers.
I've created a stub for my service using Spring Boot, with Jetty offering some REST and JMX endpoints for management. Before introducing the protobuf stuff, the stub built and ran just fine.
I've downloaded the Envoy data-plane-api and checked the API definition files (**/*.proto) into my project under src/main/proto, keeping the directory structure from the download (e.g. src/main/proto/envoy/api/v2/eds.proto). (Side question: do I need the BUILD files?)
In the end, I'd like a standalone Maven build that can read these files and generate Java classes. The build needs to work on Windows and OS X boxes so that it works for developers, and on Linux boxes so that it works in our CI (Bamboo). It should require nothing more than a JDK, a Maven install, and a Maven repo. (We have an Artifactory instance where I can upload artifacts not otherwise available online, if necessary.)
What I have so far seems like it will accomplish my portability goals:
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>discovery-service</artifactId>
<packaging>jar</packaging>
<name>Discovery Service</name>
<version>1.0.0</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<verbose>true</verbose>
<fork>true</fork>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.5.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:3.5.1-1:exe:${os.detected.classifier}</protocArtifact>
<checkStaleness>true</checkStaleness>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<extensions>
<!-- provides os.detected.classifier (i.e. linux-x86_64, osx-x86_64) property -->
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.4.1.Final</version>
</extension>
</extensions>
</build>
<dependencies>
<dependency>
<groupId>io.hydrosphere</groupId>
<artifactId>envoy-data-plane-api_2.11</artifactId>
<version>v1.5.0_1</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.5.1</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-core</artifactId>
<version>1.9.1</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>1.9.1</version>
</dependency>
<!--Unrelated deps (Spring Boot, Jetty, logging, etc) omitted for brevity -->
</dependencies>
</project>
When I build this project using mvn compile, Maven correctly downloads protoc and invokes it, but protoc errors because it can't find some external dependencies:
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Discovery Service 1.0.0
[INFO] ------------------------------------------------------------------------
Downloading: http://devutl1.sircon.com:8081/artifactory/libs-snapshot/org/glassfish/javax.el/maven-metadata.xml
Downloading: http://devutl1.sircon.com:8081/artifactory/libs-release/org/glassfish/javax.el/maven-metadata.xml
Downloaded: http://devutl1.sircon.com:8081/artifactory/libs-release/org/glassfish/javax.el/maven-metadata.xml (882 B at 547 B/s)
Downloaded: http://devutl1.sircon.com:8081/artifactory/libs-snapshot/org/glassfish/javax.el/maven-metadata.xml (882 B at 547 B/s)
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (default-cli) # discovery-service ---
[INFO]
[INFO] --- protobuf-maven-plugin:0.5.1:compile (default) # discovery-service ---
[INFO] Compiling 56 proto file(s) to /Users/jrobb/Projects/vertabrae/trunk/scaling/discovery-service/target/generated-sources/protobuf/java
[ERROR] PROTOC FAILED: validate/validate.proto: File not found.
gogoproto/gogo.proto: File not found.
envoy/api/v2/core/address.proto: Import "validate/validate.proto" was not found or had errors.
envoy/api/v2/core/address.proto: Import "gogoproto/gogo.proto" was not found or had errors.
envoy/config/metrics/v2/stats.proto: Import "envoy/api/v2/core/address.proto" was not found or had errors.
envoy/config/metrics/v2/stats.proto: Import "validate/validate.proto" was not found or had errors.
envoy/config/metrics/v2/stats.proto:148:5: "envoy.api.v2.core.Address" is not defined.
envoy/config/metrics/v2/stats.proto:167:5: "envoy.api.v2.core.Address" is not defined.
The output continues, with the same set of errors for each .proto file.
It seems that the crux of the issue is that I don't have validate/validate.proto or gogoproto/gogo.proto, which are imported at the top of (nearly?) every .proto file:
import "google/protobuf/wrappers.proto";
import "validate/validate.proto";
import "gogoproto/gogo.proto";
It seems to be finding wrappers.proto, which I think is coming from my compile-time Maven dependency on protobuf-java.
I think gogoproto/gogo.proto might be looking for this: https://github.com/gogo/protobuf/blob/master/gogoproto/gogo.proto
I'm totally stumped about where validate/validate.proto is supposed to come from. I've seen some evidence that it's specific to Envoy, but I can't find it.
I've spent the last few hours searching, and I'm coming up empty on anyone who's done this before. Where do I get those files? If I can't get them from Maven Central, I'll build them myself and upload to Artifactory.
My eyes are glazing over looking at documentation intended for people using other tech stacks, and I'm having trouble translating for my needs. Any and all help will be much appreciated, and I apologize if this is a hugely n00bish question. :)

Answer 1

Once again, the mere act of asking on SO has led me to the answer; found while adding context and detail to my question. There's a brand-new Java port of the Go-based Envoy control-plane: java-control-plane!
Somebody who, like me, would prefer to just depend on something from Maven Central, and already knew what they were looking for reported this issue, and while I was writing up this question, the maintainer answered: exactly that is coming soon!
I'm pleased to find that api/pom.xml in java-control-plane looks very similar to what I posted in my question. :)
So, if you find yourself in my situation, wanting to implement a Discovery Service for Envoy on a Java tech stack, java-control-plane is already there -- you just have to extend it with whatever backing store you want to use. I assume that, over time, implementations for that will pop up in the community as well. I'll be writing one for etcd and might end up contributing it back to the community.
Answering the finer details of my question:
Where do I get the two .proto file dependencies not included in data-plane-api?
They are in the java-control-plane source: they're checked in as part of the api module's source code, and therefore presumably not available as a separate dependency. I'm reasonably confident that the gogo.proto I found was the right one, but I'm still unclear on where I could have found validate.proto.
Do I need to include the BUILD files from data-plane-api in order to compile the *.proto files with protoc?
Nope! The api module in java-control-plane doesn't include them.