Gradle protobuf task not picking up definitions from dependencies - gradle

I have a situation where I have two Gradle subprojects that are basically dumb protobuf containers. One sub project needs to import the definitions from the other, but I can't seem to figure out how to get the protobuf Gradle plugin to work correctly (I'm fairly new to Gradle).
Here's an example.
Directory layout:
build.gradle
settings.gradle
gradle.properties
dependency/
|
- build.gradle
- src/main/proto/dependency.proto
main/
|
- build.gradle
- src/main/proto/main.proto
build.gradle (top level)
plugins {
id "com.google.protobuf" version "0.8.12"
id "java"
}
configure (allprojects) {
apply plugin: 'java'
apply plugin: 'com.google.protobuf'
}
settings.gradle
rootProject.name = 'sample'
include 'main'
include 'dependency'
gradle.properties
group=sample
version=0.1.0
dependency/build.gradle
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.11.0"
}
}
dependencies {
compile 'com.google.protobuf:protobuf-java:3.11.4'
compile 'com.google.protobuf:protobuf-java-util:3.11.4'
}
dependency/src/main/proto/dependency.proto
syntax = "proto3";
package dependency;
option java_package = "dependency";
message DependencyRequest {
string foo = 1;
}
message DependencyResponse {
string bar = 1;
}
main/build.gradle
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.11.0"
}
plugins {
grpc {
artifact = 'io.grpc:protoc-gen-grpc-java:1.27.2'
}
}
generateProtoTasks {
all()*.plugins {
grpc {}
}
}
}
dependencies {
compile 'com.google.protobuf:protobuf-java:3.11.4'
compile 'com.google.protobuf:protobuf-java-util:3.11.4'
protobuf project(':dependency')
}
main/src/main/proto/main.proto
syntax = "proto3";
package main;
option java_package = "main";
import "dependency/dependency.proto";
service MainService {
rpc CallDependency(DependencyRequest) returns (DependencyResponse) {}
}
When I try to build, I get this error:
$ ./gradlew :main:compileJava
> Task :main:generateProto FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':main:generateProto'.
> protoc: stdout: . stderr: dependency/dependency.proto: File not found.
main.proto:7:1: Import "dependency/dependency.proto" was not found or had errors.
main.proto:11:24: "DependencyRequest" is not defined.
main.proto:11:52: "DependencyResponse" is not defined.
I read documentation for the plugin and this answer, but I haven't been able to get it to work. Any idea why the protobuf task isn't picking up dependency.proto?

Apparently the protobuf files were getting set up correctly, but there was a mismatch in the package names that cause the main.proto to not be able to find dependency.proto. I was able to get this working by removing the package names in the protobuf definitions.
If anyone runs into this problem in the future, check your package names and imports to make sure they're correct.

Related

How to import code from outside into gradle build file?

My build files are large and messy, making them difficult to read. like below:
plugins {
...
id "com.google.protobuf" version "0.8.17"
}
dependencies {
implementation "androidx.datastore:datastore-core:1.0.0"
implementation "com.google.protobuf:protobuf-javalite:3.18.0"
...
}
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.14.0"
}
// Generates the java Protobuf-lite code for the Protobufs in this project. See
// https://github.com/google/protobuf-gradle-plugin#customizing-protobuf-compilation
// for more information.
generateProtoTasks {
all().each { task ->
task.builtins {
java {
option 'lite'
}
}
}
}
}
I want to define the above code into an external file, and then introduce it into the build file, how should I do it?
According to the Gradle documentation, as of now it is not possible to move the plugin block to other file than the project’s build script or settings.gradle file.
For the other sections, let's say dependencies or protobuf, then you can move these sections on a separate gradle files and import them by using the following statement:
apply from: "${project.rootDir}/your-gradle-file"
Of course the path of your-gradle-file should be adjusted according to the project's folder structure you decide.
If you want to split the dependencies into multiple gradle file you can do the following:
on your main gradle file:
dependencies {
apply from: "${project.rootDir}/depsGroup1.gradle"
apply from: "${project.rootDir}/depsGroup2.gradle"
}
and within each depsGroup file:
dependencies {
implementation xyz
}

'Could not get unknown property 'classesDir' for main classes of type org.gradle.api.internal.tasks.DefaultSourceSetOutput.' error

Since I wanted to examine this source code, I imported it into Android Studio. I got a few other errors before and fixed them. This is annoying, now I have a new error.
I am getting this error:
Build file 'C:\Users\hange\Desktop\libgdx-demo-superjumper-master\desktop\build.gradle' line: 18
A problem occurred evaluating project ':desktop'.
> Could not get unknown property 'classesDir' for main classes of type org.gradle.api.internal.tasks.DefaultSourceSetOutput.
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
My "desktop\build.gradle" file looks like this. I specified line 18 there.
apply plugin: "java"
sourceCompatibility = 1.6
sourceSets.main.java.srcDirs = [ "src/" ]
project.ext.mainClassName = "com.badlogicgames.superjumper.desktop.DesktopLauncher"
project.ext.assetsDir = new File("../android/assets");
task run(dependsOn: classes, type: JavaExec) {
main = project.mainClassName
classpath = sourceSets.main.runtimeClasspath
standardInput = System.in
workingDir = project.assetsDir
ignoreExitValue = true
}
task dist(type: Jar) {
from files(sourceSets.main.output.classesDir) // THIS IS 18th LINE.
from files(sourceSets.main.output.resourcesDir)
from {configurations.compile.collect {zipTree(it)}}
from files(project.assetsDir);
manifest {
attributes 'Main-Class': project.mainClassName
}
}
dist.dependsOn classes
EDIT: My "build.gradle" file looks like this. But I noticed that it can't resolve the json library. Probably the error is here because we are pulling the gradle version from a json file on the internet.
import groovy.json.JsonSlurper // Cannot resolve symbol 'json'
buildscript {
ant.get(src: 'https://libgdx.com/service/versions.json', dest: 'versions.json')
def versionFile = file('versions.json')
def json
if (versionFile.exists()) {
json = new JsonSlurper().parseText(versionFile.text)
} else throw new GradleException("Unable to retrieve latest versions, please check your internet connection")
ext {
gdxVersion = json.libgdxRelease
roboVMVersion = json.robovmVersion
roboVMGradleVersion = json.robovmPluginVersion
androidToolsVersion = json.androidBuildtoolsVersion
androidSDKVersion = json.androidSDKVersion
androidGradleToolsVersion = json.androidGradleToolVersion
gwtVersion = json.gwtVersion
gwtGradleVersion = json.gwtPluginVersion
}
repositories {
mavenLocal()
mavenCentral()
maven { url "https://plugins.gradle.org/m2/" }
maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }
jcenter()
google()
}
dependencies {
classpath "org.wisepersist:gwt-gradle-plugin:$gwtGradleVersion"
classpath "com.android.tools.build:gradle:$androidGradleToolsVersion"
classpath "com.mobidevelop.robovm:robovm-gradle-plugin:$roboVMGradleVersion"
}
}
I'm posting the solution in case anyone else has the same problem:
androidToolsVersion, androidSDKVersion, androidGradleToolsVersion, gwtGradleVersion
These 4 variables are pulled from a json file on the internet, but the value pulled from the json file may not always work correctly. Make the variables compatible.
For example this code is using classesDir.The classesDir property was removed in Gradle 5.x.
If you pull to Gradle 4.8.1, you will get the error "Minimum supported Gradle version is 6.5. Current version is 4.8.1.". To fix this error you have to assign 3.2.x value compatible with 4.8.1 gradle version to the androidGradleToolsVersion variable.
androidGradleToolsVersion = "3.2.1"

Why does Gradle ignore sourceSets and/or srcDir when dealing with proto files?

My directory structure is src/ps/proto. My build.gradle file is located in the src directory. I've set sourceSets to
sourceSets {
ps {
proto {
srcDir 'ps/proto'
}
}
}
Yet, a gradlew generatePsProto gives me a slew of error, one of which is that my source directory is src/src/ps/proto.
Execution failed for task ':generatePsProto'.
> protoc: stdout: . stderr: /home/build/tree/src/src/ps/proto: warning: directory does not exist.
[libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax specified for the proto file: cldb.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
ps/proto/security.proto: File not found.
ps/proto/common.proto: File not found.
ps/proto/cli.proto: File not found.
ps/proto/volumemirrorcommon.proto: File not found.
ps/proto/metrics.proto: File not found.
cldb.proto: Import "ps/proto/security.proto" was not found or had errors.
cldb.proto: Import "ps/proto/common.proto" was not found or had errors.
mldb.proto: Import "ps/proto/cli.proto" was not found or had errors.
mldb.proto:214:12: "CredentialsMsg" is not defined.
mldb.proto:218:12: "CredentialsMsg" is not defined.
[...]
mldb.proto:3614:12: "Key" is not defined.
mldb.proto:3618:12: "CredentialsMsg" is not defined.
mldb.proto:3619:12: "ServerKeyType" is not defined.
I don't want the Gradle default of src/ps/proto. I want ps/proto. Can this be done? My goal is to remove the hard coded exec calls to protoc and use the protobuf pluging to compile the *.proto files into their respective *.h, *.cc, and eventually into *.o files.
Gradle 4.7 Build time: 2018-04-18 09:09:12 UTC Revision:
b9a962bf70638332300e7f810689cb2febbd4a6c
Groovy: 2.4.12 Ant: Apache Ant(TM) version 1.9.9
compiled on February 2 2017 JVM: 1.8.0_144 (Oracle
Corporation 25.144-b01) OS: Linux 3.10.0-514.el7.x86_64
amd64
build.gradle
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.5'
}
}
apply plugin: 'c'
apply plugin: 'cpp'
apply plugin: 'java'
apply plugin: 'com.google.protobuf'
repositories {
mavenCentral()
}
dependencies {
compile "com.google.protobuf:protobuf-java:2.4.1"
}
sourceSets {
ps {
proto {
srcDir 'ps/proto'
}
}
}
protobuf {
protoc {
artifact = 'com.google.protobuf:protoc:2.4.1'
}
generateProtoTasks {
all().each { task ->
task.plugins {
cpp {}
}
}
}
}
Gradle has a widely applicable convention that collection properties can be configured with methods of the same name. The key difference between setting a property using assignment (=) and using a method is that the latter appends its values.
So, to override the default source directory for the source set you need to use assignment:
sourceSets {
ps {
proto {
srcDirs = ['ps/proto']
}
}
}
I don't really understand why you're getting an error message about src/src/ps/proto, so there may be some other issue. That error seems to be coming from protoc rather than Gradle.
EDIT
To solve your import problem, you should use
import security.proto;
i.e. without the source directory path. This is because the Protobuf Gradle Plugin appears to add each source directory as an "include directory". protoc only looks in such include directories for imports.
This is at least what I have garnered from the documentation and source code.
You seemed to have typo either for 'fs' or 'ps'.
The problem is the Import.
because you have for example
cldb.proto: Import "ps/proto/security.proto"
mldb.proto: Import "ps/proto/cli.proto"
Edited:
you have to put every .proto file in src/yourCustomSourceSetName/proto/ps/proto/ and add
sourceSets {
yourCustomSourceSetName {}
}
dependencies {
compile ... // dependencies for compileJava task
yourCustomSourceSetNameCompile ... // dependencies for custom sourceSet
compile sourceSets.yourCustomSourceSetName.output // compileJava uses output of the custom sourceSet
}
Then the import will look for other .proto files in the right place.

gRPC gradle :generateProto fails with directory not found when importing other proto definitions

I'm trying to compile some protobuf definitions as a gradle task, but get the following error with no source generation:
* What went wrong:
Execution failed for task ':generateProto'.
> protoc: stdout: . stderr: /Users/ash/IdeaProjects/kotlin/grpc/build/extracted-protos/main: warning: directory does not exist.
service-definitions.proto:10:17: "Empty" is not defined.
service-definitions.proto:10:33: "Empty" is not defined.
service-definitions.proto:15:17: "SimpleRequest" is not defined.
service-definitions.proto:15:41: "SimpleResponse" is not defined.
service-definitions.proto:21:27: "StreamingOutputCallRequest" is not defined.
service-definitions.proto:21:71: "StreamingOutputCallResponse" is not defined.
service-definitions.proto:27:33: "StreamingInputCallRequest" is not defined.
service-definitions.proto:27:69: "StreamingInputCallResponse" is not defined.
service-definitions.proto:34:29: "StreamingOutputCallRequest" is not defined.
service-definitions.proto:34:73: "StreamingOutputCallResponse" is not defined.
service-definitions.proto: warning: Import empty.proto but not used.
service-definitions.proto: warning: Import messages.proto but not used.
This only occurs when I try to compile a definition that imports other definitions. If I remove the service-definitions.proto definition, the others compile and sources are generated.
The protobuf definitions are available from the vertx-grpc examples and I have them in $projectDir/src/main/proto. I've not changed their default build location, so sources are being generated to $projectDir/build/generated/source/proto. Instead of an extracted-protos directory, however, I have extracted-include-protos (probably explains the directory not found error).
Here's my gradle.build file, should it be of use. Note that the gRPC plugin uses the Vert.x artifact rather than the version from io.grpc.. (it's actually just a wrapper over the top of it). I have tried swapping the Vert.x version out for the io.grpc version and get the same error.
group 'grpc'
version '1.0-SNAPSHOT'
buildscript {
ext.kotlin_version = '1.1.3'
repositories {
jcenter()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"
}
}
apply plugin: 'kotlin'
apply plugin: "java"
apply plugin: "com.google.protobuf"
apply plugin: "idea"
protobuf {
//generatedFilesBaseDir = "$projectDir/build/generated/source/proto"
protoc {
artifact = "com.google.protobuf:protoc:3.3.0"
}
plugins {
grpc {
artifact = "io.vertx:protoc-gen-grpc-java:1.3.0"
}
}
generateProtoTasks {
all().each { task ->
task.plugins {
grpc {}
}
}
}
}
clean {
delete protobuf.generatedFilesBaseDir
}
idea {
module {
sourceDirs += file("${protobuf.generatedFilesBaseDir}/main/java")
sourceDirs += file("${protobuf.generatedFilesBaseDir}/main/grpc")
}
}
sourceSets {
main {
java {
srcDirs += "${protobuf.generatedFilesBaseDir}/main/java"
srcDirs += "${protobuf.generatedFilesBaseDir}/main/grpc"
}
}
}
repositories {
jcenter()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"
compile "io.vertx:vertx-core:3.4.2"
compile "io.vertx:vertx-web:3.4.2"
compile "io.vertx:vertx-grpc:3.4.2"
}
compileKotlin {
kotlinOptions {
jvmTarget = "1.8"
apiVersion = "1.1"
languageVersion = "1.1"
}
}
compileTestKotlin {
kotlinOptions {
jvmTarget = "1.8"
apiVersion = "1.1"
languageVersion = "1.1"
}
}
I've not changed any settings, why is it (a) looking for extracted files in the wrong location, and (b) looking for extracted files at all when importing, given that the necessary definitions are all in the src/main/proto directory?
Okay, after much head scratching, the clue was in the last two lines of the error message:
service-definitions.proto: warning: Import empty.proto but not used.
service-definitions.proto: warning: Import messages.proto but not used.
In my proto definitions, I had specified messages and empty to be in the messages package but hadn't qualified the types when used in the importing definition, which is in the services package. Alas, a simple change and it works:
package services;
service EmptyPingPongService {
// One empty request followed by one empty response.
rpc EmptyCall(Empty) returns (Empty);
}
Becomes:
package services;
service EmptyPingPongService {
// One empty request followed by one empty response.
rpc EmptyCall(messages.Empty) returns (messages.Empty);
}

gradle protobuf plugin not functioning

I am using the below mentioned protobuf gradle plugin in one project where its working fine but when I referenced the same plugin in a different project, 'gradle clean' is consistently giving me the error copied below:
relevant parts of build.grade (v3.4)
apply plugin: 'com.google.protobuf'
buildscript {
repositories {
mavenCentral()
mavenLocal()
jcenter()
}
dependencies {
// classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.0.9"
// classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.4'
classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.0'
}
}
def grpcVersion = '1.1.2'
dependencies {
compile "io.grpc:grpc-netty:${grpcVersion}"
compile "io.grpc:grpc-protobuf:${grpcVersion}"
compile "io.grpc:grpc-stub:${grpcVersion}"
}
protobuf {
protoc {
Artifact = 'com.google.protobuf:protoc:3.2.0'
}
plugins {
grpc {
Artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
}
}
generateProtoTasks {
all()*.plugins {
grpc {
// To generate deprecated interfaces and static bindService method,
// turn the enable_deprecated option to true below:
option 'enable_deprecated=false'
}
}
}
}
error when I run gradle clean
* What went wrong:
Could not compile build file '/xyz/xyz/build.gradle'.
> startup failed:
build file '/xyz/xyz/build.gradle': 102: you tried to assign a value to the class 'org.gradle.api.component.Artifact'
# line 102, column 9.
Artifact = 'com.google.protobuf:protoc:3.2.0'
^
build file '/xyz/xyz/build.gradle': 106: you tried to assign a value to the class 'org.gradle.api.component.Artifact'
# line 106, column 13.
Artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
I have tried protobuf plugins 0.8.0. and 0.8.1 but both give the same error. v0.8.0 works as is in a different project. Any thoughts on how to troubleshoot this further would be appreciated.
It should be artifact, not Artifact. The latter is a class that you try to assign to which will not work, the former is a property you assign to.

Resources