gRPC gradle :generateProto fails with directory not found when importing other proto definitions - gradle

I'm trying to compile some protobuf definitions as a gradle task, but get the following error with no source generation:
* What went wrong:
Execution failed for task ':generateProto'.
> protoc: stdout: . stderr: /Users/ash/IdeaProjects/kotlin/grpc/build/extracted-protos/main: warning: directory does not exist.
service-definitions.proto:10:17: "Empty" is not defined.
service-definitions.proto:10:33: "Empty" is not defined.
service-definitions.proto:15:17: "SimpleRequest" is not defined.
service-definitions.proto:15:41: "SimpleResponse" is not defined.
service-definitions.proto:21:27: "StreamingOutputCallRequest" is not defined.
service-definitions.proto:21:71: "StreamingOutputCallResponse" is not defined.
service-definitions.proto:27:33: "StreamingInputCallRequest" is not defined.
service-definitions.proto:27:69: "StreamingInputCallResponse" is not defined.
service-definitions.proto:34:29: "StreamingOutputCallRequest" is not defined.
service-definitions.proto:34:73: "StreamingOutputCallResponse" is not defined.
service-definitions.proto: warning: Import empty.proto but not used.
service-definitions.proto: warning: Import messages.proto but not used.
This only occurs when I try to compile a definition that imports other definitions. If I remove the service-definitions.proto definition, the others compile and sources are generated.
The protobuf definitions are available from the vertx-grpc examples and I have them in $projectDir/src/main/proto. I've not changed their default build location, so sources are being generated to $projectDir/build/generated/source/proto. Instead of an extracted-protos directory, however, I have extracted-include-protos (probably explains the directory not found error).
Here's my gradle.build file, should it be of use. Note that the gRPC plugin uses the Vert.x artifact rather than the version from io.grpc.. (it's actually just a wrapper over the top of it). I have tried swapping the Vert.x version out for the io.grpc version and get the same error.
group 'grpc'
version '1.0-SNAPSHOT'
buildscript {
ext.kotlin_version = '1.1.3'
repositories {
jcenter()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"
}
}
apply plugin: 'kotlin'
apply plugin: "java"
apply plugin: "com.google.protobuf"
apply plugin: "idea"
protobuf {
//generatedFilesBaseDir = "$projectDir/build/generated/source/proto"
protoc {
artifact = "com.google.protobuf:protoc:3.3.0"
}
plugins {
grpc {
artifact = "io.vertx:protoc-gen-grpc-java:1.3.0"
}
}
generateProtoTasks {
all().each { task ->
task.plugins {
grpc {}
}
}
}
}
clean {
delete protobuf.generatedFilesBaseDir
}
idea {
module {
sourceDirs += file("${protobuf.generatedFilesBaseDir}/main/java")
sourceDirs += file("${protobuf.generatedFilesBaseDir}/main/grpc")
}
}
sourceSets {
main {
java {
srcDirs += "${protobuf.generatedFilesBaseDir}/main/java"
srcDirs += "${protobuf.generatedFilesBaseDir}/main/grpc"
}
}
}
repositories {
jcenter()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"
compile "io.vertx:vertx-core:3.4.2"
compile "io.vertx:vertx-web:3.4.2"
compile "io.vertx:vertx-grpc:3.4.2"
}
compileKotlin {
kotlinOptions {
jvmTarget = "1.8"
apiVersion = "1.1"
languageVersion = "1.1"
}
}
compileTestKotlin {
kotlinOptions {
jvmTarget = "1.8"
apiVersion = "1.1"
languageVersion = "1.1"
}
}
I've not changed any settings, why is it (a) looking for extracted files in the wrong location, and (b) looking for extracted files at all when importing, given that the necessary definitions are all in the src/main/proto directory?

Okay, after much head scratching, the clue was in the last two lines of the error message:
service-definitions.proto: warning: Import empty.proto but not used.
service-definitions.proto: warning: Import messages.proto but not used.
In my proto definitions, I had specified messages and empty to be in the messages package but hadn't qualified the types when used in the importing definition, which is in the services package. Alas, a simple change and it works:
package services;
service EmptyPingPongService {
// One empty request followed by one empty response.
rpc EmptyCall(Empty) returns (Empty);
}
Becomes:
package services;
service EmptyPingPongService {
// One empty request followed by one empty response.
rpc EmptyCall(messages.Empty) returns (messages.Empty);
}

Related

Gradle protobuf task not picking up definitions from dependencies

I have a situation where I have two Gradle subprojects that are basically dumb protobuf containers. One sub project needs to import the definitions from the other, but I can't seem to figure out how to get the protobuf Gradle plugin to work correctly (I'm fairly new to Gradle).
Here's an example.
Directory layout:
build.gradle
settings.gradle
gradle.properties
dependency/
|
- build.gradle
- src/main/proto/dependency.proto
main/
|
- build.gradle
- src/main/proto/main.proto
build.gradle (top level)
plugins {
id "com.google.protobuf" version "0.8.12"
id "java"
}
configure (allprojects) {
apply plugin: 'java'
apply plugin: 'com.google.protobuf'
}
settings.gradle
rootProject.name = 'sample'
include 'main'
include 'dependency'
gradle.properties
group=sample
version=0.1.0
dependency/build.gradle
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.11.0"
}
}
dependencies {
compile 'com.google.protobuf:protobuf-java:3.11.4'
compile 'com.google.protobuf:protobuf-java-util:3.11.4'
}
dependency/src/main/proto/dependency.proto
syntax = "proto3";
package dependency;
option java_package = "dependency";
message DependencyRequest {
string foo = 1;
}
message DependencyResponse {
string bar = 1;
}
main/build.gradle
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.11.0"
}
plugins {
grpc {
artifact = 'io.grpc:protoc-gen-grpc-java:1.27.2'
}
}
generateProtoTasks {
all()*.plugins {
grpc {}
}
}
}
dependencies {
compile 'com.google.protobuf:protobuf-java:3.11.4'
compile 'com.google.protobuf:protobuf-java-util:3.11.4'
protobuf project(':dependency')
}
main/src/main/proto/main.proto
syntax = "proto3";
package main;
option java_package = "main";
import "dependency/dependency.proto";
service MainService {
rpc CallDependency(DependencyRequest) returns (DependencyResponse) {}
}
When I try to build, I get this error:
$ ./gradlew :main:compileJava
> Task :main:generateProto FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':main:generateProto'.
> protoc: stdout: . stderr: dependency/dependency.proto: File not found.
main.proto:7:1: Import "dependency/dependency.proto" was not found or had errors.
main.proto:11:24: "DependencyRequest" is not defined.
main.proto:11:52: "DependencyResponse" is not defined.
I read documentation for the plugin and this answer, but I haven't been able to get it to work. Any idea why the protobuf task isn't picking up dependency.proto?
Apparently the protobuf files were getting set up correctly, but there was a mismatch in the package names that cause the main.proto to not be able to find dependency.proto. I was able to get this working by removing the package names in the protobuf definitions.
If anyone runs into this problem in the future, check your package names and imports to make sure they're correct.

How to compile Kotlin domain module with client module with Gradle?

I have the following project structure:
- parent
- - client (written in Kotlin but compiled to JS)
- - server (written in Kotlin)
- - model (written in Kotlin)
client module has a dependency on model. So when I compile client to JS it also should compile model with it. For now I have the following Gradle configurations which are not doing the desired thing:
project/parent.gradle
group 'com.vchernogorov.tycher'
version '1.0-SNAPSHOT'
buildscript {
ext.kotlin_version = '1.1.3-2'
repositories {
mavenCentral()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
apply plugin: 'kotlin'
repositories {
mavenCentral()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"
compile project(":model")
compile project(":client")
compile project(":server")
}
compileKotlin {
kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
kotlinOptions.jvmTarget = "1.8"
}
project/settings.gradle
rootProject.name = 'parent'
include ':model'
include ':server'
include ':client'
project/model/build.gradle
group 'com.vchernogorov.tycher'
version '1.0-SNAPSHOT'
buildscript {
ext.kotlin_version = '1.1.3-2'
repositories {
mavenCentral()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
apply plugin: 'kotlin'
repositories {
mavenCentral()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"
}
compileKotlin {
kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
kotlinOptions.jvmTarget = "1.8"
}
project/client/build.gradle
group 'com.vchernogorov.tycher'
version '1.0-SNAPSHOT'
buildscript {
ext.kotlin_version = '1.1.3-2'
repositories {
mavenCentral()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
apply plugin: 'kotlin2js'
repositories {
mavenCentral()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib-js:$kotlin_version"
compile project(":model")
}
build.doLast {
configurations.compile.each { File file ->
copy {
includeEmptyDirs = false
from zipTree(file.absolutePath)
into "${projectDir}/web"
include { fileTreeElement ->
def path = fileTreeElement.path
path.endsWith(".js") && (path.startsWith("META-INF/resources/") || !path.startsWith("META-INF/"))
}
}
}
}
When I run gradle build command on client I get this message:
:model:compileKotlin
Using kotlin incremental compilation
:model:compileJava UP-TO-DATE
:model:copyMainKotlinClasses
:model:processResources UP-TO-DATE
:model:classes UP-TO-DATE
:model:jar
:client:compileJava UP-TO-DATE
:client:compileKotlin2Js
e: project/client/src/main/kotlin/HelloWorld.kt: (30, 19): Unresolved reference: Position
e: project/client/src/main/kotlin/HelloWorld.kt: (50, 5): Unresolved reference: Test
e: project/client/src/main/kotlin/SocketHandler.kt: (10, 36): Unresolved reference: User
:client:compileKotlin2Js FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':client:compileKotlin2Js'.
> Compilation error. See log for more details
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 13.891 secs
These 3 classes are defined in model.
So what should I do to successfully compile client and not changing the code?
I have developed the following architecture for my application:
I have client,server and model modules but also I have a separate source directory with DTOs called dto (not module btw). Here's the dependency hierarchy for the build:
client depends on dto
server depends on model
model depends on dto
With such hierarchy model can still use stdlib functionality and server will share all DTOs with client module. Meanwhile dto module will use the stdlib if it's compiled as module dependency and stdlib-js if it's compiled as client dependency, so it important to remember which classes you are able to use there.
In order to achieve this, you need to add
sourceSets.main.kotlin.srcDirs += '../dto/src/main/kotlin
config to both client and model build.gradle files. For Gradle Kotlin DSL:
the<JavaPluginConvention>().sourceSets {
"main" {
java {
srcDirs("../dto/src/main/kotlin")
}
}
}

gradle protobuf plugin not functioning

I am using the below mentioned protobuf gradle plugin in one project where its working fine but when I referenced the same plugin in a different project, 'gradle clean' is consistently giving me the error copied below:
relevant parts of build.grade (v3.4)
apply plugin: 'com.google.protobuf'
buildscript {
repositories {
mavenCentral()
mavenLocal()
jcenter()
}
dependencies {
// classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.0.9"
// classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.4'
classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.0'
}
}
def grpcVersion = '1.1.2'
dependencies {
compile "io.grpc:grpc-netty:${grpcVersion}"
compile "io.grpc:grpc-protobuf:${grpcVersion}"
compile "io.grpc:grpc-stub:${grpcVersion}"
}
protobuf {
protoc {
Artifact = 'com.google.protobuf:protoc:3.2.0'
}
plugins {
grpc {
Artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
}
}
generateProtoTasks {
all()*.plugins {
grpc {
// To generate deprecated interfaces and static bindService method,
// turn the enable_deprecated option to true below:
option 'enable_deprecated=false'
}
}
}
}
error when I run gradle clean
* What went wrong:
Could not compile build file '/xyz/xyz/build.gradle'.
> startup failed:
build file '/xyz/xyz/build.gradle': 102: you tried to assign a value to the class 'org.gradle.api.component.Artifact'
# line 102, column 9.
Artifact = 'com.google.protobuf:protoc:3.2.0'
^
build file '/xyz/xyz/build.gradle': 106: you tried to assign a value to the class 'org.gradle.api.component.Artifact'
# line 106, column 13.
Artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
I have tried protobuf plugins 0.8.0. and 0.8.1 but both give the same error. v0.8.0 works as is in a different project. Any thoughts on how to troubleshoot this further would be appreciated.
It should be artifact, not Artifact. The latter is a class that you try to assign to which will not work, the former is a property you assign to.

How to build Google protocol buffers and Kotlin using Gradle?

I'm trying to build a project that uses both Google protocol buffers and Kotlin using Gradle. I want the proto files to compile into Java source, which is then called from my Kotlin code.
My source files are arranged like this:
src/main/proto/*.proto
src/main/kotlin/*.kt
src/test/kotlin/*.kt
Here's my build.gradle file:
version '1.0-SNAPSHOT'
apply plugin: 'kotlin'
apply plugin: 'java'
apply plugin: 'com.google.protobuf'
repositories {
mavenCentral()
maven { url "http://dl.bintray.com/kotlin/kotlin-eap-1.1" }
}
buildscript {
ext.kotlin_version = '1.1-M02'
repositories {
mavenCentral()
maven { url "http://dl.bintray.com/kotlin/kotlin-eap-1.1" }
}
dependencies {
classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.0'
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
protobuf {
protoc {
artifact = 'com.google.protobuf:protoc:3.0.0'
}
}
dependencies {
compile 'com.google.protobuf:protobuf-java:3.0.0'
compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
testCompile 'junit:junit:4.12'
}
When I run ./gradlew assemble I get a bunch of "Unresolved reference" errors during :compileKotlin. Afterwards I can see that there are no Java source files generated, so it appears that the proto compiler is not being invoked at all.
If I remove the apply plugin: 'kotlin' line, then ./gradlew assemble successfully generates the Java source, but of course my Kotlin source is never compiled.
How do I fix my build.gradle so that I can call my protobuf code from Kotlin?
To get protobuf-gradle-plugin and kotlin-gradle-plugin to cooperate, you need to ensure that the Java code is (re)generated before invoking the Kotlin compiler.
For Gradle's default source sets, main and test, you can do that like this:
compileKotlin.dependsOn ':generateProto'
compileTestKotlin.dependsOn ':generateTestProto'
If you are using other source sets, you'll need to make adjustments.
Older versions of protobuf-gradle-plugin also required updating sourceSets, but newer versions do not seem to require this.
// Don't do this with protobuf-gradle-plugin 0.9.0 or higher
sourceSets.main.java.srcDirs += "${protobuf.generatedFilesBaseDir}/main/java"
sourceSets.test.java.srcDirs += "${protobuf.generatedFilesBaseDir}/test/java"
For Kotlin and Android:
android {
sourceSets {
debug.java.srcDirs += 'build/generated/source/proto/debug/java'
release.java.srcDirs += 'build/generated/source/proto/release/java'
}
}
An additional source directory has to be added for every build type. In this sample there are two build types: debug and release.
If you're using grpc, another line has to be added per build type:
android {
sourceSets {
debug.java.srcDirs += 'build/generated/source/proto/debug/java'
debug.java.srcDirs += 'build/generated/source/proto/debug/grpc'
release.java.srcDirs += 'build/generated/source/proto/release/java'
release.java.srcDirs += 'build/generated/source/proto/release/grpc'
}
}
At least with Kotlin 1.0.6, protobuf-gradle-plugin 0.8.0, protobuf 3.2.x and grpc 1.x it's not required to fiddle with the task order.
if you are working with multiple build types and flavors in android and with protobuf-lite use below with kotlin.
for example I have debug and release builds with demo and prod flavors it will create demoDebug, demoRelease and prodDebug and prodRelease variants.
then use
`
android{
sourceSets {
debug.java.srcDirs += 'build/generated/source/proto/demoDebug/javalite'
debug.java.srcDirs += 'build/generated/source/proto/prodDebug/javalite'
release.java.srcDirs += 'build/generated/source/proto/demoRelease/javalite'
release.java.srcDirs += 'build/generated/source/proto/prodRelease/javalite'
}
}
`
tie the different compileKotlin with generateProto
tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile).all {
if (getName() == 'compileDemoDebugKotlin')
dependsOn(':app:generateDemoDebugProto')
if (getName() == 'compileDemoReleaseKotlin')
dependsOn(':app:generateDemoReleaseProto')
if (getName() == 'compileProdDebugKotlin')
dependsOn(':app:generateProdDebugProto')
if (getName() == 'compileProdReleaseKotlin')
dependsOn(':app:generateProdReleaseProto')
}
For the gradle setup :
plugins {
id 'com.android.application'
id 'kotlin-android'
id 'com.google.protobuf' version "0.8.17"
}
Then at the bottom of the build.gradle
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.10.0"
}
// Generates the java Protobuf-lite code for the Protobufs in this project. See
// https://github.com/google/protobuf-gradle-plugin#customizing-protobuf-compilation
// for more information.
generateProtoTasks {
all().each { task ->
task.builtins {
java {
option 'lite'
}
}
}
}
}

avro gradle plugin sample usage

I am trying to use the avro-gradle-plugin on github, but have not gotten any luck getting it to work. Does anyone have any sample code on how they get it to work?
I figured out how to do it myself. The following is a snippet that I would like to share for people who might run into the same issues as I did:
apply plugin: 'java'
apply plugin: 'avro-gradle-plugin'
sourceCompatibility = "1.6"
targetCompatibility = "1.6"
buildscript {
repositories {
maven {
// your maven repo information here
}
}
dependencies {
classpath 'org.apache.maven:maven-artifact:2.2.1'
classpath 'org.apache.avro:avro-compiler:1.7.1'
classpath 'org.apache.avro.gradle:avro-gradle-plugin:1.7.1'
}
}
compileAvro.source = 'src/main/avro'
compileAvro.destinationDir = file("$buildDir/generated-sources/avro")
sourceSets {
main {
java {
srcDir compileAvro.destinationDir
}
}
}
dependencies {
compileAvro
}
I found "com.commercehub.gradle.plugin.avro" gradle plugin to work better.
use the folllowing:
// Gradle 2.1 and later
plugins {
id "com.commercehub.gradle.plugin.avro" version "VERSION"
}
// Earlier versions of Gradle
buildscript {
repositories {
jcenter()
}
dependencies {
classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:VERSION"
}
}
apply plugin: "com.commercehub.gradle.plugin.avro"
more details at https://github.com/commercehub-oss/gradle-avro-plugin
When evaluating a plugin the following questions needs to be asked:
Are generated files included into source jar?
Is plugin fast? Good plugin use avro tools api instead of forking VM for every file. For large amount of files creating VM for every file can take 10min to compile.
Do you need intermediate avsc files?
Is build incremental (i.e. do not regenerate all files unless one of the sources changed)?
Is plugin flexible enough to give access to generated schema files, so further actions, such as registration schema in schema repository can be made?
It is easy enough to implement without any plugin if you are not happy with plugin or need more flexibility.
//
// define source and destination
//
def avdlFiles = fileTree('src/Schemas').include('**/*.avdl')
// Do NOT generate into $buildDir, because IntelliJ will ignore files in
// this location and will show errors in source code
def generatedJavaDir = "generated/avro/java"
sourceSets.main.java.srcDir generatedJavaDir
//
// Make avro-tools available to the build script
//
buildscript {
dependencies {
classpath group:'org.apache.avro', name:'avro-tools' ,version: avro_version
}
}
//
// Define task's input and output, compile idl to schema and schema to java
//
task buildAvroDtos(){
group = "build"
inputs.files avdlFiles
outputs.dir generatedJavaDir
doLast{
avdlFiles.each { avdlFile ->
def parser = new org.apache.avro.compiler.idl.Idl(avdlFile)
parser.CompilationUnit().getTypes().each { schema ->
def compiler = new org.apache.avro.compiler.specific.SpecificCompiler(schema)
compiler.compileToDestination(avdlFile, new File(generatedJavaDir))
}
}
}
}
//
// Publish source jar, including generated files
//
task sourceJar(type: Jar, dependsOn: buildAvroDtos) {
from sourceSets.main.allSource
// Package schemas into source jar
into("Schemas") { from avdlFiles }
}
// Clean "generated" folder upon "clean" task
clean {
delete('generated')
}
Configuration for avro with gradle as build tool need to add along with applying java plugin.
below changes in settings.gradle
pluginManagement {
repositories {
gradlePluginPortal()
mavenCentral()
}
}
below changes in build.gradle
plugins {
id "com.github.davidmc24.gradle.plugin.avro" version "1.3.0"
}
repositories {
mavenCentral()
}
dependencies {
implementation "org.apache.avro:avro:1.11.0"
}
generateAvroJava {
source("${projectDir}/src/main/resources/avro")//sourcepath avrofile
}
if you want to generate setter methods too add this task too in build.gradle
avro {
createSetters = true
}
link for reference

Resources