Accessing Custom Options from .desc file in Protobuf - protocol-buffers

I have a proto with the following definitions.
import "google/protobuf/descriptor.proto";
extend google.protobuf.FieldOptions{
optional bool is_key = 50002;
}
message Foo{
int64 id = 1 [(is_key) = true];
}
I generated a .desc file for the above. I was able to access all the Fields and Message defined by the FieldDescriptorProto and DescriptorProto types but not sure how to access the options defined and the value provided to it in this case is_key.
Could anyone provide me with a java version that could access the options from the .desc file

Not sure If this is the right way but all I was trying to do was read custom options from a descriptor_set.desc file.
I parsed the desc file, the problem is that it returns fileDescriptorProto Types. So I parsed them and built a dependencies graph. This helped me to create FileDescriptor Types.
Through FileDescriptor Types I was able to get Extensions and add them to extensionRegistry.
Post Which I parsed the .desc file once again but this time with extensionRegistry.
val filePathToFileDescriptorMap = mutableMapOf <String, FileDescriptor>(DescriptorProtos.getDescriptor().file.name to
DescriptorProtos.getDescriptor())
val extensionRegistry = ExtensionRegistry.newInstance()
fun buildDependencies(fileDescriptorProto: DescriptorProtos.FileDescriptorProto,
filePathToFileDescriptorProtoMap: Map<String, DescriptorProtos.FileDescriptorProto>) : FileDescriptor {
val dependencies = fileDescriptorProto.dependencyList.map { dependency ->
if(filePathToFileDescriptorMap.containsKey(dependency)) {
filePathToFileDescriptorMap[dependency]
}
else if(!filePathToFileDescriptorProtoMap.containsKey(dependency)) {
throw Exception("dependency not found $dependency")
}
else {
buildDependencies(filePathToFileDescriptorProtoMap[dependency]!!, filePathToFileDescriptorProtoMap)
}
}
filePathToFileDescriptorMap[fileDescriptorProto.name] =
FileDescriptor.buildFrom(fileDescriptorProto, dependencies.toTypedArray())
filePathToFileDescriptorMap[fileDescriptorProto.name]!!.extensions.forEach {
if(it.type == Descriptors.FieldDescriptor.Type.MESSAGE){
extensionRegistry.add(it, DynamicMessage.newBuilder(it.messageType).build())
}else{
extensionRegistry.add(it)
}
}
return filePathToFileDescriptorMap[fileDescriptorProto.name]!!
}
fun registerExtensions(){
val fileDescriptorSet = DescriptorProtos.FileDescriptorSet.parseFrom(
FileInputStream("src/generated/resources/desc_set.desc"),
)
val filePathToFileDescriptorProtoMap = fileDescriptorSet.fileList.associateBy { it.name }
fileDescriptorSet.fileList.forEach { fileDescriptorProto ->
if(!filePathToFileDescriptorMap.containsKey(fileDescriptorProto.name)){
buildDependencies(fileDescriptorProto, filePathToFileDescriptorProtoMap)
}
}
val fileDescriptorSetWithOptions = DescriptorProtos.FileDescriptorSet.parseFrom(
FileInputStream("src/generated/resources/desc_set.desc"),
extensionRegistry
)
println(fileDescriptorSetWithOptions)
}

Related

TransformerAction is not triggered on dependency resolution

I would like to transform some dependencies when Gradle resolves them.
I have followed the 'Transforming dependency artifacts on resolution' documentation, but my TransformAction is not triggered.
I have
created a custom configuration, gitHub
created custom attributes, and both
applied them to the gitHub configuration,
and registered them in the dependencies {} block)
created and registered a custom TransformAction.
created a task, resolveGitHubDependencies, that will resolve the gitHub configuration.
I have tried a few different adjustments, but nothing seems to have an impact.
What am I missing? Or is this a bug?
Here's the code I've created. It can be tested by running ./gradlew resolveGitHubDependencies. I expect that MyTransformer will print some log statements, but it doesn't print anything.
import org.slf4j.LoggerFactory
plugins {
base
}
repositories {
ivy("https://github.com/") {
name = "GitHub"
patternLayout {
artifact("/[organisation]/[module]/archive/[revision].[ext]")
artifact("/[organisation]/[module]/releases/download/[revision]/[module](-[revision]).[ext]")
}
metadataSources { artifact() }
}
}
// from https://github.com/gradle/gradle/issues/19707
val transformed = Attribute.of("my-library.transformed", Boolean::class.javaObjectType)
val dependencySource = Attribute.of("dependency source", String::class.java)
// my custom configuration - I want to transform all dependencies from this configuration
val gitHub by configurations.creating {
isCanBeConsumed = false
isCanBeResolved = true
attributes {
attribute(dependencySource, "gitHub")
attribute(transformed, false)
}
}
// this is the transformer that I expect to be triggered on dependency resolution
abstract class MyTransformer : TransformAction<MyTransformer.Parameters> {
private val logger = LoggerFactory.getLogger(this::class.java)
interface Parameters : TransformParameters {
#get:Input
val sourceAttributes: MapProperty<Attribute<*>, Any?>
}
#get:PathSensitive(PathSensitivity.NAME_ONLY)
#get:InputArtifact
abstract val inputArtifact: Provider<FileSystemLocation>
override fun transform(outputs: TransformOutputs) {
val sourceAttributes = parameters.sourceAttributes.get()
logger.error("sourceAttributes: $sourceAttributes")
logger.error("sourceAttributes: ${sourceAttributes.keys}")
logger.error("sourceAttributes: ${sourceAttributes.values.joinToString { it.toString() }}")
val input = inputArtifact.get().asFile
outputs.file(input)
}
}
dependencies {
// this is the dependency I want to transform
gitHub("emcrisostomo:fswatch:1.17.1#tar.gz")
attributesSchema {
attribute(dependencySource)
attribute(transformed)
}
// artifactTypes.configureEach {
// attributes.attribute(transformed, false)
// }
registerTransform(MyTransformer::class) {
// from.attributes.attribute(dependencySource, "gitHub")
// to.attributes.attribute(dependencySource, "gitHub")
from.attributes.attribute(dependencySource, "gitHub").attribute(transformed, false)
to.attributes.attribute(dependencySource, "gitHub").attribute(transformed, true)
parameters {
sourceAttributes.set(
from.attributes.keySet().associateWith { key ->
from.attributes.getAttribute(key)
}
)
}
}
}
val resolveGitHubDependencies by tasks.registering(Sync::class) {
group = "gh transform"
from(gitHub)
into(temporaryDir)
}
Gradle version: 7.6

How to add arguments to a org.gradle.api.Action?

I'm writing a custom Gradle plugin that registers an extension and uses Actions for configuring it so you can use it like
myExtension {
info {
someConfig("a") {
// config for someConfig["a"]
}
someConfig("b") {
// config for someConfig["b"]
}
}
}
Based on how much someConfig you define, the plugin created new Gradle tasks dynamically. I've seen some scenarios in which it'd useful to configure all someConfig with the same values, so instead of writing
myExtension {
info {
someConfig("a") {
someValue = x
}
someConfig("b") {
someValue = x
}
}
}
I was thinking in providing something like
myExtension {
info {
someConfig("a")
someConfig("b")
allConfigs { configName ->
someValue = x
}
}
}
When trying to code it, I've used an objectFactory for creating instances of Action so I can configure the extension values
val allMyConfigs = mutableMapOf<String, SomeConfig>()
fun someConfig(configName: String, init: Action<in EnvironmentInfo>) {
val myConfig = allMyConfigs.computeIfAbsent(env) {
objectFactory.newInstance(SomeConfig::class.java, env)
}
init.execute(environmentInfo)
}
but since Action is a (SomeConfig) -> Unit (i.e. a function receiving a SomeConfig instance) I'm unable to pass the config name (an String) as param
Is there a way to create an Action, so it's compatible with both build.gradle and build.gradle.kts files, that receives an extra parameter and maintain the SomeConfig as delegate?

How to import a helper class in buildSrc/build.gradle.kts, settings.gradle.kts, and build.gradle.kts?

I'd like to create a class to help me loading different types of properties (local.properties, gradle.properties, $GRADLE_HOME/gradle.properties, environment variables, system properties, and custom properties files (maybe in other formats like yml, xml, etc.).
Also, I'd like to use this in my buildSrc/build.gradle.kts, settings.gradle.kts, and build.gradle.kts.
Please consider that we are using Gradle 6.+.
A simple implementation of this class would be (the full implementation would be a lot of more powerful):
plugins/properties/build.gradle.kts:
package com.example
object Properties {
val environmentVariables = System.getenv()
}
How can we successfully import this Properties class in all of those files (buildSrc/build.gradle.kts, settings.gradle.kts, build.gradle.kts) and use it from there? Something like:
println(com.example.Properties.environmentVariables["my.property"])
Can we do that creating this class inside of a plugin and applying it from there? Without pre-compiling and releasing the plugin? Maybe something like:
apply("plugins/properties/build.gradle.kts")
How would it be a minimal implementation for this?
I tried different approaches but I'm not being able to find a way that work with those 3 files altogether.
I'm not completely satisfied with this approach but maybe it can help others.
I wasn't able to reuse a class but a function in all places, like this:
settings.gradle.kts
apply("plugin/properties/build.gradle.kts")
#Suppress("unchecked_cast", "nothing_to_inline")
inline fun <T> uncheckedCast(target: Any?): T = target as T
val getProperty = uncheckedCast<(key: String) -> String>(extra["getProperty"])
println(getProperty("group"))
buildSrc/build.gradle.kts
apply("../plugin/properties/build.gradle.kts")
#Suppress("unchecked_cast", "nothing_to_inline")
inline fun <T> uncheckedCast(target: Any?): T = target as T
val getProperty = uncheckedCast<(key: String) -> String>(extra["getProperty"])
println(getProperty("group"))
build.gradle.kts
// Can be used inside of the file
apply("plugin/properties/build.gradle.kts")
#Suppress("unchecked_cast", "nothing_to_inline")
inline fun <T> uncheckedCast(target: Any?): T = target as T
val getProperty = uncheckedCast<(key: String) -> String>(extra["getProperty"])
println(getProperty("group"))
buildScript {
// Since the other getProperty is not visible here we need to do this again.
apply("plugin/properties/build.gradle.kts")
#Suppress("unchecked_cast", "nothing_to_inline")
inline fun <T> uncheckedCast(target: Any?): T = target as T
val getProperty = uncheckedCast<(key: String) -> String>(extra["getProperty"])
println(getProperty("group"))
}
plugin/properties/build.gradle.kts
import java.io.File
import java.nio.file.Path
import java.nio.file.Paths
import java.util.Properties as JavaProperties
import org.gradle.api.initialization.ProjectDescriptor
object Properties {
lateinit var rootProjectAbsolutePath : String
val local: JavaProperties by lazy {
loadProperties(JavaProperties(), Paths.get(rootProjectAbsolutePath, "local.properties").toFile())
}
val environment by lazy {
System.getenv()
}
val system: JavaProperties = JavaProperties()
val gradle: JavaProperties by lazy {
loadProperties(JavaProperties(), Paths.get(rootProjectAbsolutePath, "gradle.properties").toFile())
}
val globalGradle: JavaProperties by lazy {
loadProperties(JavaProperties(), Paths.get(System.getProperty("user.home"), ".gradle", "gradle.properties").toFile())
}
fun containsKey(vararg keys: String): Boolean {
if (keys.isNullOrEmpty()) return false
keys.forEach {
when {
local.containsKey(it) -> return true
environment.containsKey(it) -> return true
system.containsKey(it) -> return true
gradle.containsKey(it) -> return true
globalGradle.containsKey(it) -> return true
}
}
return false
}
fun get(vararg keys: String): String {
return this.getAndCast<String>(*keys) ?: throw IllegalArgumentException("Property key(s) ${keys} not found.")
}
fun getOrNull(vararg keys: String): String? {
return getAndCast<String>(*keys)
}
inline fun <reified R> getOrDefault(vararg keys: String, default: R?): R? {
return getAndCast<R>(*keys) ?: default
}
inline fun <reified R> getAndCast(vararg keys: String): R? {
if (keys.isNullOrEmpty()) return null
keys.forEach {
val value = when {
local.containsKey(it) -> local[it]
environment.containsKey(it) -> environment[it]
system.containsKey(it) -> system[it]
gradle.containsKey(it) -> gradle[it]
globalGradle.containsKey(it) -> globalGradle[it]
else -> null
}
// TODO Improve the casting using Jackson
if (value != null) return value as R
}
return null
}
private fun loadProperties(target: JavaProperties, file: File): JavaProperties {
if (file.canRead()) {
file.inputStream().use { target.load(it) }
}
return target
}
}
if (rootProject.name == "buildSrc") {
Properties.rootProjectAbsolutePath = rootDir.parent
} else {
Properties.rootProjectAbsolutePath = rootDir.absolutePath
}
extra["getProperty"] = {key: String -> Properties.get(key)}

In Spring Webflux how to go from an `OutputStream` to a `Flux<DataBuffer>`?

I'm building a tarball dynamically, and would like to stream it back directly, which should be 100% possible with a .tar.gz.
The below code is the closest thing I could get to a dataBuffer, through lots of googling. Basically, I need something that implements an OutputStream and provides, or publishes, to a Flux<DataBuffer> so that I can return that from my method, and have streaming output, instead of buffering the entire tarball in ram (which I'm pretty sure is what is happening here). I'm using apache Compress-commons, which has a wonderful API, but it's all OutputStream based.
I suppose another way to do it would be to directly write to the response, but I don't think that would be properly reactive? Not sure how to get an OutputStream out of some sort of Response object either.
This is kotlin btw, on Spring Boot 2.0
#GetMapping("/cookbook.tar.gz", "/cookbook")
fun getCookbook(): Mono<DefaultDataBuffer> {
log.info("Creating tarball of cookbooks: ${soloConfig.cookbookPaths}")
val transformation = Mono.just(soloConfig.cookbookPaths.stream()
.toList()
.flatMap {
Files.walk(Paths.get(it)).map(Path::toFile).toList()
})
.map { files ->
//Will make one giant databuffer... but oh well? TODO: maybe use some kind of chunking.
val buffer = DefaultDataBufferFactory().allocateBuffer()
val outputBufferStream = buffer.asOutputStream()
//Transform my list of stuff into an archiveOutputStream
TarArchiveOutputStream(GzipCompressorOutputStream(outputBufferStream)).use { taos ->
taos.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU)
log.info("files to compress: ${files}")
for (file in files) {
if (file.isFile) {
val entry = "cookbooks/" + file.name
log.info("Adding ${entry} to tarball")
taos.putArchiveEntry(TarArchiveEntry(file, entry))
FileInputStream(file).use { fis ->
fis.copyTo(taos) //Copy that stuff!
}
taos.closeArchiveEntry()
}
}
}
buffer
}
return transformation
}
I puzzled through this, and have an effective solution. You implement an OutputStream and take those bytes and publish them into a stream. Be sure to override close, and send an onComplete. Works great!
#RestController
class SoloController(
val soloConfig: SoloConfig
) {
val log = KotlinLogging.logger { }
#GetMapping("/cookbooks.tar.gz", "/cookbooks")
fun streamCookbook(serverHttpResponse: ServerHttpResponse): Flux<DataBuffer> {
log.info("Creating tarball of cookbooks: ${soloConfig.cookbookPaths}")
val publishingOutputStream = PublishingOutputStream(serverHttpResponse.bufferFactory())
//Needs to set up cookbook path as a parent directory, and then do `cookbooks/$cookbook_path/<all files>` for each cookbook path given
Flux.just(soloConfig.cookbookPaths.stream().toList())
.doOnNext { paths ->
//Transform my list of stuff into an archiveOutputStream
TarArchiveOutputStream(GzipCompressorOutputStream(publishingOutputStream)).use { taos ->
taos.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU)
paths.forEach { cookbookDir ->
if (Paths.get(cookbookDir).toFile().isDirectory) {
val cookbookDirFile = Paths.get(cookbookDir).toFile()
val directoryName = cookbookDirFile.name
val entryStart = "cookbooks/${directoryName}"
val files = Files.walk(cookbookDirFile.toPath()).map(Path::toFile).toList()
log.info("${files.size} files to compress")
for (file in files) {
if (file.isFile) {
val relativePath = file.toRelativeString(cookbookDirFile)
val entry = "$entryStart/$relativePath"
taos.putArchiveEntry(TarArchiveEntry(file, entry))
FileInputStream(file).use { fis ->
fis.copyTo(taos) //Copy that stuff!
}
taos.closeArchiveEntry()
}
}
}
}
}
}
.subscribeOn(Schedulers.parallel())
.doOnComplete {
publishingOutputStream.close()
}
.subscribe()
return publishingOutputStream.publisher
}
class PublishingOutputStream(bufferFactory: DataBufferFactory) : OutputStream() {
val publisher: UnicastProcessor<DataBuffer> = UnicastProcessor.create(Queues.unbounded<DataBuffer>().get())
private val bufferPublisher: UnicastProcessor<Byte> = UnicastProcessor.create(Queues.unbounded<Byte>().get())
init {
bufferPublisher
.bufferTimeout(4096, Duration.ofMillis(100))
.doOnNext { intList ->
val buffer = bufferFactory.allocateBuffer(intList.size)
buffer.write(intList.toByteArray())
publisher.onNext(buffer)
}
.doOnComplete {
publisher.onComplete()
}
.subscribeOn(Schedulers.newSingle("publisherThread"))
.subscribe()
}
override fun write(b: Int) {
bufferPublisher.onNext(b.toByte())
}
override fun close() {
bufferPublisher.onComplete() //which should trigger the clean up of the whole thing
}
}
}

Adding a property to an existing enum in Kotlin

Given the following enum defined in an external api.
public enum Status {
COMPLETE,
RUNNING,
WAITING
}
I would like a way to add a int flag to each enum value. I know that I can extend the enum:
fun Status.flag(): Int {
when(this) {
RUNNING -> return 1;
WAITING -> return 2;
else -> return 0;
}
}
However I would like to define those int flag values as constants. Maybe a companion object, but I don't think I can extend an existing enum and add a companion object.
Any ideas?
Unless you are using a field that already exists in the original enum (like ordinal), you won't be able to do what you're asking without wrapping the external enum in your own enum.
Sure you could use ordinal, but a newer version of the external API may change the order of the items in the enum, so I wouldn't recommend it. But, if you REALLY want to, you could do something like this (again, this is NOT recommended):
val Status.flag: Int
get() = this.ordinal
But I'd definitely recommend wrapping it. That way you guarantee that the flag integers you define won't change.
enum class MyStatus(val status: Status, val flag: Int) {
COMPLETE(Status.COMPLETE, 0),
RUNNING(Status.RUNNING, 1),
WAITING(Status.WAITING, 2);
companion object {
private val STATUS_TO_MYSTATUS = values().associateBy { it.status }
fun fromStatus(status: Status): MyStatus {
return STATUS_TO_MYSTATUS[status] ?: throw Exception("No MyStatus found for status ${status.name}")
}
}
}
You can then convert Status to MyStatus by using MyStatus.fromStatus(...). Or you can add an extension function to Status to easily convert to MyStatus.
fun Status.toMyStatus() = MyStatus.fromStatus(this)
You can add extension properties/methods to the companion object of enum/class/etc. if one exists:
val Status.Companion.COMPLETE_INT = 0
val Status.Companion.RUNNING_INT = 1
but indeed you can't currently "create" the companion object if it doesn't. So just put the constants into your own non-companion object:
object StatusFlags {
const val COMPLETE_INT = 0
const val RUNNING_INT = 1
const val WAITING_INT = 2
}
fun Status.flag(): Int {
when(this) {
RUNNING -> return StatusFlags.RUNNING_INT
...
}
}

Resources