Custom protoc plugin parsing not working for custom options - protocol-buffers

I am trying to write a protoc plugin that requires me to use custom options. I defined my custom option as shown in the example (https://developers.google.com/protocol-buffers/docs/proto#customoptions):
import "google/protobuf/descriptor.proto";
extend google.protobuf.MessageOptions {
string my_option = 51234;
}
I use it as follows:
message Hello {
bool greeting = 1;
string name = 2;
int32 number = 3;
option (my_option) = "telephone";
}
However, when I read the parsed request, the options field is empty for the "Hello" message.
I am doing the following to read
data = sys.stdin.read()
request = plugin.CodeGeneratorRequest()
request.ParseFromString(data)
When I print "request," it just gives me this
message_type {
name: "Hello"
field {
name: "greeting"
number: 1
label: LABEL_REQUIRED
type: TYPE_BOOL
json_name: "greeting"
}
field {
name: "name"
number: 2
label: LABEL_REQUIRED
type: TYPE_STRING
json_name: "name"
}
field {
name: "number"
number: 3
label: LABEL_OPTIONAL
type: TYPE_INT32
json_name: "number"
}
options {
}
}
As seen, the options field is empty even though I defined options in my .proto file. Is my syntax incorrect for defining custom options? Or could it be a problem with my version of protoc?

I'm making my protobuf python plugin.
I also got the problem like yours and i have found a solution for that.
Put your custom options to a file my_custom.proto
Use protoc to gen a python file from my_custom.proto => my_custom_pb2.py
In your python plugin code, import my_custom_pb2.py import my_custom_pb2

Turns out you need to have the _pb2.py file imported for the .proto file in which the custom option is defined. For example, it you are parsing a file (using ParseFromString) called example.proto which uses a custom option defined in option.proto, you must import option_pb2.py in the Python file that calls ParseFromString.

Related

gql codegen generate file within the same folder as in query/mutation file path

Using this guide https://the-guild.dev/graphql/codegen/docs/advanced/generated-files-colocation
It works as intended for "operation-types" file, but how about the "types.ts" file itself, is it possible to generate separate type file depending for each operation needs, or just create the object types inside the ".generated.tsx" file?
my config is as follow, similar to the docs, its just putting into a folder called __generated__. Thank you.
const config: CodegenConfig = {
schema: 'http://localhost:4000/graphql',
documents: ['src/**/*.{gql,graphql}'],
generates: {
'src/codegen/types.ts': {
plugins: ['typescript'],
},
'src/': {
preset: 'near-operation-file',
presetConfig: { extension: '.generated.tsx', baseTypesPath: 'codegen/types.ts', folder: '__generated__' },
plugins: ['typescript-operations'],
}
}
}

Terraform: set max_buckets in ElasticSearch

I need to set max_buckets in elasticsearch aws. So far I've tried using a max_buckets key right in the module block, but that didn't work. Next try was using advanced_options
module "elasticsearch" {
es_version = "6.3"
advanced_options = {
"search.max_buckets" = "123456"
}
But this causes:
Error: Unsupported argument
on elasticsearch.tf line 14, in module "elasticsearch":
14: advanced_options = {
How can I set max_buckets?
Which module are you using? The aws_elasticsearch_domain resource has the advanced_options argument.
advanced_options - Key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes).
resource "aws_elasticsearch_domain" "es" {
domain_name = "${var.domain}"
elasticsearch_version = "6.3"
advanced_options = {
"rest.action.multi.allow_explicit_index" = "true"
}
}
Could you provide more details about your implementation? It seems in your example that a double-quote is missing for search.max_buckets and if you're using a module, I think you should pass the source argument.

Specify the webpack "mainFields" on a case by case basis

Webpack has a resolve.mainFields configuration: https://webpack.js.org/configuration/resolve/#resolvemainfields
This allows control over what package.json field should be used as an entrypoint.
I have an app that pulls in dozens of different 3rd party packages. The use case is that I want to specify what field to use depending on the name of the package. Example:
For package foo use the main field in node_modules/foo/package.json
For package bar use the module field in node_modules/bar/package.json
Certain packages I'm relying on are not bundled in a correct manner, the code that the module field is pointing to does not follow these rules: https://github.com/dherman/defense-of-dot-js/blob/master/proposal.md This causes the app to break if I wholesale change the webpack configuration to:
resolve: {
mainFields: ['module']
}
The mainFields has to be set to main to currently get the app to work. This causes it to always pull in the CommonJS version of every dependency and miss out on treeshaking. Hoping to do something like this:
resolve: {
foo: {
mainFields: ['main']
},
bar: {
mainFields: ['module'],
}
Package foo gets bundled into my app via its main field and package bar gets bundled in via its module field. I realize the benefits of treeshaking with the bar package, and I don't break the app with foo package (has a module field that is not proper module syntax).
One way to achieve this would be instead of using resolve.mainFields you can make use of resolve.plugins option and write your own custom resolver see https://stackoverflow.com/a/29859165/6455628 because by using your custom resolver you can programmatically resolve different path for different modules
I am copy pasting the Ricardo Stuven's Answer here
Yes, it's possible. To avoid ambiguity and for easier implementation,
we'll use a prefix hash symbol as marker of your convention:
require("#./components/SettingsPanel");
Then add this to your configuration file (of course, you can refactor
it later):
var webpack = require('webpack');
var path = require('path');
var MyConventionResolver = {
apply: function(resolver) {
resolver.plugin('module', function(request, callback) {
if (request.request[0] === '#') {
var req = request.request.substr(1);
var obj = {
path: request.path,
request: req + '/' + path.basename(req) + '.js',
query: request.query,
directory: request.directory
};
this.doResolve(['file'], obj, callback);
}
else {
callback();
}
});
}
};
module.exports = {
resolve: {
plugins: [
MyConventionResolver
]
}
// ...
};
resolve.mainFields not work in my case, but resolve.aliasFields works.
More details in https://stackoverflow.com/a/71555568/7534433

Go structs to OpenAPI to JSONSchema generation automatically

I have a Go struct for which I want to generate an OpenAPI schema automatically. Once I have an OpenAPI definition of that struct I wanna generate JSONSchema of it, so that I can validate the input data that comes and is gonna be parsed into those structs.
The struct looks like the following:
// mySpec: io.myapp.MinimalPod
type MinimalPod struct {
Name string `json:"name"`
// k8s: io.k8s.kubernetes.pkg.api.v1.PodSpec
v1.PodSpec
}
Above struct is clearly an augmentation of what Kubernetes PodSpec is.
Now the approach that I have used is to generate definition part for my struct MinimalPod, the definition for PodSpec will come from upstream OpenAPI spec of Kubernetes. PodSpec has a key io.k8s.kubernetes.pkg.api.v1.PodSpec in the upstream OpenAPI spec, this definition is injected from there in my Properties. Now in my code that parses above struct I have templates of what to do if struct field is string.
If the field has a comment that starts with k8s: ... the next part is Kubernetes object's OpenAPI definition key. In our case the OpenAPI definition key is io.k8s.kubernetes.pkg.api.v1.PodSpec. So I retrieve that field's definition from the upstream OpenAPI definition and embed it into the definition of my struct.
Once I have generated an OpenAPI definition for this struct which is injected in Kubernetes OpenAPI schema's definition with key being io.myapp.MinimalPod. Now I can use the tool openapi2jsonschema to generate JSONSchema out of this one. Which generates a JSONSchema file named MinimalPod.json.
Now jsonschema tool and the file MinimalPod.json can be used for validating input given to my tool parser to see if all fields were given right.
Is this the right approach of doing things, or is there a tool/library and if I feed Go structs to it, it gives me OpenAPI schema? It would be fine if it does not identify where to inject Kubernetes OpenAPI schema from even automatic parsing of Go structs and giving OpenAPI definition would be much appreciated.
Update 1
After following #mehdy 's instructions, this is what I have tried:
I have used this import path github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1 to import the PodSpec definition instead of k8s.io/api/core/v1 and code looks like this:
package foomodel
import "github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1"
// MinimalPod is a minimal pod.
// +k8s:openapi-gen=true
type MinimalPod struct {
Name string `json:"name"`
v1.PodSpec
}
Now when I generate the same with flag -i changed from k8s.io/api/core/v1 to github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1
$ go run example/openapi-gen/main.go -i k8s.io/kube-openapi/example/model,github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1 -h example/foomodel/header.txt -p k8s.io/kube-openapi/example/foomodel
This is what is generated:
$ cat openapi_generated.go
// +build !ignore_autogenerated
/*
======
Some random text
======
*/
// This file was autogenerated by openapi-gen. Do not edit it manually!
package foomodel
import (
spec "github.com/go-openapi/spec"
common "k8s.io/kube-openapi/pkg/common"
)
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
"k8s.io/kube-openapi/example/model.Container": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Container defines a single application container that you want to run within a pod.",
Properties: map[string]spec.Schema{
"health": {
SchemaProps: spec.SchemaProps{
Description: "One common definitions for 'livenessProbe' and 'readinessProbe' this allows to have only one place to define both probes (if they are the same) Periodic probe of container liveness and readiness. Container will be restarted if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes",
Ref: ref("k8s.io/client-go/pkg/api/v1.Probe"),
},
},
"Container": {
SchemaProps: spec.SchemaProps{
Ref: ref("k8s.io/client-go/pkg/api/v1.Container"),
},
},
},
Required: []string{"Container"},
},
},
Dependencies: []string{
"k8s.io/client-go/pkg/api/v1.Container", "k8s.io/client-go/pkg/api/v1.Probe"},
},
}
}
I get only this much of the configuration generated. While when I switch back to "k8s.io/api/core/v1" I get config code auto generated which is more than 8k lines. What am I missing here?
Here definition of k8s.io/client-go/pkg/api/v1.Container and k8s.io/client-go/pkg/api/v1.Probe is missing while when I use k8s.io/api/core/v1 as import everything is generated.
Note: To generate above steps, please git clone https://github.com/kedgeproject/kedge in GOPATH.
You can use kube-openapi package for this. I am going to add a sample to the repo but I've tested this simple model:
// Car is a simple car model.
// +k8s:openapi-gen=true
type Car struct {
Color string
Capacity int
// +k8s:openapi-gen=false
HiddenFeature string
}
If you assume you created this file in
go run example/openapi-gen/main.go -h example/model/header.txt -i k8s.io/kube-openapi/example/model -p k8s.io/kube-openapi/example/model
(you also need to add a header.txt file). You should see a new file created in example/model folder called openapi_generated.go. This is an intermediate generated file that has your OpenAPI model in it:
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
"k8s.io/kube-openapi/example/model.Car": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Car is a simple car model.",
Properties: map[string]spec.Schema{
"Color": {
SchemaProps: spec.SchemaProps{
Type: []string{"string"},
Format: "",
},
},
"Capacity": {
SchemaProps: spec.SchemaProps{
Type: []string{"integer"},
Format: "int32",
},
},
},
Required: []string{"Color", "Capacity"},
},
},
Dependencies: []string{},
},
}
}
From there you should be able to call the generated method, get the model for your Type and get its Schema.
With some go get magic and changing the command line a little, I was able to generate the model for your model. Here is what you should change in your code:
package model
import "k8s.io/api/core/v1"
// MinimalPod is a minimal pod.
// +k8s:openapi-gen=true
type MinimalPod struct {
Name string `json:"name"`
v1.PodSpec
}
and then change the run command a little to include PodSpec in the generation:
go run example/openapi-gen/main.go -h example/model/header.txt -i k8s.io/kube-openapi/example/model,k8s.io/api/core/v1 -p k8s.io/kube-openapi/example/model
Here is what I got: https://gist.github.com/mbohlool/e399ac2458d12e48cc13081289efc55a

Gant: Copy with filtering

I have a 'doc' directory containing HTML documentation and each HTML contains placeholders for the application version and the SVN revision:
Welcome to the ... V${version} r${buildNumber}
In my Grails/Gant build script we create a doc package for which we first copy the doc directory to a staging area before zipping it up. I now wanna replace these placeholders with values like this (assume the variables appVersion and svnRevision are set properly:
ant.mkdir(dir: "${baseDocDir}")
ant.copy(todir: "${baseDocDir}") {
fileset(dir: "./src/main/doc", includes: '*.html')
filterset {
filter ( token : 'version' , value: appVersion )
filter ( token : 'buildNumber' , value : svnRevision )
}
}
The copy works but somehow the filter does not!
I can answer the question myself now. The following code works:
ant.copy(todir: "${baseDocDir}") {
filterset(begintoken: "\${", endtoken: "}") {
filter(token: "version", value: appVersion)
filter(token: "buildNumber", value: svnRevision)
}
fileset(dir: "./src/main/doc/", includes: "**/*.html")
}

Resources