YAML Schema Validation? - validation

Is there a schema validation language for YAML? I've googled but couldn't find anything useful.
Something like XSD format, using the language itself to describe the schema, would be the best choice in my case.

JSON Schema can be used with most YAML documents resulting in a more portable and better documented solution than Rx or Kwalify. JSON Schema is the only of the three for which I have been able to find editor support.
More information on using YAML and JSON Schema including tools and editor support is tracked on this page. At the time of writing, there is editor support in Visual Studio Code and a command-line based validation tool available via npm.
Full Disclosure: I authored the web site linked above to try to make the application of JSON Schema to YAML more discoverable. I also wrote an editor extension for VS Code that has since been superseded by the RedHat extension linked above.

Try Kwalify (Ruby and Java only), or Rx (many languages)

I wonder if it would make sense to reuse JSON schema for this. YAML can be easily converted to JSON without loosing any information (?), so in theory YAML could be validated by the same tool chain, allowing open source community to concentrate on one good schema tool chain. The schema itself could also be written in YAML and converted to JSON.

Good idea. Googled this up because I was looking for the same.
It's possible to convert YAML to XML in a defined manner (similarly to JSON <-> XML) and validate with a standard XML validator.
Depending on your platform, there are tools or snippets for that conversion: JavaScript (NPM), Ruby, Java (Jackson), Java (TestNG) (you'll need to see the source for what parameters it wants).
If done using an API, the error positions can even be mapped back to the original YAML file.

You can use this python ysd project to validate your yaml files. https://github.com/yonahd/ysd
Super simple to use
python yaml-validator/main.py -v yamls/example-values.yaml -r yamls/example-rules.yaml
Example rule file:
required: // field must exist and type must match
env: str
enabled: bool
replicas: int
optional: // if field exists type must match
disk: str
Example yaml file (helm values file):
network:
service:
port: 8060
enabled: true
image:
app: my-app:build
replicas: 1
env: dev
disk: local

If your project is in C++, you can also use the yaml-schema-cpp library. It allows to validate (and complete) the .yaml input files using schema files (YAML files with extension .schema) from your C++ code.

Related

I am looking for code policy enforcement tool for xml and Python

I have projects that are developed with xml and python code mostly (Odoo modules). There is a bit of .po files for translation and csv fields for data.
I would like to enforce specific policies in xml files, for example:
No duplicate id attributes.
A specific attribute must be present if child elements contain a specific tags.
On python, I want to enforce rules like:
Look for SQL queries, and make sure that they use specific parameter methods to prevent SQL injection
Follow a specific naming convention
Some attributes are required in classes that inherit a specific class
I hope that the idea is clear.
Is there any open source solution for this? Preferably linked with github and checks on every commit!
I found a python package made specifically for this, pylint-odoo, here.
It can also be installed with pip install pylint-odoo.
An example .pylintrc config file can be found at the web OCA module, here. They also have another file named .pylintrc-mandatory.
There is even a warning for duplicate xml id attribute W7902.

How to generate swagger3 (OpenAPI3) spec in (.json/.yaml) from protobuf (.proto) files?

My original use-case:
I am building an application in GO with a gRPC server (using protobuf), and wrapping it inside an HTTPS server (using gin). Only the HTTPS server is being published to the clients for use (by which I mean that my application can be accessed via REST API, that actually then dials on the gRPC endpoint), and I am publishing it using Swagger OpenAPI3 (version 3 is the main requirement here) specification. Both gRPC and HTTPS is required, and any solution should adhere to this architecture.
I don't want to maintain my server specification at two places, that is I don't want to maintain both proto files (.proto) and swagger spec (.json/.yaml). Since I absolutely have to write proto files to generate gRPC server, I want to automate the swagger spec generation (OpenAPI3).
Where I am:
I am able to generate swagger OpenAPI2 spec from protobuf files (.proto) using grpc-gateway library something like so: grpc-rest-go-example. But my requirement is OpenAPI3; more specifically I want to use the oneOf feature in OpenAPI3 and map to it from oneof feature of proto. This is not possible with OpenAPI2, as it does not allow an API to have a request/response body of multiple type definitions, which was a feature added in OpenAPI3 by enabling oneOf, anyOf and allOf constructs.
While trying to do so, I stumbled upon this library by GoogleAPIs googleapis/gnostic, the description for which is:
This repository contains a Go command line tool which converts JSON and YAML OpenAPI descriptions to and from equivalent Protocol Buffer representations.
At first look, this seems to exactly solve my problem, but as it turns out, this library only interconverts between protocol buffer (protobuf) binary (.pb) and swagger OpenAPI2/OpenAPI3 (.json/.yaml) files, which brings me to my new problem.
So for example for the following .pb file:
�3.0.1�…�
�Example service��Example service description*�
�Example contact2=
Apache 2.0�/http://www.apache.org/licenses/LICENSE-2.0.html:�1.0�!
�//localhost:9999/example/api/v1"â�
�
�/exampleResource��"���Example API��Example API description*�example-operation2B
#
example-query��query��example-query description �R�
Ê��stringBÇ��œ�
�200�”�
‘�
�OK�Š�
C
�application/json�/
-�+
)#/components/schemas/common.StatusMessage
C
�application/yaml�/
-�+
)#/components/schemas/common.StatusMessage�¥�
�400���
š�
�Bad Request�Š�
C
�application/json�/
-�+
)#/components/schemas/common.StatusMessage
C
�application/yaml�/
-�+
)#/components/schemas/common.StatusMessage*Y
W
U
�common.StatusMessage�=
;Ê��objectú�/
�
�message��
��string
�
�status��
��string
it generates the following swagger file:
openapi: 3.0.1
info:
title: Example service
description: Example service description
contact:
name: Example contact
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
version: "1.0"
servers:
- url: //localhost:9999/example/api/v1
paths:
/exampleResource:
get:
summary: Example API
description: Example API description
operationId: example-operation
parameters:
- name: example-query
in: query
description: example-query description
required: true
schema:
type: string
responses:
200:
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/common.StatusMessage'
application/yaml:
schema:
$ref: '#/components/schemas/common.StatusMessage'
400:
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/common.StatusMessage'
application/yaml:
schema:
$ref: '#/components/schemas/common.StatusMessage'
components:
schemas:
common.StatusMessage:
type: object
properties:
message:
type: string
status:
type: string
The .pb may not view properly, access it here. So something like:
�status��
��string
looks like:
<0x06>status<0x12><0x0b>
Ê<0x01><0x06>string
For above example, I first wrote the swagger spec and then generated the .pb but same can be done the other way around.
Current state:
If I have a way to convert between (.pb) and (.proto) files, the conversion loop will be closed and complete (.proto -> .pb -> .json/.yaml -> .pb -> .proto).
I am sure there has to be a way to achieve this, and hence there exists a solution to my original problem. But I could not find any article or piece of code that does it. Are there sane ways to interconvert between .pb and .proto files?
If you have an entirely different solution to my original use-case, please feel free to share that as well. It would help a lot.
Thanks in advance!
Edits:
(1) Thanks to recent comments, it is clear that a "conversion" between .pb and .proto is an absurd ask in the first place. But the original problem remains the same i.e. how to generate swagger3 (OpenAPI3) spec from protobuf file (.proto), either using annotations, tags or otherwise. Changing question title accordingly.
(2) Just the next day after I posted this, I bumped into gnostic-grpc repository, the description of which says:
This tool converts an OpenAPI v3.0 API description into a description of a gRPC service that can be used to implement that API using gRPC-JSON Transcoding.
Again, this got me exited too soon. Actually, this was a GSOC project, and as amazing as the idea of this repository is, it does not fulfill the requirements. Moreover, this is not an interconversion library, and very immature for any production use. In fact, it fails to deliver some of the basic fundamental features of the OpenAPI3 spec.
But this repository is headed in the right direction. My conclusion is to have a custom plugin that does this, mostly by extending the annotations library in GO.
(3) Apparently there are no good and obvious candidates for converting from .proto to OpenAPI3 spec (.yaml/.json), except gnostic-grpc which is very immature and highly work in progress for any kind of real use.
But for the reverse conversion, i.e. OpenAPI3 spec (.yaml/.json) to .proto, there is a good library called openapi-generator under OpenAPITools, which converts OpenAPI v2/3 spec to client/server stubs for almost all platforms. But since this is not the original ask, the question still remains open.
Although, your requirement seems to be to get a YAML (OpenAPNv3) specification from a PROTO, you may checkout this plugin - gnostic-grpc - for gnostic which does the reverse i.e. converts from a YAML/JSON spec to a proto along with gRPC service calls.
We failed to find the direct way to convert .proto file to OpenAPI3/Swagger3. Here is one workaround way
convert .proto files to Swagger2 through protoc-gen-openapiv2
then convert the Swagger2 files to Swagger3 through this link
Swagger Editor
Swagger Converter
Swagger Codegen version 3.x
Sample
protoc --proto_path=${PROTO_PATH} --swagger_out=logtostderr=true:./swagger.json
swagger-codegen generate -i ./swagger.json -l openapi-yaml -o swaggerv3.yaml

Validating configurations files with viper

I was looking for a configuration parser for go and https://github.com/spf13/viper seems to come highly recommended.
I am very surprised to find that configuration files are not validated by default.
Viper parses files and extracts requested values from them but I cannot find a way to detect bad configuration.
For instance I if create a (Java style) .properties file containing just "???" and nothing else. This is accepted without any error.
I can understand the philosophy that you should ignore irrelevant configuration items but I desire more rigor. I would also like to reject anything that does not match the X=Y format in a properties file.
To me this is a fatal flaw that suggests I should use a different package (or roll my own as usual).
Have I missed something? Does viper in fact support detecting and rejecting bad configuration keys?
I think the answer is no. viper does not validate java .properties files.
I posted a bug report (or feature request depending on your point of view) as https://github.com/spf13/viper/issues/790
You can try https://github.com/num30/config library which is based on Viper. It has built-in validation.

Paw: Possible to copy/paste data from Hash/Dictionary source in target language?

I'm using Paw to integrate with an API and it's a great tool, but the codebase I'm working in has lots of data in Ruby Hashes already that I would like to bring into Paw for testing. Moreover, I would like to be able to copy the data out of a Paw response for use in Ruby code rather than having to copy JSON and either store it as a String and parse it in my code or manually convert it to a native dictionary data structure. Is this possible?

Output all language strings in Revel?

I'm developing an API Server in Go and the server (at the moment) handles all translations for clients. When an API client fetches particular data it also asks for the translations that are available for the given section.
Ideally I want to have the following folder structure:
/messages
/home.en
/home.fr
/home.sv
/news.en
/news.fr
/news.sv
Where news and home are distinct modules.
Now the question I have for Revel is is it possible to fetch ALL language strings for a given module and given locale? For example pull all home strings for en-US.
EDIT:
I would like the output (something I can return to the client) a key:value string of translations.
Any guidance would be appreciated.
It seems to me that revel uses messaged based translation (just like gettext does), so you need
the original string to get the translation. These strings are stored in Config objects,
which are themselves stored in messages of i18n.go, sorted by language.
As you can see, this mapping is not exported, so you can't access it. The best way
to fix this is to write a function for what you want (getting the config by supplying a language)
or exporting one of the existing functions and create a pull request for revel.
You may workaround this by copying the code of loadMessageFile or by forking your version
of revel and exporting loadMessageFile or parseMessagesFile. This also is a great opportunity
to create a pull request.
Note that the localizations are stored in a INI file format parsed by robfig/config,
so manually parsing is also an option (although not recommended).

Resources