type MyRequest struct {
email string `json:"email" valid:"email,required"`
}
json is used to (un)marshal JSON structures, but what valid is for?
It's for anything that reads it. There's no official registry of tags (unless you count the ones used by the standard library).
Based on the semantics of the tag's value, one can reasonably assume it's used by some (likely more than one) validation library of some sort. The best way to find out its intended use is to look at the code that sets the tag, and see if it uses (or is used by) some library that sets that tag. Unfortunately, there's no mechanical way to do this--you'll probably be looking through "import" lines and grepping package source code.
I think it is used by this package: https://github.com/asaskevich/govalidator
Related
Warning: This question is a more philosophical question than practical, but I find it well as to be asked and answered in practical contexts (forums like StackOverflow here, instead of the SoftwareEngineering stack-exchange website), due to the native development in the actual use de-facto of YAML and the way the way it's specification has evolved and features have been added to it over time. Let's ask:
As opposed to formats/languages/protocols such as JSON, the YAML format allows you (according to this link, that seems pretty official, or at least accurate and reliable source to understand the YAML specification) to embed multiple 'Documents' within one file/stream, using the three-dashes marking ("---").
If so, it's hard to ignore the fact that the concept/model/idea of 'Document' in YAML, is no longer an external definition, or "meta"-directive that helps the human/parser to organize multiple/distincted documents along each other (similar to the way file-systems defining the concept of "file" to organize different files, but each file in itself - does not necessarily recognize that it's a file, or that it's being part of a file system that wraps it, by definition, AFAIK.
However, when YAML allows for a multi-Document YAML files, that gather collections of Documents in a single YAML file (and perhaps in a way that is similar/analogous to HTTP Pipelining approach of HTTP protocol), the concept/model/idea/goal of Document receives a new, wider definition/character de-facto, as a part of the YAML grammar and it's produces, and not just of the YAML specification as an assistive concept or format description that helps to describe the specification.
If so, being a Document part of the language itself, what is the added value of this data-structure, compared to the existing, familiar and well-used good old data-structure of Collection (array of items)?
I'm asking it, because I've seen in this link (here) some snippet (in the second example), which describes a YAML sequence that is actually a collection of logs. For some reason, the author of the example, chose to prefer to present each log as a separate "Document" (separated with three-dashes), gathered together in the same YAML sequence/file, instead of writing a file that has a "Collection" of logs represented with the data-type of array. Why did he choose to do this? Is his choice fit, correct, ideal?
I can speculate that the added value of the distinction between a Document and a Collection become relevant when using more advanced features of the YAML grammar, such as Anchors, Tags, References. I guess every Document provide a guarantee that all these identifiers will be a unique set, and there is no collision or duplicates among them. Am I right? And if so, is this the only advantage, or maybe there are any more justifications for the existence of these two pretty-similar data structures?
My best for now, is to see Document as a "meta"-Collection, that is more strict, and lack of high-level logic, or as two different layers of collection schemes. Is it correct, accurate way of view?
And even if I am right, why in the above example (of the logs document from the link), when there's no use and not imply or expected to use duplications or collisions or even identifiers/anchors or compound structures at all - the author is still choosing to represent the collection's items as separate documents? Is this just not so successful selection of an example? Or maybe I'm missing something, and this is a redundancy in the specification, or an evolving syntactic-sugar due to practical needs?
Because the example was written on a website that looks serious with official information written by professionals who dealt with the essence of the language and its definition, theory and philosophy behind (as opposed to practical uses in the wild), and also in light of other provided examples I have seen in it and the added value of them being meticulous, I prefer not to assume that the example is just simply imperfect/meticulous/fit, and that there may be a good reason to choose to write it this way over another, in the specific case exampled.
First, let's look at the technical difference between the list of documents in a YAML stream and a YAML sequence (which is a collection of ordered items). For this, I'll discuss YAML tags, which are an advanced feature so I'll provide a quick overview:
YAML nodes can have tags, such as !!str (the official tag for string values) or !dice (a local tag that can be interpreted by your application but is unknown to others). This applies to all nodes: Scalars, mappings and sequences. Nodes that do not have such a tag set in the source will be assigned the non-specific tag ?, except for quoted scalars which get ! instead. These non-specific tags are later resolved to specific tags, thereby defining to which kind of data structure the node will be deserialized into.
YAML implementations in scripting languages, such as PyYAML, usually only implement resolution by looking at the node's value. For example, a scalar node containing true will become a boolean value, 42 will become an integer, and droggeljug will become a string.
YAML implementations for languages with static types, however, do this differently. For example, assume you deserialize your YAML into a Java class
public class Config {
String name;
int count;
}
Assume the YAML is
name: 42
count: five
The 42 will become a String despite the fact that it looks like a number. Likewise, five will generate an error because it is not a number; it won't be deserialized into a string. This means that not the content of the node defines how it will be deserialized, but the path to the node.
What does this have to do with documents? Well, the YAML spec says:
Resolving the tag of a node must only depend on the following three parameters: (1) the non-specific tag of the node, (2) the path leading from the root to the node and (3) the content (and hence the kind) of the node.)
So, the technical difference is: If you put your data into a single document with a collection at the top, the YAML processor is allowed to take into account the position of the data in the top-level collection when resolving a tag. However, when you put your data in different documents, the YAML processor must not depend on the position of the document in the YAML stream for resolving the tag.
What does this mean in practice? It means that YAML documents are structurally disjoint from one another. Whether a YAML document is valid or not must not depend on any preceeding or succeeding documents. Consequentially, even when deserialization runs into a semantic problem (such as with the five above) in one document, a following document may still be deserialized successfully.
The goal of this design is to be able to concatenate arbitrary YAML documents together without altering their semantics: A middleware component may, without understanding the semantics of the YAML documents, collect multiple streams together or split up a single stream. As long as they are syntactically correct, stream splitting and merging are sound operations that do not invalidate a YAML document even if another document is structurally invalid.
This design primary focuses on sending and receiving data over networks. Of course, nowadays, YAML is primarily used as configuration language. This is why this feature is seldom used and of rather little importance.
Edit: (Reply to comment)
What about end-cases like a string-tagged Document starts with a folded-string, making even its following "---" and "..." just a characters of the global string?
That is not the case, see rules l-bare-document and c-forbidden. A line containing un-indented ... not followed by non-whitespace will always end a document if one is open.
Moreover, ... doesn't do anything if no document is open. This ensures that a stream merger can always append ... to a document to ensure that the current document is closed, but no additional one is created.
--- has widely been adopted as separator between YAML documents (and, perhaps more prominently, between YAML front matter and content in tools like Jekyll) where ... would have been more appropriate, particularly in Jekyll. This gives the false impression that --- should be used by tooling to separate documents, when in reality ... is the syntactic element designed for that use-case.
In the docs for FieldMask the paths use the field names (e.g., foo.bar.buzz), which means renaming the message field names can result in a breaking change.
Why doesn't FieldMask use the field numbers to define the path?
Something like 1.3.1?
You may want to consider filing an issue on the GitHub protocolbuffers repo for a definitive answer from the code's authors.
Your proposal seems logical. Using names may be a historical artifact. There's a possibly relevant comment on an issue thread in that repo:
https://github.com/protocolbuffers/protobuf/issues/3793#issuecomment-339734117
"You are right that if you use FieldMasks then you can't safely rename fields. But for that matter, if you use the JSON format or text format then you have the same issue that field names are significant and can't be changed easily. Changing field names really only works if you use the binary format only and avoid FieldMasks."
The answer for your question lies in the fact FieldMasks are a convention/utility developed on top of the proto3 schema definition language, and not a feature of it (and that utility is not present in all of the language bindings)
While you’re right in your observation that it can break easily (as schemas tend evolve and change), you need to consider this design choice from a user friendliness POV:
If you’re building an API and want to allow the user to select the field set present inside the response payload (the common use case for field masks), it’ll be much more convenient for you to allow that using field paths, rather then binary fields indices, as the latter would force the user of the gRPC/protocol generated code to be “aware” of the schema. That’s not always the desired case when providing API as a code software packages.
While implementing this as a proto schema feature can allow the user to have the best of both worlds (specify field paths, have them encoded as binary indices) for binary encoding, it would also:
Complicate code generation requirements
Still be an issue for plain text encoding.
So, you can understand why it was left as an “external utility”.
I'm looking for a string representation of arbitrary fields inside protocol buffer messages. Is there any library that implements this? I've looked at using field masks, however they don't have a strong support for repeated fields.
Protocol buffer message and field descriptors provide field access by name. This allows you to find a particular field using a path and to erase it, if that's what you are asking for (if not, I'd recommend to expand the question to include an example for what you'd like to do).
One corresponding Java method is getDescriptorForType (the return type is a message descriptor, where you'll find field descriptors).
There is a similar descriptor API for C++ (in Java, you could theoretically also use reflection).
This API is not available in light mode.
I've seen advice around the Internet that if you want a YAML scalar value to be processed as a string, you should quote it:
foo : "2018-04-17"
In the example above, this advice is intended to tell me that the value 2018-04-17 will be processed by any given YAML parser as its native language's string type. For example, SnakeYAML would, if this advice were true, interpret this as a java.lang.String, and not as a java.util.Date. (As it happens, SnakeYAML interprets this as a java.util.Date, quotes or not, which is why I'm asking this question.)
But although this advice may happen to work with any given parser, I can't see where in the YAML 1.2. specification this advice might come from. The closest thing I can find is the following sentence:
YAML allows scalars to be presented in several formats. For example, the integer “11” might also be written as “0xB”. Tags must specify a mechanism for converting the formatted content to a canonical form for use in equality testing. Like node style, the format is a presentation detail and is not reflected in the serialization tree and representation graph.
And this one:
The scalar style is a presentation detail and must not be used to convey content information, with the exception that plain scalars are distinguished for the purpose of tag resolution.
And this one:
Note that resolution must not consider presentation details such as comments, indentation and node style.
Nevertheless, I see lots of YAML documents that rely on the double-quoting-the-value-means-it-will-be-parsed-as-a-string advice, which makes me think I'm misreading something. Is there contention on this subject?
Relevant section from the YAML 1.1 spec (note that SnakeYaml is YAML 1.1 and therefore, the 1.2 spec does not necessarily apply):
It is not required that all the tags of the complete representation be explicitly specified in the character stream. During parsing, nodes that omit the tag are given a non-specific tag: “?” for plain scalars and “!” for all other nodes. [...]
It is recommended that nodes having the “!” non-specific tag should be resolved as “tag:yaml.org,2002:seq”, “tag:yaml.org,2002:map” or “tag:yaml.org,2002:str” depending on the node’s kind. This convention allows the author of a YAML character stream to exert some measure of control over the tag resolution process. By explicitly specifying a plain scalar has the “!” non-specific tag, the node is resolved as a string, as if it was quoted or written in a block style. Note, however, that each application may override this behavior. For example, an application may automatically detect the type of programming language used in source code presented as a non-plain scalar and resolve it accordingly.
So to sum up, a YAML processor is not required to parse quoted scalars as string, and YAML also does not dictate which native type tag:yaml.org,2002:str does map to. And in fact, most YAML implementations do only follow parts of that advice. For example, if you deserialise YAML into a POJO/JavaBean with SnakeYaml, you typically do not use any explicit tags in your YAML, but your mappings are resolved to the corresponding Java classes in the root class' structure, instead of the generic Map which is what this advice suggests (since all mappings without explicit tags get the ! non-specific tag).
Note that this has been changed in YAML 1.2:
During parsing, nodes lacking an explicit tag are given a non-specific tag: “!” for non-plain scalars, and “?” for all other nodes.
That's closer to most implementations, but for example, if you deserialise into a class class Foo { String bar; }, this will still load although bar is not a string, but a field name:
"bar": some value
So the advice for using YAML is to specify the desired structure on the application side – in SnakeYaml, you would set the root class type, and then every value will be mapped to the required type at its point in the hierarchy, as long as it is able to map there, regardless of whether it is quoted or unquoted. In general, it makes more sense for the application to specify which kind of value it expects throughout the hierarchy instead of the YAML author to do that via quoting. This is also conformant with the YAML spec, which says
Resolving the tag of a node must only depend on the following three parameters: (1) the non-specific tag of the node, (2) the path leading from the root to the node, and (3) the content (and hence the kind) of the node.
Resolving a tag is the YAML term for determining the target type. And it is allowed to determine the target type based on its position in the hierarchy: The root type is determined by the fact that the element is the root of the YAML document and in the case of SnakeYaml, may be fed in via the API. All other types are determined by the fact that they are descendants from the root type.
Final note: If you really really want something to be a string, !!str 2018-04-17 will do since it sets a specific tag for the node.
In the protobuf documentation (https://developers.google.com/protocol-buffers/docs/proto#customoptions) it says this about custom options:
One last thing: Since custom options are extensions, they must be
assigned field numbers like any other field or extension. In the
examples above, we have used field numbers in the range 50000-99999.
This range is reserved for internal use within individual
organizations, so you can use numbers in this range freely for
in-house applications. If you intend to use custom options in public
applications, however, then it is important that you make sure that
your field numbers are globally unique. To obtain globally unique
field numbers, please send a request to
protobuf-global-extension-registry#google.com. Simply provide your
project name (e.g. Object-C plugin) and your project website (if
available). Usually you only need one extension number.
Why do the options field numbers have to be globally unique for public applications? In what way can collisions be a problem?
Basically, because you wouldn't know whether the data you get is correct.
The protobuf binary wire format only stores the field numbers and the payload (which is itself, for complex types, just field numbers and sub-payloads). There is no name data. So: when you store and retrieve an extension field, all you're saying is "fetch field {field number}, interpret it as {type}". If two different systems have extended the same data using the same field number, then you don't have any way of knowing whether the data you're fetching was actually in that format.
Normally this isn't a problem - as it is rare to conflict like this on the same data; but custom options are different! I'm a library author; I might want to add a custom option that my schema parsing tools recognize, by extending (say) MessageOptions. MessageOptions is the extension point for DescriptorProto, which is to say: that's what option (foo) = "bar"; goes to inside a message.
To do that, I need to assign a number for foo. I choose 5000 arbitrarily (MessageOptions defines extensions 1000 to max;, so that's fine). All is good. My tooling works.
Unknown to me, another library author has chosen to do something similar and has also used 5000. Once the schema is compiled (by protoc or similar), all I have is numbers. If I ask for the data from field 5000, I don't know whether I'm getting my extension, or the other one. The meaning is lost. OK, at a push I could also check the dependency list on the FileDescriptorProto, but ... that's hit and miss.
I don't know whether the presense of value 1 in field 5000 is:
option (.mystuff.someext) = 1;
vs
option (.anotherlib.whatever) = -1; // stored as sint32
vs
option (.yetanother.library.option) = true;
If those extensions all have number 5000, they appear identically on the wire.