I am starting to use Go and have found that to parse an html or xml document does not have a powerful tool by itself (excuse my ignorance if not).
The fact is that I have the following map (I converted the xml to a map using github.com/clbanning/mxj). It's a XML and HTML Clover report.
What I want is access to the different values: XML pastebin
My code:
xmlFile, err := os.Open(dir + "\\clover.xml")
if err != nil {
fmt.Println("Error opening file:", err)
return
}
b, _ := ioutil.ReadAll(xmlFile)
defer xmlFile.Close()
fmt.Println("xmldata:", string(b))
// get all image tag values - []interface{}
mapvalue, merr := mxj.NewMapXml(b)
if merr != nil {
fmt.Println("merr:", merr.Error())
return
}
Even in the following way I can subdivide it:
sources, err := mapvalue.ValuesForKey("metrics")
if err != nil {
fmt.Println("err:", err.Error())
return
}
for i, src := range sources {
fmt.Println(i, src)
}
fmt.Println(sources[0])
It's that particular one I need. But now I can't access every one of the inner values.
map[-loc:459 -statements:71 -coveredmethods:14 -ncloc:307 -files:12
-conditionals:6 -coveredelements:45 -packages:8 -elements:110 -complexity:37 -classes:12 -coveredconditionals:1 -coveredstatements:30 -methods:33]
Is there an easier way to work with the XML and html I have in local?
I added the XML to map result: pastebin
GO has an xml parsing library https://golang.org/pkg/encoding/xml/
I have a runnable example here: https://play.golang.org/p/kVG3w4iu3Kl
package main
import (
"fmt"
"encoding/xml"
)
var rawXml = "<metrics coveredelements=\"45\" complexity=\"37\" loc=\"459\" methods=\"33\"/>"
type Metrics struct {
CoveredElements string `xml:"coveredelements,attr"`
Complexity string `xml:"complexity,attr"`
Loc string `xml:"loc,attr"`
Methods string `xml:"methods,attr"`
}
func main() {
data := &Metrics{}
xml.Unmarshal([]byte(rawXml), data)
fmt.Printf("%+v", data)
}
I think that you will find it much easier to work with structs.
I have found this utility to allow me to implement the necessary structures to parse the JSON answers (you can use a converter ). I hope you find it useful too:
https://mholt.github.io/json-to-go/
You can use a JSON/XML converter online like this: http://www.utilities-online.info/xmltojson/#.W1cSCNIzZPY.
For the rest I use the same what #Rossiar said in his comment.
Related
When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.
This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
// ... marshal and write the response ...
}
(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)
Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.
The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.
For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.
lastEvaluatedKey := map[string]types.AttributeValue{
"year": &types.AttributeValueMemberN{Value: "1993"},
"title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}
bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
panic(err)
}
decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
panic(err)
}
What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.
I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.
Has anyone found a generic way to do this?
OK, I figured something out.
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
eskMap := map[string]string{}
json.Unmarshal(params.NextToken, &eskMap)
esk, _ = dynamodbattribute.MarshalMap(eskMap)
dynamoIn.ExclusiveStartKey = esk
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
lek := map[string]string{}
dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
response.NextToken := json.Marshal(lek)
// ... marshal and write the response ...
}
(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)
As #buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.
I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.
Inspired by Dan, here is a solution to serialize and deserialize to/from base64
package dynamodb_helpers
import (
"encoding/base64"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func Serialize(input map[string]types.AttributeValue) (*string, error) {
var inputMap map[string]interface{}
err := attributevalue.UnmarshalMap(input, &inputMap)
if err != nil {
return nil, err
}
bytesJSON, err := json.Marshal(inputMap)
if err != nil {
return nil, err
}
output := base64.StdEncoding.EncodeToString(bytesJSON)
return &output, nil
}
func Deserialize(input string) (map[string]types.AttributeValue, error) {
bytesJSON, err := base64.StdEncoding.DecodeString(input)
if err != nil {
return nil, err
}
outputJSON := map[string]interface{}{}
err = json.Unmarshal(bytesJSON, &outputJSON)
if err != nil {
return nil, err
}
return attributevalue.MarshalMap(outputJSON)
}
I have a use case where I have the code as below. I have a request coming in to hit the backend where I need to append data to a map. My question is how do I convert the below type to a []byte to unmarshal?
Any ideas would be appreciated.
type Example struct {
Category string `json:"category"`
Name string `json:"name"`
}
Incoming Postman request json looks like this:
[{"Category":"TestCategory", "Name":"Sample1"}]
but after doing
jsonString Type: []Example
if err := gc.ShouldBindJSON(&jsonString) it looks like [{TestCategory Sample1}] ; how do I convert this to a []byte?
for _, req := range blob{
var jsonString Example
if err := json.Unmarshal([]byte(jsonString), &blob); err != nil { //this does not work
logger.Fatal(err)
}
//I am checking if a key-value is present and appending it to the map
dict := make(map[string][]Example)
dict[req.Category] = append(dict[req.Category], req)
fmt.Println(dict)
if value, ok := dict["TestCategory"]; ok {
fmt.Printf("Found %d\n", value)
} else {
fmt.Println("not found")
}
}
//I was able to test the above logic by declaring the jsonString as a const and it works
There are two directions in which you can move the data:
from JSON to a Go data structure
// This is your payload coming from the request.
jsonStr := `[{"Category":"TestCategory", "Name":"Sample1"}]`
// This is the Go struct that will hold the unmarshalled data.
var examples []Example
err := json.Unmarshal([]byte(jsonStr), &examples)
if err != nil {
log.Fatal(err)
}
fmt.Println("Examples:", examples) // prints "Examples: [{TestCategory Sample1}]"
from a Go data structure to JSON (either string or []byte)
exampleBytes, err := json.Marshal(examples)
if err != nil {
log.Fatal(err)
}
fmt.Println("Example bytes:", string(exampleBytes)) // prints "Example bytes: [{"category":"TestCategory","name":"Sample1"}]"
You should check out "Go by Example" if you haven't already: https://gobyexample.com/json
Looking at your code:
You are looping on blob but instead of using the req you are trying to unmarshal onto the entire blob each time. I'm not sure what you are trying to achieve there but nothing good can come out of changing a struct you're looping over from within the loop.
The request JSON you are listing is an array of JSON objects. You are trying to unmarshal that into a single Example struct. That won't work, you need an array of those.
TL;DR: How can I flexibly decode a k8s API object and inspect its top-level metav1.ObjectMeta struct without knowing the object's Kind in advance?
I'm writing an admission controller endpoint that unmarshals a metav1.AdmissionReview object's Request.Object.Raw field into a concrete object based on the Request.Kind field - e.g.
if kind == "Pod" {
var pod core.Pod
// ...
if _, _, err := deserializer.Decode(admissionReview.Request.Object.Raw, nil, &pod); err != nil {
return nil, err
}
annotations := pod.ObjectMeta.Annotations
// inspect/validate the annotations...
This requires knowing all possible types up front, or perhaps asking a user to supply a map[kind]corev1.Object that we can use to be more flexible.
What I'd like to instead achieve is something closer to:
var objMeta core.ObjectMeta
if _, _, err := deserializer.Decode(admissionReview.Request.Object.Raw, nil, &objMeta); err != nil {
return nil, err
}
// if objMeta is populated, validate the fields, else
// assume it is an object that does not define an ObjectMeta
// as part of its schema.
Is this possible? The k8s API surface is fairly extensive, and I've crawled through the metav1 godoc, corev1 godoc & https://cs.k8s.io for prior art without a decent example.
The closest I've found is possibly the ObjectMetaAccessor interface, but I'd need to get from an AdmissionReview.Request.Object (type runtime.RawExtension) to a runtime.Object first.
I believe you can't find what you are looking for because, when decoding an object, Kubernetes uses GetObjectKind and compares the result to a Scheme to convert the object to a concrete type, rather than using some generic like approach and interacting with the fields of an object without knowing it's concrete type.
So you can use reflection instead, something like:
k8sObjValue := reflect.ValueOf(admissionReview.Request.Object.Raw).Elem()
k8sObjObjectMeta := k8sObjValue.FieldByName("ObjectMeta")
annotations, ok := k8sObjObjectMeta.FieldByName("Annotations").Interface().(map[string]string)
if !ok {
panic("failed to retrieve annotations")
}
EDIT:
Or closer to your requirements, convert to an ObjectMeta object
k8sObjValue := reflect.ValueOf(admissionReview.Request.Object.Raw).Elem()
objMeta, ok := k8sObjValue.FieldByName("ObjectMeta").Interface().(core.ObjectMeta)
if !ok {
panic("failed to retrieve object metadata")
}
There is a way to do this that I discovered recently, let me describe it here:
Quick disclaimer: I used admission/v1 and never tested with admission/v1beta1, which should work identically.
The data type of admissionReview.Request.Object is runtime.RawExtension, and the k8s.io/apimachinery/pkg/runtime provides a method that can convert the runtime.RawExtension to a runtime.Object. The method is called runtime.Convert_runtime_RawExtension_To_runtime_Object(...). From there, you can easily convert to the unstructured.Unstructured data type, which has all the fields from the MetaV1 object accessible with simple getter methods.
Here is a code snippet that lets you get this accomplished:
import (
// ...
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime"
// ...
)
// ...
func dummyFunc(ar *v1.AdmissionReview) {
// ...
var obj runtime.Object
var scope conversion.Scope // While not actually used within the function, need to pass in
err := runtime.Convert_runtime_RawExtension_To_runtime_Object(&ar.Request.Object, &obj, scope)
if err != nil {
// ...
}
innerObj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(obj)
if err != nil {
// ...
}
u := unstructured.Unstructured{Object: innerObj}
// Now the `u` variable has all the meta info available with simple getters.
// Sample:
labels := u.GetLabels()
kind := u.GetKind()
// etc.
// ...
}
References:
https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured
https://godoc.org/k8s.io/apimachinery/pkg/runtime#Convert_runtime_RawExtension_To_runtime_Object
https://godoc.org/k8s.io/api/admission/v1#AdmissionRequest
It seems there are two possibilities:
Either the field Object should already contain the correct object instance, when using Go client, check code here.
Try using the converters here
I'm trying to parse this yaml and I want to get the values of the run entry (test1 or test2) without success, here is my working example.
im a bit get lost with the map inside map :( ,
this is given yaml which I couldent change ...
any idea how could I got those values
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
version: "3.2"
run-parameters:
before:
run-parameters:
run: test1
after:
run-parameters:
run: test2
`)
type FTD struct {
Version string `yaml:"version,omitempty"`
BuildParams *RunParams `yaml:"run-parameters,omitempty"`
}
type RunParams struct {
BeforeExec map[string]interface{} `yaml:"before,omitempty"`
AfterExec map[string]interface{} `yaml:"after,omitempty"`
}
func main() {
runners := &FTD{}
// parse mta yaml
err := yaml.Unmarshal(runContent, runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
for k, v := range runners.BuildParams.BeforeExec {
fmt.Println(k, v.(interface{}))
}
}
This is working example
https://play.golang.org/p/qTqUJy3md0c
also I've tried with
this is working
run := runners.BuildParams.BeforeExec["run-parameters"].(map[interface{}]interface{})["run"]
fmt.Println("run: ", run)
what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?
what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?
You can do
runParams, ok := runners.BuildParams.BeforeExec["run-parameters"]
if !ok {
// handle lack of "run-parameters" in BeforeExec
}
runParamsMap, ok := runParams.(map[interface{}]interface{})
if !ok {
// handle "run-parameters" not being a map
}
run, ok := runParamsMap["run"]
if !ok {
// handle lack of "run" inside "run-parameters"
}
runStr, ok := run.(string)
if !ok {
// handle "run" not being a string
}
fmt.Println("run: ", runStr)
This is quite verbose so you could use something like https://github.com/jmoiron/jsonq, where you can specify a "path" to the desired value nested inside several levels of maps. Despite the "json" in the name, this library works with map[string]interface{} and not json files. But note that the library you use for yaml unmarshalling results in map[interface{}]interface{} instead of map[string]interface{} and you will have to use a different one in order for it to work with jsonq.
run, err := jsonq.NewQuery(runners.BuildParams.BeforeExec).String("run-parameters", "run")
if err != nil {
// handle all possible errors in one place
}
fmt.Println("run: ", run)
I have the following piece of code:
func GetUUIDValidator(text string) bool {
r, _ := regexp.Compile("/[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89aAbB][a-f0-9]{3}-[a-f0-9]{12}/")
return r.Match([]byte(text))
}
But when I pass fbd3036f-0f1c-4e98-b71c-d4cd61213f90 as a value, I got false, while indeed it is an UUID v4.
What am I doing wrong?
Regex is expensive. The following approach is ~18x times faster than the regex version.
Use something like https://godoc.org/github.com/google/uuid#Parse instead.
import "github.com/google/uuid"
func IsValidUUID(u string) bool {
_, err := uuid.Parse(u)
return err == nil
}
Try with...
func IsValidUUID(uuid string) bool {
r := regexp.MustCompile("^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-4[a-fA-F0-9]{3}-[8|9|aA|bB][a-fA-F0-9]{3}-[a-fA-F0-9]{12}$")
return r.MatchString(uuid)
}
Live example: https://play.golang.org/p/a4Z-Jn4EvG
Note: as others have said, validating UUIDs with regular expressions can be slow. Consider other options too if you need better performance.
You can utilize satori/go.uuid package to accomplish this:
import "github.com/satori/go.uuid"
func IsValidUUID(u string) bool {
_, err := uuid.FromString(u)
return err == nil
}
This package is widely used for UUID operations: https://github.com/satori/go.uuid
In case you would be validating it as attribute of a struct, there is an awesome golang library straight from the Go called validator https://godoc.org/gopkg.in/go-playground/validator.v9 which you can use to validate all kinds of fields nested structures by provided built-in validators as well as complete custom validation methods. All you need to do is just add proper tags to the fields
import "gopkg.in/go-playground/validator.v9"
type myObject struct {
UID string `validate:"required,uuid4"`
}
func validate(obj *myObject) {
validate := validator.New()
err := validate.Struct(obj)
}
It provides structured field errors and other relevant data from it.