I'm using https://github.com/go-yaml/yaml to parse yaml files:
type TestConfig struct {
Test string `yaml:"test"`
}
yaml file:
test: 123
---
test: 456
But yaml.Unmarshal() only parses the first segment, how can I parse the rest of it?
But yaml.Unmarshal() only parses the first segment, how can I parse the rest of it?
yaml.Unmarshal's doc says (emphasis mine):
Unmarshal decodes the first document found within the in byte slice and assigns decoded values into the out value.
If you want to decode a series of documents, call yaml.NewDecoder() on a stream of your data and then call your decoder's .Decode(...) multiple times. Use io.EOF to identify the end of records.
I usually use an infinite for loop with a break condition for this:
decoder := yaml.NewDecoder(bytes.NewBufferString(data))
for {
var d Doc
if err := decoder.Decode(&d); err != nil {
if err == io.EOF {
break
}
panic(fmt.Errorf("Document decode failed: %w", err))
}
fmt.Printf("%+v\n", d)
}
fmt.Printf("All documents decoded")
(https://go.dev/play/p/01xdzDN0qB7)
Related
I am trying to decode proto data. But the proto is not properly decoded.
This is how I am doing:
decodedStr, err := base64.StdEncoding.DecodeString(request.Body)
if err != nil {
panic("malformed input")
}
data := &tracepb.ExportTraceServiceRequest{}
if err := proto.Unmarshal(decodedStr, data); err != nil {
log.Fatalln("Failed to parse:", err)
}
log.Printf("Response - %v", data)
The output is like this:
Response - resource_spans:{resource:{attributes:{key:"service.name" value:{string_value:"node_app"}} attributes:{key:"telemetry.sdk.language" value:{string_value:"nodejs"}} attributes:{key:"telemetry.sdk.name" value:{string_value:"opentelemetry"}} attributes:{key:"telemetry.sdk.version" value:{string_value:"1.8.0"}} attributes:{key:"process.pid" value:{int_value:1}} attributes:{key:"process.executable.name" value:{string_value:"node"}} attributes:{key:"process.command" value:{string_value:"/usr/app/index.js"}} attributes:{key:"process.command_line" value:{string_value:"/usr/local/bin/node /usr/app/index.js"}} attributes:{key:"process.runtime.version" value:{string_value:"18.13.0"}} attributes:{key:"process.runtime.name" value:{string_value:"nodejs"}} attributes:{key:"process.runtime.description" value:{string_value:"Node.js"}}} scope_spans:{scope:{name:"#opentelemetry/instrumentation-express" version:"0.32.0"} spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0" span_id:"T\x06\x89m\x1ex\xf9A" parent_span_id:"?\xbc\x18`O\xa5\xb8\xe1" name:"middleware - query" kind:SPAN_KIND_INTERNAL start_time_unix_nano:1673434036590614272 end_time_unix_nano:1673434036590671104 attributes:{key:"http.route" value:{string_value:"/"}} attributes:{key:"express.name" value:{string_value:"query"}} attributes:{key:"express.type" value:{string_value:"middleware"}} status:{}} spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0" span_id:"\xd5c\xf7>\xf6Cxz" parent_span_id:"?\xbc\x18`O\xa5\xb8\xe1" name:"middleware - expressInit" kind:SPAN_KIND_INTERNAL start_time_unix_nano:1673434036590760704
Not sure why traceId is shown like this:
spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0"
Am new to GoLang. Any help would be greatly appreciated
The trace_id field appears to contain the ID as binary data instead of hex. The generated proto String method will render the binary data as a string. Hence non-printable characters as displayed as ASCII escape sequences.
If you want to display the data in other format (eg, Hex) you will need to implement your own function to render the proto.
Used encoding/hex module's hex.EncodeToString() function to convert bytes to hex
I have an ECDSA public key that that is returned to me from an HSM in ASN.1 DER format. I need to create a bitcoin compatible key 33 byte. When I print out key in hex.EncodeToString(pubkey) I get the following output:
3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I use an online decoder here: https://holtstrom.com/michael/tools/asn1decoder.php
And it outputs:
0x049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I can then take that and hex.DecodeString(str) which gives me the necessary format to input this into addrPubKey, err := btcutil.NewAddressPubKey(bs, &chaincfg.TestNet3Params).
How do I decode this in golang to get the 0x049... output?
Thanks
The first thing we need is to use the encoding/asn1 package from the standard library.
You only have to give go the right struct to decode into. From your link we can see that we have a SEQUENCE that contains another SEQUENCE with two OBJECTIDENTIFIER and a BITSTRING. In go this will be:
type Ids struct {
OBi1 asn1.ObjectIdentifier
OBi2 asn1.ObjectIdentifier
}
type PubKey struct {
Id Ids
Bs asn1.BitString
}
Now we only have to UnMarshall the data to this structure:
str := `3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec`
bstring, err := hex.DecodeString(str)
if (err != nil) {
panic(err)
}
var decode PubKey
_, err = asn1.Unmarshal(bstring, &decode)
if (err != nil) {
panic(err)
}
fmt.Println(hex.EncodeToString(decode.Bs.Bytes))
Note that you don't have to encode the string to hex and back again, since Unmarshall accepts a byte array
This will print the expected result:
049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
Once again you probably don't need to encode to string.
I have a Go project where I'm using Zap structured logging to log the contents of structs. That's how I initialise the logger:
zapLog, err := zap.NewProductionConfig().Build()
if err != nil {
panic(err)
}
Initially I started with my own structs with json tags and it all worked perfectly:
zapLog.Info("Event persisted", zap.Any("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
I now switched to protobuf and I'm struggling to achieve the same result. Initially I just got the "reflected map" version, when using zap.Any():
zapLog.Info("Event persisted", zap.Any("event", &event))
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"sourceType:TYPE_X sourceId:\"some-source-id\",
type:\"updated\" value:{...}, context:<key: foo, value:bar>}
I tried marshalling the object with the jsonpb marshaller, which generated the correct output on itself, however, when I use it in zap.String(), the string is escaped, so I get an extra set of '\' in front of each quotation mark. Since there's processing of the logs at a later point, this causes problems there and hence I want to avoid it:
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, msg); err != nil {
// handle error
}
zapLog.Info("Event persisted", zap.ByteString("event", buf.Bytes()))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"{\"sourceType\":\"TYPE_X\", \"sourceId\":\"some-source-id\",
\"type\":\"updated\", \"value\":\"{...}\", \"context\":{\"foo\":"bar\"}}"}
I then tried using zap.Reflect() instead of zap.Any() which was the closest thing I could get to what I need, except that enums are rendered as their numerical values (the initial solution did not have enums, so that didn't work in the pre-protobuf solution either):
zapLog.Info("Event persisted", zap.Reflect("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
The only option I see so far is to write my own MarshalLogObject() function:
type ZapEvent struct {
event *Event
}
func (z *ZapEvent) MarshalLogObject(encoder zapcore.ObjectEncoder) error {
encoder.AddString("sourceType", z.event.SourceType.String()
// implement encoder for each attribute
}
func processEvent(e Event) {
...
zapLog.Info("Event persisted", zap.Object("event", &ZapEvent{event: &e}))
}
But since it's a complex struct, I would rather use a less error prone and maintenance heavy solution. Ideally, I would tell zap to use the jsonpb marshaller somehow, but I don't know if that's possible.
Use zap.Any with a json.RawMessage. You can convert directly the byte output of jsonpb.Marshaler:
foo := &pb.FooMsg{
Foo: "blah",
Bar: 1,
}
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, foo); err != nil {
// handle error
}
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes())))
The bytes will be printed as:
Event persisted {"event": {"foo":"blah","bar":"1"}}`
I believe that's the easiest way, however I'm also aware of a package kazegusuri/go-proto-zap-marshaler (I'm not affiliated to it) that generates MarshalLogObject() implementations as a protoc plugin. You may want to take a look at that too.
I used another way to jsonify protos.
Since protos can be naturally marshaled, I just wrapped them in the strict-to-json marshaler.
And you can modify the internals to use protojson (newer jsonpb).
Unlike the marshaler in the previous solution, this one doesn't require ahead-of-logging processing.
type jsonObjectMarshaler struct {
obj any
}
func (j *jsonObjectMarshaler) MarshalJSON() ([]byte, error) {
bytes, err := json.Marshal(j.obj)
// bytes, err := protojson.Marshal(j.obj)
if err != nil {
return nil, fmt.Errorf("json marshaling failed: %w", err)
}
return bytes, nil
}
func ZapJsonable(key string, obj any) zap.Field {
return zap.Reflect(key, &jsonObjectMarshaler{obj: obj})
}
Then to use it, just
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", ZapJsonable("event", buf))
I'm trying to build a translation feature for my webapp. There are multiple packages in my app. Each package(directory) contains a translation folder and yaml files inside. But I have a problem with parsing and assign it to messages.
en.yaml
msgLogin : "You've login successfully"
msgProducts:
0: "You don't have any product."
1: "You have %d product."
2: "You have %d products."
errLogin: "Wrong password or username"
my code:
type TranslationEntry struct {
Key struct {
Value interface{}
}
}
func parseTranslations(dir string) {
files, _ := ioutil.ReadDir(dir)
for _, f := range files {
yamlFile, _ := ioutil.ReadFile(dir + "/" + f.Name())
var data translation
if err := yaml.Unmarshal(yamlFile, &data); err != nil {
return nil, err
}
lang := strings.Split(f.Name(), ".")[0]
switch msg := data.Key.Value.(type) {
case string:
message.SetString(language.Make(lang), cast.ToString(data.Key), cast.ToString(data.Key.Value))
case map[int]string:
message.Set(language.Make(lang), cast.ToString(data.Key),
plural.Selectf(1, "%d",
"=0", cast.ToString(data.Key.Value["0"]),
"=1", cast.ToString(data.Key.Value["1"]),
"=2", cast.ToString(data.Key.Value["2"]),
))
}
translations[lang] = &dictionary{Data: data}
}
}
I'm totally lost about how to design my struct or parse yaml file.
Thank you in advanced
If you're using the YAML library I think you're using (https://godoc.org/gopkg.in/yaml.v2), to make a Golang struct which can use Unmarshal to map from the YAML file in your post you can do this:
type TranslationEntry struct {
MsgLogin string `yaml:"msgLogin"`
MsgProducts map[int]string `yaml:"msgProducts"`
ErrLogin string `yaml:"errLogin"`
}
The things inside the `` after the field declarations are called tags. They're the way field names are usually specified when mapping between some datatype and a Golang struct (in my case usually I map a struct to JSON, but I've also done YAML). If you're using the same YAML parser I mentioned above, this is how it works.
Basically the text inside the double quotes is the YAML key to which your struct field will be mapped. In the above code the only difference between the YAML key name and the struct field name is capitalization, but here is an example using totally different names:
type ExampleStruct struct {
MyAbcField string `yaml:"abc"`
}
This will set the value of MyAbcField to "my data" when using Unmarshal with ExampleStruct and the following YAML:
abc: "my data"
This allows you to design a Golang struct which matches however you decide to structure your YAML.
Here's my above code in Go Playground: https://play.golang.org/p/Q9FvNsw-BOx
Now, if you are unable to fix a structure for your YAML files, you can also parse into nested maps. You can do this by passing a variable of type interface{} (empty interface) to Unmarshal instead of a struct. However, this requires a lot of boilerplate because you will need to do type assertions to access your data. Thus I recommend using a fixed structure instead unless you absolutely can't avoid it.
Here's an example where I parse the YAML you posted and then get the msgLogin field:
var data interface{}
if err := yaml.Unmarshal([]byte(yamlFile), &data); err != nil {
// handle error
}
// a type assertion that data is a map is needed in order to get keys or iterate
topLevel, ok := data.(map[interface{}]interface{})
if !ok {
// handle error
}
fmt.Println(topLevel["msgLogin"])
And here's the Go Playground of my struct example changed to use parsing into a nested map instead: https://play.golang.org/p/ERBjClSazkz
How do I append output from a twitter search to the field Data in the SearchTwitterOutput{} struct.
Thanks!
I am using a twitter library to search twitter base on a query input. The search returns an array of strings(I believe), I am able to fmt.println the data but I need the data as a struct.
type SearchTwitterOutput struct {
Data string
}
func (SearchTwitter) execute(input SearchTwitterInput) (*SearchTwitterOutput, error) {
credentials := Credentials{
AccessToken: input.AccessToken,
AccessTokenSecret: input.AccessTokenSecret,
ConsumerKey: input.ConsumerKey,
ConsumerSecret: input.ConsumerSecret,
}
client, err := GetUserClient(&credentials)
if err != nil {
return nil, err
}
// search through the tweet and returns a
search, _ , err := client.Search.Tweets(&twitter.SearchTweetParams{
Query: input.Text,
})
if err != nil {
println("PANIC")
panic(err.Error())
return &SearchTwitterOutput{}, err
}
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
}
return &SearchTwitterOutput{
Data: "test", //data is a string for now it can be anything
}, nil
}
//Data field is a string type for now it can be anything
//I use "test" as a placeholder, bc IDK...
Result from fmt.Printf("Tweet %d - %s\n", k, v.Text):
Tweet 0 - You know I had to do it to them! #JennaJulien #Jenna_Marbles #juliensolomita #notjulen Got my first hydroflask ever…
Tweet 1 - RT #brenna_hinshaw: I was in J2 today and watched someone fill their hydroflask with vanilla soft serve... what starts here changes the wor…
Tweet 2 - I miss my hydroflask :(
This is my second week working with go and new to development. Any help would be great.
It doesn't look like the client is just returning you a slice of strings. The range syntax you're using (for k, v := range search.Statuses) returns two values for each iteration, the index in the slice (in this case k), and the object from the slice (in this case v). I don't know the type of search.Statuses - but I know that strings don't have a .Text field or method, which is how you're printing v currently.
To your question:
Is there any particular reason to return just a single struct with a Data field rather than directly returning the output of the twitter client?
Your function signature could look like this instead:
func (SearchTwitter) execute(input SearchTwitterInput) ([]<client response struct>, error)
And then you could operate on the text in those objects in wherever this function was called.
If you're dead-set on placing the data in your own struct, you could return a slice of them ([]*SearchTwitterOutput), in which case you could build a single SearchTwitterOutput in the for loop you're currently printing the tweets in and append it to the output list. That might look like this:
var output []*SearchTwitterOutput
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
output = append(output, &SearchTwitterOutput{
Data: v.Text,
})
}
return output, nil
But if your goal really is to return all of the results concatenated together and placed inside a single struct, I would suggest building a slice of strings (containing the text you want), and then joining them with the delimiter of your choosing. Then you could place the single output string in your return object, which might look something like this:
var outputStrings []string
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
outputStrings = append(outputStrings, v.Text)
}
output = strings.Join(outputStrings, ",")
return &SearchTwitterOutput{
Data: output,
}, nil
Though I would caution, it might be tricky to find a delimiter that will never show up in a tweet..