I'm writing a small application that receives message in BSON format from network(its not MongoDB) and have to save fields in files on local machine. I'm using gopkg.in/mgo.v2/bson for message unmarshaling and it works fine.
Almost everything works except one. There "userdefined" binary field in message and I have to save it to separate file. I tried to use:
var pwr = msg["pwr"].([]byte)
but got an "error panic: interface conversion: interface is bson.Binary, not []uint8".
Can some one point me an example how to convert bson.Binary to []byte, so I can save it to file.
This does what you want:
pwr := bson.Binary(msg["pwr"].(bson.Binary)).Data
But assumes msg["pwr"] can't be anything other than a bson.Binary... if that's not an invariant you should do the type assertion first (handle the possible type mismatch case when it happens) and then cast to get the Data field.
Related
Suppose we are debugging some Go code, and somewhere in an external dependency we encounter this line:
return json.Marshal(foo)
We want to set a breakpoint and use IntelliJ's "Evaluate Expression" to inspect the JSON being produced. However, this doesn't work:
If we evaluate the expression json.Marshal(foo), we only get to see the byte array.
Evaluating string(json.Marshal(foo)) doesn't work because json.Marshal returns two values, the byte array and an error.
There is no way in Go to access one of the return values directly.
So how can I use "Evaluate Expression" to achieve my goal of just printing the produced JSON string when I'm not able to change the underlying source code?
you can print the returned bytes as a string
bytes, err := json.Marshal(foo)
// check error here
fmt.Println(string(bytes))
update based on comments
You can't change the byte slice in the debugger to a string without changing the source code.
Golang Code Here: (include 4 files here)
https://gist.github.com/kmahyyg/02a2da2970001de455f847f4e7525aff
When defined as above, compress a big file (512M here, a bin created by dd from /dev/urandom).
If you use SetWriter(out), try to pass out as a bufio.Writer but keep the struct field definition as io.Writer, and the same as Reader part.
Then try decompress, you will get an unexpected EOF error.
But if you pass out as a io.Writer, everything will be fine.
Compress function have no errors.
Why use bufio.Writer will cause unexpected EOF?
Note:
after some observation, it seems that file smaller than a specific size (here, is 337MB on my machine) will not get unexpected EOF.
The official gunzip extract the same gzip file which caused unexpected EOF will only get about the first 337M part of data, then get the "corrupted file" message.
Edit: 1. Full code attached.
2. Screen shot here: (Use zstd as an example, same result when use gzip)
#leafbebop has the correct answer. The io.Writer will not automatically flush the buffer when close. So you must manually flush it before close when use bufio.Writer as io.Writer
The struct in the .pb.go file generated by .proto file has three additional fields and some other things.like this:
When converting this struct to json, if one field is empty, the field will not appear in json. Now I know it can be done using jsonpb.Marshaler.
m := jsonpb.Marshaler{EmitDefaults: true}
Now, I coverting struct to map[string]interface{}, put it in
InfluxDB. I have to convert struct to map[string]interface{}.The function NewPoint needs. like this:
I use structs.Map(value) function in go ,The transformed map has three additional fields, and running the program causes errors，like this:
{"error":"unable to parse 'txt,severity=1 CurrentValue=\"1002\",MetricAlias=\"CPU\",XXX_sizecache=0i,XXX_unrecognized= 1552551101': missing field value"}
When I remove these three fields, the program runs OK.These three fields are automatically generated, and I have a lot of structs.
What should I do?Thank you!
Protobuf generator adds some additional fields with names starting from XXX that are intended for optimizations. You can't change this behavior of protoc-gen-go.
The problem is in the way you convert struct to map[sting]interface{}. It's hard to figure out from which package exactly structs.Map comes from. Seems like it goes from here: https://github.com/fatih/structs/blob/master/structs.go#L89 - this code uses reflect to iterate through all fields of the structure and push them to map[sting]interface{}. You just need to write your own slightly modified version of FillMap routine that will omit XXX fields.
For testing, I often see go code read byte slices, which are parsed into structs using yaml, for example here:
https://github.com/kubernetes/kubernetes/blob/master/pkg/util/strategicpatch/patch_test.go#L74m
I just got bitten by not exporting my field names, resulting in an empty list which I iterated over in my test cases, thus assuming that all tests were passing (in hindsight, that should have been a red flag :)). There are other errors which are silently ignored by yaml unmarshaling, such as a key being misspelled and not matching a struct field exactly.
Is there a way to ensure that all the data in the byte slice was actually parsed into the struct returned by yaml.Unmarshal? If not, how do others handle this situation?
go-yaml/yaml
For anyone searching for a solution to this problem, the yaml.v2 library has an UnmarshalStrict method that returns an error if there are keys in the yaml document that have no corresponding fields in the go struct.
import yaml "gopkg.in/yaml.v2"
err := yaml.UnmarshalStrict(data, destinationStruct)
BurntSushi/toml
It's not part of the question, but I'd just like to document how to achieve something similar in toml:
You can find if there were any keys in the toml file that could not be decoded by using the metadata returned by the toml.decode function.
import "github.com/BurntSushi/toml"
metadata, err := toml.Decode(data, destinationStruct)
undecodedKeys := metadata.Undecoded()
Note that metadata.Undecoded() also returns keys that have not been decoded because of a Primitive value. You can read more about it here.
Json
The default go json library does not support this currently, but there is a proposal ready to be merged. It seems that it will be a part of go 1.10.
I have a field that is of type fixed64 in a .proto file.
I want to read it as an int64 field:
score := int64(pb_obj.Score)
When I try to compile the line agove I get the error message cannot convert pb_obj.Score (type *uint64) to type int64. I tried converting the a uint64 as well, and got an almost identical message.
pb_obj.Score's type seems to be *uint64 (pointer to uint64), not uint64. You just need to access to the value the pointer is referencing:
score := int64(*pb_obj.Score)
(See the * prefix as the difference)
Based on the compile error you're working with a uint64 pointer and not a uint64 value. You may get what you want by referencing the value directly using the * operator. I've never worked with protobuf, so I could be off but that should get you moving. Here's a nice reference that may help golang pointers