I have created a function to get some data and write them to CSV and the output is stored on a buffer.
type OptIn struct {
Email string `json:"email"`
LastUpdate string `json:"opt_in_last_update"`
}
func writeCSV(data []OptIn) ([]byte, error) {
var buf bytes.Buffer
writer := csv.NewWriter(&buf)
defer writer.Flush()
for _, obj := range data {
var record []string
record = append(record, obj.Email)
record = append(record, obj.LastUpdate)
err := writer.Write(record)
if err != nil {
panic(err.Error())
}
}
return buf.Bytes(), nil
}
The problem is that the buf.Bytes() is always empty, even though the input is not empty and there are no errors thrown.
You need to call writer.Flush() before calling .Bytes() and check .Error() before returning:
// TODO: remove `defer writer.Flush()`
// ...
writer.Flush()
if err := writer.Error(); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
Your sample code does things in the following order:
call buf.Bytes()
return the generated byte slice and nil
call writer.Flush()
Clearly this is not the intended order since we need to flush (and check for any writer errors!) before accessing the generated byte slice.
Related
Let say that I have this code:
type Type1 struct {
Name string `json:"name,omitempty"`
Path string `json:"path"`
File string `json:"file"`
Tag int `json:"tag"`
Num int `json:"num"`
}
func LoadConfiguration(data []byte) (*Type1, error) {
config, err := loadConf1(data)
if err != nil {
return nil, err
}
confOther, err := loadConfOther1()
if err != nil {
return nil, err
}
// do something with confOther
fmt.Println("confOther", confOther)
if confOther.Tag == 0 {
config.Num = 5
}
// do something with config attributes of type1
if config.Tag == 0 {
config.Tag = 5
}
if config.Num == 0 {
config.Num = 4
}
return config, nil
}
func loadConf1(bytes []byte) (*Type1, error) {
config := &Type1{}
if err := json.Unmarshal(bytes, config); err != nil {
return nil, fmt.Errorf("cannot load config: %v", err)
}
return config, nil
}
func loadConfOther1() (*Type1, error) {
// return value of this specific type
flatconfig := &Type1{}
// read a file as []byte
// written as a fixed array to simplify this example
fileContent := []byte{10, 22, 33, 44, 55}
if err := json.Unmarshal(fileContent, flatconfig); err != nil {
return nil, fmt.Errorf("cannot read config %v", err)
}
return flatconfig, nil
}
The only public function is LoadConfiguration.
It's based on a real code and It's used to read a json data as a specific struct. If something seems useless, it's because I simplified the original code.
The code above is ok, but now I want to create another struct type called "Type2" and re-use the same methods to read data into Type2 without copying and pasting everything.
type Type2 struct {
Name string `json:"name,omitempty"`
Path string `json:"path"`
Map *map[string]interface{} `json:"map"`
Other string `json:"other"`
}
Basically, I want to be able to call LoadConfiguration to get also Type2. I can accept to call a specific method like LoadConfiguration2, but I don't want to copy and paste also loadConf1 and loadConfOther1.
Is there a way to do that in an idiomatic way in Go 1.18?
Actually the code shown in your question doesn't do anything more than passing a type into json.Unmarshal and format an error so you can rewrite your function to behave just like it:
func LoadConfiguration(data []byte) (*Type1, error) {
config := &Type1{}
if err := loadConf(data, config); err != nil {
return nil, err
}
// ...
}
// "magically" accepts any type
// you could actually get rid of the intermediate function altogether
func loadConf(bytes []byte, config any) error {
if err := json.Unmarshal(bytes, config); err != nil {
return fmt.Errorf("cannot load config: %v", err)
}
return nil
}
In case the code actually does something more than just passing a pointer into json.Unmarshal, it can benefit from type parameters.
type Configurations interface {
Type1 | Type2
}
func loadConf[T Configurations](bytes []byte) (*T, error) {
config := new(T)
if err := json.Unmarshal(bytes, config); err != nil {
return nil, fmt.Errorf("cannot load config: %v", err)
}
return config, nil
}
func loadConfOther[T Configurations]() (*T, error) {
flatconfig := new(T)
// ... code
return flatconfig, nil
}
In these cases you can create a new pointer of either type with new(T) and then json.Unmarshal will take care of deserializing the content of the byte slice or file into it — provided the JSON can be actually unmarshalled into either struct.
The type-specific code in the top-level function should still be different, especially because you want to instantiate the generic functions with an explicit concrete type. So I advise to keep LoadConfiguration1 and LoadConfiguration2.
func LoadConfiguration1(data []byte) (*Type1, error) {
config, err := loadConf[Type1](data)
if err != nil {
return nil, err
}
confOther, err := loadConfOther[Type1]()
if err != nil {
return nil, err
}
// ... type specific code
return config, nil
}
However if the type-specific code is a small part of it, you can probably get away with a type-switch for the specific part, though it doesn't seem a viable option in your case. I would look like:
func LoadConfiguration[T Configuration](data []byte) (*T, error) {
config, err := loadConf[T](data)
if err != nil {
return nil, err
}
// let's pretend there's only one value of type parameter type
// type-specific code
switch t := config.(type) {
case *Type1:
// ... some *Type1 specific code
case *Type2:
// ... some *Type2 specific code
default:
// can't really happen because T is restricted to Configuration but helps catch errors if you extend the union and forget to add a corresponding case
panic("invalid type")
}
return config, nil
}
Minimal example playground: https://go.dev/play/p/-rhIgoxINTZ
I am creating a buffer and then writing into that buffer via zip.NewWriter()
So my code looks like this
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)
// Create a new zip archive.
w := zip.NewWriter(buf)
// Create file in this writer
f, err := w.Create("test.json")
/ Writing data to file
_, _ = f.Write([]byte("some data"))
// Close writer
_ = w.Close()
// TODO: Return this data in the form of io.ReadCloser
My end goal is to write files into this zip writer and return the data in the form of io.ReadCloser, my calling function expects data in io.ReadCloser
I tried to find and try multiple ways but none of them succeeded.
Check if this example works for you
func ExampleWriter() {
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)
// Create a new zip archive.
w := zip.NewWriter(buf)
// Add some files to the archive.
var files = []struct {
Name, Body string
}{
{"readme.txt", "This archive contains some text files."},
{"gopher.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
{"todo.txt", "Get animal handling licence.\nWrite more examples."},
}
for _, file := range files {
f, err := w.Create(file.Name)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(file.Body))
if err != nil {
log.Fatal(err)
}
}
// Make sure to check the error on Close.
err := w.Close()
if err != nil {
log.Fatal(err)
}
}
By systematically check the error for those operations, you will see if there are any issue.
After some help from comments I got it working
So we can directly convert buf to io.ReadCloser and return it, no need for further conversion.
func ExampleWriter() (io.ReadCloser, error) {
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)
// Create a new zip archive.
w := zip.NewWriter(buf)
// Add some files to the archive.
var files = []struct {
Name, Body string
}{
{"readme.txt", "This archive contains some text files."},
{"gopher.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
{"todo.txt", "Get animal handling licence.\nWrite more examples."},
}
for _, file := range files {
f, err := w.Create(file.Name)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(file.Body))
if err != nil {
log.Fatal(err)
}
}
// Make sure to check the error on Close.
err := w.Close()
if err != nil {
log.Fatal(err)
}
return ioutil.NopCloser(buf), nil // This returns io.ReadCloser
}
I am attempting to map a response from an API to a struct using Golang.
The JSON that comes back when I view the link in the browser is below:
{
"GBP": 657.54
}
And I just want to map it to a simple struct like so:
type Price struct {
Name string
Value float64
}
Here is my current code.
func FetchCoinPrice(fsym string, tsyms string) Price {
url := fmt.Sprintf("https://min-api.cryptocompare.com/data/price?fsym=" + fsym + "&tsyms=" + tsyms)
fmt.Println("Requesting data from " + url)
price := Price{}
// getting the data using http
request, err := http.Get(url)
if err != nil {
log.Fatal(err.Error())
}
// Read the response body using ioutil
body, err := ioutil.ReadAll(request.Body)
if err != nil {
log.Fatal(err.Error())
}
defer request.Body.Close()
if request.StatusCode == http.StatusOK {
json.Unmarshal(body, &price)
}
return price
}
At the moment all I receive is an empty struct, I know the link is bringing back the correct data and I've tested it in my browser.
The mapping doesn't work that way. Instead, you should use a map:
data := []byte(`{
"GBP": 657.54
}`)
priceMap := map[string]float64{}
err := json.Unmarshal(data, &priceMap)
// Check your errors!
if err != nil {
log.Fatal(err.Error())
}
fmt.Println(priceMap)
This will print:
map[GBP:657.54]
You can then iterate over the map and build the struct you mentioned above, or just access the entry directly if you know the currency. eg: priceMap["GBP"]
You should really check your errors, especially if you're not getting the output you expect from Unmarshal.
The problem is that the unmarshaler cannot guess that keys in a JSON object should correspond to some value in a struct. Golang JSON mapping simply doesn't work that way.
However, you can make your "Price" type implement json.Unmarshaler to deserialize a message into a map of floats (map[string]float64) then ensure the shape is right and populate the struct accordingly:
func (p *Price) UnmarshalJSON(bs []byte) error {
kvs := map[string]float64{}
err := json.Unmarshal(bs, &kvs)
if err != nil {
return err
}
if len(kvs) != 1 {
return fmt.Errorf("expected 1 key, got %d", len(kvs))
}
for name, value := range kvs {
p.Name, p.Value = name, value
}
return nil
}
func main() {
jsonstr := `[{"GBP":657.54},{"USD":123.45}]`
ps := []Price{}
err := json.Unmarshal([]byte(jsonstr), &ps)
if err != nil {
panic(err)
}
// ps=[]main.Price{
// main.Price{Name:"GBP", Value:657.54},
// main.Price{Name:"USD", Value:123.45}
// }
}
I have a func in Go that simply writes to a buffer. I have no return type set on the func so I am not sure why I am seeing this error. Here is my code:
func Write(buffer *bytes.Buffer, values ...string) {
for _, val := range values
_, err := *buffer.WriteString(val)
if err != nil {
// print error
}
}
_, err := *buffer.WriteString(" ")
if err != nil {
// print error
}
}
It complains at both lines where I have buffer.WriteString. This leads me to believe it has something to do with the return types of the WriteString method on the buffer but I am not experienced enough in Go to know for sure.
Any help would be appreciated.
Edit: Updated code.
You don't need to dereference pointers to call methods in Go. The * operator before buffer.WriteString is applied to the returned values. To dereference buffer you would need to write (*buffer).WriteString, but that's not needed at all:
func Write(buffer *bytes.Buffer, values ...string) {
for _, val := range values {
_, err := buffer.WriteString(val)
if err != nil {
// print error
}
}
_, err := buffer.WriteString(" ")
if err != nil {
// print error
}
}
I need to write a generic function which can store objects as gobjects.
func hash_store(data map[string]string) {
//initialize a *bytes.Buffer
m := new(bytes.Buffer)
//the *bytes.Buffer satisfies the io.Writer interface and can
//be used in gob.NewEncoder()
enc := gob.NewEncoder(m)
//gob.Encoder has method Encode that accepts data items as parameter
enc.Encode(data)
//the bytes.Buffer type has method Bytes() that returns type []byte,
//and can be used as a parameter in ioutil.WriteFile()
err := ioutil.WriteFile("dep_data", m.Bytes(), 0600)
if err != nil {
panic(err)
}
fmt.Printf("just saved all depinfo with %v\n", data)
n,err := ioutil.ReadFile("dep_data")
if err != nil {
fmt.Printf("cannot read file")
panic(err)
}
//create a bytes.Buffer type with n, type []byte
p := bytes.NewBuffer(n)
//bytes.Buffer satisfies the interface for io.Writer and can be used
//in gob.NewDecoder()
dec := gob.NewDecoder(p)
//make a map reference type that we'll populate with the decoded gob
//e := make(map[int]string)
e := make(map[string]string)
//we must decode into a pointer, so we'll take the address of e
err = dec.Decode(&e)
if err != nil {
fmt.Printf("cannot decode")
panic(err)
}
fmt.Println("after reading dep_data printing ",e)
}
In this function I know the data type to be stored in map[string]string . But I need to write a generic function where I don't know data type and still store it as a gobject in a file.
Change your concrete type (map[string]string) to the empty interface type (interface{}).
See this related question why this works.
Encoding:
func store(data interface{}) {
m := new(bytes.Buffer)
enc := gob.NewEncoder(m)
err := enc.Encode(data)
if err != nil { panic(err) }
err = ioutil.WriteFile("dep_data", m.Bytes(), 0600)
if err != nil { panic(err) }
}
Decoding:
func load(e interface{}) {
n,err := ioutil.ReadFile("dep_data")
if err != nil { panic(err) }
p := bytes.NewBuffer(n)
dec := gob.NewDecoder(p)
err = dec.Decode(e)
if err != nil { panic(err) }
}
The value you put in load must be a pointer of the type you stored in the file using gob.
Example for map[string]string:
org := map[string]string{"foo": "bar"}
store(org)
var loadedMap map[string]string
load(&loadedMap)
fmt.Println(loadedMap["foo"]) // bar
When you encode the data, give the Encoder a *interface{}, then you can decoded with a *interface{}
var to_enc interface{} = ...;
god.NewEncoder(...).Encode(&to_enc);
...
var to_dec interface{}
god.NewDecoder(...).Decode(&to_dec);