I have a method that takes a URL to a yaml file and unmarshals it into a struct. There's nothing unique about it so I'd like to use it on another struct.
type SWConfig struct {
Name string `yaml:"name"`
Version string `yaml:"version"`
BuildType string `yaml:"buildType"`
}
func (c *SWConfig) getConfiguration(url string) {
resp, err := http.Get(url)
if err != nil {
log.Fatalf("ERROR: GET request failed: %s\n", err.Error())
}
if resp.StatusCode != http.StatusOK {
log.Fatalf("ERROR: %v: %s\n", resp.Status, url)
}
source, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("ERROR: could not read response: %s\n", err.Error())
}
err = yaml.Unmarshal(source, c)
if err != nil {
log.Fatalf("ERROR: could not read YAML: %s\n", err.Error())
}
}
var swConfig SWConfig
swConfig.getConfiguration(swURL)
fmt.Println(swConfig.Name)
The other struct would have different fields, but again that shouldn't matter for this method. Is it possible to reuse this method or do I need to convert it to a function?
Use this utility function. It works with a pointer to any type.
// fetchYAML unmarshals the YAML document at url to the value
// pointed to by v.
func fetchYAML(url string, v interface{}) error {
resp, err := http.Get(url)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("bad status: %v: %s\n", resp.Status, url)
}
source, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
err = yaml.Unmarshal(source, v)
return err
}
Call it like this:
func (c *SWConfig) getConfiguration(url string) {
if err := fetchYAML(url, c); err != nil {
log.Fatalf("ERROR: get value failed: %s\n", err.Error())
}
}
Related
I have to read let say 2 or 3 or more yamls that are different in structure and have a struct for each of those structures where I want to store them. So far I am creating separate functions for each and it works, but does not look very elegant... I think.
Here are the functions today:
// read the Yaml into struct(s)
type Config struct {...}
type ExecuteQueries struct {...}
func parseYamlConfig(pathYaml string) Config {
myConfig := Config{}
var err error
var yamlFile []byte
if pathYaml == "" {
yamlFile, err = ioutil.ReadFile("./conf/conf.yaml")
} else {
yamlFile, err = ioutil.ReadFile(pathYaml)
}
if err != nil {
log.Fatalf("error: %v", err)
}
err = yaml.Unmarshal([]byte(yamlFile), &myConfig)
if err != nil {
log.Fatalf("error: %v", err)
}
return myConfig
}
func parseYamlConfig2(pathYaml string) ExecuteQueries {
myConfig := ExecuteQueries{}
var err error
var yamlFile []byte
if pathYaml == "" {
yamlFile, err = ioutil.ReadFile("./conf/conf.yaml")
} else {
yamlFile, err = ioutil.ReadFile(pathYaml)
}
if err != nil {
log.Fatalf("error: %v", err)
}
err = yaml.Unmarshal([]byte(yamlFile), &myConfig)
if err != nil {
log.Fatalf("error: %v", err)
}
return myConfig
}
Notice that they are actually different in what they return and what they receive, but the processing of data is very similar. How should this be expressed in a more elegant way?
func unmarshalYAMLFile(path string, v interface{}) error {
if path == "" {
path = "./conf/conf.yaml"
}
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
return yaml.NewDecoder(f).Decode(v)
}
conf1 := Config{}
if err := unmarshalYAMLFile("/path/to/conf.yaml", &conf1); err != nil {
panic(err)
}
conf2 := ExecuteQueries{}
if err := unmarshalYAMLFile("/path/to/conf_2.yaml", &conf2); err != nil {
panic(err)
}
I have this function to parse HTTP results:
func (a *Admin) ResponseDecode(structName string, body io.Reader) (interface{}, error) {
content, err := ioutil.ReadAll(body)
if err != nil {
return nil, err
}
switch structName {
case "[]Cat":
var data []Cat
err = json.Unmarshal(content, &data)
if err != nil {
return nil, err
}
return data, err
case "[]Dog":
var data []Dog
err = json.Unmarshal(content, &data)
if err != nil {
return nil, err
}
return data, err
default:
log.Fatal("Can't decode " + structName)
}
return nil, nil
}
I do a type assertion after this method :
parsed, err := a.ResponseDecode("[]Cat", resp.Body)
if err != nil {
return nil, err
}
return parsed.([]Cat), nil
but how can I do to avoid the repetition of the code :
var data []Stuff
err = json.Unmarshal(content, &data)
if err != nil {
return nil, err
}
return data, err
Every time I add an object? Usually I would use generics, but this is Go. What's the good way to do that ?
You are passing in the name of the struct, and then expecting data of that type. Instead, you can simply pass the struct:
var parsed []Cat
err := a.ResponseDecode(&parsed, resp.Body)
where:
func (a *Admin) ResponseDecode(out interface{}, body io.Reader) error {
content, err := ioutil.ReadAll(body)
if err != nil {
return nil, err
}
return json.Unmarshal(content,out)
}
In fact, you can get rid of ResponseDecode function:
var parsed []Cat
err:=json.NewDecoder(body).Decode(&parsed)
Found what I was looking for here.
This function does the job :
func (a *Admin) ResponseDecode(body io.Reader, value interface{}) (interface{}, error) {
content, err := ioutil.ReadAll(body)
if err != nil {
return nil, err
}
err = json.Unmarshal(content, &value)
if err != nil {
return nil, err
}
return value, nil}
I have defined two funcs that do slightly different things but are syntactically the same.
Functions in question send POST requests to an api.
The duplication occurs in constructing the request, adding headers, etc.
How can I refactor the code to remove said duplication.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httputil"
)
type token struct {
Token string
}
type config struct {
Foo string
}
func main() {
token, err := getAuthToken()
if err != nil {
log.Fatal(err)
}
config, err := getConfig("foo", token)
if err != nil {
log.Fatal(err)
}
_ = config
}
func getAuthToken() (string, error) {
endpoint := "foo"
body := struct {
UserName string `json:"username"`
Password string `json:"password"`
}{
UserName: "foo",
Password: "bar",
}
jsnBytes, err := json.Marshal(body)
if err != nil {
return "", err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return "", fmt.Errorf("Unable to create request. %v", err)
}
req.Header.Add("Content-Type", "application/json")
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return "", fmt.Errorf("Could not dump request. ", err)
}
log.Println("Request: ", string(dump))
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("HTTP Error: %v", err)
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("Error reading response body: %v", err)
}
var token token
err = json.Unmarshal(bytes, &token)
if err != nil {
return "", fmt.Errorf("Could not unamrshal json. ", err)
}
return token.Token, nil
}
func getConfig(id string, token string) (*config, error) {
endpoint := "foo"
body := struct {
ID string `json:"id"`
}{
ID: id,
}
jsnBytes, err := json.Marshal(body)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return nil, fmt.Errorf("Unable to create request. %v", err)
}
req.Header.Add("Authorization", "Bearer "+token)
req.Header.Add("Content-Type", "application/json")
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return nil, fmt.Errorf("Could not dump request. ", err)
}
log.Println("Request: ", string(dump))
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("HTTP Error: %v", err)
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("Error reading response body: %v", err)
}
var config config
err = json.Unmarshal(bytes, &config)
if err != nil {
return nil, fmt.Errorf("Could not unamrshal json. ", err)
}
return &config, nil
}
I would say the essence of sending the request is that you are sending a body to an endpoint and parsing a result. The headers are then optional options that you can add to the request along the way. With this in mind I would make a single common function for sending the request with this signature:
type option func(*http.Request)
func sendRequest(endpoint string, body interface{}, result interface{}, options ...option) error {
Note this is using functional options which Dave Cheney did an excellent description of here:
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
The complete code then becomes:
https://play.golang.org/p/GV6FeipIybA
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httputil"
)
type token struct {
Token string
}
type config struct {
Foo string
}
func main() {
token, err := getAuthToken()
if err != nil {
log.Fatal(err)
}
config, err := getConfig("foo", token)
if err != nil {
log.Fatal(err)
}
_ = config
}
func getAuthToken() (string, error) {
endpoint := "foo"
body := struct {
UserName string `json:"username"`
Password string `json:"password"`
}{
UserName: "foo",
Password: "bar",
}
var token token
err := sendRequest(endpoint, body, &token)
if err != nil {
return "", err
}
return token.Token, nil
}
func getConfig(id string, token string) (*config, error) {
endpoint := "foo"
body := struct {
ID string `json:"id"`
}{
ID: id,
}
var config config
err := sendRequest(endpoint, body, &config, header("Content-Type", "application/json"))
if err != nil {
return nil, err
}
return &config, nil
}
type option func(*http.Request)
func header(key, value string) func(*http.Request) {
return func(req *http.Request) {
req.Header.Add(key, value)
}
}
func sendRequest(endpoint string, body interface{}, result interface{}, options ...option) error {
jsnBytes, err := json.Marshal(body)
if err != nil {
return err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return fmt.Errorf("Unable to create request. %v", err)
}
req.Header.Add("Content-Type", "application/json")
for _, option := range options {
option(req)
}
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return fmt.Errorf("Could not dump request. ", err)
}
log.Println("Request: ", string(dump))
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("HTTP Error: %v", err)
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return fmt.Errorf("Error reading response body: %v", err)
}
err = json.Unmarshal(bytes, result)
if err != nil {
return fmt.Errorf("Could not unamrshal json. ", err)
}
return nil
}
The way I would do this is to extract the two parts that are common to both request executions: 1) create a request and 2) execute the request.
Gist with new code using HTTP Bin as an example
Creating the request includes setting up the endpoint, headers and marshaling the request body to JSON. In your case, you're also dumping the request to the log, that can also go in there. This is how it would look like:
func buildRequest(endpoint string, body interface{}, extraHeaders map[string]string) (*http.Request, error) {
jsnBytes, err := json.Marshal(body)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return nil, err
}
req.Header.Add("Content-Type", "application/json")
for name, value := range extraHeaders {
req.Header.Add(name, value)
}
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return nil, err
}
log.Println("Request: ", string(dump))
return req, nil
}
If you have no extra headers, you can pass nil as the third argument here.
The second part to extract is actually executing the request and unmarshalling the data. This is how the executeRequest would look like:
func executeRequest(req *http.Request, responseBody interface{}) error {
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
log.Printf("Response is: %s\n", string(bytes))
err = json.Unmarshal(bytes, &responseBody)
return err
}
I have a below code in Go:
func (api *ApiResource) create(request *restful.Request, response *restful.Response) {
account := &DefaultAccount
err := request.ReadEntity(account)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
tmpl := data_transformer.ParseTemplate("xml/accAdd.xml")
payload := data_transformer.RenderTemplate(tmpl, account)
resp, err := http.Post(url, "application/xml", payload)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
// Body closes when either at the end of the function or at return
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
aResp := new(AResp)
err = xml.Unmarshal(body, aResp)
if err != nil {
fmt.Printf("error: %v", err)
return
}
response.WriteHeader(http.StatusCreated)
response.WriteEntity(aResp)
}
func (api *ApiResource) updateLimit(request *restful.Request, response *restful.Response) {
account := &DefaultLimit
err := request.ReadEntity(account)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
tmpl := data_transformer.ParseTemplate("xml/addLimit.xml")
payload := data_transformer.RenderTemplate(tmpl, account)
resp, err := http.Post(url, "application/xml", payload)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
// Body closes when either at the end of the function or at return
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
aResp := new(AResp)
err = xml.Unmarshal(body, aResp)
if err != nil {
fmt.Printf("error: %v", err)
return
}
response.WriteHeader(http.StatusCreated)
response.WriteEntity(aResp)
}
I want to be able to remove duplicate codes in an elegant way Go style.
If I do remove them and put them in separate function, would all defer and net/http package work as expected for different calls and xml file loads?
What is a good Go type solutions for this?
You can separate almost the whole function and just pass to it the "account" and the xml file to read.
Also it's a better practice to use xml.Decoder instead of xml.Unmarshal when reading from streams.
func updateEntity(response *restful.Response, fn string, iface interface{}) {
tmpl := data_transformer.ParseTemplate("xml/" + fn)
payload := data_transformer.RenderTemplate(tmpl, iface)
resp, err := http.Post(url, "application/xml", payload)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
defer resp.Body.Close()
dec := xml.NewDecoder(resp.Body)
body, err := ioutil.ReadAll(resp.Body)
aResp := &AResp{}
err = dec.Decode(aResp)
if err != nil {
fmt.Printf("error: %v", err)
return
}
response.WriteHeader(http.StatusCreated)
response.WriteEntity(aResp)
}
func (*ApiResource) create(request *restful.Request, response *restful.Response) {
account := &DefaultAccount
err := request.ReadEntity(account)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
updateEntity(response, "accAdd.xml", account)
}
func (*ApiResource) updateLimit(request *restful.Request, response *restful.Response) {
account := &DefaultLimit
err := request.ReadEntity(account)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
updateEntity(response, "addLimit.xml", account)
}
I would like to decompress tar-gz file by golang.
err := DecompressTarGz('xxx.tar.gz', '/Users/foobarbuzz/')
Use compress/gzip in combination with archive/tar or use os/exec to call tar and gzip directly if you don't like to implement all of that in Go.
I was looking for an answer to this one too. Implemented something that should work.
func Decompress(targetdir string, reader io.ReadCloser) error {
gzReader, err := gzip.NewReader(reader)
if err != nil {
return err
}
defer gzReader.Close()
tarReader := tar.NewReader(gzReader)
for {
header, err := tarReader.Next()
if err == io.EOF {
break
} else if err != nil {
return err
}
target := path.Join(targetdir, header.Name)
switch header.Typeflag {
case tar.TypeDir:
err = os.MkdirAll(target, os.FileMode(header.Mode))
if err != nil {
return err
}
setAttrs(target, header)
break
case tar.TypeReg:
w, err := os.Create(target)
if err != nil {
return err
}
_, err = io.Copy(w, tarReader)
if err != nil {
return err
}
w.Close()
setAttrs(target, header)
break
default:
log.Printf("unsupported type: %v", header.Typeflag)
break
}
}
return nil
}
func setAttrs(target string, header *tar.Header) {
os.Chmod(target, os.FileMode(header.Mode))
os.Chtimes(target, header.AccessTime, header.ModTime)
}