Semantic way of http.Response receiver functions in Go - go

I just started learning GO and wrote this piece of code that writes an http.Response.Body to os.Stdout or to a file, but I'm not happy about the semantics of this.
I want the http.Response struct to have these receiver functions, so I can use it more easily throughout the entire app.
I know that the answers might get flagged as opinionated, but I still wonder, is there a better way of writing this?
Is there some sort of best practice?
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
)
type httpResp http.Response
func main() {
res, err := http.Get("http://www.stackoverflow.com")
if err != nil {
fmt.Println("Error: ", err)
os.Exit(1)
}
defer res.Body.Close()
response := httpResp(*res)
response.toFile("stckovrflw.html")
response.toStdOut()
}
func (r httpResp) toFile(filename string) {
str, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
ioutil.WriteFile(filename, []byte(str), 0666)
}
func (r httpResp) toStdOut() {
_, err := io.Copy(os.Stdout, r.Body)
if err != nil {
panic(err)
}
}
On a side note, is there a way to make the http.Get method spit out a custom type that already has access to these receiver functions without the need for casting? So i could do something like this:
func main() {
res, err := http.Get("http://www.stackoverflow.com")
if err != nil {
fmt.Println("Error: ", err)
os.Exit(1)
}
defer res.Body.Close()
res.toFile("stckovrflw.html")
res.toStdOut()
}
Thanks!

You don't have to implement these functions. *http.Response already implements io.Writer:
Write writes r to w in the HTTP/1.x server response format, including the status line, headers, body, and optional trailer.
package main
import (
"net/http"
"os"
)
func main() {
r := &http.Response{}
r.Write(os.Stdout)
}
In the example above, the zero value prints:
HTTP/0.0 000 status code 0
Content-Length: 0
Playground: https://play.golang.org/p/2AUEAUPCA8j
In case you need additional business logic in the write methods, you can embed *http.Response in your defined type:
type RespWrapper struct {
*http.Response
}
func (w *RespWrapper) toStdOut() {
_, err := io.Copy(os.Stdout, w.Body)
if err != nil {
panic(err)
}
}
But then you must construct a variable of type RespWrapper with the *http.Response:
func main() {
// resp with a fake body
r := &http.Response{Body: io.NopCloser(strings.NewReader("foo"))}
// or r, _ := http.Get("example.com")
// construct the wrapper
wrapper := &RespWrapper{Response: r}
wrapper.toStdOut()
}
is there a way to make the http.Get method spit out a custom type
No, the return types of http.Get are (resp *http.Response, err error), that's part of the function signature, you can't change it.

Related

variable is empty but later has a value

I'm trying to develop a Terraform provider but I have a problem of the first request body. Here is the code:
type Body struct {
id string
}
func resourceServerCreate(d *schema.ResourceData, m interface{}) error {
key := d.Get("key").(string)
token := d.Get("token").(string)
workspace_name := d.Get("workspace_name").(string)
board_name := d.Get("board_name").(string)
resp, err := http.Post("https://api.trello.com/1/organizations?key="+key+"&token="+token+"&displayName="+workspace_name,"application/json",nil)
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()
//lettura body.
body := new(Body)
json.NewDecoder(resp.Body).Decode(body)
log.Println("[ORCA MADONNA] il log funzia "+body.id)
d.Set("board_id",body.id)
resp1, err1 := http.Post("https://api.trello.com/1/boards?key="+key+"&token="+token+"&idOrganization="+body.id+"&=&name="+board_name,"application/json",nil)
if err1 != nil {
log.Fatalln(resp1)
}
defer resp1.Body.Close()
d.SetId(board_name)
return resourceServerRead(d, m)
}
In the log is empty, but the second call have it and work fine. How is it possible?
Go doesn't force you to check error responses, therefore it's easy to make silly mistakes. Had you checked the return value from Decode(), you would have immediately discovered a problem.
err := json.NewDecoder(resp.Body).Decode(body)
if err != nil {
log.Fatal("Decode error: ", err)
}
Decode error: json: Unmarshal(non-pointer main.Body)
So your most immediate fix is to use & to pass a pointer to Decode():
json.NewDecoder(resp.Body).Decode(&body)
Also of note, some programming editors will highlight this mistake for you:
Here's a working demonstration, including a corrected Body structure as described at json.Marshal(struct) returns “{}”:
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
"net/http"
"time"
)
type JSON = map[string]interface{}
type JSONArray = []interface{}
func ErrFatal(err error, msg string) {
if err != nil {
log.Fatal(msg+": ", err)
}
}
func handleTestRequest(w http.ResponseWriter, req *http.Request) {
w.Write(([]byte)("{\"id\":\"yourid\"}"))
}
func launchTestServer() {
http.HandleFunc("/", handleTestRequest)
go http.ListenAndServe(":8080", nil)
time.Sleep(1 * time.Second) // allow server to get started
}
// Medium: "Don’t use Go’s default HTTP client (in production)"
var restClient = &http.Client{
Timeout: time.Second * 10,
}
func DoREST(method, url string, headers, payload JSON) *http.Response {
requestPayload, err := json.Marshal(payload)
ErrFatal(err, "json.Marshal(payload")
request, err := http.NewRequest(method, url, bytes.NewBuffer(requestPayload))
ErrFatal(err, "NewRequest "+method+" "+url)
for k, v := range headers {
request.Header.Add(k, v.(string))
}
response, err := restClient.Do(request)
ErrFatal(err, "DoRest client.Do")
return response
}
type Body struct {
Id string `json:"id"`
}
func clientDemo() {
response := DoREST("POST", "http://localhost:8080", JSON{}, JSON{})
defer response.Body.Close()
var body Body
err := json.NewDecoder(response.Body).Decode(&body)
ErrFatal(err, "Decode")
fmt.Printf("Body: %#v\n", body)
}
func main() {
launchTestServer()
for i := 0; i < 5; i++ {
clientDemo()
}
}

Extract filename from io.ReadCloser

I need to get the filename of certain file(s) that receives backend from the frontend. Backend (implemented in Go) will receive the file as io.ReadCloser. Is there way I could extract it from the io.ReadCloser?
Backend (implemented in Go) will receive the file as io.ReadCloser. Is there way I could extract it from the io.ReadCloser?
No.
Take a look at which methods an io.ReadCloser provides by running go doc io.ReadCloser and note that there isn't a method which will provide a name. So unless you know nothing more that that it is an io.ReadCloser you simply cannot do it.
package main
import (
"errors"
"fmt"
"io"
"os"
)
func fatalln(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
// hasName interface is an interface that expects types
// that implements it to have "Name() string" method.
type hasName interface {
Name() string
}
func open(name string) (io.ReadCloser, error) {
f, err := os.Open(name)
if err != nil {
return nil, err
}
// f implements io.ReadCloser interface as *os.File
// has Read and Close methods.
return f, nil
}
func main() {
// rc is of the type io.ReadCloser
rc, err := open("example.txt")
if err != nil {
fatalln(err)
}
defer rc.Close()
// Type assetion to check rc's underlying type has
// a method "Name() string".
f, ok := rc.(hasName)
if !ok {
fatalln(errors.New("type assertion failed"))
}
// Yay, type assertion succeeded. Print the name!
fmt.Println("Name:", f.Name())
}
The io.ReadCloser here is a reader for runtime reader which reads file from network as the frontend sends it to backend. You'll have to work on request itself to get that file name.
This is an assumption but in most such cases for file upload, the request is a multipart request. If you have the same situation, you can read the headers, typically Content-Disposition to identify the file type. Go native http.Request has ability to parse the details. You can try this :
formFile, handler, err := r.FormFile("file") // read file from network with key "file"
defer formFile.Close()
fileName := handler.Filename // Get file name
By defining an interface which embeds io.Reader you can require a Name() method up front:
package main
import (
"fmt"
"io"
"log"
"os"
)
type NamedReadCloser interface {
io.ReadCloser
Name() string
}
func doThings(f NamedReadCloser) error {
defer f.Close()
b, err := io.ReadAll(f)
if err != nil {
return err
}
fmt.Printf("Name: %s, Content: %s\n", f.Name(), b)
return nil
}
func main() {
f, err := os.Open("/etc/hosts")
if err != nil {
log.Fatal("Cannot open file: ", err)
}
err = doThings(f)
if err != nil {
log.Fatal("Error doing things: ", err)
}
}
This will only work if what is passed in has a name method, like an *os.File. If it does not, then what you are trying to do is not possible.
You'll have to cast it to a type with a Name method:
package main
import (
"io"
"os"
)
func open(name string) (io.ReadCloser, error) {
return os.Open(name)
}
func main() {
c, e := open("file.txt")
if e != nil {
panic(e)
}
defer c.Close()
f := c.(*os.File)
println(f.Name())
}

Trouble figuring out data race in goroutine

I started learning go recently and I've been chipping away at this for a while now, but figured it was time to ask for some specific help. I have my program requesting paginated data from an api and because there are about 160 pages of data. Seems like a good use of goroutines, except I have race conditions and I can't seem to figure out why. It's probably because I'm new to the language, but my impressions was that params for a function are passed as a copy of the data in the function calling it unless it's a pointer.
According to what I think I know this should be making copies of my data which leaves me free to change it in the main function, but I end up request some pages multiple times and other pages just once.
My main.go
package main
import (
"bufio"
"encoding/json"
"log"
"net/http"
"net/url"
"os"
"strconv"
"sync"
"github.com/joho/godotenv"
)
func main() {
err := godotenv.Load()
if err != nil {
log.Fatalln(err)
}
httpClient := &http.Client{}
baseURL := "https://api.data.gov/ed/collegescorecard/v1/schools.json"
filters := make(map[string]string)
page := 0
filters["school.degrees_awarded.predominant"] = "2,3"
filters["fields"] = "id,school.name,school.city,2018.student.size,2017.student.size,2017.earnings.3_yrs_after_completion.overall_count_over_poverty_line,2016.repayment.3_yr_repayment.overall"
filters["api_key"] = os.Getenv("API_KEY")
outFile, err := os.Create("./out.txt")
if err != nil {
log.Fatalln(err)
}
writer := bufio.NewWriter(outFile)
requestURL := getRequestURL(baseURL, filters)
response := requestData(requestURL, httpClient)
wg := sync.WaitGroup{}
for (page+1)*response.Metadata.ResultsPerPage < response.Metadata.TotalResults {
page++
filters["page"] = strconv.Itoa(page)
wg.Add(1)
go func() {
defer wg.Done()
requestURL := getRequestURL(baseURL, filters)
response := requestData(requestURL, httpClient)
_, err = writer.WriteString(response.TextOutput())
if err != nil {
log.Fatalln(err)
}
}()
}
wg.Wait()
}
func getRequestURL(baseURL string, filters map[string]string) *url.URL {
requestURL, err := url.Parse(baseURL)
if err != nil {
log.Fatalln(err)
}
query := requestURL.Query()
for key, value := range filters {
query.Set(key, value)
}
requestURL.RawQuery = query.Encode()
return requestURL
}
func requestData(url *url.URL, httpClient *http.Client) CollegeScoreCardResponseDTO {
request, _ := http.NewRequest(http.MethodGet, url.String(), nil)
resp, err := httpClient.Do(request)
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()
var parsedResponse CollegeScoreCardResponseDTO
err = json.NewDecoder(resp.Body).Decode(&parsedResponse)
if err != nil {
log.Fatalln(err)
}
return parsedResponse
}
I know another issue I will be running into is writing to the output file in the correct order, but I believe using channels to tell each routine what request finished writing could solve that. If I'm incorrect on that I would appreciate any advice on how to approach that as well.
Thanks in advance.
goroutines do not receive copies of data. When the compiler detects that a variable "escapes" the current function, it allocates that variable on the heap. In this case, filters is one such variable. When the goroutine starts, the filters it accesses is the same map as the main thread. Since you keep modifying filters in the main thread without locking, there is no guarantee of what the goroutine sees.
I suggest you keep filters read-only, create a new map in the goroutine by copying all items from the filters, and add the "page" in the goroutine. You have to be careful to pass a copy of the page as well:
go func(page int) {
flt:=make(map[string]string)
for k,v:=range filters {
flt[k]=v
}
flt["page"]=strconv.Itoa(page)
...
} (page)

cannot encode json.decoded request body

I have a server implementation. Now I am writing unit test to check it's functionalities.
I cannot prepare request, that would unmarshall on the server side well. Code below results with InvalidUnmarshallError. I don't know, how to debug it further.
Client side code:
body := PatchCatRequest{Adopted: true}
bodyBuf := &bytes.Buffer{}
err := json.NewEncoder(bodyBuf).Encode(body)
assert.NoError(t, err)
req, err := http.NewRequest("PATCH", URL+"/"+catId, bodyBuf)
recorder := httptest.NewRecorder()
handler.PatchCat(recorder, req.WithContext(ctx))
Server side code:
type PatchCatRequest struct {
Adopted bool `json:"adopted"`
}
func (h *Handler) PatchCat (rw http.ResponseWriter, req *http.Request) {
var patchRequest *PatchCatRequest
if err := json.NewDecoder(req.Body).Decode(patchRequest); err != nil {
rw.WriteHeader(http.StatusBadRequest)
logger.WithField("error", err.Error()).Error(ErrDocodeRequest.Error())
return
}
...
}
You are unmarshaling into a nil pointer, as the error message says:
package main
import (
"encoding/json"
"fmt"
)
type PatchCatRequest struct {
Adopted bool
}
func main() {
var patchRequest *PatchCatRequest // nil pointer
err := json.Unmarshal([]byte(`{"Adopted":true}`), patchRequest)
fmt.Println(err) // json: Unmarshal(nil *main.PatchCatRequest)
}
https://play.golang.org/p/vt7t5BgT3lA
Initialize the pointer before unmarshaling:
func main() {
patchRequest := new(PatchCatRequest) // non-nil pointer
err := json.Unmarshal([]byte(`{"Adopted":true}`), patchRequest)
fmt.Println(err) // <nil>
}
https://play.golang.org/p/BqliguktWmr

Save an image from url to file

Very new to Go (first simple project I'm working on).
Question: How do I get an image from URL and then save it to my computer?
Here's what I have so far:
package main
import (
"fmt"
"net/http"
"image"
"io/ioutil"
)
func main() {
url := "http://i.imgur.com/m1UIjW1.jpg"
// don't worry about errors
response, _ := http.Get(url);
defer response.Body.Close()
m, _, err := image.Decode(response.Body)
error := ioutil.WriteFile("/images/asdf.jpg", m, 0644)
}
However, when I run this code, I get cannot use m (type image.Image) as type []byte in function argument
I'm assuming I have to convert image.Image (variable m) into an undefined amount of bytes? Is that the correct way to go about this?
There is no need to decode the file. Simply copy the response body to a file you've opened. Here's the deal in the modified example:
response.Body is a stream of data, and implements the Reader interface - meaning you can sequentially call Read on it, as if it was an open file.
The file I'm opening here implements the Writer interface. This is the opposite - it's a stream you can call Write on.
io.Copy "patches" a reader and a writer, consumes the reader stream and writes its contents to a Writer.
This is one of my favorite things about go - implicit interfaces. You don't have to declare you're implementing an interface, you just have to implement it to be used in some context. This allows mixing and matching of code that doesn't need to know about other code it's interacting with.
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
)
func main() {
url := "http://i.imgur.com/m1UIjW1.jpg"
// don't worry about errors
response, e := http.Get(url)
if e != nil {
log.Fatal(e)
}
defer response.Body.Close()
//open a file for writing
file, err := os.Create("/tmp/asdf.jpg")
if err != nil {
log.Fatal(err)
}
defer file.Close()
// Use io.Copy to just dump the response body to the file. This supports huge files
_, err = io.Copy(file, response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println("Success!")
}
package main
import (
"io"
"net/http"
"os"
"fmt"
)
func main() {
img, _ := os.Create("image.jpg")
defer img.Close()
resp, _ := http.Get("http://i.imgur.com/Dz2r9lk.jpg")
defer resp.Body.Close()
b, _ := io.Copy(img, resp.Body)
fmt.Println("File size: ", b)
}
Try this:
package main
import (
"fmt"
"net/http"
"net/url"
"os"
"strings"
)
var (
fileName string
fullUrlFile string
)
func main() {
fullUrlFile = "https://i.imgur.com/m1UIjW1.jpg"
r, e := http.Get(fullUrlFile)
if e != nil {
panic(e)
}
defer r.Body.Close()
buildFileName()
// Create distination
f, e := os.Create(fileName) // "m1UIjW1.jpg"
if e != nil {
panic(e)
}
defer f.Close()
// Fill distination with content
n, e := f.ReadFrom(r.Body)
if e != nil {
panic(e)
}
fmt.Println("File size: ", n)
}
func buildFileName() {
fileUrl, e := url.Parse(fullUrlFile)
if e != nil {
panic(e)
}
path := fileUrl.Path
segments := strings.Split(path, "/")
fileName = segments[len(segments)-1]
println(fileName)
}
You can also use the ReadFrom method:
package main
import (
"net/http"
"os"
)
func main() {
r, e := http.Get("https://i.imgur.com/m1UIjW1.jpg")
if e != nil {
panic(e)
}
defer r.Body.Close()
f, e := os.Create("m1UIjW1.jpg")
if e != nil {
panic(e)
}
defer f.Close()
f.ReadFrom(r.Body)
}
https://golang.org/pkg/os#File.ReadFrom
What is the type of response.Body? You should just convert that into a []byte if it is not and write that to disk. There is no reason to use the image class unless you have some reason to treat the data as an image. Just treat the data as a series of bytes and write it to the disk.

Resources