Using protobuf with golang and handling []byte HTTP response body - go

I am using the Golang protobuf package and try to write some tests to ensure my API works properly.
I construct an Object on the server-side with a generated .pb.go file.
And return it with
data, err := proto.Marshal(p)
fmt.Fprint(w, data)
And in my test I do
func TestGetProduct(t *testing.T) {
log.Println("Starting server")
go startAPITestServer()
time.Sleep(0 * time.Second)
log.Println("Server started")
//rq, err := http.NewRequest("GET", "localhost:8181/product/1", nil)
client := &http.Client{}
log.Println("Starting Request")
resp, err := client.Get("http://localhost:8181/product/1")
log.Println("Finished Request")
if err != nil {
t.Log(err)
}
defer resp.Body.Close()
log.Println("Reading Request")
data, err := ioutil.ReadAll(resp.Body)
log.Println("Reading finished")
if err != nil {
t.Log(err)
}
log.Println("HTTP Resp", data)
p := &Product{}
proto.UnmarshalText(string(data), p)
proto.Unmarshal(data, p2)
}
The Problem is that the HTTP Request is correct and displays the []byte correctly, but if I do ioutil.ReadAll it interprets the HTTP Response as a string and converts it to a []byte.
For example the response is
[12 3 2 14 41]
Then ioutil.ReadAll interprets this as a string and not as a []byte.

The problem was: I tried to write binary data to the output stream with fmt.Fprint missing the important fact, that the fmt package converts (everything?) input to a "read-able" format (ie strings).
The correct way of writting data into the output of your HTTP Response is using the responsewriter directly like this:
k, err := w.Write(data)

Related

Golang bufio from websocket breaking after first read

I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}

Converting string from HTTP request to data struct in Go

I've got an HTTP Post method, which successfully posts data to an external third party API and returns a response.
I then need data returned from this response to post to my database.
The response contains a few piece of data, but I only need the 'access_token' and 'refresh_token' from it.
As a result, what I'm attempting to do is convert the response from a string into individual components in a new data struct I've created - to then pass to my database.
However, the data is showing as blank, despite it successfully being written to my browser. I'm obviously doing something fundamentally wrong, but not sure what..
Here's my code:
type data struct {
Access_token string `json:"access_token"`
Refresh_token string `json:"refresh_token"`
}
func Fetch(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
q := url.Values{}
q.Add("grant_type", "authorization_code")
q.Add("client_id", os.Getenv("ID"))
q.Add("client_secret", os.Getenv("SECRET"))
q.Add("redirect_uri", "https://callback-url.com")
q.Add("query", r.URL.Query().Get("query"))
req, err := http.NewRequest("POST", "https://auth.truelayer-sandbox.com/connect/token", strings.NewReader(q.Encode()))
if err != nil {
log.Print(err)
fmt.Println("Error was not equal to nil at first stage.")
os.Exit(1)
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to server")
os.Exit(1)
}
respBody, _ := ioutil.ReadAll(resp.Body)
d := data{}
err = json.NewDecoder(resp.Body).Decode(&d)
if err != nil {
fmt.Println(err)
}
fmt.Println(d.Access_token)
fmt.Println(d.Refresh_token)
w.WriteHeader(resp.StatusCode)
w.Write(respBody)
}
With ioutil.ReadAll you read the body, already. The second time you pass to NewDecoder(resp.Body) the stream was consumed.
You can use instead json.Unmarshal(respBody, &d).
One more advice, don't ignore the error on ioutil.ReadAll

Transform http.Response into array of bytes

I am trying to develop a tcp proxy, in this tcp proxy I will have to manipulate both http and tcp requests.
At the moment for the incoming request I detect if it is an http or tcp request, if it is an http then I parse it into an http.Request:
func (s *TcpProxy) OnMessage(c *connection.Connection, ctx interface{}, data []byte) interface{} {
reader := bytes.NewReader(data)
newReader := bufio.NewReader(reader)
req, err := http.ReadRequest(newReader)
// This is an http request
}
Now I manipulate the request conveniently since I can use the methods exposed from that interface, then eventually I will send the response back from my proxy to the service that received the inbound request.
func (s *TcpProxy) OnMessage(c *connection.Connection, ctx interface{}, data []byte) interface{} {
reader := bytes.NewReader(data)
newReader := bufio.NewReader(reader)
req, err := http.ReadRequest(newReader)
// Manipulate http request
// ...
// Proxy the request
proxyReq, err := http.NewRequest(req.Method, proxyUrl, req.Body)
// Capture the duration while making a request to the destination service.
res, err := httpClient.Do(proxyReq)
buf := res.ToBuffer() // <= How can I achieve this
c.Send(buf)
c.Close()
return nil
}
However I cannot find a way to convert back the response into an array of bytes or string, am I missing something?
An http.Request object has a Write method:
func (r *Request) Write(w io.Writer) error
Write writes an HTTP/1.1 request, which is the header and body, in wire format.
You can use this to write the bytes into a buffer object. For example:
package main
import (
"bytes"
"fmt"
"net/http"
)
func main() {
var buf bytes.Buffer
req, err := http.NewRequest("GET", "http://google.com", nil)
if err != nil {
panic(err)
}
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
if err := res.Write(&buf); err != nil {
panic(err)
}
// ...do whatever you want with the buffer here...
fmt.Println(buf.String())
}
A Buffer object has a Bytes method that will return a byte array, if that's what you want.

Returning data back to client from Go HTTP request

I've written a simple Fetch Go function which calls an API, and generates a response.
When called, it successfully logs the data to the console which is pulled from the API.
What I want to do though is take the final 'respBody' variable generated from reading the response body, and then return it back to my frontend client - but I can't figure out how.
All the examples just use Println, and I've searched the docs but can't find anything.
Can anyone tell me how to change my code so I can return the respBody back to the client?
Here's my function:
func Fetch(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest", nil)
if err != nil {
log.Print(err)
os.Exit(1)
}
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to server")
os.Exit(1)
}
respBody, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(respBody)) // This is the final bit where I want to send this back to the client.
}
Your function is a HandlerFunc, which contains the ResponseWriter interface, in your case it's w.
So, you can write data using http.ResponseWriter:
func Fetch(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest", nil)
if err != nil {
log.Print(err)
os.Exit(1)
}
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to server")
os.Exit(1)
}
respBody, _ := ioutil.ReadAll(resp.Body)
// Here:
w.WriteHeader(resp.StatusCode)
w.Write(respBody)
}
You can use use io.Copy(w, resp.Body) instead, remember to close body using defer resp.Body.Close().
You can simply copy the contents of the response body to the response writer:
io.Copy(w,resp.Body)
Since you can only read the body once, the solution above will not allow you to get the body. If you also want to log it, or process it somehow, you can read it and then write it to the response writer.
respBody, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(respBody))
w.Write(respBody)

Reading http.Response Body stream

I'm sending a http request and getting a response in text/xml format. I don't need to parse the XML, just trying to return it as a string for testing. But it seems pretty slow.
The XML response is 278kb, but it can take about 1.7 but sometimes spike to 4.5seconds just to read the response body. I can rerun this over and over and it can vary wildly. I'm not sure what's causing it as i'm new to Go.
I feel like the ioutil.ReadAll() function is letting me down. But i've also tried bufio.NewReader and reader.ReadBytes. All seem to be just as slow.
this is what my function looks like:
client := &http.Client{}
req, err := http.NewRequest("POST", url, strings.NewReader(requestBody))
if err != nil {
fmt.Println("New Request Error", err)
}
// Set Headers
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("Content-Encoding", "gzip")
req.Header.Set("Accept-Encoding", "gzip, deflate")
defer req.Body.Close()
// Get Response
startRes := time.Now()
response, err := client.Do(req)
if response != nil {
defer response.Body.Close()
}
if err != nil {
fmt.Println("POST Error", err)
}
endRes := time.Now()
fmt.Println("Response took", endRes.Sub(startRes))
// Read Response Body
startRead := time.Now()
body, readErr := ioutil.ReadAll(response.Body)
if readErr != nil {
fmt.Println("body error:", readErr)
return nil, readErr
}
endRead := time.Now()
fmt.Println("Read response took:", endRead.Sub(startRead))
The output of these println's look like this:
> go run main.go
Response took 5.230523s
Read response took: 1.7551755s
Is there a way to make this code more efficient? I'm assuming the 1.7s read time for a 278kb file is due to network lag, right?
I recommend allocating a byte.Buffer and reading the body yourself. The code at ioutil.ReadAll creates a byte.Buffer at an initial capacity of Byte.MinRead (512 bytes).
At 278KB, Buffer needs to expand, reallocate and recopy the returned byte array 10 times which is most processing time is being spent (for things you can control).
Untested code to try below:
buf := bytes.NewBuffer(make([]byte, 0, response.ContentLength))
_, readErr = buf.ReadFrom(response.Body)
body := buf.Bytes()

Resources