Transform http.Response into array of bytes - go

I am trying to develop a tcp proxy, in this tcp proxy I will have to manipulate both http and tcp requests.
At the moment for the incoming request I detect if it is an http or tcp request, if it is an http then I parse it into an http.Request:
func (s *TcpProxy) OnMessage(c *connection.Connection, ctx interface{}, data []byte) interface{} {
reader := bytes.NewReader(data)
newReader := bufio.NewReader(reader)
req, err := http.ReadRequest(newReader)
// This is an http request
}
Now I manipulate the request conveniently since I can use the methods exposed from that interface, then eventually I will send the response back from my proxy to the service that received the inbound request.
func (s *TcpProxy) OnMessage(c *connection.Connection, ctx interface{}, data []byte) interface{} {
reader := bytes.NewReader(data)
newReader := bufio.NewReader(reader)
req, err := http.ReadRequest(newReader)
// Manipulate http request
// ...
// Proxy the request
proxyReq, err := http.NewRequest(req.Method, proxyUrl, req.Body)
// Capture the duration while making a request to the destination service.
res, err := httpClient.Do(proxyReq)
buf := res.ToBuffer() // <= How can I achieve this
c.Send(buf)
c.Close()
return nil
}
However I cannot find a way to convert back the response into an array of bytes or string, am I missing something?

An http.Request object has a Write method:
func (r *Request) Write(w io.Writer) error
Write writes an HTTP/1.1 request, which is the header and body, in wire format.
You can use this to write the bytes into a buffer object. For example:
package main
import (
"bytes"
"fmt"
"net/http"
)
func main() {
var buf bytes.Buffer
req, err := http.NewRequest("GET", "http://google.com", nil)
if err != nil {
panic(err)
}
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
if err := res.Write(&buf); err != nil {
panic(err)
}
// ...do whatever you want with the buffer here...
fmt.Println(buf.String())
}
A Buffer object has a Bytes method that will return a byte array, if that's what you want.

Related

Using Channels in Go to receive Responses and Write to SQL Concurrently

I am working with Go to implement a pipeline of JSON data from an external API, process the message and then send to a SQL database.
I am trying to concurrently run API requests, then after I return a response, I'd like to send it to be inserted into the DB via another goroutine via load().
In my below code, sometimes I'll receive my log.Printf() in the load() func, other times I won't. Which indicates that I'm likely closing a channel or not properly setting up the communication.
The pattern I am attempting is something like this:
package main
import (
"encoding/json"
"io/ioutil"
"log"
"net/http"
"time"
)
type Request struct {
url string
}
type Response struct {
status int
args Args `json:"args"`
headers Headers `json:"headers"`
origin string `json:"origin"`
url string `json:"url"`
}
type Args struct {
}
type Headers struct {
accept string `json:"Accept"`
}
func main() {
start := time.Now()
numRequests := 5
responses := make(chan Response, 5)
defer close(responses)
for i := 0; i < numRequests; i++ {
req := Request{url: "https://httpbin.org/get"}
go func(req *Request) {
resp, err := extract(req)
if err != nil {
log.Fatal("Error extracting data from API")
return
}
// Send response to channel
responses <- resp
}(&req)
// Perform go routine to load data
go load(responses)
}
log.Println("Execution time: ", time.Since(start))
}
func extract(req *Request) (r Response, err error) {
var resp Response
request, err := http.NewRequest("GET", req.url, nil)
if err != nil {
return resp, err
}
request.Header = http.Header{
"accept": {"application/json"},
}
response, err := http.DefaultClient.Do(request)
defer response.Body.Close()
if err != nil {
log.Fatal("Error")
return resp, err
}
// Read response data
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal("Error")
return resp, err
}
json.Unmarshal(body, &resp)
resp.status = response.StatusCode
return resp, nil
}
type Record struct {
origin string
url string
}
func load(ch chan Response) {
// Read response from channel
resp := <-ch
// Process the response data
records := process(resp)
log.Printf("%+v\n", records)
// Load data to db stuff here
}
func process(resp Response) (record Record) {
// Process the response struct as needed to get a record of data to insert to DB
return record
}
The program has no protection against completion before the work is done. So sometimes the program terminates before the goroutine can finish.
To prevent that, use a WaitGroup:
wg:=sync.WaitGroup{}
for i := 0; i < numRequests; i++ {
...
wg.Add(1)
go func() {
defer wg.Done()
load(responses)
}()
}
wg.Wait()

Golang bufio from websocket breaking after first read

I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}

Fast Resend Response in golang lambda

I have golang lambda that prepares ES request, send it to external system and returns its response. Currently, I haven't found a better approach than an unmarshalling response to interface{}.
func HandleRequest(ctx context.Context, searchRequest SearchRequest) (interface{}, error) {
// ... some data preparation and client initalisation
resp, err := ctxhttp.Post(ctx, &client, url, "application/json", buffer)
if err != nil {
return "", err
}
var k interface{}
all, err := ioutil.ReadAll(resp.Body)
err = json.Unmarshal(all, &k)
return k, err
}
I'm not sure it is the fastest and the most performant way to forward response due to that extra ReadAll and Unmarshall. Is there a more performant approach?
I looked at events.APIGatewayProxyResponse{}, but body in it - string and same manipulations are needed
You can handle the response in many different way
If lambda implements additional search response handling, it might be worth defining the response data type contract with respective marshaling/unmarshaling and additional handling logic.
If lambda functionality is to only proxy response from ES search, you may just passed search response payload ([]byte) directly to APIGatewayProxyResponse.Body as []byte and may need to base64 if the payload has binary data.
Code:
func handleRequest(ctx context.Context, apiRequest events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
request, err := newSearchRequest(apiRequest)
if err != nil {
return handleError(err)
}
responseBody, err := proxySearch(ctx, request)
if err != nil {
return handleError(err)
}
return events.APIGatewayProxyResponse{
StatusCode: http.StatusOK,
Body: string(responseBody),
}, nil
}
func proxySearch(ctx context.Context, searchRequest SearchRequest) ([]byte, error) {
// ... some data preparation and client initalisation
resp, err := ctxhttp.Post(ctx, &client, url, "application/json", buffer)
if err != nil {
return nil, err
}
responseBody, err := ioutil.ReadAll(resp.Body)
return responseBody, err
}

Golang net/http/transport proxy CONNECT method header supporting

Golang's package net/http/transport can automatic setup Proxy-Authorization header in
func (t *Transport) dialConn(ctx context.Context, cm connectMethod) (*persistConn, error)
like
proxyURL, _ := url.Parse("http://username:password#example.com")
client := http.Client{Transport: &http.Transport{Proxy:http.ProxyURL(proxyURL)}}
But I need submit X-Header to proxy server. How Can I custom transport CONNECT method request header?
net/http/transport
how about this:
// ...
request, err := http.NewRequest("GET", "https://www.google.com", nil)
if err != nil {
// do something
}
// add header here.
request.Header.Add("X-Header", "xxx")
response, err := client.Do(request)
if err != nil {
// do something
}
// ...
http.Transport has a function which allows you to set some additional headers which will be sent during CONNECT.
Example:
var client http.Client
client.Transport = &http.Transport{
Proxy: http.ProxyURL(myProxy),
GetProxyConnectHeader: func(ctx context.Context, proxyURL *url.URL, target string) (http.Header, error) {
return http.Header{"My-Custom-Header": []string{"My-Custom-Value"}}, nil
},
}

Using protobuf with golang and handling []byte HTTP response body

I am using the Golang protobuf package and try to write some tests to ensure my API works properly.
I construct an Object on the server-side with a generated .pb.go file.
And return it with
data, err := proto.Marshal(p)
fmt.Fprint(w, data)
And in my test I do
func TestGetProduct(t *testing.T) {
log.Println("Starting server")
go startAPITestServer()
time.Sleep(0 * time.Second)
log.Println("Server started")
//rq, err := http.NewRequest("GET", "localhost:8181/product/1", nil)
client := &http.Client{}
log.Println("Starting Request")
resp, err := client.Get("http://localhost:8181/product/1")
log.Println("Finished Request")
if err != nil {
t.Log(err)
}
defer resp.Body.Close()
log.Println("Reading Request")
data, err := ioutil.ReadAll(resp.Body)
log.Println("Reading finished")
if err != nil {
t.Log(err)
}
log.Println("HTTP Resp", data)
p := &Product{}
proto.UnmarshalText(string(data), p)
proto.Unmarshal(data, p2)
}
The Problem is that the HTTP Request is correct and displays the []byte correctly, but if I do ioutil.ReadAll it interprets the HTTP Response as a string and converts it to a []byte.
For example the response is
[12 3 2 14 41]
Then ioutil.ReadAll interprets this as a string and not as a []byte.
The problem was: I tried to write binary data to the output stream with fmt.Fprint missing the important fact, that the fmt package converts (everything?) input to a "read-able" format (ie strings).
The correct way of writting data into the output of your HTTP Response is using the responsewriter directly like this:
k, err := w.Write(data)

Resources