Go-Gin read request body many times - go

I am trying to restore the context with it's data after performing validation on it's data.I need the data to keep moving as need it later on in the next function.
I am new to golang and the below code is as far I could go. any help and a better approach is much appreciated.
thanks in advance.
the validation middleware
func SignupValidator(c *gin.Context) {
// Read the Body content
// var bodyBytes []byte
// if c.Request.Body != nil {
// bodyBytes, _ = ioutil.ReadAll(c.Request.Body)
// }
var user entity.User
if err := c.ShouldBindJSON(&user); err != nil {
validate := validator.New()
if err := validate.Struct(&user); err != nil {
c.JSON(http.StatusBadRequest, gin.H{
"error": err.Error(),
})
c.Abort()
return
}
// c.Request.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))
}
// Read the Body content
var bodyBytes []byte
if c.Request.Body != nil {
bodyBytes, _ = ioutil.ReadAll(c.Request.Body)
}
fmt.Println(string(bodyBytes)) // this empty
c.Next()
}
route
auth.POST("login", gin.Logger(), validations.SignupValidator, func(ctx *gin.Context) {
ctx.JSON(200, videoController.Save(ctx))
})

You can try this.
ByteBody, _ := ioutil.ReadAll(c.Request.Body)
c.Request.Body = ioutil.NopCloser(bytes.NewBuffer(ByteBody))
You can then use ByteBody however you want without side-effects on c.Request.Body

Here is an example of reading body twice with ShouldBindBodyWith, check it:
package main
import (
"log"
"net/http"
"github.com/gin-gonic/gin"
"github.com/gin-gonic/gin/binding"
)
type ParamsOne struct {
Username string `json:"username"`
}
type ParamsTwo struct {
Username string `json:"username"`
}
func main() {
r := gin.New()
r.POST("/", func(c *gin.Context) {
var f ParamsOne
// Read ones
if err := c.ShouldBindBodyWith(&f, binding.JSON); err != nil {
log.Printf("%+v", err)
}
log.Printf("%+v", f)
var ff ParamsTwo
if err := c.ShouldBindBodyWith(&ff, binding.JSON); err != nil {
log.Printf("%+v", err)
}
log.Printf("%+v", ff)
c.IndentedJSON(http.StatusOK, f)
})
r.Run(":4000")
}
Output:
$example: ./example
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST / --> main.main.func1 (1 handlers)
[GIN-debug] Listening and serving HTTP on :4000
2020/07/05 10:47:03 {Username:somename}
2020/07/05 10:47:03 {Username:somename}

As #Philidor has shown ShouldBindBodyWith should do the trick, in my case I decided to go with something similar to #spehlivan, because of two reasons:
ShouldBindBodyWith requires that the following binds are also ShouldBindBodyWith, it means I need to change all my previous code, which uses c.Bind
You need to explicitly tell to ShouldBindBodyWith the binding type you are trying to do, JSON, Form, ProtoBuf, etc, other binds like c.Bind detects it automatically.
This is what it looks like:
var input models.SomeInput
bodyCopy := new(bytes.Buffer)
// Read the whole body
_, err := io.Copy(bodyCopy, c.Request.Body)
if err != nil {
log.Println(err)
c.JSON(http.StatusBadRequest, gin.H{"error": "Error reading API token"})
c.Abort()
return
}
bodyData := bodyCopy.Bytes()
// Replace the body with a reader that reads from the buffer
c.Request.Body = ioutil.NopCloser(bytes.NewReader(bodyData))
err = c.Bind(&input)
// Some code here...
// Replace the body with a reader that reads from the buffer
c.Request.Body = ioutil.NopCloser(bytes.NewReader(bodyData))
Pay attention I replaced the c.Request.Body twice, for the bind in that code snippet and then at the end, for the next bind, in another place of my code (this snippet is from a middleware, the next bind is called from the controller).
In my case I needed to do this because the API Token is sent in the request body, which I don't recommend, it should be sent in the request header.

Related

Golang WebHook using chi router

I am creating a service in Go with chi router and I need to implement a WebHook for one of my endpoints. I am using this library to create my webHook. However I am having problems when I try to receive it and decode it. In their sample code they used Gin framework and I have to use chi. My receivedSignature comes out as nil and I get an error: interface conversion: interface {} is nil, not string .
Could anyone please help me with a work around on this?
Here is my send function:
data := []int{1, 2, 3, 4}
// String sent in the GoHook that helps identify actions to take with data
resource := "int-list-example"
// Secret string that should be common to sender and receiver
// in order to validate the GoHook signature
saltSecret := "0014716e-392c-4120-609e-555e295faff5"
hook := &gohooks.GoHook{}
hook.Create(data, resource, saltSecret)
// Will return *http.Response and error
resp, err := hook.Send("http://localhost:3001/")
if err != nil {
fmt.Println("error: ", err)
}
fmt.Println("resp: ", resp)
And here is my receiving logic:
func main(){
r := chi.NewRouter()
r.Use(middleware.Logger)
r.Post("/", receiveWebHook)
http.ListenAndServe(":3001", r)
}
type MyWebhook struct {
Resource string `json:"resource"`
Data []int `json:"data"`
}
func receiveWebHook(w http.ResponseWriter, r *http.Request){
var request MyWebhook
err := json.NewDecoder(r.Body).Decode(&request)
if err !=nil{
fmt.Println("err: ", err)
}
// Shared secret with sender
saltSecret := "0014716e-392c-4120-609e-555e295faff5"
receivedSignature := r.Context().Value(gohooks.DefaultSignatureHeader).(string)
// Verify validity of GoHook
isValid := gohooks.IsGoHookValid(request, receivedSignature, saltSecret)
// Decide what to do if GoHook is valid or not.
if !isValid {
fmt.Println("Not valid, receivedSignature: ", receivedSignature)
}else{
fmt.Println("Valid, receivedSignature: ", receivedSignature)
}
w.WriteHeader(200)
}
The signature will be in a header. So you should also read it from the requests' header.
sig := r.Header.Get(gohooks.DefaultSignatureHeader)

Golang bufio from websocket breaking after first read

I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}

Google Sheets API: golang BatchUpdateValuesRequest

I'm trying to follow the Google Sheets API quickstart here:
https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets.values/batchUpdate
(scroll down to "Examples" then click "GO")
This is how I tried to update a spreadsheet:
package main
// BEFORE RUNNING:
// ---------------
// 1. If not already done, enable the Google Sheets API
// and check the quota for your project at
// https://console.developers.google.com/apis/api/sheets
// 2. Install and update the Go dependencies by running `go get -u` in the
// project directory.
import (
"errors"
"fmt"
"log"
"net/http"
"golang.org/x/net/context"
"google.golang.org/api/sheets/v4"
)
func main() {
ctx := context.Background()
c, err := getClient(ctx)
if err != nil {
log.Fatal(err)
}
sheetsService, err := sheets.New(c)
if err != nil {
log.Fatal(err)
}
// The ID of the spreadsheet to update.
spreadsheetId := "1diQ943LGMDNkbCRGG4VqgKZdzyanCtT--V8o7r6kCR0"
var jsonPayloadVar []string
monthVar := "Apr"
thisCellVar := "A26"
thisLinkVar := "http://test.url"
jsonRackNumberVar := "\"RACKNUM01\""
jsonPayloadVar = append(jsonPayloadVar, fmt.Sprintf("(\"range\": \"%v!%v\", \"values\": [[\"%v,%v)\"]]),", monthVar, thisCellVar, thisLinkVar, jsonRackNumberVar))
rb := &sheets.BatchUpdateValuesRequest{"ValueInputOption": "USER_ENTERED", "data": jsonPayloadVar}
resp, err := sheetsService.Spreadsheets.Values.BatchUpdate(spreadsheetId, rb).Context(ctx).Do()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%#v\n", resp)
}
func getClient(ctx context.Context) (*http.Client, error) {
// https://developers.google.com/sheets/quickstart/go#step_3_set_up_the_sample
//
// Authorize using the following scopes:
// sheets.DriveScope
// sheets.DriveFileScope
sheets.SpreadsheetsScope
return nil, errors.New("not implemented")
}
Output:
hello.go:43: invalid field name "ValueInputOption" in struct initializer
hello.go:43: invalid field name "data" in struct initializer
hello.go:58: sheets.SpreadsheetsScope evaluated but not used
There are 2 things that aren't working:
It's not obvious how to enter the fields into variable rb
I need to use sheets.SpreadsheetsScope
Can anyone provide a working example that does a BatchUpdate?
References:
This article shows how to do an update that is not a BatchUpdate: Golang google sheets API V4 - Write/Update example?
Google's API reference - see the ValueInputOption section starting at line 1437: https://github.com/google/google-api-go-client/blob/master/sheets/v4/sheets-gen.go
This article shows how to do a BatchUpdate in Java: Write data to Google Sheet using Google Sheet API V4 - Java Sample Code
How about the following sample script? This is a simple sample script for updating sheet on Spreadsheet. So if you want to do various update, please modify it. The detail of parameters for spreadsheets.values.batchUpdate is here.
Flow :
At first, in ordet to use the link in your question, please use Go Quickstart. In my sample script, the script was made using the Quickstart.
The flow to use this sample script is as follows.
For Go Quickstart, please do Step 1 and Step 2.
Please put client_secret.json to the same directory with my sample script.
Copy and paste my sample script, and create it as new script file.
Run the script.
When Go to the following link in your browser then type the authorization code: is shown on your terminal, please copy the URL and paste to your browser. And then, please authorize and get code.
Put the code to the terminal.
When Done. is displayed, it means that the update of spreadsheet is done.
Request body :
For Spreadsheets.Values.BatchUpdate, BatchUpdateValuesRequest is required as one of parameters. In this case, the range, values and so on that you want to update are included in BatchUpdateValuesRequest. The detail information of this BatchUpdateValuesRequest can be seen at godoc. When it sees BatchUpdateValuesRequest, Data []*ValueRange can be seen. Here, please be carefull that Data is []*ValueRange. Also ValueRange can be seen at godoc. You can see MajorDimension, Range and Values in ValueRange.
When above infomation is reflected to the script, the script can be modified as follows.
Sample script :
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"golang.org/x/net/context"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"google.golang.org/api/sheets/v4"
)
// getClient uses a Context and Config to retrieve a Token
// then generate a Client. It returns the generated Client.
func getClient(ctx context.Context, config *oauth2.Config) *http.Client {
cacheFile := "./go-quickstart.json"
tok, err := tokenFromFile(cacheFile)
if err != nil {
tok = getTokenFromWeb(config)
saveToken(cacheFile, tok)
}
return config.Client(ctx, tok)
}
// getTokenFromWeb uses Config to request a Token.
// It returns the retrieved Token.
func getTokenFromWeb(config *oauth2.Config) *oauth2.Token {
authURL := config.AuthCodeURL("state-token", oauth2.AccessTypeOffline)
fmt.Printf("Go to the following link in your browser then type the "+
"authorization code: \n%v\n", authURL)
var code string
if _, err := fmt.Scan(&code); err != nil {
log.Fatalf("Unable to read authorization code %v", err)
}
tok, err := config.Exchange(oauth2.NoContext, code)
if err != nil {
log.Fatalf("Unable to retrieve token from web %v", err)
}
return tok
}
// tokenFromFile retrieves a Token from a given file path.
// It returns the retrieved Token and any read error encountered.
func tokenFromFile(file string) (*oauth2.Token, error) {
f, err := os.Open(file)
if err != nil {
return nil, err
}
t := &oauth2.Token{}
err = json.NewDecoder(f).Decode(t)
defer f.Close()
return t, err
}
func saveToken(file string, token *oauth2.Token) {
fmt.Printf("Saving credential file to: %s\n", file)
f, err := os.Create(file)
if err != nil {
log.Fatalf("Unable to cache oauth token: %v", err)
}
defer f.Close()
json.NewEncoder(f).Encode(token)
}
type body struct {
Data struct {
Range string `json:"range"`
Values [][]string `json:"values"`
} `json:"data"`
ValueInputOption string `json:"valueInputOption"`
}
func main() {
ctx := context.Background()
b, err := ioutil.ReadFile("client_secret.json")
if err != nil {
log.Fatalf("Unable to read client secret file: %v", err)
}
config, err := google.ConfigFromJSON(b, "https://www.googleapis.com/auth/spreadsheets")
if err != nil {
log.Fatalf("Unable to parse client secret file to config: %v", err)
}
client := getClient(ctx, config)
sheetsService, err := sheets.New(client)
if err != nil {
log.Fatalf("Unable to retrieve Sheets Client %v", err)
}
spreadsheetId := "### spreadsheet ID ###"
rangeData := "sheet1!A1:B3"
values := [][]interface{}{{"sample_A1", "sample_B1"}, {"sample_A2", "sample_B2"}, {"sample_A3", "sample_A3"}}
rb := &sheets.BatchUpdateValuesRequest{
ValueInputOption: "USER_ENTERED",
}
rb.Data = append(rb.Data, &sheets.ValueRange{
Range: rangeData,
Values: values,
})
_, err = sheetsService.Spreadsheets.Values.BatchUpdate(spreadsheetId, rb).Context(ctx).Do()
if err != nil {
log.Fatal(err)
}
fmt.Println("Done.")
}
Result :
References :
The detail infomation of spreadsheets.values.batchUpdate is here.
The detail infomation of Go Quickstart is here.
The detail infomation of BatchUpdateValuesRequest is here.
The detail infomation of ValueRange is here.
If I misunderstand your question, I'm sorry.

How to make function work with different input types?

I have this simple generic Request struct to make get requests in my app:
package api
import (
"net/http"
"time"
"log"
"app/errors"
)
type Request struct {
Url string
}
func (request *Request) Run(responseObject *AppStatusInfo) *errors.Error {
req, requestErr := http.NewRequest(http.MethodGet, request.Url, nil)
req.Header.Set("Content-Type", "application/json")
timeout := time.Duration(5 * time.Second)
client := &http.Client{
Timeout: timeout,
}
resp, requestErr := client.Do(req)
if requestErr != nil {
return &errors.UnknownError
}
decodeError := DecodeJsonRequestBody(resp, &responseObject)
if (decodeError != nil) {
return &errors.UnknownError
}
defer resp.Body.Close()
return nil
}
Here responseObject has pointer of type AppStatusInfo which is a struct with some fields.
I run it like this to get app status information and put it inside appStatusInfo object:
var appStatusInfo AppStatusInfo
req := Request{
Url:config.Config.ApiUrl,
}
req.Run(&appStatusInfo)
So, this code runs fine.
But, when I want to generalize Request to accept other types of responses, like UserProducts, I don't know how to do it without replacing responseObject *AppStatusInfo with responseObject interface{}, then casting it with responseObject.(UserProducts) which I think can be improved.
So, as soon as there are no generics, how do I make Request.Run() accept different types and return respective objects?
Assuming that DecodeJsonRequestBody passes the second argument to json.Unmarshal or json.Decoder.Decode, then write it like this. I show the changed lines only:
func (request *Request) Run(responseObject interface{}) *errors.Error {
...
resp, requestErr := client.Do(req)
if requestErr != nil {
return &errors.UnknownError
}
defer resp.Body.Close() // defer close before doing anything else
...
decodeError := DecodeJsonRequestBody(resp, responseObject) // don't take address of responseObject
...
}
You can call it like this:
var up UserProducts
err = r.Run(&up)
var asi AppStatusInfo
err = r.Run(&asi)
Type assertions and type conversions are not required.

How to limit download speed with Go?

I'm currently developing a download server in Go. I need to limit the download speed of users to 100KB/s.
This was my code:
func serveFile(w http.ResponseWriter, r *http.Request) {
fileID := r.URL.Query().Get("fileID")
if len(fileID) != 0 {
w.Header().Set("Content-Disposition", "attachment; filename=filename.txt")
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
w.Header().Set("Content-Length", r.Header.Get("Content-Length"))
file, err := os.Open(fmt.Sprintf("../../bin/files/test.txt"))
defer file.Close()
if err != nil {
http.NotFound(w, r)
return
}
io.Copy(w, file)
} else {
io.WriteString(w, "Invalid request.")
}
}
Then I found a package on github and my code became the following:
func serveFile(w http.ResponseWriter, r *http.Request) {
fileID := r.URL.Query().Get("fileID")
if len(fileID) != 0 {
w.Header().Set("Content-Disposition", "attachment; filename=Wiki.png")
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
w.Header().Set("Content-Length", r.Header.Get("Content-Length"))
file, err := os.Open(fmt.Sprintf("../../bin/files/test.txt"))
defer file.Close()
if err != nil {
http.NotFound(w, r)
return
}
bucket := ratelimit.NewBucketWithRate(100*1024, 100*1024)
reader := bufio.NewReader(file)
io.Copy(w, ratelimit.Reader(reader, bucket))
} else {
io.WriteString(w, "Invalid request.")
}
}
But I'm getting this error:
Corrupted Content Error
The page you are trying to view cannot be shown because an error in
the data transmission was detected.
Here's my code on the Go playground: http://play.golang.org/p/ulgXQl4eQO
Rather than mucking around with getting the correct the content type and length headers yourself it'd probably be much better to use http.ServeContent which will do that for you (as well as support "If-Modified-Since", range requests, etc. If you can supply an "ETag" header it can also handle "If-Range" and "If-None-Match" requests as well).
As mentioned previously, it's often preferable to limit on the write side but it's awkward to wrap an http.ResponseWriter since various http functions also check for optional interfaces such as http.Flusher and http.Hijacker. It's much easier to wrap the io.ReadSeeker that ServeContent needs.
For example, something like this perhaps:
func pathFromID(fileID string) string {
// replace with whatever logic you need
return "../../bin/files/test.txt"
}
// or more verbosely you could call this a "limitedReadSeeker"
type lrs struct {
io.ReadSeeker
// This reader must not buffer but just do something simple
// while passing through Read calls to the ReadSeeker
r io.Reader
}
func (r lrs) Read(p []byte) (int, error) {
return r.r.Read(p)
}
func newLRS(r io.ReadSeeker, bucket *ratelimit.Bucket) io.ReadSeeker {
// Here we know/expect that a ratelimit.Reader does nothing
// to the Read calls other than add delays so it won't break
// any io.Seeker calls.
return lrs{r, ratelimit.Reader(r, bucket)}
}
func serveFile(w http.ResponseWriter, req *http.Request) {
fileID := req.URL.Query().Get("fileID")
if len(fileID) == 0 {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
path := pathFromID(fileID)
file, err := os.Open(path)
if err != nil {
http.NotFound(w, req)
return
}
defer file.Close()
fi, err := file.Stat()
if err != nil {
http.Error(w, "blah", 500) // XXX fixme
return
}
const (
rate = 100 << 10
capacity = 100 << 10
)
// Normally we'd prefer to limit the writer but it's awkward to wrap
// an http.ResponseWriter since it may optionally also implement
// http.Flusher, or http.Hijacker.
bucket := ratelimit.NewBucketWithRate(rate, capacity)
lr := newLRS(file, bucket)
http.ServeContent(w, req, path, fi.ModTime(), lr)
}
I'm not seeing the error, but I did notice some issues with the code. For this:
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
You should use the mime package's:
func TypeByExtension(ext string) string
To determine the content type. (if you end up with the empty string default to application/octet-stream)
For:
w.Header().Set("Content-Length", r.Header.Get("Content-Length"))
You need to get the content length from the file itself. By using the request content length, for a GET this basically ends up as a no-op, but for a POST you're sending back the wrong length, which might explain the error you're seeing. After you open the file, do this:
fi, err := file.Stat()
if err != nil {
http.Error(w, err.Error(), 500)
return
}
w.Header().Set("Content-Length", fmt.Sprint(fi.Size()))
One final thing, when you open the file, if there's an error, you don't need to close the file handle. Do it like this instead:
file, err := os.Open(...)
if err != nil {
http.NotFound(w, r)
return
}
defer file.Close()

Resources