How to get JSON response body from http.Get - go

I'm trying get body from response which is json and print this json or be able to put him to array. I find this post on stack: How to get JSON response from http.Get .
There is code:
var myClient = &http.Client{Timeout: 10 * time.Second}
func getJson(url string, target interface{}) error {
r, err := myClient.Get(url)
if err != nil {
return err
}
defer r.Body.Close()
return json.NewDecoder(r.Body).Decode(target)
}
but I don't undestand why there is "Decode(target)" and "target interface{}". What does it do? Why when I try just print json.NewDecoder(r.Body) there is nothing meaningful.

The function getJson in the question decodes the JSON response body for the specified URL to the value pointed to by target. The expression json.NewDecoder(r.Body).Decode(target) does the following:
create a decoder that reads from the response body
use the decoder to unmarshal the JSON to the value pointed to by target.
returns error or nil.
Here's an example use of the function. The program fetches and prints information about this answer.
func main() {
url := "https://api.stackexchange.com/2.2/answers/67655454?site=stackoverflow"
// Answers is a type the represents the JSON for a SO answer.
var response Answers
// Pass the address of the response to getJson. The getJson passes
// that address on to the JSON decoder.
err := getJson(url, &response)
// Check for errors. Adjust error handling to match the needs of your
// application.
if err != nil {
log.Fatal(err)
}
// Print the answer.
for _, item := range response.Items {
fmt.Printf("%#v", item)
}
}
type Owner struct {
Reputation int `json:"reputation"`
UserID int `json:"user_id"`
UserType string `json:"user_type"`
AcceptRate int `json:"accept_rate"`
ProfileImage string `json:"profile_image"`
DisplayName string `json:"display_name"`
Link string `json:"link"`
}
type Item struct {
Owner *Owner `json:"owner"`
IsAccepted bool `json:"is_accepted"`
Score int `json:"score"`
LastActivityDate int `json:"last_activity_date"`
LastEditDate int `json:"last_edit_date"`
CreationDate int `json:"creation_date"`
AnswerID int `json:"answer_id"`
QuestionID int `json:"question_id"`
ContentLicense string `json:"content_license"`
}
type Answers struct {
Items []*Item `json:"items"`
HasMore bool `json:"has_more"`
QuotaMax int `json:"quota_max"`
QuotaRemaining int `json:"quota_remaining"`
}
Link to complete code including code from question.

Here is an example:
package main
import (
"encoding/json"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
defer r.Body.Close()
var s struct { Name string }
json.NewDecoder(r.Body).Decode(&s)
println(s.Name == "GitHub")
}
https://golang.org/pkg/encoding/json#Decoder.Decode

Related

Changing value of field in a go struct

I'm making an http request in golang to an external api. It gives a general response of {"error":[]string, "result":changing interface{}}. depending on the function that is making the request, the Result field changes. Since I know the structure of the Result field for each function I run, I want to be able to change the value of Result before unmarshalling to json. I've tried to do this with the following code:
func GetAssets(output *Resp, resultType interface{}) error {
return publicRequest("/Assets", output, resultType)
}
func publicRequest(endPoint string, output *Resp, resultType interface{}) error {
url := Rest_url + Pub_rest_url + endPoint //"https://api.kraken.com/0/public/Assets in this case
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
output.Result = resultType
return json.NewDecoder(resp.Body).Decode(&output)
}
Here is how it's being ran in main
type Resp struct {
Error []string `json:"error"`
Result interface{} `json:"result"`
}
type AssetInfo struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
Display int `json:"display_decimals"`
}
func main() {
var result map[string]AssetInfo
jsonData := Resp{}
rest_api_client.GetAssets(&jsonData, result)
fmt.Println(jsonData)
}
The issue is that it doesn't unmarshal correctly. A map is created for each asset, but the data contained inside of each asset is also being stored inside of a map. I'm not sure if I explained this well, but here is the current response after unmarshalling to understand what I mean.
Here is the data type of Resp.Result: map[string]interface {}
{[] map[1INCH:map[aclass:currency altname:1INCH decimals:10 display_decimals:5] AAVE:map[aclass:currency altname:AAVE decimals:10 display_decimals:5] ACA:map[aclass:currency altname:ACA decimals:10 display_decimals:5] ADA:map[aclass:currency altname:ADA decimals:8 display_decimals:6]...}
The response type I'm looking for is map[string]AssetInfo. Hopefully it could be unmarshalled like this:
{[] map[1INCH:{currency 1INCH 10 5} AAVE:{currency AAVE 10 5} ACA:{currency ACA 10 5} ADA:{currency ADA 8 6} ADA.S:{currency ADA.S 8 6}...}
Any help? I'd rather keep the Resp struct as generic as possible and just change the value of the Result field (if this is even possible to do correctly) since I plan to have multiple functions that call different endpoints of the api, and they'll all have the same underlying response type of the Resp struct with different Result types
You can view a working example in the following repo:
https://github.com/alessiosavi/GoArbitrage/blob/e107af466852b1ed30c2413eb4401595f7412b4f/markets/kraken/kraken.go
Basically, I've defined the following structure:
type Tickers struct {
Error []interface{} `json:"error"`
Result map[string]Ticker `json:"result"`
}
type Ticker struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
DisplayDecimals int `json:"display_decimals"`
}
Than I execute the request in the following way:
const KRAKEN_TICKERS_URL string = `https://api.kraken.com/0/public/Assets`
type Kraken struct {
PairsNames []string `json:"pairs_name"`
Pairs map[string]datastructure.KrakenPair `json:"pairs"`
OrderBook map[string]datastructure.KrakenOrderBook `json:"orderbook"`
MakerFee float64 `json:"maker_fee"`
TakerFees float64 `json:"taker_fee"`
// FeePercent is delegated to save if the fee is in percent or in coin
FeePercent bool `json:"fee_percent"`
Tickers []string
}
// Init is delegated to initialize the maps for the kraken
func (k *Kraken) Init() {
k.Pairs = make(map[string]datastructure.KrakenPair)
k.OrderBook = make(map[string]datastructure.KrakenOrderBook)
k.SetFees()
}
// SetFees is delegated to initialize the fee type/amount for the given market
func (k *Kraken) SetFees() {
k.MakerFee = 0.16
k.TakerFees = 0.26
k.FeePercent = true
}
func (k *Kraken) GetTickers() error {
res := datastructure.Tickers{}
var err error
var request req.Request
var data []byte
var tickers []string
resp := request.SendRequest(KRAKEN_TICKERS_URL, "GET", nil, nil, false, 10*time.Second)
if resp.Error != nil {
zap.S().Debugw("Error during http request. Err: " + resp.Error.Error())
return resp.Error
}
if resp.StatusCode != 200 {
zap.S().Warnw("Received a non 200 status code: " + strconv.Itoa(resp.StatusCode))
return errors.New("NON_200_STATUS_CODE")
}
data = resp.Body
if err = json.Unmarshal(data, &res); err != nil {
zap.S().Warn("ERROR! :" + err.Error())
return err
}
zap.S().Infof("Data: %v", res.Result)
tickers = make([]string, len(res.Result))
i := 0
for key := range res.Result {
tickers[i] = res.Result[key].Altname
i++
}
k.Tickers = tickers
return nil
}

How to omit empty json fields using json.decoder

I try to understand why both functions return the same output.
As far as I understood, the point of omit empty is to not add that key to the result struct.
I wrote this example, I was expecting the first output not to have the "Empty" key, but for some reason its value still shows as 0.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type agentOmitEmpty struct {
Alias string `json:"Alias,omitempty"`
Skilled bool `json:"Skilled,omitempty"`
FinID int32 `json:"FinId,omitempty"`
Empty int `json:"Empty,omitempty"`
}
type agent struct {
Alias string `json:"Alias"`
Skilled bool `json:"Skilled"`
FinID int32 `json:"FinId"`
Empty int `json:"Empty"`
}
func main() {
jsonString := `{
"Alias":"Robert",
"Skilled":true,
"FinId":12345
}`
fmt.Printf("output with omit emtpy: %v\n", withEmpty(strings.NewReader(jsonString)))
// output with omit emtpy: {Robert true 12345 0}
fmt.Printf("output regular: %v\n", withoutEmpty(strings.NewReader(jsonString)))
// output without omit: {Robert true 12345 0}
}
func withEmpty(r *strings.Reader) agentOmitEmpty {
dec := json.NewDecoder(r)
body := agentOmitEmpty{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
func withoutEmpty(r *strings.Reader) agent {
dec := json.NewDecoder(r)
body := agent{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
You need to define Empty as *int so it will be replaced with nil when there is no value. Then it will not be saved in the database.

Convert URL.Query (map of slices) to struct golang

It would be awesome to have a straight forward mapping from the standard library URL.Query() to an struct.
Query() returns a map like:
map[a:[aaaa] b:[bbbb] c:[cccc]]
The struct looks like:
type Thing struct {
A string
B string
C string
}
I've no idea why URL.Query returns a map with array elements inside tough. (well.. I know why but a GET is not likely to have duplicated params)
Please find below the complete example of parsing get query params directly in a golang struct and then sending the struct back as response
package main
import (
"log"
"net/http"
"encoding/json"
"github.com/gorilla/schema"
)
var decoder = schema.NewDecoder()
type EmployeeStruct struct {
MemberId string `schema:"memberId"`
ActivityType string `schema:"activityType"`
BusinessUnitCode int `schema:"businessUnitCode"`
}
func GetEmployee(w http.ResponseWriter, r *http.Request) {
var employeeStruct EmployeeStruct
err := decoder.Decode(&employeeStruct, r.URL.Query())
if err != nil {
log.Println("Error in GET parameters : ", err)
} else {
log.Println("GET parameters : ", employeeStruct)
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(employeeStruct)
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/GetEmployee", GetEmployee)
log.Fatal(http.ListenAndServe(":8080", mux))
}
Steps to execute & Test (Assuming you are saving above code in employee.go) :
Step 1 : go run employee.go
Step 2 : Open in browser http://localhost:8080/GetEmployee?memberId=123&activityType=Call&businessUnitCode=56
Step 3 : You should get below response in browser window
{
"MemberId": "123",
"ActivityType": "Call",
"BusinessUnitCode": 56
}
Step 4 : On console you should see below
GET parameters : {123 Call 56}
example:
filters={"reference":["docker.io/library/alpine:latest"]}
need url encode to:
filters=%7B%22reference%22%3A%5B%22docker.io%2Flibrary%2Falpine%3Alatest%22%5D%7D
and could use "github.com/gorilla/schema"
query := struct {
All bool
Filters map[string][]string `schema:"filters"`
Digests bool
Filter string
}{}
decoder := schema.NewDecoder()
decoder.Decode(&query, r.URL.Query())
As pointed out by #mh-cbon gorilla schema is the ultimate solution here.
Instead for obtaining the queryParams from the URL attribute.
func handleRequest(w http.ResponseWriter, r *http.Request) {
queryString := r.URL.Query()
//...parsing the Values -> map[string][]string
}
The approach of gorilla schema is to ship r.PostForm to the decode function.
func handleRequest(w http.ResponseWriter, r *http.Request) {
err := decoder.Decode(person, r.PostForm)
//...using reflect each struct's property can be called using
// the PostForm(url string, data url.Values) signature
fmt.Print(person.GoodJobGorilla)
}
Just parse the string to URL and after you can use the lib github.com/gorilla/schema to parse it :)
// Example to parse querystring to struct
package main
import (
"log"
"net/url"
"github.com/gorilla/schema"
)
type URLParams struct {
Code string `schema:"code"`
State string `schema:"state"`
}
func main() {
var (
params URLParams
decoder = schema.NewDecoder()
)
p := "https://www.redirect-url.com?code=CODE&state=RANDOM_ID"
u, _ := url.Parse(p)
err := decoder.Decode(&params, u.Query())
if err != nil {
log.Println("Error in Decode parameters : ", err)
} else {
log.Printf("Decoded parameters : %#v\n", params)
}
}
https://go.dev/play/p/CmuPhdKh6Yg
Using ggicci/httpin
Disclaimer: I'm the creator and maintainer of this package.
httpin helps you easily decoding HTTP request data from
Query parameters, e.g. ?name=john&is_member=true
Headers, e.g. Authorization: xxx
Form data, e.g. username=john&password=******
JSON/XML Body, e.g. POST {"name":"john"}
Path variables, e.g. /users/{username}
File uploads
How to use?
type ListUsersInput struct {
Page int `in:"query=page"`
PerPage int `in:"query=per_page"`
IsMember bool `in:"query=is_member"`
}
func ListUsers(rw http.ResponseWriter, r *http.Request) {
input := r.Context().Value(httpin.Input).(*ListUsersInput)
if input.IsMember {
// Do sth.
}
// Do sth.
}
httpin is:
well documented: at https://ggicci.github.io/httpin/
well tested: coverage over 98%
open integrated: with net/http, go-chi/chi, gorilla/mux, gin-gonic/gin, etc.
extensible (advanced feature): by adding your custom directives. Read httpin - custom directives for more details.
awesome mentioned: https://github.com/avelino/awesome-go#forms
You can use the graceful package of Echo.
I write some codes as an example, with self-explanatory comments
package main
import (
"log"
"github.com/labstacks/echo"
)
// Declare your struct with form: "" tag
type Employee struct {
MemberId string `form:"memberId"`
ActivityType string `form:"activityType"`
BusinessUnitCode int `form:"businessUnitCode"`
}
// Your handlers should look like this method
// Which takes an echo.Context and returns an error
func GetEmployee(ctx echo.Context) error{
var employee Employee
// With Bind, you can get the Post Body or query params from http.Request
// that is wrapped by echo.Context here
if err := ctx.Bind(&employee);err != nil {
return err
}
// now you can use your struct , e.g
return ctx.json(200, employee.MemberId)
}
// now use the handler in your main function or anywhere you need
func main() {
e := echo.New()
e.Get("/employee", GetEmployee)
log.Fatal(e.Start(":8080"))
}
If anyone is using Echo, query struct tag will be useful for this case.
Example Struct
type struct Name {
FirstName string `query:"first_name"`
LastName string `query:"last_name"`
}
Example Query Param
?first_name="shahriar"&last_name="sazid"
Code
var name Name
err := c.Bind(&name); if err != nil {
return c.String(http.StatusBadRequest, "bad request")
}

Beego validation accepts invalid data

I am trying to validate some forms using Beego validation, but it is not working at all: invalid data passes without errors.
This the relevant code, I don't know what is wrong. Can you point me at the mistake?
https://github.com/dionyself/golang-cms/blob/master/models/form.go
package models
import (
"github.com/astaxie/beego"
"github.com/astaxie/beego/validation"
)
type BaseForm struct {
Errors map[string]string
}
func (form *BaseForm) Validate() bool {
valid := validation.Validation{}
b, err := valid.Valid(form)
if err != nil {
beego.Error(err)
}
if !b {
for _, err := range valid.Errors {
form.Errors[err.Key] = err.Message
beego.Debug(err.Key, err.Message)
}
}
return b
}
type RegisterForm struct {
BaseForm
Username string `form:"username" valid:"Required; AlphaNumeric; MinSize(4); MaxSize(300)"`
Password string `form:"password" valid:"Required; MinSize(4); MaxSize(30)"`
PasswordRe string `form:"passwordre" valid:"Required; MinSize(4); MaxSize(30)"`
}
func (form *RegisterForm) Valid(v *validation.Validation) {
// Check if passwords of two times are same.
if form.Password != form.PasswordRe {
v.SetError("PasswordRe", "Passwords did not match")
return
}
}
type ArticleForm struct {
BaseForm
Id int `form:"-"`
Title string `form:"title" valid:"Required;MinSize(4);MaxSize(300)"`
Category int `form:"category"`
Content string `form:"content" valid:"Required; MinSize(50); MaxSize(2000)"`
TopicTags string `form:"topic-tags" valid:"MinSize(4); MaxSize(300)"`
TaggedUsers string `form:"tagged-users" valid:"MinSize(4); MaxSize(300)"`
AllowReviews bool `form:"allow-reviews" valid:"Required"`
AllowComments bool `form:"allow-comments" valid:"Required"`
Errors map[string]string
}
func (form *ArticleForm) Valid(v *validation.Validation) {
if form.Category >= 0 {
v.SetError("Category", "Invalid category")
return
}
}
Some documentation:
http://beego.me/docs/mvc/controller/validation.md
This is the code that parses the form:
func (this *ArticleController) Post() {
form := models.ArticleForm{}
Art := new(models.Article)
if err := this.ParseForm(&form); err != nil {
this.Abort("401")
} else {
db := this.GetDB()
if !form.Validate() {
this.Data["form"] = form
var cats []*models.Category
db.QueryTable("category").All(&cats)
this.Data["Categories"] = cats
this.ConfigPage("article-editor.html")
for key, msg := range form.Errors {
fmt.Println(key, msg)
}
} else {
db.Insert(Art)
this.Data["Article"] = Art
this.ConfigPage("article.html")
}
}
}
Note: FormData is always accepted (even an empty form), form.Validate() is always returning 'true'... 0 errors on logs.
It's because your struct has a data type with map[string]interface{} which accepts any data type and converting it into a string try to be specific in data type

Iterate Over String Fields in Struct

I'm looking to iterate over the string fields of a struct so I can do some clean-up/validation (with strings.TrimSpace, strings.Trim, etc).
Right now I have a messy switch-case that's not really scalable, and as this isn't in a hot spot of my application (a web form) it seems leveraging reflect is a good choice here.
I'm at a bit of a roadblock for how to implement this however, and the reflect docs are a little confusing to me (I've been digging through some other validation packages, but they're way too heavyweight + I'm using gorilla/schema for the unmarshalling part already):
Iterate over the struct
For each field of type string, apply whatever I need to from the strings package i.e. field = strings.TrimSpace(field)
If there exists a field.Tag.Get("max"), we'll use that value (strconv.Atoi, then unicode.RuneCountInString)
Provide an error slice that's also compatible with the error interface type
type FormError []string
type Listing struct {
Title string `max:"50"`
Location string `max:"100"`
Description string `max:"10000"`
ExpiryDate time.Time
RenderedDesc template.HTML
Contact string `max:"255"`
}
// Iterate over our struct, fix whitespace/formatting where possible
// and return errors encountered
func (l *Listing) Validate() error {
typ := l.Elem().Type()
var invalid FormError
for i = 0; i < typ.NumField(); i++ {
// Iterate over fields
// For StructFields of type string, field = strings.TrimSpace(field)
// if field.Tag.Get("max") != "" {
// check max length/convert to int/utf8.RuneCountInString
if max length exceeded, invalid = append(invalid, "errormsg")
}
if len(invalid) > 0 {
return invalid
}
return nil
}
func (f FormError) Error() string {
var fullError string
for _, v := range f {
fullError =+ v + "\n"
}
return "Errors were encountered during form processing: " + fullError
}
Thanks in advance.
What you want is primarily the methods on reflect.Value called NumFields() int and Field(int). The only thing you're really missing is the string check and SetString method.
package main
import "fmt"
import "reflect"
import "strings"
type MyStruct struct {
A,B,C string
I int
D string
J int
}
func main() {
ms := MyStruct{"Green ", " Eggs", " and ", 2, " Ham ", 15}
// Print it out now so we can see the difference
fmt.Printf("%s%s%s%d%s%d\n", ms.A, ms.B, ms.C, ms.I, ms.D, ms.J)
// We need a pointer so that we can set the value via reflection
msValuePtr := reflect.ValueOf(&ms)
msValue := msValuePtr.Elem()
for i := 0; i < msValue.NumField(); i++ {
field := msValue.Field(i)
// Ignore fields that don't have the same type as a string
if field.Type() != reflect.TypeOf("") {
continue
}
str := field.Interface().(string)
str = strings.TrimSpace(str)
field.SetString(str)
}
fmt.Printf("%s%s%s%d%s%d\n", ms.A, ms.B, ms.C, ms.I, ms.D, ms.J)
}
(Playground link)
There are two caveats here:
You need a pointer to what you're going to change. If you have a value, you'll need to return the modified result.
Attempts to modify unexported fields generally will cause reflect to panic. If you plan on modifying unexported fields, make sure to do this trick inside the package.
This code is rather flexible, you can use switch statements or type switches (on the value returned by field.Interface()) if you need differing behavior depending on the type.
Edit: As for the tag behavior, you seem to already have that figured out. Once you have field and have checked that it's a string, you can just use field.Tag.Get("max") and parse it from there.
Edit2: I made a small error on the tag. Tags are part of the reflect.Type of a struct, so to get them you can use (this is a bit long-winded) msValue.Type().Field(i).Tag.Get("max")
(Playground version of the code you posted in the comments with a working Tag get).
I got beat to the punch, but since I went to the work, here's a solution:
type FormError []*string
type Listing struct {
Title string `max:"50"`
Location string `max:"100"`
Description string `max:"10000"`
ExpiryDate time.Time
RenderedDesc template.HTML
Contact string `max:"255"`
}
// Iterate over our struct, fix whitespace/formatting where possible
// and return errors encountered
func (l *Listing) Validate() error {
listingType := reflect.TypeOf(*l)
listingValue := reflect.ValueOf(l)
listingElem := listingValue.Elem()
var invalid FormError = []*string{}
// Iterate over fields
for i := 0; i < listingElem.NumField(); i++ {
fieldValue := listingElem.Field(i)
// For StructFields of type string, field = strings.TrimSpace(field)
if fieldValue.Type().Name() == "string" {
newFieldValue := strings.TrimSpace(fieldValue.Interface().(string))
fieldValue.SetString(newFieldValue)
fieldType := listingType.Field(i)
maxLengthStr := fieldType.Tag.Get("max")
if maxLengthStr != "" {
maxLength, err := strconv.Atoi(maxLengthStr)
if err != nil {
panic("Field 'max' must be an integer")
}
// check max length/convert to int/utf8.RuneCountInString
if utf8.RuneCountInString(newFieldValue) > maxLength {
// if max length exceeded, invalid = append(invalid, "errormsg")
invalidMessage := `"`+fieldType.Name+`" is too long (max allowed: `+maxLengthStr+`)`
invalid = append(invalid, &invalidMessage)
}
}
}
}
if len(invalid) > 0 {
return invalid
}
return nil
}
func (f FormError) Error() string {
var fullError string
for _, v := range f {
fullError = *v + "\n"
}
return "Errors were encountered during form processing: " + fullError
}
I see you asked about how to do the tags. Reflection has two components: a type and a value. The tag is associated with the type, so you have to get it separately than the field: listingType := reflect.TypeOf(*l). Then you can get the indexed field and the tag from that.
I don't know if it's a good way, but I use it like this.
https://play.golang.org/p/aQ_hG2BYmMD
You can send the address of a struct to this function.
Sorry for My English is not very good.
trimStruct(&someStruct)
func trimStruct(v interface{}) {
bytes, err := json.Marshal(v)
if err != nil {
fmt.Println("[trimStruct] Marshal Error :", err)
}
var mapSI map[string]interface{}
if err := json.Unmarshal(bytes, &mapSI); err != nil {
fmt.Println("[trimStruct] Unmarshal to byte Error :", err)
}
mapSI = trimMapStringInterface(mapSI).(map[string]interface{})
bytes2, err := json.Marshal(mapSI)
if err != nil {
fmt.Println("[trimStruct] Marshal Error :", err)
}
if err := json.Unmarshal(bytes2, v); err != nil {
fmt.Println("[trimStruct] Unmarshal to b Error :", err)
}
}
func trimMapStringInterface(data interface{}) interface{} {
if values, valid := data.([]interface{}); valid {
for i := range values {
data.([]interface{})[i] = trimMapStringInterface(values[i])
}
} else if values, valid := data.(map[string]interface{}); valid {
for k, v := range values {
data.(map[string]interface{})[k] = trimMapStringInterface(v)
}
} else if value, valid := data.(string); valid {
data = strings.TrimSpace(value)
}
return data
}

Resources