I have made an Url type that should contain the response body.
type Url struct {
Address string
Refresh string
Watch string
Found bool
Body bytes.Buffer // bytes.Buffer needs no initialization
}
An Url object is created with the right Address, and then I do
resp, err := http.Get(url.Address)
Now I would like to do something like the following, but I cannot get out of it:
io.Copy(url.Body, b) // Copy that to the Url buffer
As for now, the Url.Body field can be modified to another type if needed.
Afterwards, I want to get the string from that Buffer/Writer/whatever, but I assume this will be easy as soon as I will manage the former copy.
Regards,
Le Barde.
I guess you want to use ioutil.ReadAll which returns []byte:
resp, err := http.Get(url.Address)
if err != nil {
// handle error
}
defer resp.Body.Close()
url.Body, err = ioutil.ReadAll(resp.Body)
Related
I'm using range to loop through an array of structs to extract data which will be used as a URL parameter for my API calls. Within this loop, I'm trying to push response data from one struct to another.
I'm able to get everything working, except for moving data from one struct to another, but not entirely sure how to solve for the errors I keep getting. I've tried multiple methods and seem to be stuck in the mud here for something I don't consider to be too hard, until now... In my code I'm using the append method but I'm not so sure that might be the correct way to proceed.
Presenting my code:
models.go
//Here is my existing struct, with populated data that I get from a CSV
type TravelItenaries struct {
Origin string
Destination string
Flight_num string
Origin_latitude string
Origin_longitude string
Destination_latitude string
Destination_longitude string
Origin_weather string
Destination_weather string
Coordinates_ori string
Coordinates_dest string
Temp_c_ori string
Temp_f_ori string
Temp_c_dest string
Temp_f_dest string
}
//Here is the response data that I'm expected to get from my API calls.
//I'm trying to "push" Temp_c_dest and Temp_f_dest data into TravelItenaries.Temp_f_dest and TravelItenaries.Temp_c_dest
//While also changing the data types to fit above.
type Response struct {
Current struct {
LastUpdatedEpoch int `json:"last_updated_epoch"`
LastUpdated string `json:"last_updated"`
Temp_c_dest float64 `json:"temp_c"`
Temp_c_dest float64 `json:"temp_f"`
IsDay int `json:"is_day"`
} `json:"current"
}
weather.go
func (s *Server) getWeather(w http.ResponseWriter, r *http.Request) {
// open file
f, err := os.Open("challenge_dataset.csv")
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
// remember to close the file at the end of the program
defer f.Close()
// read csv values using csv.Reader
csvReader := csv.NewReader(f)
data, err := csvReader.ReadAll()
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
// convert records to array of structs
travelItenaries := createTravelItenaries(data)
// remove duplicate flight records
cleanTravelItenaries:= remDupKeys(travelItenaries)
// set up params for API get request
params := url.Values{
"key": []string{"xxx"},
"q": []string{""},
}
// Construct URL for API request
u := &url.URL{
Scheme: "https",
Host: "api.weatherapi.com",
Path: "/v1/current.json",
RawQuery: params.Encode(),
}
client := &http.Client{}
// Will need this to populate the params using a range over a struct
values := u.Query()
// loop through cleaned data set
for _, service := range cleanTravelItenaries {
// dynamically acquire data from struct to pass as parameter
values.Set("q", service.Coordinates_dest)
u.RawQuery = values.Encode()
req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
resp, err := client.Do(req)
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
body, err := ioutil.ReadAll(resp.Body)
// create empty struct to parse response data with using Inmarshal
var responseData models.Response
json.Unmarshal(body, &responseData)
// Here is the issue, I don't think append might be the correct procedure here?
// I simply just need to pass this response data to my already existing struct
service.Temp_c_dest = append(responseData.Current.Temp_c_dest , cleanTravelItenaries )
service.Temp_f_dest = append(responseData.Current.Temp_f_dest , cleanTravelItenaries )
}
}
The errors I get are related to both append statements at the end of the range function.
first argument to append must be slice; have float64
first argument to append must be slice; have float64
for both append methods.
Also, note how type TravelItenaries struct uses string type for:
Temp_c_dest string
Temp_f_dest string
Hence why I also need to do some field type conversion from Float64 to string.
How can I extract the fields Temp_c_dest and Temp_f_dest from API response struct to TravelItenaries struct fields while changing datatypes?
EDIT:
I've managed to get this somewhat working, but only inside the for loop. The data is not being saved outside the function.
service.Temp_f_dest = strconv.FormatFloat(responseData.Current.Temp_f_dest, 'g', -1, 64)
service.Temp_c_dest = strconv.FormatFloat(responseData.Current.Temp_c_dest, 'g', -1, 64)
When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.
This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
// ... marshal and write the response ...
}
(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)
Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.
The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.
For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.
lastEvaluatedKey := map[string]types.AttributeValue{
"year": &types.AttributeValueMemberN{Value: "1993"},
"title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}
bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
panic(err)
}
decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
panic(err)
}
What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.
I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.
Has anyone found a generic way to do this?
OK, I figured something out.
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
eskMap := map[string]string{}
json.Unmarshal(params.NextToken, &eskMap)
esk, _ = dynamodbattribute.MarshalMap(eskMap)
dynamoIn.ExclusiveStartKey = esk
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
lek := map[string]string{}
dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
response.NextToken := json.Marshal(lek)
// ... marshal and write the response ...
}
(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)
As #buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.
I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.
Inspired by Dan, here is a solution to serialize and deserialize to/from base64
package dynamodb_helpers
import (
"encoding/base64"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func Serialize(input map[string]types.AttributeValue) (*string, error) {
var inputMap map[string]interface{}
err := attributevalue.UnmarshalMap(input, &inputMap)
if err != nil {
return nil, err
}
bytesJSON, err := json.Marshal(inputMap)
if err != nil {
return nil, err
}
output := base64.StdEncoding.EncodeToString(bytesJSON)
return &output, nil
}
func Deserialize(input string) (map[string]types.AttributeValue, error) {
bytesJSON, err := base64.StdEncoding.DecodeString(input)
if err != nil {
return nil, err
}
outputJSON := map[string]interface{}{}
err = json.Unmarshal(bytesJSON, &outputJSON)
if err != nil {
return nil, err
}
return attributevalue.MarshalMap(outputJSON)
}
I have a use case where I have the code as below. I have a request coming in to hit the backend where I need to append data to a map. My question is how do I convert the below type to a []byte to unmarshal?
Any ideas would be appreciated.
type Example struct {
Category string `json:"category"`
Name string `json:"name"`
}
Incoming Postman request json looks like this:
[{"Category":"TestCategory", "Name":"Sample1"}]
but after doing
jsonString Type: []Example
if err := gc.ShouldBindJSON(&jsonString) it looks like [{TestCategory Sample1}] ; how do I convert this to a []byte?
for _, req := range blob{
var jsonString Example
if err := json.Unmarshal([]byte(jsonString), &blob); err != nil { //this does not work
logger.Fatal(err)
}
//I am checking if a key-value is present and appending it to the map
dict := make(map[string][]Example)
dict[req.Category] = append(dict[req.Category], req)
fmt.Println(dict)
if value, ok := dict["TestCategory"]; ok {
fmt.Printf("Found %d\n", value)
} else {
fmt.Println("not found")
}
}
//I was able to test the above logic by declaring the jsonString as a const and it works
There are two directions in which you can move the data:
from JSON to a Go data structure
// This is your payload coming from the request.
jsonStr := `[{"Category":"TestCategory", "Name":"Sample1"}]`
// This is the Go struct that will hold the unmarshalled data.
var examples []Example
err := json.Unmarshal([]byte(jsonStr), &examples)
if err != nil {
log.Fatal(err)
}
fmt.Println("Examples:", examples) // prints "Examples: [{TestCategory Sample1}]"
from a Go data structure to JSON (either string or []byte)
exampleBytes, err := json.Marshal(examples)
if err != nil {
log.Fatal(err)
}
fmt.Println("Example bytes:", string(exampleBytes)) // prints "Example bytes: [{"category":"TestCategory","name":"Sample1"}]"
You should check out "Go by Example" if you haven't already: https://gobyexample.com/json
Looking at your code:
You are looping on blob but instead of using the req you are trying to unmarshal onto the entire blob each time. I'm not sure what you are trying to achieve there but nothing good can come out of changing a struct you're looping over from within the loop.
The request JSON you are listing is an array of JSON objects. You are trying to unmarshal that into a single Example struct. That won't work, you need an array of those.
My API server has middle ware which is getting token from request header.
If it access is correct, its go next function.
But request went to middle ware and went to next function, c.Request.Body become 0.
middle ware
func getUserIdFromBody(c *gin.Context) (int) {
var jsonBody User
length, _ := strconv.Atoi(c.Request.Header.Get("Content-Length"))
body := make([]byte, length)
length, _ = c.Request.Body.Read(body)
json.Unmarshal(body[:length], &jsonBody)
return jsonBody.Id
}
func CheckToken() (gin.HandlerFunc) {
return func(c *gin.Context) {
var userId int
config := model.NewConfig()
reqToken := c.Request.Header.Get("token")
_, resBool := c.GetQuery("user_id")
if resBool == false {
userId = getUserIdFromBody(c)
} else {
userIdStr := c.Query("user_id")
userId, _ = strconv.Atoi(userIdStr)
}
...
if ok {
c.Nex()
return
}
}
next func
func bindOneDay(c *gin.Context) (model.Oneday, error) {
var oneday model.Oneday
if err := c.BindJSON(&oneday); err != nil {
return oneday, err
}
return oneday, nil
}
bindOneDay return error with EOF. because maybe c.Request.Body is 0.
I want to get user_id from request body in middle ware.
How to do it without problem that c.Request.Body become 0
You can only read the Body from the client once. The data is streaming from the user, and they're not going to send it again. If you want to read it more than once, you're going to have to buffer the whole thing in memory, like so:
bodyCopy := new(bytes.Buffer)
// Read the whole body
_, err := io.Copy(bodyCopy, req.Body)
if err != nil {
return err
}
bodyData := bodyCopy.Bytes()
// Replace the body with a reader that reads from the buffer
req.Body = ioutil.NopCloser(bytes.NewReader(bodyData))
// Now you can do something with the contents of bodyData,
// like passing it to json.Unmarshal
Note that buffering the entire request into memory means that a user can cause you to allocate unlimited memory -- you should probably either block this at a frontend proxy or use an io.LimitedReader to limit the amount of data you'll buffer.
You also have to read the entire body before Unmarshal can start its work -- this is probably no big deal, but you can do better using io.TeeReader and json.NewDecoder if you're so inclined.
Better, of course, would be to figure out a way to restructure your code so that buffering the body and decoding it twice aren't necessary.
Gin provides a native solution to allow you to get data multiple times from c.Request.Body. The solution is to use c.ShouldBindBodyWith. Per the gin documentation
ShouldBindBodyWith ... stores the
request body into the context, and reuse when it is called again.
For your particular example, this would be implemented in your middleware like so,
func getUserIdFromBody(c *gin.Context) (int) {
var jsonBody User
if err := c.ShouldBindBodyWith(&jsonBody, binding.JSON); err != nil {
//return error
}
return jsonBody.Id
}
After the middleware, if you want to bind to the body again, just use ctx.ShouldBindBodyWith again. For your particular example, this would be implemented like so
func bindOneDay(c *gin.Context) (model.Oneday, error) {
var oneday model.Oneday
if err := c.ShouldBindBodyWith(&oneday); err != nil {
return error
}
return oneday, nil
}
The issue we're fighting against is that gin has setup c.Request.Body as an io.ReadCloser object -- meaning that it is intended to be read from only once. So, if you access c.Request.Body in your code at all, the bytes will be read (consumed) and c.Request.Body will be empty thereafter. By using ShouldBindBodyWith to access the bytes, gin saves the bytes into another storage mechanism within the context, so that it can be reused over and over again.
As a side note, if you've consumed the c.Request.Body and later want to access c.Request.Body, you can do so by tapping into gin's storage mechanism via ctx.Get(gin.BodyBytesKey). Here's an example of how you can obtain the gin-stored Request Body as []byte and then convert it to a string,
var body string
if cb, ok := ctx.Get(gin.BodyBytesKey); ok {
if cbb, ok := cb.([]byte); ok {
body = string(cbb)
}
}
I'm using the following code to get a response from a server after a post request:
type ResponseFromPost struct {
N_expediente string
Enviar string
}
func main(){
......
res, err := client.Do(req)
if err != nil {
return
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
var re ResponseFromPost
err = json.Unmarshal(body, &re)
fmt.Println(re.Enviar);
}
With this I get:
error: &{%!e(string=array) %!e(*reflect.rtype=&{32 2509985895 0 8 8 25 0x608170
[0x7703c0 <nil>] 0x730b80 0x69acb0 0x6116c0 0x7732c0})}
The value sent by the server is:
[{"n_expediente":"9","enviar":"2"}]
How can I use the json variables?
The json is an array of those objects having the strings n_expediente and enviar on each instance. In the Go you'll need an array of your type;
re := []ResponseFromPost{}
err := json.Unmarshal([]byte(`[{"n_expediente":"9","enviar":"2"}]`), &re)
fmt.Println(re[0].Enviar);
Here's an example to show what your model needs to be https://play.golang.org/p/d64Sict4AG
You'll probably want to make some other changes to your code based on that. Like a loop bound by the length of the slice ( for i := range re { print i } ) after the unmarshal and a different name for the type.