I am trying to create a buildURL function that can generate an url. However I have run into a problem when facing a dynamic url.
type ShopifyDownloader struct {
Domain string
AccessToken string
}
func (sd *ShopifyDownloader) BuildURL(Path string, Query map[string]string) (string, error) {
u, err := url.Parse("https://123.myshopify.com/admin/orders/count.json?access_token=123")
if err != nil {
fmt.Println("Cannot parse", err)
return "", err
}
u.Host = sd.Domain
u.Path = Path
params := url.Values{}
for key, value := range Query {
params.Add(key, value)
}
u.RawQuery = params.Encode()
nu := u.String()
var buffer bytes.Buffer
buffer.WriteString(nu)
buffer.WriteString("?access_token=")
buffer.WriteString(sd.AccessToken)
bu := buffer.String()
return bu, err
}
It takes domain, token, path and queries and generate an url in string. However it does not work with a dynamic path (with id) like
this /admin/orders/#{id}.json or this /admin/products/#{id}/images/#{id}.json
Does anybody have any suggestion?
Why not just build a url with something simpler at the point where you use it?
url := fmt.Sprintf("%s/admin/orders/count.json?access_token=%s&custom=%s",sd.Domain,sd.Token,customparam)
Then use it. This has the virtue of being completely clear as to intent and the url used in each case while sharing the important stuff (shopify domain etc). Then for your question about ids it becomes easier:
url := fmt.Sprintf("%s/admin/orders/%d.json?access_token=%s",sd.Domain,orderID,sd.Token)
and
url := fmt.Sprintf("%s/admin/products/%d/images/%d.json?access_token=%s",sd.Domain,productID,imageID,sd.Token)
if you feel your params are super complex (for me this would have to be at least 5), just put them in an url.Values (including the token):
params := url.Values{"myparam":{"123"}, "token": {sd.Token}}
url := fmt.Sprintf("%s/admin/orders/count.json?%s", sd.Domain,
params.Encode())
but this is only worthwhile if you have a significant number of params.
If you want to add functions to ShopifyDownloader, why force the caller to define the url structure? Try to make it simpler for them. You don't need to parse urls or write to a bytes buffer, http.Get expects a string, so work with strings. If you want to build functions, consider instead having separate functions which know about the url formats, to abstract this away from the caller, so :
func (sd *ShopifyDownloader) OrderURL(orderID int, params url.Values) string {
params["token"] = []string{sd.AccessToken}
return fmt.Sprintf("%s/admin/orders/%d.json?%s",sd.Domain,orderID,params)
}
The caller should not know about the specific format of Shopify urls, which may change (just as the endpoint may change).
Related
I'm using range to loop through an array of structs to extract data which will be used as a URL parameter for my API calls. Within this loop, I'm trying to push response data from one struct to another.
I'm able to get everything working, except for moving data from one struct to another, but not entirely sure how to solve for the errors I keep getting. I've tried multiple methods and seem to be stuck in the mud here for something I don't consider to be too hard, until now... In my code I'm using the append method but I'm not so sure that might be the correct way to proceed.
Presenting my code:
models.go
//Here is my existing struct, with populated data that I get from a CSV
type TravelItenaries struct {
Origin string
Destination string
Flight_num string
Origin_latitude string
Origin_longitude string
Destination_latitude string
Destination_longitude string
Origin_weather string
Destination_weather string
Coordinates_ori string
Coordinates_dest string
Temp_c_ori string
Temp_f_ori string
Temp_c_dest string
Temp_f_dest string
}
//Here is the response data that I'm expected to get from my API calls.
//I'm trying to "push" Temp_c_dest and Temp_f_dest data into TravelItenaries.Temp_f_dest and TravelItenaries.Temp_c_dest
//While also changing the data types to fit above.
type Response struct {
Current struct {
LastUpdatedEpoch int `json:"last_updated_epoch"`
LastUpdated string `json:"last_updated"`
Temp_c_dest float64 `json:"temp_c"`
Temp_c_dest float64 `json:"temp_f"`
IsDay int `json:"is_day"`
} `json:"current"
}
weather.go
func (s *Server) getWeather(w http.ResponseWriter, r *http.Request) {
// open file
f, err := os.Open("challenge_dataset.csv")
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
// remember to close the file at the end of the program
defer f.Close()
// read csv values using csv.Reader
csvReader := csv.NewReader(f)
data, err := csvReader.ReadAll()
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
// convert records to array of structs
travelItenaries := createTravelItenaries(data)
// remove duplicate flight records
cleanTravelItenaries:= remDupKeys(travelItenaries)
// set up params for API get request
params := url.Values{
"key": []string{"xxx"},
"q": []string{""},
}
// Construct URL for API request
u := &url.URL{
Scheme: "https",
Host: "api.weatherapi.com",
Path: "/v1/current.json",
RawQuery: params.Encode(),
}
client := &http.Client{}
// Will need this to populate the params using a range over a struct
values := u.Query()
// loop through cleaned data set
for _, service := range cleanTravelItenaries {
// dynamically acquire data from struct to pass as parameter
values.Set("q", service.Coordinates_dest)
u.RawQuery = values.Encode()
req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
resp, err := client.Do(req)
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
body, err := ioutil.ReadAll(resp.Body)
// create empty struct to parse response data with using Inmarshal
var responseData models.Response
json.Unmarshal(body, &responseData)
// Here is the issue, I don't think append might be the correct procedure here?
// I simply just need to pass this response data to my already existing struct
service.Temp_c_dest = append(responseData.Current.Temp_c_dest , cleanTravelItenaries )
service.Temp_f_dest = append(responseData.Current.Temp_f_dest , cleanTravelItenaries )
}
}
The errors I get are related to both append statements at the end of the range function.
first argument to append must be slice; have float64
first argument to append must be slice; have float64
for both append methods.
Also, note how type TravelItenaries struct uses string type for:
Temp_c_dest string
Temp_f_dest string
Hence why I also need to do some field type conversion from Float64 to string.
How can I extract the fields Temp_c_dest and Temp_f_dest from API response struct to TravelItenaries struct fields while changing datatypes?
EDIT:
I've managed to get this somewhat working, but only inside the for loop. The data is not being saved outside the function.
service.Temp_f_dest = strconv.FormatFloat(responseData.Current.Temp_f_dest, 'g', -1, 64)
service.Temp_c_dest = strconv.FormatFloat(responseData.Current.Temp_c_dest, 'g', -1, 64)
When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.
This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
// ... marshal and write the response ...
}
(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)
Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.
The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.
For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.
lastEvaluatedKey := map[string]types.AttributeValue{
"year": &types.AttributeValueMemberN{Value: "1993"},
"title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}
bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
panic(err)
}
decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
panic(err)
}
What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.
I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.
Has anyone found a generic way to do this?
OK, I figured something out.
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
eskMap := map[string]string{}
json.Unmarshal(params.NextToken, &eskMap)
esk, _ = dynamodbattribute.MarshalMap(eskMap)
dynamoIn.ExclusiveStartKey = esk
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
lek := map[string]string{}
dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
response.NextToken := json.Marshal(lek)
// ... marshal and write the response ...
}
(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)
As #buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.
I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.
Inspired by Dan, here is a solution to serialize and deserialize to/from base64
package dynamodb_helpers
import (
"encoding/base64"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func Serialize(input map[string]types.AttributeValue) (*string, error) {
var inputMap map[string]interface{}
err := attributevalue.UnmarshalMap(input, &inputMap)
if err != nil {
return nil, err
}
bytesJSON, err := json.Marshal(inputMap)
if err != nil {
return nil, err
}
output := base64.StdEncoding.EncodeToString(bytesJSON)
return &output, nil
}
func Deserialize(input string) (map[string]types.AttributeValue, error) {
bytesJSON, err := base64.StdEncoding.DecodeString(input)
if err != nil {
return nil, err
}
outputJSON := map[string]interface{}{}
err = json.Unmarshal(bytesJSON, &outputJSON)
if err != nil {
return nil, err
}
return attributevalue.MarshalMap(outputJSON)
}
I am trying to implement an oauth server and the package I am using needs the complete http.ResponseWriter and http.Request types.
c.Response does not contain all the methods that http.ResponseWriter does and c.Request gives error incompatible type.
How do I get http.ResponseWriter and http.Request in a Revel controller?
type client struct {
ClientId string
ClientSecret string
}
type App struct {
*revel.Controller
}
func (c App) TokenRequest() {
r := c.Request
w := c.Response
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
log.Println(string(body))
var cli client
err = json.Unmarshal(body, &cli)
if err != nil {
panic(err)
}
log.Println(cli.ClientId)
err = OauthSrv.HandleTokenRequest(w, r)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
Warning
I am generally not fond of frameworks like Revel in Go, for reasons that I hope demonstrate themselves on this page. My first recommendation would be that you examine closely what you are actually getting out of Revel that merits the use of such a heavy abstraction layer, and if it's really that valuable, you may want to ask questions going in the other direction, such as how one might make OauthSrv work within Revel's customized ecosystem.
Using a Revel controller as a ResponseWriter
For something to be an http.ResponseWriter, it just needs to have these methods.
Header
You need a method named Header() that returns an http.Header, which you can build out of any map[string][]string. Revel provides similar functionality, but through several layers of abstraction. You will need to unravel them:
c.Response is a *Response, so it has a field named Out containing an OutResponse.
An OutResponse has a Header() method—but it doesn't return an http.Header. Instead, it returns a *RevelHeader.
A *RevelHeader has a GetAll(key string) []string method—which is very similar to the API already provided by the built-in map type, but isn't exactly the same. So, you will need to copy the returned values into a new map every time Header() is called, in order to fully satisfy the function signature requirements.
Also, GetAll() requires you to know the key name you are interested in, and *RevelHeader on its own does not provide a way to look up which keys are available. For now we can rely on the fact that the current implementation only has one field, a ServerHeader that does provide a GetKeys() []string method.
Putting all this together, we can build our Header method:
func (rrw RevelResponseWrapper) Header() http.Header {
revelHeader := rrw.Response.Out.Header()
keys := revelHeader.Server.GetKeys()
headerMap := make(map[string][]string)
for _, key := range keys {
headerMap[key] = revelHeader.GetAll(key)
}
return http.Header(headerMap)
}
Write and WriteHeader
You would use similar anti-patterns to expose rrw.Write([]byte) (int, error) so that it calls through to c.Response.Out.Write(data []byte) (int, error), and rrw.WriteHeader(int) error so that it calls c.Response.WriteHeader(int, string). Depending on what is considered appropriate for the framework, either panic on errors or fail silently, since their API doesn't expect WriteHeader errors to be handle-able.
Getting an http.Request from Revel
Unfortunately, the http.Request type is a struct, so you can't just simulate it. You basically have two options: reconstruct it using the net/http package from all the properties you are able to access, or hope that the *revel.Request you have is secretly an http.Request under the hood. In the latter case, you can use a type assertion:
revelReq, ok := c.Request.In.(*revel.GoRequest)
if !ok {
// handle this somehow
}
r := revelReq.Original
My API server has middle ware which is getting token from request header.
If it access is correct, its go next function.
But request went to middle ware and went to next function, c.Request.Body become 0.
middle ware
func getUserIdFromBody(c *gin.Context) (int) {
var jsonBody User
length, _ := strconv.Atoi(c.Request.Header.Get("Content-Length"))
body := make([]byte, length)
length, _ = c.Request.Body.Read(body)
json.Unmarshal(body[:length], &jsonBody)
return jsonBody.Id
}
func CheckToken() (gin.HandlerFunc) {
return func(c *gin.Context) {
var userId int
config := model.NewConfig()
reqToken := c.Request.Header.Get("token")
_, resBool := c.GetQuery("user_id")
if resBool == false {
userId = getUserIdFromBody(c)
} else {
userIdStr := c.Query("user_id")
userId, _ = strconv.Atoi(userIdStr)
}
...
if ok {
c.Nex()
return
}
}
next func
func bindOneDay(c *gin.Context) (model.Oneday, error) {
var oneday model.Oneday
if err := c.BindJSON(&oneday); err != nil {
return oneday, err
}
return oneday, nil
}
bindOneDay return error with EOF. because maybe c.Request.Body is 0.
I want to get user_id from request body in middle ware.
How to do it without problem that c.Request.Body become 0
You can only read the Body from the client once. The data is streaming from the user, and they're not going to send it again. If you want to read it more than once, you're going to have to buffer the whole thing in memory, like so:
bodyCopy := new(bytes.Buffer)
// Read the whole body
_, err := io.Copy(bodyCopy, req.Body)
if err != nil {
return err
}
bodyData := bodyCopy.Bytes()
// Replace the body with a reader that reads from the buffer
req.Body = ioutil.NopCloser(bytes.NewReader(bodyData))
// Now you can do something with the contents of bodyData,
// like passing it to json.Unmarshal
Note that buffering the entire request into memory means that a user can cause you to allocate unlimited memory -- you should probably either block this at a frontend proxy or use an io.LimitedReader to limit the amount of data you'll buffer.
You also have to read the entire body before Unmarshal can start its work -- this is probably no big deal, but you can do better using io.TeeReader and json.NewDecoder if you're so inclined.
Better, of course, would be to figure out a way to restructure your code so that buffering the body and decoding it twice aren't necessary.
Gin provides a native solution to allow you to get data multiple times from c.Request.Body. The solution is to use c.ShouldBindBodyWith. Per the gin documentation
ShouldBindBodyWith ... stores the
request body into the context, and reuse when it is called again.
For your particular example, this would be implemented in your middleware like so,
func getUserIdFromBody(c *gin.Context) (int) {
var jsonBody User
if err := c.ShouldBindBodyWith(&jsonBody, binding.JSON); err != nil {
//return error
}
return jsonBody.Id
}
After the middleware, if you want to bind to the body again, just use ctx.ShouldBindBodyWith again. For your particular example, this would be implemented like so
func bindOneDay(c *gin.Context) (model.Oneday, error) {
var oneday model.Oneday
if err := c.ShouldBindBodyWith(&oneday); err != nil {
return error
}
return oneday, nil
}
The issue we're fighting against is that gin has setup c.Request.Body as an io.ReadCloser object -- meaning that it is intended to be read from only once. So, if you access c.Request.Body in your code at all, the bytes will be read (consumed) and c.Request.Body will be empty thereafter. By using ShouldBindBodyWith to access the bytes, gin saves the bytes into another storage mechanism within the context, so that it can be reused over and over again.
As a side note, if you've consumed the c.Request.Body and later want to access c.Request.Body, you can do so by tapping into gin's storage mechanism via ctx.Get(gin.BodyBytesKey). Here's an example of how you can obtain the gin-stored Request Body as []byte and then convert it to a string,
var body string
if cb, ok := ctx.Get(gin.BodyBytesKey); ok {
if cbb, ok := cb.([]byte); ok {
body = string(cbb)
}
}
I have made an Url type that should contain the response body.
type Url struct {
Address string
Refresh string
Watch string
Found bool
Body bytes.Buffer // bytes.Buffer needs no initialization
}
An Url object is created with the right Address, and then I do
resp, err := http.Get(url.Address)
Now I would like to do something like the following, but I cannot get out of it:
io.Copy(url.Body, b) // Copy that to the Url buffer
As for now, the Url.Body field can be modified to another type if needed.
Afterwards, I want to get the string from that Buffer/Writer/whatever, but I assume this will be easy as soon as I will manage the former copy.
Regards,
Le Barde.
I guess you want to use ioutil.ReadAll which returns []byte:
resp, err := http.Get(url.Address)
if err != nil {
// handle error
}
defer resp.Body.Close()
url.Body, err = ioutil.ReadAll(resp.Body)