I have the following code:
func init() {
today := time.Now()
// If ENDPOINT is empty, It'll use this hardcoded endpoint. The ENDPOINT variable should not contain any text after "ModifiedDate gt". The actual date is currentDay-1
if ENDPOINT == "" {
ENDPOINT = "http://localhost:8000/Contacts/Export/?$select=Firstname,Lastname,Email,SubaccountId&$filter=EEA eq '' and ModifiedDate gt"
}
// Append CurrentDay-1 in YYY`enter code here`Y-MM-DDTHH:MM:SSZ format.
// The time is NOT in UTC. It's the local time of the machine on which lambda function was running
ENDPOINT = fmt.Sprintf("%s %s", ENDPOINT, today.AddDate(0, 0, -1).Format("2006-01-02T15:04:05Z"))
var err error
// parse the url
PARSED_ENDPOINT, err = url.Parse(ENDPOINT)
if err != nil {
log.Fatalln("Invalid $ENDPOINT", err)
}
// parse the query parameters
parsedQueryParams, err := url.ParseQuery(PARSED_ENDPOINT.RawQuery)
if err != nil {
log.Fatalln("error in parsing query parameters", err)
}
// URLEncode query parameters
PARSED_ENDPOINT.RawQuery = parsedQueryParams.Encode()
}
When I output the URL, I get:
'http://localhost:8000/Contacts/Export/?%24filter=EEA+eq+%27%27+and+ModifiedDate+gt+2018-10-22T08%3A45%3A45Z&%24select=Email%2CFirstname%2CLastname%2CSubaccountId%2CEEA'
How do I return:
'http://localhost:8000/Contacts/Export/?$filter=EEA%20eq%20%27%27%20and%20ModifiedDate%20gt%202018-10-22T00:00:00Z&$select=Email,Firstname,Lastname,SubaccountId,EEA'
Any advice is much appreciated
Golang provide url package to manage this problem and pass the query string with key values to the browser and parse it accordingly after encoding the string which will resolve special characters issue:
package main
import (
"fmt"
"net/url"
)
func main() {
query := make(url.Values)
query.Add("key", "value")
url := &url.URL{RawQuery: query.Encode(), Host: "foo", Scheme: "http"}
fmt.Println(url)
}
Avoid using string query and adding values using fmt package Sprintf methods.This is not a proper way to manage query strings and creating a dynamic url.
Related
I'm using range to loop through an array of structs to extract data which will be used as a URL parameter for my API calls. Within this loop, I'm trying to push response data from one struct to another.
I'm able to get everything working, except for moving data from one struct to another, but not entirely sure how to solve for the errors I keep getting. I've tried multiple methods and seem to be stuck in the mud here for something I don't consider to be too hard, until now... In my code I'm using the append method but I'm not so sure that might be the correct way to proceed.
Presenting my code:
models.go
//Here is my existing struct, with populated data that I get from a CSV
type TravelItenaries struct {
Origin string
Destination string
Flight_num string
Origin_latitude string
Origin_longitude string
Destination_latitude string
Destination_longitude string
Origin_weather string
Destination_weather string
Coordinates_ori string
Coordinates_dest string
Temp_c_ori string
Temp_f_ori string
Temp_c_dest string
Temp_f_dest string
}
//Here is the response data that I'm expected to get from my API calls.
//I'm trying to "push" Temp_c_dest and Temp_f_dest data into TravelItenaries.Temp_f_dest and TravelItenaries.Temp_c_dest
//While also changing the data types to fit above.
type Response struct {
Current struct {
LastUpdatedEpoch int `json:"last_updated_epoch"`
LastUpdated string `json:"last_updated"`
Temp_c_dest float64 `json:"temp_c"`
Temp_c_dest float64 `json:"temp_f"`
IsDay int `json:"is_day"`
} `json:"current"
}
weather.go
func (s *Server) getWeather(w http.ResponseWriter, r *http.Request) {
// open file
f, err := os.Open("challenge_dataset.csv")
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
// remember to close the file at the end of the program
defer f.Close()
// read csv values using csv.Reader
csvReader := csv.NewReader(f)
data, err := csvReader.ReadAll()
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
// convert records to array of structs
travelItenaries := createTravelItenaries(data)
// remove duplicate flight records
cleanTravelItenaries:= remDupKeys(travelItenaries)
// set up params for API get request
params := url.Values{
"key": []string{"xxx"},
"q": []string{""},
}
// Construct URL for API request
u := &url.URL{
Scheme: "https",
Host: "api.weatherapi.com",
Path: "/v1/current.json",
RawQuery: params.Encode(),
}
client := &http.Client{}
// Will need this to populate the params using a range over a struct
values := u.Query()
// loop through cleaned data set
for _, service := range cleanTravelItenaries {
// dynamically acquire data from struct to pass as parameter
values.Set("q", service.Coordinates_dest)
u.RawQuery = values.Encode()
req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
resp, err := client.Do(req)
if err != nil {
responses.ERROR(w, http.StatusBadRequest, fmt.Errorf("helpful error"))
return
}
body, err := ioutil.ReadAll(resp.Body)
// create empty struct to parse response data with using Inmarshal
var responseData models.Response
json.Unmarshal(body, &responseData)
// Here is the issue, I don't think append might be the correct procedure here?
// I simply just need to pass this response data to my already existing struct
service.Temp_c_dest = append(responseData.Current.Temp_c_dest , cleanTravelItenaries )
service.Temp_f_dest = append(responseData.Current.Temp_f_dest , cleanTravelItenaries )
}
}
The errors I get are related to both append statements at the end of the range function.
first argument to append must be slice; have float64
first argument to append must be slice; have float64
for both append methods.
Also, note how type TravelItenaries struct uses string type for:
Temp_c_dest string
Temp_f_dest string
Hence why I also need to do some field type conversion from Float64 to string.
How can I extract the fields Temp_c_dest and Temp_f_dest from API response struct to TravelItenaries struct fields while changing datatypes?
EDIT:
I've managed to get this somewhat working, but only inside the for loop. The data is not being saved outside the function.
service.Temp_f_dest = strconv.FormatFloat(responseData.Current.Temp_f_dest, 'g', -1, 64)
service.Temp_c_dest = strconv.FormatFloat(responseData.Current.Temp_c_dest, 'g', -1, 64)
I am facing an issue where i have made an api in Go every thing work fine but i am not getting data in postman. When i print the data in logs i am getting the data properly but it is showing blank data in postman.
authorizeModel.go
func GetSkillList() map[string]interface{} {
db := GetDB()
var (
// id int
skillName string
)
type SkillList struct {
name string
}
skillList := SkillList{}
skillArr := []SkillList{}
rows, err := db.Query("select DISTINCT(name) as name from skills where company_id IN ('2') and name != 'Skill Needed' order by name")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
err := rows.Scan(&skillName)
if err != nil {
log.Fatal(err)
}
skillList.name = skillName
skillArr = append(skillArr, skillList)
}
response := u.Message(true, "Skill list retrieved successfully")
response["data"] = skillArr
log.Println(skillArr)
response["authorization"] = false
return response
}
authController.go
var SkillTagList = func(w http.ResponseWriter, r *http.Request) {
resp := models.GetSkillList()
u.Respond(w, resp)
}
routes.go
router.HandleFunc("/api/v1/authorize/skillTagList", controllers.SkillTagList).Methods("POST")
If you see authorizeModel.go i have printed my data in logs i am getting that data successfully in logs. But see the postman screenshot below.
You have to rename name to Name
I'm not sure what is u.Respond(), so I will assume it's a helper function of some framework that you are using, and I will assume u.Respond() is internally using json.Marshal.
If your struct has unexported fields(fields name starting with lowercase letter, in your case name), json.Marshal cannot access those field, and the result won't have name field. That is why you are getting empty objects in JSON.
Update
Since I'm not able to achieve this using the approach in this question, I created my own library to do the same thing (link). It doesn't rely on go-ethereum package but use the normal net/http package to do JSON RPC request.
I still love to know what I did wrong in my approach below.
Definitions:
owner = public variable in contract with address type
contract = smart-contract that has owner
This is the curl request to get the owner of a contract. I managed to get the owner. (JSON RPC docs)
curl localhost:8545 -X POST \
--header 'Content-type: application/json' \
--data '{"jsonrpc":"2.0", "method":"eth_call", "params":[{"to": "0x_MY_CONTRACT_ADDRESS", "data": "0x8da5cb5b"}, "latest"], "id":1}'
{"jsonrpc":"2.0","id":1,"result":"0x000000000000000000000000_OWNER"}
But when I try to replicate it in Golang (code below), I got json: cannot unmarshal string into Go value of type main.response error. (go-ethereum code that I use)
package main
import (
"fmt"
"log"
"os"
"github.com/ethereum/go-ethereum/rpc"
)
func main() {
client, err := rpc.DialHTTP(os.Getenv("RPC_SERVER"))
if err != nil {
log.Fatal(err)
}
defer client.Close()
type request struct {
To string `json:"to"`
Data string `json:"data"`
}
type response struct {
Result string
}
req := request{"0x_MY_CONTRACT_ADDRESS", "0x8da5cb5b"}
var resp response
if err := client.Call(&resp, "eth_call", req, "latest"); err != nil {
log.Fatal(err)
}
fmt.Printf("%v\n", resp)
}
What did I miss here?
Expected result:
Address in string format. E.g. 0x3ab17372b25154400738C04B04f755321bB5a94b
P/S — I'm aware of abigen and I know it's better and easier to do this using abigen. But I'm trying to solve this specific issue without using abigen method.
You can solve the problem best using the go-ethereum/ethclient:
package main
import (
"context"
"log"
"github.com/ethereum/go-ethereum"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/ethclient"
)
func main() {
client, _ := ethclient.Dial("https://mainnet.infura.io")
defer client.Close()
contractAddr := common.HexToAddress("0xCc13Fc627EFfd6E35D2D2706Ea3C4D7396c610ea")
callMsg := ethereum.CallMsg{
To: &contractAddr,
Data: common.FromHex("0x8da5cb5b"),
}
res, err := client.CallContract(context.Background(), callMsg, nil)
if err != nil {
log.Fatalf("Error calling contract: %v", err)
}
log.Printf("Owner: %s", common.BytesToAddress(res).Hex())
}
If you look at the client library code, you'll see that the JSON RPC response object is already disassembled and either an error is returned on failure, or the actual result parsed: https://github.com/ethereum/go-ethereum/blob/master/rpc/client.go#L277
The parser however already unwrapped the containing "result" field. Your type still wants to do an additional unwrap:
type response struct {
Result string
}
Drop the outer struct, simply pass a string pointer to the client.Call's first parameter.
Your response struct doesn't show the data that the json of the response has
try this
type response struct {
Jsonrpc string `json:"jsonrpc"`
ID int `json:"id"`
Result string `json:"result"`
}
json: cannot unmarshal string into Go value of type main.response error. I got similar type error when i was unmarshaling a response. It was because the response was actually json string, i mean it had Quotation " as first character. So to be sure you also encountered the same problem, please printf("%v",resp.Result) before unmarshaling in here https://github.com/ethereum/go-ethereum/blob/1ff152f3a43e4adf030ac61eb5d8da345554fc5a/rpc/client.go#L278.
I'm trying to ensure that URLs passed to my go program are valid. However, I can't seem to work out how to. I thought I could just feed it through url.Parse, but that doesn't seem to do the job.
package main
import (
"fmt"
"net/url"
)
func main() {
url, err := url.Parse("http:::/not.valid/a//a??a?b=&&c#hi")
if err != nil {
panic(err)
}
fmt.Println("It's valid!", url.String())
}
playground
Is there anything along the lines of filter_var I can use?
You can check that your URL has a Scheme, Host, and/or a Path.
If you inspect the URL returned, you can see that the invalid part is inserted into the Opaque data section (so in a sense, it is valid).
url.URL{Scheme:"http", Opaque:"::/not.valid/a//a", Host:"", Path:"", RawQuery:"?a?b=&&c", Fragment:"hi"}
If you parse a URL and don't have a Scheme, Host and Path you can probably assume it's not valid. (though a host without a path is often OK, since it implies /, so you need to check for that)
u, err := url.Parse("http:::/not.valid/a//a??a?b=&&c#hi")
if err != nil {
log.Fatal(err)
}
if u.Scheme == "" || u.Host == "" || u.Path == "" {
log.Fatal("invalid URL")
}
have a try of this package go validator and the IsURL func is what you want.(you can use regexp package as well)
package main
import (
"fmt"
"regexp"
)
const URL string = `^((ftp|http|https):\/\/)?(\S+(:\S*)?#)?((([1-9]\d?|1\d\d|2[01]\d|22[0-3])(\.(1?\d{1,2}|2[0-4]\d|25[0-5])){2}(?:\.([0-9]\d?|1\d\d|2[0-4]\d|25[0-4]))|(((([a-z\x{00a1}-\x{ffff}0-9]+-?-?_?)*[a-z\x{00a1}-\x{ffff}0-9]+)\.)?)?(([a-z\x{00a1}-\x{ffff}0-9]+-?-?_?)*[a-z\x{00a1}-\x{ffff}0-9]+)(?:\.([a-z\x{00a1}-\x{ffff}]{2,}))?)|localhost)(:(\d{1,5}))?((\/|\?|#)[^\s]*)?$`
func Matches(str, pattern string) bool {
match, _ := regexp.MatchString(pattern, str)
return match
}
func main() {
u1 := "http:::/not.valid/a//a??a?b=&&c#hi"
u2 := "http://golang.fastrl.com"
func(us ...string) {
for _, u := range us {
fmt.Println(Matches(u, URL))
}
}(u1, u2)
}
The url.Parse() function will return an error mainly if viaRequest is true, meaning if the URL is assumed to have arrived via an HTTP request.
Which is not the case when you call url.Parse() directly: see the source code.
if url, err = parse(u, false); err != nil {
return nil, err
}
And func parse(rawurl string, viaRequest bool) (url *URL, err error) { only returns err if viaRequest is true.
That is why you never see any error when using url.Parse() directly.
In that latter case, where no err is ever returned, you can check the fields of the URL object returned.
An empty url.Scheme, or an url.Path which isn't the one expected would indicate an error.
Im using web.go (http://webgo.io/) for writing a simple web app that accepts json in a POST request and after parsing it returns the result. Im having trouble reading the json from ctx.Params object.
Below is the code i have so far
package main
import (
"github.com/hoisie/web";
"encoding/json"
)
func parse(ctx *web.Context, val string) string {
for k,v := range ctx.Params {
println(k, v)
}
//Testing json parsing
mapB := map[string]int{"apple": 5, "lettuce": 7}
mapD, _ := json.Marshal(mapB)
return string(mapD)
}
func main() {
web.Post("/(.*)", parse)
web.Run("0.0.0.0:9999")
}
Though the post request gets registered i dont see anything printed on the command line for the json i posted. How can i fix this ?
Thank You
The reason you're not getting any JSON data from the body of the POST request is because hoisie/web reads form data into .Params, as seen here:
req.ParseForm()
if len(req.Form) > 0 {
for k, v := range req.Form {
ctx.Params[k] = v[0]
}
}
In order to fix this, you'll need to add something that can parse the raw body of the response. You should just be able to use ctx.Body to access the raw body, since it implements *http.Request and doesn't redefine Body in the Context struct.
For example, this should work:
json := make(map[string]interface{})
body, err := ioutil.ReadAll(ctx.Body)
if err != nil {
// Handle Body read error
return
}
err = json.Unmarshal(body, &json)
if err != nil {
// Handle JSON parsing error
return
}
// Use `json`