Output of GET request different to view source - go

I'm trying to extract match data from whoscored.com. When I view the source on firefox, I find on line 816 a big json string with the data I want for that matchid. My goal is to eventually get this json.
In doing this, I've tried to download every page of https://www.whoscored.com/Matches/ID/Live where ID is the id of the match. I wrote a little Go program to GET request each ID up to a certain point:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
// http://www.whoscored.com/Matches/614052/Live is the match for
// Eveton vs Manchester
const match_address = "http://www.whoscored.com/Matches/"
// the max id we get
const max_id = 300
const num_workers = 10
// function that get the bytes of the match id from the website
func match_fetch(matchid int) {
url := fmt.Sprintf("%s%d/Live", match_address, matchid)
resp, err := http.Get(url)
if err != nil {
fmt.Println(err)
return
}
// if we sucessfully got a response, store the
// body in memory
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
// write the body to memory
pwd, _ := os.Getwd()
filepath := fmt.Sprintf("%s/match_data/%d", pwd, matchid)
err = ioutil.WriteFile(filepath, body, 0644)
if err != nil {
fmt.Println(err)
return
}
}
// data type to send to the workers,
// last means this job is the last one
// matchid is the match id to be fetched
// a matchid of -1 means don't fetch a match
type job struct {
last bool
matchid int
}
func create_worker(jobs chan job) {
for {
next_job := <-jobs
if next_job.matchid != -1 {
match_fetch(next_job.matchid)
}
if next_job.last {
return
}
}
}
func main() {
// do the eveton match as a reference
match_fetch(614052)
var joblist [num_workers]chan job
var v int
for i := 0; i < num_workers; i++ {
job_chan := make(chan job)
joblist[i] = job_chan
go create_worker(job_chan)
}
for i := 0; i < max_id; i = i + num_workers {
for index, c := range joblist {
if i+index < max_id {
v = i + index
} else {
v = -1
}
c <- job{false, v}
}
}
for _, c := range joblist {
c <- job{true, -1}
}
}
The code seems to work in that it fills a directory called match_data with html. The problem is that this html is completely different to what I get in the browser! Here is the section which I think does this: (from the body of the GET request of http://www.whoscored.com/Matches/614052/Live.
(function() {
var z="";var b="7472797B766172207868723B76617220743D6E6577204461746528292E67657454696D6528293B766172207374617475733D227374617274223B7661722074696D696E673D6E65772041727261792833293B77696E646F772E6F6E756E6C6F61643D66756E6374696F6E28297B74696D696E675B325D3D22723A222B286E6577204461746528292E67657454696D6528292D74293B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B69662877696E646F772E584D4C4874747052657175657374297B7868723D6E657720584D4C48747470526571756573747D656C73657B7868723D6E657720416374697665584F626A65637428224D6963726F736F66742E584D4C4854545022297D7868722E6F6E726561647973746174656368616E67653D66756E6374696F6E28297B737769746368287868722E72656164795374617465297B6361736520303A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374206E6F7420696E697469616C697A656420223B627265616B3B6361736520313A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2073657276657220636F6E6E656374696F6E2065737461626C6973686564223B627265616B3B6361736520323A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374207265636569766564223B627265616B3B6361736520333A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2070726F63657373696E672072657175657374223B627265616B3B6361736520343A7374617475733D22636F6D706C657465223B74696D696E675B315D3D22633A222B286E6577204461746528292E67657454696D6528292D74293B6966287868722E7374617475733D3D323030297B706172656E742E6C6F636174696F6E2E72656C6F616428297D627265616B7D7D3B74696D696E675B305D3D22733A222B286E6577204461746528292E67657454696D6528292D74293B7868722E6F70656E2822474554222C222F5F496E63617073756C615F5265736F757263653F535748414E45444C3D313536343032333530343538313538333938362C31373139363833393832313930303534313833392C31333935303737313737393531363432383234342C3132363636222C66616C7365293B7868722E73656E64286E756C6C297D63617463682863297B7374617475732B3D6E6577204461746528292E67657454696D6528292D742B2220696E6361705F6578633A20222B633B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B";for (var i=0;i<b.length;i+=2){z=z+parseInt(b.substring(i, i+2), 16)+",";}z = z.substring(0,z.length-1); eval(eval('String.fromCharCode('+z+')'));})();
The reason I think this is the case is that the javascript in the page fetches and edits the DOM to what I see on view source. How can I get golang to run the javascript? Is there are library to do this? Better still, could I directly grab the JSON from the servers?

This can be done with https://godoc.org/github.com/sourcegraph/webloop#View.EvaluateJavaScript
Read their main example https://github.com/sourcegraph/webloop
What you need is a "headless browser" in general.

In general it is better to use an Web API vs. scraping. For example, whoscored themselves use OPTA which you should be able to access directly.
http://www.jokecamp.com/blog/guide-to-football-and-soccer-data-and-apis/#opta

Related

Changing value of field in a go struct

I'm making an http request in golang to an external api. It gives a general response of {"error":[]string, "result":changing interface{}}. depending on the function that is making the request, the Result field changes. Since I know the structure of the Result field for each function I run, I want to be able to change the value of Result before unmarshalling to json. I've tried to do this with the following code:
func GetAssets(output *Resp, resultType interface{}) error {
return publicRequest("/Assets", output, resultType)
}
func publicRequest(endPoint string, output *Resp, resultType interface{}) error {
url := Rest_url + Pub_rest_url + endPoint //"https://api.kraken.com/0/public/Assets in this case
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
output.Result = resultType
return json.NewDecoder(resp.Body).Decode(&output)
}
Here is how it's being ran in main
type Resp struct {
Error []string `json:"error"`
Result interface{} `json:"result"`
}
type AssetInfo struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
Display int `json:"display_decimals"`
}
func main() {
var result map[string]AssetInfo
jsonData := Resp{}
rest_api_client.GetAssets(&jsonData, result)
fmt.Println(jsonData)
}
The issue is that it doesn't unmarshal correctly. A map is created for each asset, but the data contained inside of each asset is also being stored inside of a map. I'm not sure if I explained this well, but here is the current response after unmarshalling to understand what I mean.
Here is the data type of Resp.Result: map[string]interface {}
{[] map[1INCH:map[aclass:currency altname:1INCH decimals:10 display_decimals:5] AAVE:map[aclass:currency altname:AAVE decimals:10 display_decimals:5] ACA:map[aclass:currency altname:ACA decimals:10 display_decimals:5] ADA:map[aclass:currency altname:ADA decimals:8 display_decimals:6]...}
The response type I'm looking for is map[string]AssetInfo. Hopefully it could be unmarshalled like this:
{[] map[1INCH:{currency 1INCH 10 5} AAVE:{currency AAVE 10 5} ACA:{currency ACA 10 5} ADA:{currency ADA 8 6} ADA.S:{currency ADA.S 8 6}...}
Any help? I'd rather keep the Resp struct as generic as possible and just change the value of the Result field (if this is even possible to do correctly) since I plan to have multiple functions that call different endpoints of the api, and they'll all have the same underlying response type of the Resp struct with different Result types
You can view a working example in the following repo:
https://github.com/alessiosavi/GoArbitrage/blob/e107af466852b1ed30c2413eb4401595f7412b4f/markets/kraken/kraken.go
Basically, I've defined the following structure:
type Tickers struct {
Error []interface{} `json:"error"`
Result map[string]Ticker `json:"result"`
}
type Ticker struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
DisplayDecimals int `json:"display_decimals"`
}
Than I execute the request in the following way:
const KRAKEN_TICKERS_URL string = `https://api.kraken.com/0/public/Assets`
type Kraken struct {
PairsNames []string `json:"pairs_name"`
Pairs map[string]datastructure.KrakenPair `json:"pairs"`
OrderBook map[string]datastructure.KrakenOrderBook `json:"orderbook"`
MakerFee float64 `json:"maker_fee"`
TakerFees float64 `json:"taker_fee"`
// FeePercent is delegated to save if the fee is in percent or in coin
FeePercent bool `json:"fee_percent"`
Tickers []string
}
// Init is delegated to initialize the maps for the kraken
func (k *Kraken) Init() {
k.Pairs = make(map[string]datastructure.KrakenPair)
k.OrderBook = make(map[string]datastructure.KrakenOrderBook)
k.SetFees()
}
// SetFees is delegated to initialize the fee type/amount for the given market
func (k *Kraken) SetFees() {
k.MakerFee = 0.16
k.TakerFees = 0.26
k.FeePercent = true
}
func (k *Kraken) GetTickers() error {
res := datastructure.Tickers{}
var err error
var request req.Request
var data []byte
var tickers []string
resp := request.SendRequest(KRAKEN_TICKERS_URL, "GET", nil, nil, false, 10*time.Second)
if resp.Error != nil {
zap.S().Debugw("Error during http request. Err: " + resp.Error.Error())
return resp.Error
}
if resp.StatusCode != 200 {
zap.S().Warnw("Received a non 200 status code: " + strconv.Itoa(resp.StatusCode))
return errors.New("NON_200_STATUS_CODE")
}
data = resp.Body
if err = json.Unmarshal(data, &res); err != nil {
zap.S().Warn("ERROR! :" + err.Error())
return err
}
zap.S().Infof("Data: %v", res.Result)
tickers = make([]string, len(res.Result))
i := 0
for key := range res.Result {
tickers[i] = res.Result[key].Altname
i++
}
k.Tickers = tickers
return nil
}

Problems with map, goroutine and mutex

This project is made to receive POST routes that will finally count as access to later write to a database. The intuition is to save interaction with the database of another project in production. I decided to do it in go, but I'm new to the language and I'm struggling to understand. I'm trying to make it so that there is no loss or that there are more accesses.
The project basically consists of a controller, a service and two models, just enough to meet the need for which it was created. In my controller I have the function that will be responsible for receiving the POST.
controllers/views.go:
func StoreViews(c *fiber.Ctx) error {
var songview models.SongView
err := c.BodyParser(&songview)
if err != nil {
return c.Status(403).JSON(fiber.Map{
"errors": fiber.Map{"request": err.Error()},
})
}
songview.Date = time.Now()
errs := utils.ValidateStruct(songview)
if len(errs) > 0 {
return c.Status(403).JSON(map[string]interface{}{"errors": errs})
}
go services.StoreViews(songview)
return c.SendStatus(fiber.StatusOK)
}
To handle the received data I made these three functions in my service:
services/views.go
var (
StoreViewsMap = make(map[string]*models.SongView)
StoreControl sync.RWMutex
)
func StoreViews(sview models.SongView) bool {
nameKey := strconv.Itoa(int(sview.SongId)) + sview.Lang + sview.Date.Format("2006-01-02")
songview := getSongView(nameKey)
initSongView(nameKey, songview, sview)
return true
}
func getSongView(name string) *models.SongView {
StoreControl.RLock()
defer StoreControl.RUnlock()
return StoreViewsMap[name]
}
func initSongView(name string, songview *models.SongView, sview models.SongView) bool {
StoreControl.Lock()
defer StoreControl.Unlock()
if songview == nil {
insert := models.SongView{
SongId: sview.SongId,
Lang: sview.Lang,
Date: sview.Date,
Views: 0,
}
songViewNew := &insert // see if & is needed
StoreViewsMap[name] = songViewNew
} else {
songview.Views = songview.Views + 1
}
return true
}
I tried to implement RWMutex to get it to do everything without overlapping anything, but it's not working as it should, sometimes it disappears with views, other times it rescues "songview" in the getSongView function wrongly, among several other problems that I found modifying and reviewing my code. The current code is not in the version that I managed to get closer to the expected result, but I didn't save this version so I decided to bring the current code to exemplify what I'm facing.
I would like you to help me understand how I can deal with several concurrent requests disputing, how I can interact with the data in the best possible way and if there is an error in the use of a pointer I am open to understand. To simulate a POST "attack" to my code I'm using this code in another main.go I made for this test.
var limit int = 10
func main() {
channel := make(chan string)
for i := 0; i < limit; i++ {
go func(i int) {
post("http://localhost:3000/views/store", "lang=pt&song_id=296", i)
channel <- "ok"
}(i)
go func(i int) {
post("http://localhost:3000/views/store", "lang=en&song_id=3016", i)
channel <- "ok"
}(i)
go func(i int) {
post("http://localhost:3000/views/store", "lang=pt&song_id=3016", i)
channel <- "ok"
}(i)
}
for i := 0; i < limit*3; i++ {
<-channel
}
}
func post(url string, json string, index int) {
payload := strings.NewReader(json)
client := &http.Client{}
req, err := http.NewRequest("POST", url, payload)
if err != nil {
fmt.Println(err)
return
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer res.Body.Close()
_, err = ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
return
}
if res.StatusCode != 200 {
fmt.Println(res.StatusCode)
}
}
My song-view model is this: (I'm just using it to sort the data, although the project is connected to the bank of the project in production, it is read-only)
type SongView struct {
Id int64 `json:"id"`
SongId int64 `json:"song_id" form:"song_id" gorm:"notNull" validate:"required,number"`
ArtistId int64 `json:"artist_id"`
Lang string `json:"lang" validate:"required,oneof=pt en es de fr"`
Date time.Time `json:"date" gorm:"column:created_at" validate:"required"`
Views int64 `json:"views"`
}
I believe that this code can be written a little more easily in Go, but that is not the question. From your description, it appears that the data is lost somewhere. Have you tried the Go data race detector tool? Below is a link
https://go.dev/doc/articles/race_detector
Can you provide examples of input data where errors/missing items appear?
It happens because your code has a race condition in between read and write to map.
Example:
G1 - goroutine 1
G2 - goroutine 2
G1: ReadLock and Read songview named "MySong"
G1: MySong doesn't exists. Nil will be inserted and returned.
G1: Unlock
G2: ReadLock and Read songview named "MySong"
G2: MySong exist Nil. Nil will be returned.
G1: WriteLock.
G1: songView = nil, so create a new one. Set counter to 1.
G1: set counter to 1. Insert to map on key "MySong"
G1: Unlock
G2: WriteLock: songView = nil(because you read it on step 2). Create new SongView. Set counter to 1.
G2: unlock
As a result you have 1 "MySong" with counter 1 because you rewrite a previous value.
The idea of locking - Atomicity. So, all your operation should be atomic.
func initSongView(name string, sview models.SongView) bool {
StoreControl.Lock()
defer StoreControl.Unlock()
songview := StoreViewsMap[name]
if songview == nil {
insert := models.SongView{
SongId: sview.SongId,
Lang: sview.Lang,
Date: sview.Date,
Views: 1, // counter should be 1 because it's a first view
}
StoreViewsMap[name] = &insert
} else {
songview.Views = songview.Views + 1
}
return true
}

Parsing prometheus metrics from file and updating counters

I've a go application that gets run periodically by a batch. Each run, it should read some prometheus metrics from a file, run its logic, update a success/fail counter, and write metrics back out to a file.
From looking at How to parse Prometheus data as well as the godocs for prometheus, I'm able to read in the file, but I don't know how to update app_processed_total with the value returned by expfmt.ExtractSamples().
This is what I've done so far. Could someone please tell me how should I proceed from here? How can I typecast the Vector I got into a CounterVec?
package main
import (
"fmt"
"net/http"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/prometheus/common/model"
)
var (
fileOnDisk = prometheus.NewRegistry()
processedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "app_processed_total",
Help: "Number of times ran",
}, []string{"status"})
)
func doInit() {
prometheus.MustRegister(processedTotal)
}
func recordMetrics() {
go func() {
for {
processedTotal.With(prometheus.Labels{"status": "ok"}).Inc()
time.Sleep(5 * time.Second)
}
}()
}
func readExistingMetrics() {
var parser expfmt.TextParser
text := `
# HELP app_processed_total Number of times ran
# TYPE app_processed_total counter
app_processed_total{status="ok"} 300
`
parseText := func() ([]*dto.MetricFamily, error) {
parsed, err := parser.TextToMetricFamilies(strings.NewReader(text))
if err != nil {
return nil, err
}
var result []*dto.MetricFamily
for _, mf := range parsed {
result = append(result, mf)
}
return result, nil
}
gatherers := prometheus.Gatherers{
fileOnDisk,
prometheus.GathererFunc(parseText),
}
gathering, err := gatherers.Gather()
if err != nil {
fmt.Println(err)
}
fmt.Println("gathering: ", gathering)
for _, g := range gathering {
vector, err := expfmt.ExtractSamples(&expfmt.DecodeOptions{
Timestamp: model.Now(),
}, g)
fmt.Println("vector: ", vector)
if err != nil {
fmt.Println(err)
}
// How can I update processedTotal with this new value?
}
}
func main() {
doInit()
readExistingMetrics()
recordMetrics()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe("localhost:2112", nil)
}
I believe you would need to use processedTotal.WithLabelValues("ok").Inc() or something similar to that.
The more complete example is here
func ExampleCounterVec() {
httpReqs := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
},
[]string{"code", "method"},
)
prometheus.MustRegister(httpReqs)
httpReqs.WithLabelValues("404", "POST").Add(42)
// If you have to access the same set of labels very frequently, it
// might be good to retrieve the metric only once and keep a handle to
// it. But beware of deletion of that metric, see below!
m := httpReqs.WithLabelValues("200", "GET")
for i := 0; i < 1000000; i++ {
m.Inc()
}
// Delete a metric from the vector. If you have previously kept a handle
// to that metric (as above), future updates via that handle will go
// unseen (even if you re-create a metric with the same label set
// later).
httpReqs.DeleteLabelValues("200", "GET")
// Same thing with the more verbose Labels syntax.
httpReqs.Delete(prometheus.Labels{"method": "GET", "code": "200"})
}
This is taken from the Promethus examples on Github
To use the value of vector you can do the following:
vectorFloat, err := strconv.ParseFloat(vector[0].Value.String(), 64)
if err != nil {
panic(err)
}
processedTotal.WithLabelValues("ok").Add(vectorFloat)
This is assuming you will only ever get a single vector value in your response. The value of the vector is stored as a string but you can convert it to a float with the strconv.ParseFloat method.

Understand this code (golang), double parantheses ()()

I was wondering if someone can explain this syntax to me. In the google maps go api, they have
type Client struct {
httpClient *http.Client
apiKey string
baseURL string
clientID string
signature []byte
requestsPerSecond int
rateLimiter chan int
}
// NewClient constructs a new Client which can make requests to the Google Maps WebService APIs.
func NewClient(options ...ClientOption) (*Client, error) {
c := &Client{requestsPerSecond: defaultRequestsPerSecond}
WithHTTPClient(&http.Client{})(c) //???????????
for _, option := range options {
err := option(c)
if err != nil {
return nil, err
}
}
if c.apiKey == "" && (c.clientID == "" || len(c.signature) == 0) {
return nil, errors.New("maps: API Key or Maps for Work credentials missing")
}
// Implement a bursty rate limiter.
// Allow up to 1 second worth of requests to be made at once.
c.rateLimiter = make(chan int, c.requestsPerSecond)
// Prefill rateLimiter with 1 seconds worth of requests.
for i := 0; i < c.requestsPerSecond; i++ {
c.rateLimiter <- 1
}
go func() {
// Wait a second for pre-filled quota to drain
time.Sleep(time.Second)
// Then, refill rateLimiter continuously
for _ = range time.Tick(time.Second / time.Duration(c.requestsPerSecond)) {
c.rateLimiter <- 1
}
}()
return c, nil
}
// WithHTTPClient configures a Maps API client with a http.Client to make requests over.
func WithHTTPClient(c *http.Client) ClientOption {
return func(client *Client) error {
if _, ok := c.Transport.(*transport); !ok {
t := c.Transport
if t != nil {
c.Transport = &transport{Base: t}
} else {
c.Transport = &transport{Base: http.DefaultTransport}
}
}
client.httpClient = c
return nil
}
}
And this is the line I don't understand in NewClient
WithHTTPClient(&http.Client{})(c)
Why are there two ()()?
I see that WithHTTPClient takes in a *http.Client which that line does, but then it also passes in a pointer to the client struct declared above it?
WithHTTPClient returns a function, ie:
func WithHTTPClient(c *http.Client) ClientOption {
return func(client *Client) error {
....
return nil
}
}
WithHTTPClient(&http.Client{})(c) is just calling that function with c (a pointer to a Client) as parameter. It could be written as:
f := WithHTTPClient(&http.Client{})
f(c)

Go url parameters mapping

Is there a native way for inplace url parameters in native Go?
For Example, if I have a URL: http://localhost:8080/blob/123/test I want to use this URL as /blob/{id}/test.
This is not a question about finding go libraries. I am starting with the basic question, does go itself provide a basic facility to do this natively.
There is no built in simple way to do this, however, it is not hard to do.
This is how I do it, without adding a particular library. It is placed in a function so that you can invoke a simple getCode() function within your request handler.
Basically you just split the r.URL.Path into parts, and then analyse the parts.
// Extract a code from a URL. Return the default code if code
// is missing or code is not a valid number.
func getCode(r *http.Request, defaultCode int) (int, string) {
p := strings.Split(r.URL.Path, "/")
if len(p) == 1 {
return defaultCode, p[0]
} else if len(p) > 1 {
code, err := strconv.Atoi(p[0])
if err == nil {
return code, p[1]
} else {
return defaultCode, p[1]
}
} else {
return defaultCode, ""
}
}
Well, without external libraries you can't, but may I recommend two excellent ones:
httprouter - https://github.com/julienschmidt/httprouter - is extremely fast and very lightweight. It's faster than the standard library's router, and it creates 0 allocations per call, which is great in a GCed language.
Gorilla Mux - http://www.gorillatoolkit.org/pkg/mux -
Very popular, nice interface, nice community.
Example usage of httprouter:
func Hello(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
fmt.Fprintf(w, "hello, %s!\n", ps.ByName("name"))
}
func main() {
router := httprouter.New()
router.GET("/hello/:name", Hello)
log.Fatal(http.ListenAndServe(":8080", router))
}
What about trying using regex, and find a named group in your url, like playground:
package main
import (
"fmt"
"net/url"
"regexp"
)
var myExp = regexp.MustCompile(`/blob/(?P<id>\d+)/test`) // use (?P<id>[a-zA-Z]+) if the id is alphapatic
func main() {
s := "http://localhost:8080/blob/123/test"
u, err := url.Parse(s)
if err != nil {
panic(err)
}
fmt.Println(u.Path)
match := myExp.FindStringSubmatch(s) // or match := myExp.FindStringSubmatch(u.Path)
result := make(map[string]string)
for i, name := range myExp.SubexpNames() {
if i != 0 && name != "" {
result[name] = match[i]
}
}
fmt.Printf("id: %s\n", result["id"])
}
output
/blob/123/test
id: 123
Below full code to use it with url, that is receiving http://localhost:8000/hello/John/58 and returning http://localhost:8000/hello/John/58:
package main
import (
"fmt"
"net/http"
"regexp"
"strconv"
)
var helloExp = regexp.MustCompile(`/hello/(?P<name>[a-zA-Z]+)/(?P<age>\d+)`)
func hello(w http.ResponseWriter, req *http.Request) {
match := helloExp.FindStringSubmatch(req.URL.Path)
if len(match) > 0 {
result := make(map[string]string)
for i, name := range helloExp.SubexpNames() {
if i != 0 && name != "" {
result[name] = match[i]
}
}
if _, err := strconv.Atoi(result["age"]); err == nil {
fmt.Fprintf(w, "Hello, %v year old named %s!", result["age"], result["name"])
} else {
fmt.Fprintf(w, "Sorry, not accepted age!")
}
} else {
fmt.Fprintf(w, "Wrong url\n")
}
}
func main() {
http.HandleFunc("/hello/", hello)
http.ListenAndServe(":8090", nil)
}
How about writing your own url generator (extend net/url a little bit) as below.
// --- This is how does it work like --- //
url, _ := rest.NewURLGen("http", "stack.over.flow", "1234").
Pattern(foo/:foo_id/bar/:bar_id).
ParamQuery("foo_id", "abc").
ParamQuery("bar_id", "xyz").
ParamQuery("page", "1").
ParamQuery("offset", "5").
Do()
log.Printf("url: %s", url)
// url: http://stack.over.flow:1234/foo/abc/bar/xyz?page=1&offset=5
// --- Your own url generator would be like below --- //
package rest
import (
"log"
"net/url"
"strings"
"straas.io/base/errors"
"github.com/jinzhu/copier"
)
// URLGen generates request URL
type URLGen struct {
url.URL
pattern string
paramPath map[string]string
paramQuery map[string]string
}
// NewURLGen new a URLGen
func NewURLGen(scheme, host, port string) *URLGen {
h := host
if port != "" {
h += ":" + port
}
ug := URLGen{}
ug.Scheme = scheme
ug.Host = h
ug.paramPath = make(map[string]string)
ug.paramQuery = make(map[string]string)
return &ug
}
// Clone return copied self
func (u *URLGen) Clone() *URLGen {
cloned := &URLGen{}
cloned.paramPath = make(map[string]string)
cloned.paramQuery = make(map[string]string)
err := copier.Copy(cloned, u)
if err != nil {
log.Panic(err)
}
return cloned
}
// Pattern sets path pattern with placeholder (format `:<holder_name>`)
func (u *URLGen) Pattern(pattern string) *URLGen {
u.pattern = pattern
return u
}
// ParamPath builds path part of URL
func (u *URLGen) ParamPath(key, value string) *URLGen {
u.paramPath[key] = value
return u
}
// ParamQuery builds query part of URL
func (u *URLGen) ParamQuery(key, value string) *URLGen {
u.paramQuery[key] = value
return u
}
// Do returns final URL result.
// The result URL string is possible not escaped correctly.
// This is input for `gorequest`, `gorequest` will handle URL escape.
func (u *URLGen) Do() (string, error) {
err := u.buildPath()
if err != nil {
return "", err
}
u.buildQuery()
return u.String(), nil
}
func (u *URLGen) buildPath() error {
r := []string{}
p := strings.Split(u.pattern, "/")
for i := range p {
part := p[i]
if strings.Contains(part, ":") {
key := strings.TrimPrefix(p[i], ":")
if val, ok := u.paramPath[key]; ok {
r = append(r, val)
} else {
if i != len(p)-1 {
// if placeholder at the end of pattern, it could be not provided
return errors.Errorf("placeholder[%s] not provided", key)
}
}
continue
}
r = append(r, part)
}
u.Path = strings.Join(r, "/")
return nil
}
func (u *URLGen) buildQuery() {
q := u.URL.Query()
for k, v := range u.paramQuery {
q.Set(k, v)
}
u.RawQuery = q.Encode()
}
With net/http the following would trigger when calling localhost:8080/blob/123/test
http.HandleFunc("/blob/", yourHandlerFunction)
Then inside yourHandlerFunction, manually parse r.URL.Path to find 123.
Note that if you don't add a trailing / it won't work. The following would only trigger when calling localhost:8080/blob:
http.HandleFunc("/blob", yourHandlerFunction)
As of 19-Sep-22, with go version 1.19, instance of http.request URL has a method called Query, which will return a map, which is a parsed query string.
func helloHandler(res http.ResponseWriter, req *http.Request) {
// when request URL is `http://localhost:3000/?first=hello&second=world`
fmt.Println(req.URL.Query()) // outputs , map[second:[world] first:[hello]]
res.Write([]byte("Hello World Web"))
}
No way without standard library. Why you don't want to try some library? I think its not so hard to use it, just go get bla bla bla
I use Beego. Its MVC style.
how about a simple utility function ?
func withURLParams(u url.URL, param, val string) url.URL{
u.Path = strings.ReplaceAll(u.Path, param, val)
return u
}
you can use it like this:
u, err := url.Parse("http://localhost:8080/blob/:id/test")
if err != nil {
return nil, err
}
u := withURLParams(u, ":id","123")
// now u.String() is http://localhost:8080/blob/123/test
If you need a framework and you think it will be slow because it's 'bigger' than a router or net/http, then you 're wrong.
Iris is the fastest go web framework that you will ever find, so far according to all benchmarks.
Install by
go get gopkg.in/kataras/iris.v6
Django templates goes easy with iris:
import (
"gopkg.in/kataras/iris.v6"
"gopkg.in/kataras/iris.v6/adaptors/httprouter"
"gopkg.in/kataras/iris.v6/adaptors/view" // <-----
)
func main() {
app := iris.New()
app.Adapt(iris.DevLogger())
app.Adapt(httprouter.New()) // you can choose gorillamux too
app.Adapt(view.Django("./templates", ".html")) // <-----
// RESOURCE: http://127.0.0.1:8080/hi
// METHOD: "GET"
app.Get("/hi", hi)
app.Listen(":8080")
}
func hi(ctx *iris.Context){
ctx.Render("hi.html", iris.Map{"Name": "iris"})
}

Resources