How to reuse HTTP request instance in Go - go

I'm building an API that scrapes some data off a webpage.
To do so, i need to send a GET request to a home page, scrape a 'RequestVerificationToken' from the HTML, then send another POST request to the same URL with a username, password, and the RequestVerificationToken.
I've been able to do this previously with Python:
session_requests = requests.session()
result = session_requests.get(LOGIN_URL)
parser = createBS4Parser(result.text)
return parser.find('input', attrs={'name': '__RequestVerificationToken'})["value"]
pageDOM = session_requests.post(
LOGIN_URL,
data=requestPayload, //RequestVerificationToken is in here
headers=requestHeaders
)
It seems like when i reuse the session_requests variable in Python, it's reusing the previous instance of the HTTP request.
However, when i try to do this in Go, I get an error due to an invalid token. I assume that this is because for the POST request, Go is using a new instance.
Is there any way I can get the same behavior from Go as I was with Python?

package main
import (
"fmt"
"log"
"github.com/gocolly/colly"
"github.com/gocolly/colly/proxy"
)
func main() {
//initiates the configuration
c := colly.NewCollector(colly.AllowURLRevisit())
//defining the proxy chain
revpro, err := proxy.RoundRobinProxySwitcher("socks5://127.0.0.1:9050", "socks5://127.0.0.1:9050")
if err != nil {
log.Fatal(err)
}
c.SetProxyFunc(revpro)
//parsing the required field from html we are extracting the csrf_token required for the login
c.OnHTML("form[role=form] input[type=hidden][name=CSRF_TOKEN]", func(e *colly.HTMLElement) {
csrftok := e.Attr("value")
fmt.Println(csrftok)
//posting the csrf value along with password
err := c.Post("https://www.something.com/login.jsp", map[string]string{"CSRF_TOKEN": csrftok, "username": "username", "password": "password"})
if err != nil {
log.Fatal(err)
}
return
})
//The website to visit
c.Visit("https://www.something.com/login.jsp")
//maintaining the connection using clone not initiating a callback request
d := c.Clone()
d.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Attr("href")
fmt.Printf("Link found: %q -> %s\n", e.Text, link)
})
d.Visit("https://skkskskskk.htm")
}

Related

How to set scope for token in Google Play Developer Reporting API

I started to use Google Play Developer Reporting API with using Golang(https://pkg.go.dev/google.golang.org/api#v0.79.0/playdeveloperreporting/v1beta1), and faced with issue related to API scopes:
Code:
package main
import (
"context"
"encoding/json"
"fmt"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"golang.org/x/oauth2/jwt"
"google.golang.org/api/option"
"google.golang.org/api/playdeveloperreporting/v1beta1"
"io/ioutil"
)
const (
GoogleApplicationCredentials = "path_to_service_account_credentials"
ProjectID = "apps/{project_id}"
)
func main() {
tokenSource, err := getTokenSource(GoogleApplicationCredentials)
if err !=nil {
panic(err)
}
if err := getAnomalies(tokenSource, ProjectID); err != nil {
panic(err)
}
}
func getAnomalies(tokenSource oauth2.TokenSource, projectID string) error {
ctx := context.Background()
service, err := playdeveloperreporting.NewService(ctx, option.WithTokenSource(tokenSource))
if err != nil {
return err
}
anomaliesCall := service.Anomalies.List(projectID)
result, err := anomaliesCall.Do()
if err != nil {
return err
}
fmt.Printf("\nStatus: %d", result.HTTPStatusCode)
return nil
}
func getTokenSource(credentialFile string) (oauth2.TokenSource, error) {
ctx := context.Background()
b, err := ioutil.ReadFile(credentialFile)
if err != nil {
return nil, err
}
var c = struct {
Email string `json:"client_email"`
PrivateKey string `json:"private_key"`
}{}
if err := json.Unmarshal(b, &c); err != nil {
return nil, err
}
fmt.Printf("\nClient email: %s\n", c.Email)
config := &jwt.Config{
Email: c.Email,
PrivateKey: []byte(c.PrivateKey),
Scopes: []string{
"?????????",
},
TokenURL: google.JWTTokenURL,
}
return config.TokenSource(ctx), nil
}
My question is what scope I need to use? I didn't find in - https://developers.google.com/identity/protocols/oauth2/scopes
Thanks
When you access an api in this case Google Play Developer Reporting API most of the data is private user data. In order to access private user data your application needs the permissions of the owner of that data or someone with access.
To get that access we use Oauth2, not all methods are created equal depending upon which method within the Google Play Developer Reporting API you are trying to use will dictate which scope of access the user will need to grant you.
The easest way to tell which scope is to check the documentation
Lets look at the anomalies.list method for an example. If we scroll down to the bottom it tells you exactly which scope your user needs to authorize.
After a bit of checking i think there is only one scope for this api it is https://www.googleapis.com/auth/playdeveloperreporting. So if you request that scope then you should have access to the full api.
I had the same problem
scope
https://www.googleapis.com/auth/playdeveloperreporting
You are receiving this error either because your input OAuth2 scope name is invalid or it refers to a newer scope that is outside the domain of this legacy API.
This API was built at a time when the scope name format was not yet standardized. This is no longer the case and all valid scope names (both old and new) are catalogued at https://developers.google.com/identity/protocols/oauth2/scopes. Use that webpage to lookup (manually) the scope name associated with the API you are trying to call and use it to craft your OAuth2 request.

How do I generate a token with TTL 8h from the below golang function?

I am new in programming and have no idea about using the the token generate client api function in the source code from my client side golang program. Looking for some advice. Thank you so much.
Source code package: https://pkg.go.dev/github.com/gravitational/teleport/api/client#Client.UpsertToken
Function Source Code:
func (c *Client) UpsertToken(ctx context.Context, token types.ProvisionToken) error {
tokenV2, ok := token.(*types.ProvisionTokenV2)
if !ok {
return trace.BadParameter("invalid type %T", token)
}
_, err := c.grpc.UpsertToken(ctx, tokenV2, c.callOpts...)
return trail.FromGRPC(err)
}
My code:
package main
import (
"context"
"crypto/tls"
"fmt"
"log"
"os"
"strings"
"time"
"github.com/gravitational/teleport/api/client"
"github.com/gravitational/teleport/api/client/proto"
"google.golang.org/grpc"
)
// Client is a gRPC Client that connects to a Teleport Auth server either
// locally or over ssh through a Teleport web proxy or tunnel proxy.
//
// This client can be used to cover a variety of Teleport use cases,
// such as programmatically handling access requests, integrating
// with external tools, or dynamically configuring Teleport.
type Client struct {
// c contains configuration values for the client.
//c Config
// tlsConfig is the *tls.Config for a successfully connected client.
tlsConfig *tls.Config
// dialer is the ContextDialer for a successfully connected client.
//dialer ContextDialer
// conn is a grpc connection to the auth server.
conn *grpc.ClientConn
// grpc is the gRPC client specification for the auth server.
grpc proto.AuthServiceClient
// closedFlag is set to indicate that the connnection is closed.
// It's a pointer to allow the Client struct to be copied.
closedFlag *int32
// callOpts configure calls made by this client.
callOpts []grpc.CallOption
}
/*
type ProvisionToken interface {
Resource
// SetMetadata sets resource metatada
SetMetadata(meta Metadata)
// GetRoles returns a list of teleport roles
// that will be granted to the user of the token
// in the crendentials
GetRoles() SystemRoles
// SetRoles sets teleport roles
SetRoles(SystemRoles)
// GetAllowRules returns the list of allow rules
GetAllowRules() []*TokenRule
// GetAWSIIDTTL returns the TTL of EC2 IIDs
GetAWSIIDTTL() Duration
// V1 returns V1 version of the resource
V2() *ProvisionTokenSpecV2
// String returns user friendly representation of the resource
String() string
}
type ProvisionTokenSpecV2 struct {
// Roles is a list of roles associated with the token,
// that will be converted to metadata in the SSH and X509
// certificates issued to the user of the token
Roles []SystemRole `protobuf:"bytes,1,rep,name=Roles,proto3,casttype=SystemRole" json:"roles"`
Allow []*TokenRule `protobuf:"bytes,2,rep,name=allow,proto3" json:"allow,omitempty"`
AWSIIDTTL Duration `protobuf:"varint,3,opt,name=AWSIIDTTL,proto3,casttype=Duration" json:"aws_iid_ttl,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
*/
func main() {
ctx := context.Background()
args := os.Args[1:]
nodeType := ""
if len(args) > 0 {
nodeType = args[0]
}
proxyAddress := os.Getenv("TELEPORT_PROXY")
if len(proxyAddress) <= 0 {
proxyAddress = "proxy.teleport.example.local:443"
}
clt, err := client.New(ctx, client.Config{
Addrs: []string{
"proxy.teleport.example.local:443",
"proxy.teleport.example.local:3025",
"proxy.teleport.example.local:3024",
"proxy.teleport.example.local:3080",
},
Credentials: []client.Credentials{
client.LoadProfile("", ""),
},
})
if err != nil {
log.Fatalf("failed to create client: %v", err)
}
defer clt.Close()
ctx, err, token, err2 := clt.UpsertToken(ctx, token)
if err || err2 != nil {
log.Fatalf("failed to get tokens: %v", err)
}
now := time.Now()
t := 0
fmt.Printf("{\"tokens\": [")
for a, b := range token {
if strings.Contains(b.GetRoles(), b.Allow().String(), b.GetAWSIIDTTL(), nodeType) {
if t >= 1 {
fmt.Printf(",")
} else {
panic(err)
}
expiry := "never" //time.Now().Add(time.Hour * 8).Unix()
_ = expiry
if b.Expiry().Unix() > 0 {
exptime := b.Expiry().Format(time.RFC822)
expdur := b.Expiry().Sub(now).Round(time.Second)
expiry = fmt.Sprintf("%s (%s)", exptime, expdur.String())
}
fmt.Printf("\"count\": \"%1d\",", a)
fmt.Printf(b.Roles(), b.GetAllowRules(), b.GetAWSIIDTTL(), b.GetMetadata().Labels)
}
}
}
Output:
Syntax error instead of creating a token
It's seems your code have many mistake. And, It's very obvious you are getting syntax error. I am sure you would have got the line number in the console where actually these syntax error has occurred.
Please understand the syntax of Golang and also how to call the functions and how many parameter should i pass to those functions.
There are few mistakes i would like to point out after reviewing your code.
//It shouldn't be like this
ctx, err, token, err2 := clt.UpsertToken(ctx, token)
//Instead it should be like this
err := clt.UpsertToken(ctx, token)
//The return type of UpsertToken() method is error, you should use only one variable to receive this error.
strings.Contains() function takes two argument but you are passing four.
Refer this document for string.Contains()
You are assigning t := 0 and checking it with if condition inside for loop and never incremented.
Refer this document for fmt.Printf()
Refer this for function
Remove all the syntax error then only your code will run also cross check your logic.
If you want to see the example of syntax error then check here : https://go.dev/play/p/Hhu48UqlPRF

golang cgi + apache login session management

I am new to golang. I want to use golang as cgi for my apache server. I dont want to run my golang program as a listener/server. I write .go files for each action such as login.go, logout.go, welcome.go and compile them into individual exes. I rename the exes into cgi and put them in the cgi-bin folder of apache and post from my html file to cgi bin folder. I want my go lang to serve through apache.
This is my simple html form
<html>
<head><title>Go Test</title></head>
<body>
<form action='cgi-bin/login.cgi' method='post'>
username : <input name='username' value='vimal'><br>
password : <input name='password' value='hello'><br>
<input type='submit' value='submit'>
</form>
</body>
</html>
This is my golang source for login. I compile into login.exe rename it to login.cgi and put it in cgibin folder of apache.
package main
import (
"net/http/cgi"
"net/http"
"fmt"
)
var (
// key must be 16, 24 or 32 bytes long (AES-128, AES-192 or AES-256)
// key = []byte("super-secret-key")
key = uuid.NewV4().String()
store = sessions.NewCookieStore(key)
)
func errorResponse(code int, msg string) {
fmt.Printf("Status:%d %s\r\n", code, msg)
fmt.Printf("Content-Type: text/plain\r\n")
fmt.Printf("\r\n")
fmt.Printf("%s\r\n", msg)
}
func main() {
var req *http.Request
var w *http.ResponseWriter
var err error
req, err = cgi.Request()
if err != nil {
errorResponse(500, "cannot get cgi request" + err.Error())
return
}
if err := req.ParseForm(); err != nil {
errorResponse(500, "cannot get cgi request" + err.Error())
}
username := req.FormValue("username")
password := req.FormValue("password")
//create session
//store session variables like user id, user name etc., that can be accessed through other .cgi files like welcome.cgi
// Use req to handle request
fmt.Printf("Content-type: text/html\n\n")
fmt.Printf("<!DOCTYPE html>\n")
fmt.Printf("<p>username: %s\n",username)
fmt.Printf("<p>password: %s\n",password)
fmt.Printf("req=%v\r\n", req)
}
I need help to create a session and store session variables like user id, username etc.,
If my another golang cgi file welcome.cgi is called it should be able to access the session variables like user id, name and print them
Please provide me with complete code. I am new to golang.
I dont want to use my golang as server. I just want to make small cgi pages.
Thanks
You can use normal handlers like the ones used in several examples on the internet, that way you don't need to use a series of fmt.Printf to send data to the CGI server. One way that I found easy to work is by using the cgi.Serve function which can be used in conjunction with an http.HandlerFunc. Using http.HandlerFunc or http.Handler will allow you to use any session manager implementation in go that uses http.ResponseWriter and http.Request. Note however that you should not initialize your store with a randomly generated key from inside your program, and also, if you use a database back-end you will not be able to use connection polling directly from go, since your program main function will be executed on each request and it should open an close connections on each invocation.
Following is a simple example that uses the approach discussed above:
package main
import (
"encoding/hex"
"fmt"
"io"
"net/http"
"net/http/cgi"
"os"
"github.com/gorilla/sessions"
)
func main() {
// Use Serve function with an anonymous function, you can also use instances
// of http.Handler. The function does not need to be an anonymous function, it
// just has to match the HandlerFunc signature
err := cgi.Serve(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
// Get the session auth key from a trusted source,
// in this example an env var is used. You can use any mechanism you want
// so long that you don't generate it from within this program
key, _ := hex.DecodeString(os.Getenv("SESSION_AUTH_KEY"))
store := sessions.NewCookieStore(key)
// Get a session from the store if any
sess, err := store.Get(r, "mysess")
if err != nil {
io.WriteString(rw, fmt.Sprintf("error reading cookies: %v", err))
return
}
if sess.IsNew {
sess.Values["val"] = 1
//Save your store before writing anything to the client
if err := sess.Save(r, rw); err != nil {
io.WriteString(rw, "error writing sesion")
} else {
io.WriteString(rw, "new session")
}
return
}
//Write session values to validate that your session is working
io.WriteString(rw, "Values:")
for name, value := range sess.Values {
io.WriteString(rw, fmt.Sprintf("name: %s, val: %v", name, value))
}
}))
if err != nil {
fmt.Fprintf(os.Stderr, "Error :%v", err)
}
}

Unable to obtain cookies from cookiejar

I've constructed a simple script which goes on a website, and does a few things on there (automating a checkout process on lego.com)
This is all done using the go http client with a cookiejar on it, but when I try to print out the cookies after all the activity on the site, nothing prints - not sure if I'm doing something wrong here.
type Program struct {
taskInfo task.Task
client http.Client
jar *cookiejar.Jar
// Task Specific Variables
maxOrderQty string
sessionID string
sku string
shipMethodUID string
}
The HTTP client is initialized below
func (pr *Program) initializeClient() {
pr.jar, _ = cookiejar.New(nil)
pr.client = http.Client{
Timeout: time.Second * 10,
Jar: pr.jar,
}
}
After that, i make a few calls to the same domain (mixture of GET and POST) using said client. When I try to print out the cookies, nothing prints.
func (pr *Program) getSessionCookies() {
log.Debug("Getting Cookies")
u, _ := url.Parse("https://www.lego.com/")
for _, cookie := range pr.jar.Cookies(u) {
fmt.Printf(" %s: %s\n", cookie.Name, cookie.Value)
}
}
Go's http package support redirects by default, but the cookie jar must be refreshed, specially if the domain changes.
I spent a great deal of time debugging it some time ago, the solution I found was to specify a redirect handler:
client.CheckRedirect = getRedirectHandler(req, &client)
Which had the following code:
func getRedirectHandler(req *Request, client *http.Client) redirectHandler {
fn := func(request *http.Request, via []*http.Request) error {
if req.FollowRedirect {
log.Debug(fmt.Sprintf("Redirecting to %s", request.URL.String()))
jar, err := getCookieJarWithURL(request.URL.String(), req.Cookies)
if err != nil {
return err
}
client.Jar = jar
return nil
}
return errors.New("Redirect not allowed")
}
return fn
}
The full code can be found in this gist:
I hope that saves you some time :)

How to retrieve the final URL destination while using the http package in Go?

Most sites redirect to another URL during a request. For example: http://example.com might might redirects to http://mobile.example.com
Is there a way to retrieve the final destination? In case of cURL, they call this the effective URL.
For example,
package main
import (
"fmt"
"net/http"
)
func main() {
getURL := "http://pkgdoc.org/"
fmt.Println("getURL:", getURL)
resp, err := http.Get(getURL)
if err != nil {
fmt.Println(err)
return
}
finalURL := resp.Request.URL.String()
fmt.Println("finalURL:", finalURL)
}
Output:
getURL: http://pkgdoc.org/
finalURL: http://godoc.org/

Resources