I'm using golang chromedp as headless webdriver and ccproxy as proxy server, according to this document i'm set credentials headers but chrome also show basic authentication popup.
chromeDP (set headers):
func (c *Browser) setHeaders() chromedp.Tasks {
authData := base64.StdEncoding.EncodeToString([]byte(c.Proxy.User + ":" + c.Proxy.Password))
headers := map[string]interface{}{
"Proxy-Authorization": "Basic " + authData,
}
return chromedp.Tasks{
network.Enable(),
network.SetExtraHTTPHeaders(headers),
}
}
...
apply:
if err := chromedp.Run(c.ctx,
c.setHeaders(),
chromedp.Navigate(c.NavigationUrl),
chromedp.Reload(),
); err != nil {
log.Println("error# ", err)
return c
}
Consider the examples repo which has an example of proxy authentication: https://github.com/chromedp/examples/blob/master/proxy/main.go.
Related
I am attempting to get a signed url from Google cloud where we can upload a document.
func GetSignedURL(bucketName string, objectName string) string {
ctx := context.Background()
// Get a connection to gcloud
client, err := storage.NewClient(ctx, option.WithCredentialsFile(config.GetSettings().GoogleCloudKey))
if err != nil {
msg := err.Error() + " # " + whereami.WhereAmI() + " on " + helpers.GetTimeInTimezone()
panic(msg)
}
defer client.Close()
opts := &storage.SignedURLOptions{
Scheme: storage.SigningSchemeV4,
Method: "PUT",
PrivateKey: []byte(getPrivateKey()), //<- a Google service account private key, obtainable from the Google Developers Console
Expires: time.Now().Add(15 * time.Minute),
Insecure: false,
}
//Use connection to get url
url, err := client.Bucket(bucketName).SignedURL(objectName, opts)
if err != nil {
msg := err.Error() + " # " + whereami.WhereAmI() + " on " + helpers.GetTimeInTimezone()
panic(msg)
}
return url
}
Per https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#storage-signed-url-object-go
I do sign the request with my Private key, I get a url similar to this:
https://storage.googleapis.com/yyy-vv-upload-xxx/aa41dcaed3a24f65b8d5a9ac94b4c0a6?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx-assets-gcs-yyy-devops%40yyy-devops.iam.gserviceaccount.com%2F20226667%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20221227T183246Z&X-Goog-Expires=899&X-Goog-Signature=260b2e695999ccdea5d5d5e46f0f83d204609dbfc0efcdf1fa8381b9ec748eacad36b525e695c6fc2fb02277a56e3c10674a75de2ba83c668c2052a89192345ddbdd0fe841ea28dc7df01648f2ca420b43d6fa162540e1c34865847d34c832ded157f2ec58bd8d57c47eef32d632ff02dc1927dc94f7509795e78c357dc556e2c96ea8e86dfb5015c796627f327caf04f45560bad4d5cdb714c5bc97d442cbc87639f44b2a102ce9d9a3fc83e81bcaf9fde38ce56efd02decd3fe8a1cf5fc823e46c85b39d7ada41618993e80b5c2e61a0380d403d0d6763310350ab1cf22d539c80fdaafc2320cbdeb3f1e7c21b4c40e3dc8889b348f67573852d5e7986167e&X-Goog-SignedHeaders=host
When I load it in a browser I get the following:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GOOG4-RSA-SHA256 20221227T183246Z 20221227/auto/storage/goog4_request 78f5677e7572233dc56657f7b055601eee26e7913bb6426194c888367c521990</StringToSign>
<CanonicalRequest>GET /yyy-vv-upload-xxx/aa41dcaed3a24f65b8d5a9ac94b4c0a6 X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx-assets-gcs-yyy-devops%40yyy-devops.iam.gserviceaccount.com%2F20221227%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20221227T183246Z&X-Goog-Expires=899&X-Goog-SignedHeaders=host host:storage.googleapis.com host UNSIGNED-PAYLOAD</CanonicalRequest>
</Error>
Any advice?
The error is completely misleading, it should have complained about the http method. Once I put the generated url in Postman and made a PUT request it worked. I could upload an image of a goldfish. The error above is generated for a GET request.
We have a Github repository that stores image files.
https://github.com/rollthecloudinc/ipe-objects/tree/dev/media
We would like to serve those image files via golang. The golang api runs on aws api gateway as a lambda function. The function in its current state which goes to a blank screen is below.
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Printf("requested media site: " + siteName)
log.Printf("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Text string
IsBinary bool
ByteSize int
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String("rollthecloudinc"),
"name": githubv4.String(siteName + suffix),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
// log.Printf(q.Repository.Object.ObjectFragment.Text)
// json.Unmarshal([]byte(q.Repository.Object.ObjectFragment.Text), &obj)
// log.Printf("END GithubFileUploadAdaptor::LOAD %s", id)
log.Print("content")
log.Print(q.Repository.Object.ObjectFragment.Text)
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = q.Repository.Object.ObjectFragment.Text //base64.StdEncoding.EncodeToString([]byte(q.Repository.Object.ObjectFragment.Text))
res.IsBase64Encoded = true
return res, nil
}
The full api file can viewed below but excludes the changes above for migration to Github. This api has been running fine using s3. However, we are now trying to migrate to Github for object storage instead. Have successfully implemented write but are having difficulties described above for reading the file using our lambda.
https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go
Help requested to figure out how to serve image files from our Github repo using the golang lambda on aws which can be accessed here as a blank screen.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
However, this repo is also a pages site which serves the image just fine.
https://rollthecloudinc.github.io/ipe-objects/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
Thanks
Further debugging the Text property appears to be empty inside the log.
The IsBinary property value being false lead use to the discovery of a typo. The name input for the graph QL invocation was missing -objects. Once the typo was corrected IsBinary started showing up true. However, the Text property value is still empty.
Having managed to find some similar issues but for uploading many have suggested that graph QL isn't the right tool for uploading binary data to begin with. Therefore, rather than chase tail we have decided to try the Github REST v3 api. Specifically, the go-github package for golang instead.
https://github.com/google/go-github
Perhaps using the REST api instead will lead to successful results.
An additional step was necessary to fetch the blob contents of the object queried via the graph QL api. Once this was achieved the media file was served with success. This required using the go-github blob api to fetch the blob base64 contents from github.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
GetMediaFile lambda
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Print("requested media site: " + siteName)
log.Print("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
owner := "rollthecloudinc"
repo := siteName + "-objects" + suffix
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Oid githubv4.GitObjectID
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String(owner),
"name": githubv4.String(repo),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
oid := q.Repository.Object.ObjectFragment.Oid
log.Print("Github file object id " + oid)
blob, _, err := ac.GithubRestClient.Git.GetBlob(context.Background(), owner, repo, string(oid))
if err != nil {
log.Print("Github get blob failure.")
log.Panic(err)
}
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = blob.GetContent()
res.IsBase64Encoded = true
return res, nil
}
Full Source: https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go
I'm building an API that scrapes some data off a webpage.
To do so, i need to send a GET request to a home page, scrape a 'RequestVerificationToken' from the HTML, then send another POST request to the same URL with a username, password, and the RequestVerificationToken.
I've been able to do this previously with Python:
session_requests = requests.session()
result = session_requests.get(LOGIN_URL)
parser = createBS4Parser(result.text)
return parser.find('input', attrs={'name': '__RequestVerificationToken'})["value"]
pageDOM = session_requests.post(
LOGIN_URL,
data=requestPayload, //RequestVerificationToken is in here
headers=requestHeaders
)
It seems like when i reuse the session_requests variable in Python, it's reusing the previous instance of the HTTP request.
However, when i try to do this in Go, I get an error due to an invalid token. I assume that this is because for the POST request, Go is using a new instance.
Is there any way I can get the same behavior from Go as I was with Python?
package main
import (
"fmt"
"log"
"github.com/gocolly/colly"
"github.com/gocolly/colly/proxy"
)
func main() {
//initiates the configuration
c := colly.NewCollector(colly.AllowURLRevisit())
//defining the proxy chain
revpro, err := proxy.RoundRobinProxySwitcher("socks5://127.0.0.1:9050", "socks5://127.0.0.1:9050")
if err != nil {
log.Fatal(err)
}
c.SetProxyFunc(revpro)
//parsing the required field from html we are extracting the csrf_token required for the login
c.OnHTML("form[role=form] input[type=hidden][name=CSRF_TOKEN]", func(e *colly.HTMLElement) {
csrftok := e.Attr("value")
fmt.Println(csrftok)
//posting the csrf value along with password
err := c.Post("https://www.something.com/login.jsp", map[string]string{"CSRF_TOKEN": csrftok, "username": "username", "password": "password"})
if err != nil {
log.Fatal(err)
}
return
})
//The website to visit
c.Visit("https://www.something.com/login.jsp")
//maintaining the connection using clone not initiating a callback request
d := c.Clone()
d.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Attr("href")
fmt.Printf("Link found: %q -> %s\n", e.Text, link)
})
d.Visit("https://skkskskskk.htm")
}
In Golang we can launch a browser window to open an URL using exec.Command method. An example can be found here
My question is how can we open that URL with a header?
If you're using Chrome, you could use Chrome DevTools Protocol to attach to a running Chrome instance and issue a command to navigate to a URL with specific headers.
First, Launch Chrome with Chrome Devtools Protocol enabled by using the flag --remote-debugging-port=9222
You'll get a response similar to DevTools listening on ws://127.0.0.1:9222/devtools/browser/2393d6e8-a85d-40a2-a79e-13f1585ff336
Pass that ws://... URL into the program below:
package main
import (
"context"
"flag"
"log"
"github.com/chromedp/cdproto/network"
"github.com/chromedp/chromedp"
)
func main() {
var devToolWsURL string
flag.StringVar(&devToolWsURL, "devtools-ws-url", "", "DevTools Websocket URL")
flag.Parse()
// Create contexts.
actxt, cancelActxt := chromedp.NewRemoteAllocator(context.Background(), devToolWsURL)
defer cancelActxt()
// Create new tab.
ctxt, _ := chromedp.NewContext(actxt)
// Custom header.
headers := map[string]interface{}{
"X-Header": "my request header",
}
task := chromedp.Tasks{
network.Enable(),
network.SetExtraHTTPHeaders(network.Headers(headers)),
chromedp.Navigate("http://google.com"),
}
// Run task.
err := chromedp.Run(ctxt, task)
if err != nil {
log.Fatal(err)
}
}
Notes:
9222 is the default port for this protocol but you can use any port
you want.
I didn't include the exec.Command code for brevity.
References:
Header example
Remote Chrome control example
UPDATE
Found a simpler way. You can just launch Chrome straight from chromedp by overriding the default headless option:
func main() {
// Create contexts.
opts := append(chromedp.DefaultExecAllocatorOptions[:], chromedp.Flag("headless", false))
actx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
ctx, cancel := chromedp.NewContext(actx)
// Call cancel() to close Chrome on some condition.
if false {
cancel()
}
// Custom header.
headers := map[string]interface{}{
"X-Header": "my request header",
}
task := chromedp.Tasks{
network.Enable(),
network.SetExtraHTTPHeaders(network.Headers(headers)),
chromedp.Navigate("http://tested.com"),
}
// Run task.
err := chromedp.Run(ctx, task)
if err != nil {
log.Fatal(err)
}
}
I'm starting to build a regular web app with golang and Angular2, and most importantly I'm trying to secure my login with the help of auth0.com. I download the quickstart code from here and try to run the code, it worked for a while and then next time I run it, the /tmp/session file cannot be found any more.
Here are some basic idea of the code auth0.com provides.
1. Initialize the gorilla sessions filesystemstore
2. Then start the authentification process
code is provided below
app.go
var (
Store *sessions.FilesystemStore
)
func Init() error {
Store = sessions.NewFilesystemStore("", []byte("something-very-secret"))
gob.Register(map[string]interface{}{})
return nil
}
login.go
func LoginHandler(w http.ResponseWriter, r *http.Request) {
domain := os.Getenv("AUTH0_DOMAIN")
aud := os.Getenv("AUTH0_AUDIENCE")
conf := &oauth2.Config{
ClientID: os.Getenv("AUTH0_CLIENT_ID"),
ClientSecret: os.Getenv("AUTH0_CLIENT_SECRET"),
RedirectURL: os.Getenv("AUTH0_CALLBACK_URL"),
Scopes: []string{"openid", "profile"},
Endpoint: oauth2.Endpoint{
AuthURL: "https://" + domain + "/authorize",
TokenURL: "https://" + domain + "/oauth/token",
},
}
if aud == "" {
aud = "https://" + domain + "/userinfo"
}
// Generate random state
b := make([]byte, 32)
rand.Read(b)
state := base64.StdEncoding.EncodeToString(b)
session, err := app.Store.Get(r, "state")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
session.Values["state"] = state
err = session.Save(r, w)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
audience := oauth2.SetAuthURLParam("audience", aud)
url := conf.AuthCodeURL(state, audience)
http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}
The Error log is
open /tmp/session_46CNLLHBKD... : no such file or directory
I try to understand the code and findout that error log comes from login.go line 39(session, err := app.Store.Get(r, "state")). And I started to track down the code and find out.
login.go:39 -->store.go: 180-->session.go:132-->store.go:186-->store.go:272
you can find store.go and session.go here.
The error log comes from this line: fdata, err := ioutil.ReadFile(filename)
Through the whole process I have not found any function call to save the session file.
I don't understand this error and I don't know why I can run the code at the very beginning, please help me with this problem.
Your any suggestion will be greatly appreciated.Thanks a lot.
I figure out the answer myself
It turns out that I changed my secret key while initializing the gorilla session filesystemstore, but I have not deleted my cookie file in chrome, so it cannot find the tmp sesiion file needed.
I change the key, then delete the coorsponding cookie file and everything is ok now.