I am trying login to my amazon buyer account for getting tracking info. I made wordpress-woocommerce login and getting infos but I could not for Amazon.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly"
)
func main() {
// create a new collector
c := colly.NewCollector()
login_link := "https://www.amazon.de/-/en/ap/signin?openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.de%2F%3Flanguage%3Den_GB%26ref_%3Dnav_signin&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=deflex&openid.mode=checkid_setup&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&"
// authenticate
err := c.Post(login_link, map[string]string{"username": "mail#example.com", "password": "123qwerty"})
if err != nil {
log.Fatal(err)
}
// attach callbacks after login
c.OnResponse(func(r *colly.Response) {
log.Println("response received", r.StatusCode) //response received 200
})
c.OnHTML("div", func(h *colly.HTMLElement) {
fmt.Println("PRINT ALL: ", h.Text)
})
// start scraping
c.Visit("https://www.amazon.de/-/en/gp/your-account/order-history?ref_=ya_d_c_yo")
}
Wordpress Login One Page - Amazon Login Two Page. We probably need to scroll 2 pages for Amazon
https://i.stack.imgur.com/4TNj5.png -> Wordpress Login (One Page)
https://i.stack.imgur.com/bhE4m.png -> Amazon Login(Page #1 - Mail)
https://i.stack.imgur.com/0BFcA.png -> Amazon Login(Page #1 - Password)
chromedp is a very useful library in such cases. You may try the following snipet;
package main
import (
"context"
"os"
"time"
"github.com/chromedp/chromedp"
)
func main() {
var res []byte
ctx, cancel := chromedp.NewContext(context.Background(), chromedp.WithBrowserOption())
defer cancel()
err := chromedp.Run(ctx,
chromedp.Navigate("https://www.amazon.com"),
chromedp.WaitReady("body"),
chromedp.Click(`a[data-nav-role="signin"]`, chromedp.ByQuery),
chromedp.Sleep(time.Second*2),
chromedp.SetValue(`ap_email`, "youramazonemail", chromedp.ByID),
chromedp.Click(`continue`, chromedp.ByID),
chromedp.Sleep(time.Second*1),
chromedp.SetValue(`ap_password`, "youramazonpassword", chromedp.ByID),
chromedp.Click(`signInSubmit`, chromedp.ByID),
chromedp.Sleep(time.Second*2),
chromedp.CaptureScreenshot(&res),
)
if err != nil {
log.Fatal(err)
}
os.WriteFile("loggedin.png", res, 0644)
}
The example given above is basically navigates trough all steps required for login process. After successful login, you can use context (ctx) to navigate and get the information whatever you want by using same function.
chromedp.Run(ctx,
chromedp.Navigate(url),
...)
Related
I managed to create a script that creates an email draft using Gmail in GO. I would like a solution to be able to attach a file to this draft.
p.s.: It needs to be a draft because someone else will review the email before sending it.
The code is here:
package main
import (
"bytes"
"log"
"os"
"time"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
func main() {
log.Println("Connecting to server...")
// Connect to server
c, err := client.DialTLS(os.Getenv("IMAP_SERVER"), nil)
if err != nil {
log.Fatal(err)
}
log.Println("Connected")
// Don't forget to logout
defer c.Logout()
// Login
if err := c.Login(os.Getenv("IMAP_USER"), os.Getenv("IMAP_PASSWORD")); err != nil {
log.Fatal(err)
}
log.Println("Logged in")
// Write the message to a buffer
var b bytes.Buffer
b.WriteString("From: <...#gmail.com>\r\n")
b.WriteString("To: <...#gmail.com>\r\n")
b.WriteString("Subject: Append test\r\n")
b.WriteString("\r\n")
// Message body
b.WriteString("Append test using Gmail IMAP and Draft folder")
// Append it to Drafts
if err := c.Append("[Gmail]/Drafts", nil, time.Now(), &b); err != nil {
log.Fatal(err)
}
My org is new to Google Auth and we have poured countless hours into documentation readings. The mission was simple: view members in our google groups through the Directory API.
Our setup: The cloud function deploys and runs with a service account that has been granted domain-wide access with the proper scopes, and impersonates an admin user detailed here:
https://developers.google.com/admin-sdk/directory/v1/guides/delegation
When I run the function locally and pull the service account key from a file path I get the error: "Error 403: Not Authorized to access this resource/api, forbidden"
I noticed that when deploying the Cloud Function via inline text or an uploaded zip it was unable to read a .json or .text file type when I included it in the package. I know this is bad practice but just to see I put in marshaled the JSON key in the main file.
And still got a "Error 403: Not Authorized to access this resource/api, forbidden"
Where am I going wrong?
import (
"encoding/json"
"fmt"
_"io/ioutil"
"log"
"net/http"
_ "os"
"time"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
admin "google.golang.org/api/admin/directory/v1"
"google.golang.org/api/option"
)
var User_email = <user>
func createAdminDirectoryService(serviceAccountFilePath, gsuiteAdminUserEmail string) *admin.Service {
jsonCredentials,_ := json.Marshal(map[string]string{<SERVICE KEY FILE>})
log.Println("Json creds: ", jsonCredentials)
config, err := google.JWTConfigFromJSON(
jsonCredentials,
"https://www.googleapis.com/auth/admin.directory.group.member.readonly",
)
if err != nil {
log.Printf("Json Config error:%v", err.Error())
}
config.Subject = gsuiteAdminUserEmail
fmt.Println(serviceAccountFilePath)//vestigial of previous version reading key from file
fmt.Println(gsuiteAdminUserEmail)
ctx, _ := context.WithDeadline(context.Background(), time.Now().Add(20*time.Second))
ts := config.TokenSource(ctx)
srv, err := admin.NewService(ctx, option.WithTokenSource(ts))
if err != nil {
log.Println("Admin Service error:", err.Error())
}
return srv
}
func listUsersInGroup(srv *admin.Service, groupEmail string) ([]string, error) {
membersEmails := make([]string, 1)
members, err := srv.Members.List(groupEmail).Do()
if err != nil {
log.Fatal("fatalerror list users: ", err)
membersEmails[0] = "Nope"
} else {
membersEmails := make([]string, len(members.Members))
for i, member := range members.Members {
membersEmails[i] = member.Email
}
}
return membersEmails, err
}
func Main(w http.ResponseWriter, r *http.Request) {
groupEmail := <groupemail>
path := "./key.json" //vestigial of previous version reading key from file
fmt.Println("Path:", path)
srv := createAdminDirectoryService(
path,
User_email,
)
members, err := listUsersInGroup(srv, groupEmail)
if err != nil {
log.Println(members)
} else {
log.Println("sorry bud")
}
}
I'm trying to identify that an alert popped up after navigating to a URL using chromedp. I tried using a listener as follows but I'm new to Golang so I'm not sure why it didn't work.
package main
import (
"context"
"log"
"fmt"
"github.com/chromedp/chromedp"
"github.com/chromedp/cdproto/page"
)
func main() {
// create context
url := "https://grey-acoustics.surge.sh/?__proto__[onload]=alert('hello')"
ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()
chromedp.ListenTarget(ctx, func(ev interface{}) {
if ev, ok := ev.(*page.EventJavascriptDialogOpening); ok {
fmt.Println("Got an alert: %s", ev.Message)
}
})
// run task list
err := chromedp.Run(ctx,
chromedp.Navigate(url),
)
if err != nil {
log.Fatal(err)
}
}
For your specific URL it helped to wait for the iframe to load to receive the event, otherwise chromedp seems to stop because it is finished with its task list.
// run task list
err := chromedp.Run(ctx,
chromedp.Navigate(url),
chromedp.WaitVisible("iframe"),
)
}
So I am new to go and I currently try to build a little REST-API using chi (and I love it). Yesterday I run into a problem, that I cannot quite understand.
In my little test-project I have a main.go file which contains the main function with router instantiation, adding middlewares and starting the server:
func main() {
router := chi.NewRouter()
// Middleware
router.Use(middleware.RequestID)
router.Use(middleware.RealIP)
router.Use(middleware.Logger)
router.Use(middleware.Recoverer)
// Routes
router.Post("/login", users.Login)
router.Post("/register", users.Register)
router.With(users.LoginRequired).Route("/users", func(r chi.Router) {
r.Get("/{user_id}", users.GetUser)
})
// Start Server
port := ":8080"
log.Printf("Server starting at port %v\n", port)
log.Fatal(http.ListenAndServe(port, router))
}
First the problem didn't exist because I defined all the handler functions within my main.go file and the GetUser-function worked as expected and returned a user from my "Database" (array with 3 users):
func GetUser(w http.ResponseWriter, r *http.Request) {
uID := chi.URLParam(r, "user_id") // Problem when not in main -> uID = ""
id, err := strconv.Atoi(uID)
if err != nil {
log.Printf("Error while parsing int: %v\n", err)
// TODO: return error 400
}
user := DataBase[id-1]
response, err := json.Marshal(user)
if err != nil {
log.Printf("Error while marshalling user: %v\n", err)
}
w.Write(response)
}
As soon as I moved this function out of the main.go file into another package called users the chi.URLParam function returns an empty string and cannot find the URLParam anymore. I read it has something to do with the context, but I cannot wrap my head around that I have to place functions inside the main-file if I want to use the chi functions.
Am I missing something?
UPDATE
As requested I removed everything except the GetUser function. My main.go file currently looks like this:
package main
import (
"encoding/json"
"log"
"net/http"
"strconv"
"github.com/MyUserName/MyProjectName/internals/users"
"github.com/go-chi/chi/v5"
)
func GetUser(w http.ResponseWriter, r *http.Request) {
id, err := strconv.Atoi(chi.URLParam(r, "user_id"))
if err != nil {
log.Printf("Error while parsing int: %v\n", err)
// TODO: return error 400
}
log.Printf("ID=%v, Current Database=%v\n", id, users.DataBase)
user := users.DataBase[id-1]
response, err := json.Marshal(user)
if err != nil {
log.Printf("Error while marshalling user: %v\n", err)
}
w.Write(response)
}
func main() {
router := chi.NewRouter()
// Routes
router.Get("/users/{user_id}", GetUser)
// Start Server
port := ":8080"
log.Printf("Server starting at port %v\n", port)
log.Fatal(http.ListenAndServe(port, router))
}
and my users package looks like this:
package users
import (
"encoding/json"
"log"
"net/http"
"strconv"
"github.com/MyUserName/MyProjectName/internals/models"
"github.com/go-chi/chi"
)
var (
DataBase = make([]models.User, 0)
)
func GetUser(w http.ResponseWriter, r *http.Request) {
id, err := strconv.Atoi(chi.URLParam(r, "user_id"))
if err != nil {
log.Printf("Error while parsing int: %v\n", err)
// TODO: return error 400
}
log.Printf("ID=%v, Current Database=%v\n", id, DataBase)
user := DataBase[id-1]
response, err := json.Marshal(user)
if err != nil {
log.Printf("Error while marshalling user: %v\n", err)
}
w.Write(response)
}
func init() {
initUser := []models.User{
{
ID: 1,
UserName: "john",
Password: "doe",
},
{
ID: 2,
UserName: "max",
Password: "mustermann",
},
{
ID: 3,
UserName: "jane",
Password: "doe",
},
}
for _, user := range initUser {
DataBase = append(DataBase, user)
}
log.Println("Initializing Database")
}
When I use the function from the users package it does not work and is still an empty string, if I use the function from the main.go file it works.
UPDATE
So apparently I am to stupid to import the same packages twice. In my main file I used "github.com/go-chi/chi/v5" and in my users package I used "github.com/go-chi/chi". Using the same resolved the issue, thanks a lot
Adding answer because the comments just saved me!
Check that all files in your go solution have the same version of chi in use. If you're using VSCode it may import a different version than you expect. In my code I had one file with
import(
"github.com/go-chi/chi"
)
and in the other
import(
"github.com/go-chi/chi/v5"
)
This meant that when I was calling into middleware function to extract URLParams the context was not finding a value.
TL;DR
Check that all files use same version of Chi!
I'm setting up an OAuth2 flow from a CLI application, I'm working on. I need to create a temporary HTTP server for the provider to send the callback to, e.g. localhost:8080/callback
Once the provider has sent the details I need, I want to be able to shut the HTTP server down, just to keep everything clean. I think what I'm looking for is Routines and Wait Groups, but I'm still quite new to this area.
This is what I have so far. I have redacted the part that sends the user to the provider, as my main issue is simply how to shut down the HTTP server once the token variable has been captured.
Server starts
User is directed to authorization URL at the provider site
User approves the request
Provider directs user back to localhost:8080/callback
URL includes client-side only params so I have to server HTML to use JS to capture the values and send it back to the server
Server receives token and can then shutdown
package main
import (
"fmt"
"log"
"net/http"
"sync"
)
func main() {
// Start local HTTP serevr to listen for response
serverDone := &sync.WaitGroup{}
serverDone.Add(1)
Start(serverDone)
// ... Process to start OAuth2 flow
// User is directed to provider website
// User approves
// Provider direct user back to localhost/callback
serverDone.Wait()
}
func Start(wg *sync.WaitGroup) {
srv := &http.Server{Addr: ":8080"}
http.HandleFunc("/callback", func(w http.ResponseWriter, r *http.Request) {
token := r.URL.Query().Get("token")
if token != "" {
fmt.Println("Found Token:", token)
// Shut down server here
} else {
// Server HTML page to fetch token and return to server at /callback
}
})
go func() {
// let main know we are done cleaning up
defer wg.Done()
// ErrServerClosed on graceful close
if err := srv.ListenAndServe(); err != http.ErrServerClosed {
log.Fatalf("ListenAndServe(): %v", err)
}
}()
}
Use:
var ctxShutdown, cancel = context.WithCancel(context.Background())
Then:
cancel() // to say sorry, above.
// graceful-shutdown
err := srv.Shutdown(context.Background())
Try this:
package main
import (
"context"
"fmt"
"log"
"net/http"
"sync"
)
func main() {
serverDone := &sync.WaitGroup{}
serverDone.Add(1)
Start(serverDone)
serverDone.Wait()
fmt.Println("Done that.")
}
var ctxShutdown, cancel = context.WithCancel(context.Background())
func Start(wg *sync.WaitGroup) {
srv := &http.Server{Addr: ":8080"}
http.HandleFunc("/callback", func(w http.ResponseWriter, r *http.Request) {
select {
case <-ctxShutdown.Done():
fmt.Println("Sorry: Shuting down ...")
return
default:
}
token := r.URL.Query().Get("token")
if token != "" {
fmt.Println("Found Token:", token)
fmt.Println("Shuting down ...")
// Shut down server here
cancel() // to say sorry, above.
// graceful-shutdown
err := srv.Shutdown(context.Background())
if err != nil {
log.Println("server.Shutdown:", err)
}
} else {
fmt.Fprintln(w, "Hi") // Server HTML page to fetch token and return to server at /callback
}
})
go func() {
defer wg.Done()
if err := srv.ListenAndServe(); err != http.ErrServerClosed {
log.Fatalf("ListenAndServe(): %v", err)
}
fmt.Println("Bye.")
}()
}
Run and open http://127.0.0.1:8080/callback?token=2
Output:
Found Token: 2
Shuting down ...
Bye.
Done that.