What I'm looking to do, given a URL and take a screenshot of the website using Golang. I searched for results but I didn't get any. Can anyone please help me.
You can use a Go version of Selenium if you want to go that route. https://godoc.org/github.com/tebeka/selenium
There is no pure golang way to do at the moment this since it must involve a browser is some form.
The easiest path to achieve this functionality is probably:
Find a nice NodeJS library to take website screenshots
Create a NodeJS script that is suits your needs for taking screenshots (i/o and settings)
Execute this NodeJS script from Golang and handle the results in your Golang code
Not the cleanest method to get this done though - if you want it cleaner you probably have to build/find a golang package that controls a browser so you can skip the NodeJS middleman.
I solved this issue using https://github.com/mafredri/cdp and a Chrome headless docker container.
You can see my service example here: https://gist.github.com/efimovalex/9f9b815b0d5b1b7889a51d46860faf8a
A few more tools using Go and Chrome/Chromium include:
gowitness CLI app
screenshot library
web2image CLI app based on chromedp
I was writing a program for this specific task. Here is a sample code that browse google.com and takes a screenshot.
package main
import (
"time"
driver "github.com/dreygur/webdriver"
)
func main() {
url := `https://google.com`
driver.RunServer("./geckodriver")
driver.GetSession()
driver.Get(url)
time.Sleep(8 * time.Second)
driver.Screenshot("google")
time.Sleep(8 * time.Second)
defer driver.Kill()
}
To install the module, run go get github.com/dreygur/webdriver
You can use chromedp.
But you need install chrome browser!
Example :
package main
import (
"context"
"fmt"
"os"
"time"
"github.com/chromedp/chromedp"
)
func TackScreenShot(ctx context.Context, url string) ([]byte, error) {
context, cancel := chromedp.NewContext(ctx)
defer cancel()
var filebyte []byte
if err := chromedp.Run(context, chromedp.Tasks{
chromedp.Navigate(url),
chromedp.Sleep(3 * time.Second),
chromedp.CaptureScreenshot(&filebyte),
}); err != nil {
return nil, err
}
return filebyte, nil
}
func main() {
url := "https://google.com"
ctx := context.TODO()
data, err := TackScreenShot(ctx, url)
if err != nil {
panic(err)
}
defer ctx.Done()
pngFile, err := os.Create("./shot.png")
if err != nil {
panic(err)
}
defer pngFile.Close()
pngFile.Write(data)
fmt.Println("screen shot tacked!")
}
Related
I'm starting to take a look on Selenium with Go language, but I didn't find too much info.
I'm using github.com/tebeka/selenium.
In Python, I just install (pip install selenium) and code like this to open a browser:
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'./chromedriver.exe')
driver.get('http://www.hp.com')
How do I do the same in go?
I'm trying this, but it does not open the browser like Python does:
package main
import (
"fmt"
"github.com/tebeka/selenium"
)
func main() {
selenium.ChromeDriver("./chromedriver.exe")
caps := selenium.Capabilities{"browserName": "chrome"}
selenium.NewRemote(caps, fmt.Sprintf("http://www.google.com", 80))
}
Is there a simple way in go to just open the browser in my machine like that 3 Python lines do?
Thanks!
In python selenium, it automatically starts the browser. But Golang doesn't. You have to run the browser (service) explicitly. This is a simple example of using Chromedriver with Golang.
package main
import (
"github.com/tebeka/selenium"
"github.com/tebeka/selenium/chrome"
)
func main() error {
// Run Chrome browser
service, err := selenium.NewChromeDriverService("./chromedriver", 4444)
if err != nil {
panic(err)
}
defer service.Stop()
caps := selenium.Capabilities{}
caps.AddChrome(chrome.Capabilities{Args: []string{
"window-size=1920x1080",
"--no-sandbox",
"--disable-dev-shm-usage",
"disable-gpu",
// "--headless", // comment out this line to see the browser
}})
driver, err := selenium.NewRemote(caps, "")
if err != nil {
panic(err)
}
driver.Get("https://www.google.com")
}
I've recently shifted from python to golang. I had been using python to work with GCP.
I used to pass in the scopes and mention the discovery client I wanted to create like this :
def get_client(scopes, api, version="v1"):
service_account_json = os.environ.get("SERVICE_ACCOUNT_KEY_JSON", None)
if service_account_json is None:
sys.exit("Exiting !!! No SSH_KEY_SERVICE_ACCOUNT env var found.")
credentials = service_account.Credentials.from_service_account_info(
json.loads(b64decode(service_account_json)), scopes=scopes
)
return discovery.build(api, version, credentials=credentials, cache_discovery=False)
And this would create my desired discovery client, whether it be compute engine service or sqladmin
However in go I don't seem to find this.
I found this : https://pkg.go.dev/google.golang.org/api/discovery/v1
For any client that I want to create I would've to import that and then create that, like this :
https://cloud.google.com/resource-manager/reference/rest/v1/projects/list#examples
package main
import (
"fmt"
"log"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/api/cloudresourcemanager/v1"
)
func main() {
ctx := context.Background()
c, err := google.DefaultClient(ctx, cloudresourcemanager.CloudPlatformScope)
if err != nil {
log.Fatal(err)
}
cloudresourcemanagerService, err := cloudresourcemanager.New(c)
if err != nil {
log.Fatal(err)
}
req := cloudresourcemanagerService.Projects.List()
if err := req.Pages(ctx, func(page *cloudresourcemanager.ListProjectsResponse) error {
for _, project := range page.Projects {
// TODO: Change code below to process each `project` resource:
fmt.Printf("%#v\n", project)
}
return nil
}); err != nil {
log.Fatal(err)
}
}
So I've to import each client library to get the client for that.
"google.golang.org/api/cloudresourcemanager/v1"
There's no dynamic creation of it.
Is it even possible, cause go is strict type checking 🤔
Thanks.
No, this is not possible with the Golang Google Cloud library.
You've nailed the point on the strict type checking, as it would definitely defeat the benefits of compile time type checking. It would also be a bad Golang practice to return different objects with different signatures, as we don't do duck typing and instead we rely on interface contracts.
Golang is boring and verbose, and it's like that by design :)
I did make Scraping for Amazon Product Titles but Amazon captcha catches my scraper. I tried 10 times- go run main.go(8 times catches me - 2 times I scraped the product title)
I researched this but I did not find any solution for golang(there is just python) is there any solution for me?
package main
import (
"fmt"
"strings"0
"github.com/gocolly/colly"
)
func main() {
// Create a Collector specifically for Shopify
c := colly.NewCollector(
colly.AllowedDomains("www.amazon.com", "amazon.com"),
)
c.OnHTML("div", func(h *colly.HTMLElement) {
capctha := h.Text
title := h.ChildText("span#productTitle")
fmt.Println(strings.TrimSpace(title))
fmt.Println(strings.TrimSpace(capctha))
})
// Start the collector
c.Visit("https://www.amazon.com/Bluetooth-Over-Ear-Headphones-Foldable-Prolonged/dp/B07K5214NZ")
}
Output:
Enter the characters you see below Sorry, we just need to make sure
you're not a robot. For best results, please make sure your browser is
accepting cookies.
If you don't mind a different package, I wrote a package to search HTML
(essentially thin wrapper around github.com/tdewolff/parse):
package main
import (
"github.com/89z/parse/html"
"net/http"
"os"
)
func main() {
req, err := http.NewRequest(
"GET", "https://www.amazon.com/dp/B07K5214NZ", nil,
)
req.Header = http.Header{
"User-Agent": {"Mozilla"},
}
res, err := new(http.Transport).RoundTrip(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
lex := html.NewLexer(res.Body)
lex.NextAttr("id", "productTitle")
os.Stdout.Write(lex.Bytes())
}
Result:
Bluetooth Headphones Over-Ear, Zihnic Foldable Wireless and Wired Stereo
Headset Micro SD/TF, FM for Cell Phone,PC,Soft Earmuffs &Light Weight for
Prolonged Waring(Rose Gold)
https://github.com/89z/parse
I'm creating some trivial apps to learn Firestore.
I started the local Firestore Emulator with:
$ gcloud beta emulator firestore start
After starting the emulator, I ran tests with "go test"
I populated the Firestore with data and created a function that queried some of the records/documents added.
I deleted some of the Documents from my app but they continue to show up in Queries.
I tried:
stoping with ctrl-c and ctrl d
$ gcloud beta emulator firestore stop
restarted my Macbook but the Documents persist.
I don't understand how the datastore is persisting after restarting the computer, I'm guessing that the data is stored in a JSON file or something like that.
I searched but was not able to find any documentation on the emulator.
Am I supposed to start the emulator and then run tests against the emulated Firestore?
How do I flush the Firestore?
The emulator supports an endpoint to clear the database (docs):
curl -v -X DELETE "http://localhost:PORT/emulator/v1/projects/PROJECT_NAME/databases/(default)/documents"
Fill in the PORT and PROJECT_NAME.
Since you're using Go, here's a little test helper I implemented that helps with starting the emulator, waiting for it to come up, purging existing data, initializing a client, and shutting down the operator after it is completed.
It uses the technique in Juan's answer (which you should mark as the answer).
To use this utility, you need to just say:
client := startFirestoreEmulator(t)
Source code:
// Copyright 2021 Ahmet Alp Balkan
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package firestoretestutil contains test utilities for starting a firestore
// emulator locally for unit tests.
package firestoretestutil
import (
"bytes"
"context"
"fmt"
"net"
"net/http"
"os"
"os/exec"
"sync"
"testing"
"time"
firestore "cloud.google.com/go/firestore"
)
const firestoreEmulatorProj = "dummy-emulator-firestore-project"
// cBuffer is a buffer safe for concurrent use.
type cBuffer struct {
b bytes.Buffer
sync.Mutex
}
func (c *cBuffer) Write(p []byte) (n int, err error) {
c.Lock()
defer c.Unlock()
return c.b.Write(p)
}
func StartEmulator(t *testing.T, ctx context.Context) *firestore.Client {
t.Helper()
port := "8010"
addr := "localhost:" + port
ctx, cancel := context.WithCancel(ctx)
t.Cleanup(func() {
t.Log("shutting down firestore operator")
cancel()
})
// TODO investigate why there are still java processes hanging around
// despite we kill the exec'd command, suspecting /bin/bash wrapper that gcloud
// applies around the java process.
cmd := exec.CommandContext(ctx, "gcloud", "beta", "emulators", "firestore", "start", "--host-port="+addr)
out := &cBuffer{b: bytes.Buffer{}}
cmd.Stderr, cmd.Stdout = out, out
if err := cmd.Start(); err != nil {
t.Fatalf("failed to start firestore emulator: %v -- out:%s", err, out.b.String())
}
dialCtx, clean := context.WithTimeout(ctx, time.Second*10)
defer clean()
var connected bool
for !connected {
select {
case <-dialCtx.Done():
t.Fatalf("emulator did not come up timely: %v -- output: %s", dialCtx.Err(), out.b.String())
default:
c, err := (&net.Dialer{Timeout: time.Millisecond * 200}).DialContext(ctx, "tcp", addr)
if err == nil {
c.Close()
t.Log("firestore emulator started")
connected = true
break
}
time.Sleep(time.Millisecond * 200) //before retrying
}
}
os.Setenv("FIRESTORE_EMULATOR_HOST", addr)
cl, err := firestore.NewClient(ctx, firestoreEmulatorProj)
if err != nil {
t.Fatal(err)
}
os.Unsetenv("FIRESTORE_EMULATOR_HOST")
truncateDB(t, addr)
return cl
}
func truncateDB(t *testing.T, addr string) {
t.Helper()
// technique adopted from https://stackoverflow.com/a/58866194/54929
req, _ := http.NewRequest(http.MethodDelete, fmt.Sprintf("http://%s/emulator/v1/projects/%s/databases/(default)/documents",
addr, firestoreEmulatorProj), nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
if resp.StatusCode != http.StatusOK {
t.Fatalf("failed to clear db: %v", resp.Status)
}
}
You can use this:
module.exports.teardown = async () => {
Promise.all(firebase.apps().map(app => app.delete()));
};
Now, each time you call teardown, you will delete all data from the Firestore emulator.
I'm using GoLand IDE and I'm getting a problem when running my Go web app. The code isn't compiling when the Terminal is used.
Here is the problem: The terminal duplicated the command prompt when I make an attempt at running the code.
C:\Users\Evan\go\src\awesomeProject9>go run main.go
C:\Users\Evan\go\src\awesomeProject9>
package main
import (
"fmt"
"html/template"
"net/http"
)
var tpl *template.Template
func init(){
template.Must(template.ParseGlob("templates/*.html"))
}
func main() {
http.HandleFunc("templates/index", idx)
http.ListenAndServe("8000", nil)
fmt.Println("hello World")
}
func idx(w http.ResponseWriter, r *http.Request){
tpl.ExecuteTemplate(w, "templates/index.html", nil)
}
Thanks to #zerkms for pointing out, that I was wrong. I simply ran into the exact mistake I tried to warn you later on:
you really should use the err returned by called functions, since these really help you a lot! For startes simply:
err := http.ListenAndServe("8000", nil)
if err != nil {
log.Fatal(err)
}
This panics with:
2018/12/18 10:43:16 listen tcp: address 8000: missing port in address
the correct line should be
err := http.ListenAndServe(":8000", nil)
WRONG only for documentation
ListenAndServe doesn't block the further code execution....