I'm playing around with chromedp and been trying to replicate the functionality in puppeteer node.js but in golang.
I'm finding that the same JSON payload to chromium is causing an error when using chromedp
package main
import (
"context"
"io/ioutil"
"log"
"github.com/chromedp/cdproto/page"
"github.com/chromedp/chromedp"
)
func main() {
// create context
ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()
// capture pdf
var buf []byte
if err := chromedp.Run(ctx, printToPDF(`<html><body><h1>Yeeeew!</h1></body></html>`, &buf)); err != nil {
log.Fatal(err)
}
if err := ioutil.WriteFile("sample.pdf", buf, 0644); err != nil {
log.Fatal(err)
}
}
// https://github.com/puppeteer/puppeteer/blob/4d9dc8c0e613f22d4cdf237e8bd0b0da3c588edb/src/common/PDFOptions.ts#L74
// https://github.com/puppeteer/puppeteer/blob/4d9dc8c0e613f22d4cdf237e8bd0b0da3c588edb/src/common/Page.ts#L3366
//https://github.com/chromedp/chromedp/issues/836
func printToPDF(html string, res *[]byte) chromedp.Tasks {
return chromedp.Tasks{
chromedp.Navigate("about:blank"),
chromedp.ActionFunc(func(ctx context.Context) error {
frameTree, err := page.GetFrameTree().Do(ctx)
if err != nil {
return err
}
return page.SetDocumentContent(frameTree.Frame.ID, html).Do(ctx)
}),
chromedp.ActionFunc(func(ctx context.Context) error {
buf, _, err := page.PrintToPDF().
// WithPrintBackground(false).
WithMarginTop(20.0).
// WithMarginLeft(20).
// WithMarginBottom(20.0).
// WithMarginRight(20).
Do(ctx)
if err != nil {
return err
}
*res = buf
return nil
}),
}
}
I've vendored the modules and edited cdproto/page/page.go to print out the JSON being sent to chromium
{"marginTop":20,"marginBottom":0,"marginLeft":0,"marginRight":0,"transferMode":"ReturnAsStream"}
I've also done this in node.js and logged out the json to compare
node index.js
PDF command: 'Page.printToPDF' {
transferMode: 'ReturnAsStream',
marginTop: 20,
marginBottom: 20,
marginLeft: 0,
marginRight: 0
}
I'm not sure why I'm getting this error? Any ideas?
TL;DR
The margin value is too big. I think you meant to pass 20.0/96 inches:
- WithMarginTop(20.0).
+ WithMarginTop(20.0/96).
Explanation
The error message returned from chromium is misleading. I guess here is what happened: the provided margin settings is invalid, and since chromium is running in headless mode, it can not show the error dialog. The error message can be interpreted as "I can not show a dialog to alert the user that the provided printer settings are invalid".
"marginTop":20 in the raw CDP message means 20 inches on the top, which is too big for an A4 page (A4: 8.27in x 11.7in). Please note that a number in puppeteer is treated as pixels and will be converted to inches before sending to chromium (see https://github.com/puppeteer/puppeteer/blob/4d9dc8c0e613f22d4cdf237e8bd0b0da3c588edb/src/common/Page.ts#L3366-L3402). So the fix is obvious.
BTW, there are easy ways to see the raw CDP messages:
chromedp: use the chromedp.WithDebugf option:
ctx, cancel = chromedp.NewContext(context.Background(), chromedp.WithDebugf(log.Printf))`
puppeteer: env DEBUG="puppeteer:*" node script.js. See https://github.com/puppeteer/puppeteer#debugging-tips
Related
I am calling this code:
package multicall
import (
"fmt"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/ethclient"
"github.com/ethereum/go-ethereum/accounts/abi/bind"
)
var multicallContractAddress = common.HexToAddress("0x5e227AD1969Ea493B43F840cfF78d08a6fc17796")
var multicallContractEthBalanceSelector = "4d2301cc"
func GetBalances(addresses []string, ETHProviderURL string) ([]string, error) {
ethProvider, err := ethclient.Dial(ETHProviderURL)
if err != nil {
panic(err)
}
multicallContract, err := NewMulticallCaller(multicallContractAddress, ethProvider)
if err != nil {
panic(err)
}
fmt.Println("multicallContract: ", multicallContract)
var calls = []MulticallCall{}
for _, address := range addresses {
hashAddress := common.HexToHash(address)
call := MulticallCall{multicallContractAddress, []byte("0x"+multicallContractEthBalanceSelector+hashAddress.String()[2:])}
calls = append(calls, call)
}
fmt.Println(string(calls[0].CallData))
var results []byte
err = multicallContract.contract.Call(&bind.CallOpts{}, results, "aggregate", common.Hex2Bytes(addresses[0]))
if err != nil {
panic(err)
}
fmt.Println("Result: ", &results)
return nil, nil
}
like this:
multicall.GetBalances([]string{"<MY_ADDRESS>"}, "<INFURA_API_KEY>")
And it returns this error:
# github.com/mteam88/keyswarm/multicall
multicall/multicall.go:37:58: cannot use results (variable of type []byte) as type *[]interface{} in argument to multicallContract.contract.Call
It seems like results should be a specific type/interface (maybe []uint8) but I can't seem to pass it like that to the Call function.
I have tried tons of things and I can't seem to get it. My Google diving has uncovered this Why am I getting a compile error 'cannot use ... as type uint8 in argument to ...' when the parameter is an int which led me to my previous assumption.
If you -1 please comment (:
Please provide links to relevant documentation in your answer.
NOTE: https://github.com/mteam88/keyswarm/blob/multicall/multicall/multicallContract.go is the location of the defenitions of MulticallCall and other Multicall prefixed defenitions.
I've a go application that gets run periodically by a batch. Each run, it should read some prometheus metrics from a file, run its logic, update a success/fail counter, and write metrics back out to a file.
From looking at How to parse Prometheus data as well as the godocs for prometheus, I'm able to read in the file, but I don't know how to update app_processed_total with the value returned by expfmt.ExtractSamples().
This is what I've done so far. Could someone please tell me how should I proceed from here? How can I typecast the Vector I got into a CounterVec?
package main
import (
"fmt"
"net/http"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/prometheus/common/model"
)
var (
fileOnDisk = prometheus.NewRegistry()
processedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "app_processed_total",
Help: "Number of times ran",
}, []string{"status"})
)
func doInit() {
prometheus.MustRegister(processedTotal)
}
func recordMetrics() {
go func() {
for {
processedTotal.With(prometheus.Labels{"status": "ok"}).Inc()
time.Sleep(5 * time.Second)
}
}()
}
func readExistingMetrics() {
var parser expfmt.TextParser
text := `
# HELP app_processed_total Number of times ran
# TYPE app_processed_total counter
app_processed_total{status="ok"} 300
`
parseText := func() ([]*dto.MetricFamily, error) {
parsed, err := parser.TextToMetricFamilies(strings.NewReader(text))
if err != nil {
return nil, err
}
var result []*dto.MetricFamily
for _, mf := range parsed {
result = append(result, mf)
}
return result, nil
}
gatherers := prometheus.Gatherers{
fileOnDisk,
prometheus.GathererFunc(parseText),
}
gathering, err := gatherers.Gather()
if err != nil {
fmt.Println(err)
}
fmt.Println("gathering: ", gathering)
for _, g := range gathering {
vector, err := expfmt.ExtractSamples(&expfmt.DecodeOptions{
Timestamp: model.Now(),
}, g)
fmt.Println("vector: ", vector)
if err != nil {
fmt.Println(err)
}
// How can I update processedTotal with this new value?
}
}
func main() {
doInit()
readExistingMetrics()
recordMetrics()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe("localhost:2112", nil)
}
I believe you would need to use processedTotal.WithLabelValues("ok").Inc() or something similar to that.
The more complete example is here
func ExampleCounterVec() {
httpReqs := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
},
[]string{"code", "method"},
)
prometheus.MustRegister(httpReqs)
httpReqs.WithLabelValues("404", "POST").Add(42)
// If you have to access the same set of labels very frequently, it
// might be good to retrieve the metric only once and keep a handle to
// it. But beware of deletion of that metric, see below!
m := httpReqs.WithLabelValues("200", "GET")
for i := 0; i < 1000000; i++ {
m.Inc()
}
// Delete a metric from the vector. If you have previously kept a handle
// to that metric (as above), future updates via that handle will go
// unseen (even if you re-create a metric with the same label set
// later).
httpReqs.DeleteLabelValues("200", "GET")
// Same thing with the more verbose Labels syntax.
httpReqs.Delete(prometheus.Labels{"method": "GET", "code": "200"})
}
This is taken from the Promethus examples on Github
To use the value of vector you can do the following:
vectorFloat, err := strconv.ParseFloat(vector[0].Value.String(), 64)
if err != nil {
panic(err)
}
processedTotal.WithLabelValues("ok").Add(vectorFloat)
This is assuming you will only ever get a single vector value in your response. The value of the vector is stored as a string but you can convert it to a float with the strconv.ParseFloat method.
I have a program with multiple multiple loops, each one ran in a Goroutine. I need to plug or unplug monitors while my program runs, so I have to restart the sdl to let it find my new monitors, I do it by sdl.quit() to quit the last sdl and sdl.init(sdl.InitEverything) to initialize it again.
My problem is that I have to handle the sdl events in a loop, If I don't it will become unresponsive, but this loop will block my Main code. I don't need to handle any events like mouse clicks, I just want to show some simple picture and manipulate them, is there any way to stop events or run the whole sdl in a goroutine? I tried but get weird results. This is my struct:
type SDLstruct{
window *sdl.Window
renderer *sdl.Renderer
texture *sdl.Texture
src, dst sdl.Rect
event sdl.Event
Close bool
winWidth, winHeight int32
}
This function starts a window and a renderer:
func (sdlVars *SDLstruct)StartWindow() (err error) {
sdlVars.winWidth, sdlVars.winHeight = 1920,1080
sdl.Init(sdl.INIT_VIDEO)
Num,err :=sdl.GetNumVideoDisplays()
if err!=nil {
return err
}
var rect sdl.Rect
rect,err = sdl.GetDisplayBounds(0)
if err!=nil {
return err
}
for i:=Num-1;i>=0;i--{
Mod , _:=sdl.GetDisplayMode(i,0)
if Mod.W ==info.winWidth && Mod.H==info.winHeight{
rect,err = sdl.GetDisplayBounds(i)
if err!=nil {
return err
}
break
}
}
sdlVars.window, err = sdl.CreateWindow(winTitle, rect.X, rect.Y,
rect.W, rect.H, sdl.WINDOW_SHOWN)
sdlVars.window.SetBordered(false)
sdlVars.window.SetFullscreen(1)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create window: %s\n", err)
return err
}
sdlVars.renderer, err = sdl.CreateRenderer(sdlVars.window, -1, sdl.RENDERER_ACCELERATED)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create renderer: %s\n", err)
return err
}
sdlVars.renderer.Clear()
sdlVars.renderer.SetDrawColor(0, 0, 0, 255)
sdlVars.renderer.FillRect(&sdl.Rect{0, 0, int32(info.winWidth), int32(info.winHeight)})
sdlVars.renderer.Present()
sdl.ShowCursor(0)
return nil
}
This function handle events:
func (sdlVars *SDLstruct)HandleEvents() {
for sdlVars.event = sdl.PollEvent(); sdlVars.event != nil; sdlVars.event = sdl.PollEvent() {
switch t := sdlVars.event.(type) {
case *sdl.QuitEvent:
sdlVars.Close = true
}
}
}
and this one closes everything:
func (sdlVars *SDLstruct)CloseSDL() (err error) {
err =sdlVars.renderer.Destroy()
if err!=nil{
return err
}
err = sdlVars.window.Destroy()
if err!=nil{
return err
}
sdl.Quit()
return nil
}
This is my Main function:
func main() {
var wg sync.WaitGroup
SdlVars:= new(SDLstruct)
wg.Add(1)
go SdlVars.StartSDL()
time.Sleep(time.Second*5)
SdlVars.Close = true
time.Sleep(time.Second*15)
}
In my Main function I tell it to start the sdl and after 5 seconds close everything and wait 15 seconds, but it doesn't close window.
Be careful as using SDL from multiple threads is not entirely trivial. Typically, your SDL loop must sit in the main thread. There is a good reason for that and it's because the event loop is part of the main thread and many of your objects are created and mutated on the main thread...or at least they should be.
When you introduce multiple threads weird undefined and dangerous behavior could happen. This is why a good rule of thumb is to use runtime.LockOSThread to ensure the Go runtime doesn't pin your main goroutine to other threads and keeps it on the main thread.
See this for more details: enter link description here
I have some code (see below) written in Go which is supposed to "fan-out" HTTP requests, and collate/aggregate the details back.
I'm new to golang and so expect me to be a nOOb and my knowledge to be limited
The output of the program is currently something like:
{
"Status":"success",
"Components":[
{"Id":"foo","Status":200,"Body":"..."},
{"Id":"bar","Status":200,"Body":"..."},
{"Id":"baz","Status":404,"Body":"..."},
...
]
}
There is a local server running that is purposely slow (sleeps for 5 seconds and then returns a response). But I have other sites listed (see code below) that sometime trigger an error as well (if they error, then that's fine).
The problem I have at the moment is how best to handle these errors, and specifically the "timeout" related errors; in that I'm not sure how to recognise if a failure is a timeout or some other error?
At the moment I get a blanket error back all the time:
Get http://localhost:8080/pugs: read tcp 127.0.0.1:8080: use of closed network connection
Where http://localhost:8080/pugs will generally be the url that failed (hopefully by timeout!). But as you can see from the code (below), I'm not sure how to determine the error code is related to a timeout nor how to access the status code of the response (I'm currently just blanket setting it to 404 but obviously that's not right - if the server was to error I'd expect something like a 500 status code and obviously I'd like to reflect that in the aggregated response I send back).
The full code can be seen below. Any help appreciated.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"sync"
"time"
)
type Component struct {
Id string `json:"id"`
Url string `json:"url"`
}
type ComponentsList struct {
Components []Component `json:"components"`
}
type ComponentResponse struct {
Id string
Status int
Body string
}
type Result struct {
Status string
Components []ComponentResponse
}
var overallStatus string = "success"
func main() {
var cr []ComponentResponse
var c ComponentsList
b := []byte(`{"components":[{"id":"local","url":"http://localhost:8080/pugs"},{"id":"google","url":"http://google.com/"},{"id":"integralist","url":"http://integralist.co.uk/"},{"id":"sloooow","url":"http://stevesouders.com/cuzillion/?c0=hj1hfff30_5_f&t=1439194716962"}]}`)
json.Unmarshal(b, &c)
var wg sync.WaitGroup
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
for i, v := range c.Components {
wg.Add(1)
go func(i int, v Component) {
defer wg.Done()
resp, err := client.Get(v.Url)
if err != nil {
fmt.Printf("Problem getting the response: %s\n", err)
cr = append(cr, ComponentResponse{
v.Id,
404,
err.Error(),
})
} else {
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("Problem reading the body: %s\n", err)
}
cr = append(cr, ComponentResponse{
v.Id,
resp.StatusCode,
string(contents),
})
}
}(i, v)
}
wg.Wait()
j, err := json.Marshal(Result{overallStatus, cr})
if err != nil {
fmt.Printf("Problem converting to JSON: %s\n", err)
return
}
fmt.Println(string(j))
}
If you want to fan out then aggregate results and you want specific timeout behavior the net/http package isn't giving you, then you may want to use goroutines and channels.
I just watched this video today and it will walk you through exactly those scenarios using the concurrency features of Go. Plus, the speaker Rob Pike is quite the authority -- he explains it much better than I could.
https://www.youtube.com/watch?v=f6kdp27TYZs
I am adding this for completes, as the correct answer was provided by Dave C in the comments of the accepted answer.
We can try to cast the error to a net.Error and check if it is a timeout.
resp, err := client.Get(url)
if err != nil {
// if there is an error check if its a timeout error
if e, ok := err.(net.Error); ok && e.Timeout() {
// handle timeout
return
}
// otherwise handle other types of error
}
The Go 1.5 release solved this issue by being more specific about the type of error it has handled.
So if you see this example https://github.com/Integralist/Go-Requester/blob/master/requester.go#L38 you'll see that I'm able to apply a regex pattern to the error message to decipher if the error was indeed a timeout or not
status := checkError(err.Error())
func checkError(msg string) int {
timeout, _ := regexp.MatchString("Timeout", msg)
if timeout {
return 408
}
return 500
}
package main
import (
"bytes"
"code.google.com/p/go.net/html"
"fmt"
"log"
"strings"
)
func main() {
s := "Blah. <b>Blah.</b> Blah."
n, err := html.Parse(strings.NewReader(s))
if err != nil {
log.Fatalf("Parse error: %s", err)
}
var buf bytes.Buffer
if err := html.Render(&buf, n); err != nil {
log.Fatalf("Render error: %s", err)
}
fmt.Println(buf.String())
}
Output:
<html><head></head><body>Blah. <b>Blah.</b> Blah.</body></html>
Is there a way to stop html.Parse from making a document out of fragments (ie avoid adding <html>, <body> etc.)? I'm aware of html.ParseFragment but it seems to exhibit the same behaviour.
You can get around it by wrapping the text to be parsed with a parent element such as <span> then doing something like the following:
n = n.FirstChild.LastChild.FirstChild
but that seems, well, kludgy to say the least.
Ideally I'd like to: accept input, manipulate or remove nodes found within it, and write the result back to a string, even if the result is an incomplete document.
You need to provide a context to ParseFragment. The following program prints out the original text:
package main
import (
"bytes"
"code.google.com/p/go.net/html"
"code.google.com/p/go.net/html/atom"
"fmt"
"log"
"strings"
)
func main() {
s := "Blah. <b>Blah.</b> Blah."
n, err := html.ParseFragment(strings.NewReader(s), &html.Node{
Type: html.ElementNode,
Data: "body",
DataAtom: atom.Body,
})
if err != nil {
log.Fatalf("Parse error: %s", err)
}
var buf bytes.Buffer
for _, node := range n {
if err := html.Render(&buf, node); err != nil {
log.Fatalf("Render error: %s", err)
}
}
fmt.Println(buf.String())
}
You want http://godoc.org/code.google.com/p/go.net/html#ParseFragment. Pass in a fake Body element as your context and the fragment will be returned as a slice of just the elements in your fragment.
You can see an example in the Partial* functions for go-html-transform's go.net/html wrapper package. https://code.google.com/p/go-html-transform/source/browse/h5/h5.go#32