For Google Cloud Functions triggered from HTTP, it is possible to retrieve the execution id by inspecting the headers of the HTTP request ("Function-Execution-Id") :
package p
import (
"fmt"
"net/http"
)
func F(w http.ResponseWriter, r *http.Request) {
executionID := r.Header.Get("Function-Execution-Id")
fmt.Println(executionID)
}
However, for GCF triggered by PubSub events, I can't find how to retrieve this execution ID :
package p
import (
"context"
)
type PubSubMessage struct {
Data []byte `json:"data"`
}
func F(ctx context.Context, m PubSubMessage) error {
executionID := "" // ???
fmt.Println(executionID)
return nil
}
I have looked into the PubSubMessage (https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage), but it only contains data + an empty attributes map.
I have also checked if execution ID is in the metadata handled by the context. However, from my tests, and the docs (https://godoc.org/cloud.google.com/go/functions/metadata#FromContext), only EventID, Timestamp, EventType and Resource are present.
How can I retrieve the execution id of a GCF function triggered by a PubSub event?
No, it is currently not possible to get execution id from pubsub triggered events.
You can get the event id from context as Lauren stated but that does not match execution id.
Also, pubsub triggered events do have execution ids. You can see this by using a default logger to log the event id. In stackdriver there will be an attached execution id label and it will not match the event id. We have observed that event id is numeric while execution id is alphanumeric.
Further, if the function is retried, it will keep the same event id but get a different execution id.
This is a recent (undocumented) change but can be easily observed.
EDIT: This appears to no longer be accurate. See the other answer from ProGirlXOXO.
A Pub/Sub-triggered event does not have an execution ID; instead it has an EventID contained in the context metadata, which is a unique ID for the event.
You can access the EventID as follows:
import (
"context"
"log"
"cloud.google.com/go/functions/metadata"
)
func F(ctx context.Context, m PubSubMessage) error {
ctxMetadata, err := metadata.FromContext(ctx)
if err != nil {
log.Fatal(err);
}
log.Println("EventID: " + ctxMetadata.EventID)
return nil
}
Related
What is the default mode in which network requests are executed in GoColly? Since we have the Async method in the collector I would assume that the default mode is synchronous.
However, I see no particular difference when I execute these 8 requests in the program other than I need to use Wait for async mode. It seems as if the method only controls how the program is executed (the other code) and the requests are always asynchronous.
package main
import (
"fmt"
"github.com/gocolly/colly/v2"
)
func main() {
urls := []string{
"http://webcode.me",
"https://example.com",
"http://httpbin.org",
"https://www.perl.org",
"https://www.php.net",
"https://www.python.org",
"https://code.visualstudio.com",
"https://clojure.org",
}
c := colly.NewCollector(
colly.Async(true),
)
c.OnHTML("title", func(e *colly.HTMLElement) {
fmt.Println(e.Text)
})
for _, url := range urls {
c.Visit(url)
}
c.Wait()
}
The default collection is synchronous.
The confusing bit is probably the collector option colly.Async() which ignores the actual param. In fact the implementation at the time of writing is:
func Async(a ...bool) CollectorOption {
return func(c *Collector) {
c.Async = true // uh-oh...!
}
}
Based on this issue, it was done this way for backwards compatibility, so that (I believe) you can pass an option with no param at it'll still work, e.g.:
colly.NewCollector(colly.Async()) // no param, async collection
If you remove the async option altogether and instantiate with just colly.NewCollector(), the network requests will be clearly sequential — i.e. you can also remove c.Wait() and the program won't exit right away.
I'm trying to access messageId of the Pub/Sub message triggering my Golang function. To do so, I'm trying to modify the PubSubMessage struct from the Cloud Functions documentation:
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
MessageId string `json:"messageId"`
}
The function compiles OK but the MessageID value comes empty. Changing the type doesn't help.
I wonder if there's a way to get the triggering message Id from within a function. Or maybe that's not passed to functions at all?
In the document you refer,
Event structure
Cloud Functions triggered from a Pub/Sub topic will be
sent events conforming to the PubsubMessage type, with the caveat that
publishTime and messageId are not directly available in the
PubsubMessage. Instead, you can access publishTime and messageId via
the event ID and timestamp properties of the event metadata. This
metadata is accessible via the context object that is passed to your
function when it is invoked.
You can get messageId like this.
import "cloud.google.com/go/functions/metadata"
func YourFunc(ctx context.Context, m PubSubMessage) error {
metadata, err := metadata.FromContext(ctx)
if err != nil {
// Handle Error
}
messageId := metadata.EventID
// Rest of your code below here.
}
I want to automate OTP submission on a bank page. I will get the OTP in my database only after webdriver have clicked confirm on the bank page. After confirming, I need to fetch OTP from the db and then automate OTP submission.
ctx, cancel := chromedp.NewContext(context.Background(), chromedp.WithDebugf(log.Printf))
defer cancel()
// run chromedp tasks
err := chromedp.Run(ctx,
chromedp.Navigate(bankUrl),
chromedp.WaitVisible(`#username`),
chromedp.SendKeys(`#username`, `usernameXXX`),
chromedp.WaitVisible(`#label2`, ),
chromedp.SendKeys(`#label2`, `passwordxxx` ),
chromedp.Click(`//input[#title="Login"]`),
chromedp.WaitVisible(`#Go`),
chromedp.Click(`#Go`),
chromedp.WaitVisible(`#confirmButton`),
chromedp.Click(`#confirmButton`),
chromedp.WaitVisible(`//input[#type="password"]`),
// perform fetch OTP below, this raise error
otp := fetchOTPFromDb()
chromedp.SendKeys(`//input[#type="password"]`, otp),
chromedp.WaitVisible(`#confirmButton`),
chromedp.Click(`#confirmButton`))
if err != nil {
log.Fatal(err)
}
The problem is that chromedp.Run expects all args to be of the type chromedp.Tasks, so I can't call custom functions there and I get error while fetching the OTP from db. How do I go around this?
The solution is to wrap the otp fetch within an Action.Do call, then return the result of a call to chromdp.SendKeys to set the HTML input value.
It is required to work that way because the One Time Password does not exist until the page was fetched, thus, its read must happen while manipulating the resource.
Like this
package main
import "context"
type OTPAction struct {
// DB ....
}
func (a OTPAction) Do(ctx context.Context) error {
// fetch OTP here
otp := "otp test"
return chromedp.SendKeys(`//input[#id="user-message"]`, otp).Do(ctx)
}
I have a web app whose server creates a Client for each websocket connection. A Client acts as an intermediary between the websocket connection and a single instance of a Hub. The Hub maintains a set of registered clients and broadcasts messages to the clients. This works pretty well but the problem is that a client might miss events between when the server generates the initial state bundle that the client receives on connection and when the client is registered with the hub and starts receiving broadcast events.
My idea is to register the client with the hub before any information is fetched from the db. That would ensure that the client doesn't miss any broadcasts, though now it could receive messages that are already applied to the initial state it receives. To allow the client to disregard these messages I could include a monotonic timestamp in both the initial state bundle as well as broadcast events.
Can you think of a more elegant/simpler solution?
I have used a write-ahead-log in the past to do something like this. In short, keep a ring buffer of messages in the hub. Then replay messages that where send to existing clients while the new one was initialized.
You can expose this concept to the clients too if you wish. That way you can implement efficient re-connects (particularly nice for mobile connections). When clients loose the websocket connection they can reconnect and say "Hey there, it's me again. Looks like we got interrupted. The last message I've seen was number 42. What's new?"
The following is from memory, so take this only as an illustration of the idea, not a finished implementation. In the intererest of brevity I've omited the select statements around client.send, for instance.
package main
import (
"container/list"
"sync"
"github.com/gorilla/websocket"
)
type Client struct { // all unchanged
hub *Hub
conn *websocket.Conn
send chan []byte
}
type Hub struct {
mu *sync.RWMutex
wal list.List // List if recent messages
clients map[*Client]bool // Registered clients.
register chan Registration // not a chan *Client anymore
broadcast chan []byte
unregister chan *Client
}
type Registration struct {
client *Client
// init is a function that is executed before the client starts to receive
// broadcast messages. All messages that are broadcast while init is
// running will be sent after init returns.
init func()
}
func (h *Hub) run() {
for {
select {
case reg := <-h.register:
// Take note of the most recent message as of right now.
// initClient will replay all later messages
h.mu.RLock()
head := h.wal.Back()
h.mu.RUnlock()
go h.initClient(reg, head)
case client := <-h.unregister:
h.mu.Lock()
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
h.mu.Unlock()
case message := <-h.broadcast:
h.mu.Lock()
h.wal.PushBack(message)
// TODO: Trim list if too long by some metric (e.g. number of
// messages, age, total message size, etc.)
clients := make([]*Client, len(h.clients))
copy(clients, h.clients)
h.mu.Unlock()
for client := range clients {
// TODO: deal with backpressure
client.send <- message
}
}
}
}
func (h *Hub) initClient(reg Registration, head *list.Element) {
reg.init()
// send messages in h.wal after head
for {
h.mu.RLock()
head = head.Next()
if head == nil {
// caught up
h.clients[reg.client] = true
h.mu.RUnlock()
return
}
h.mu.RUnlock()
// TODO: deal with backpressure
reg.client.send <- head.Value.([]byte)
}
}
Coming from a Java background, I have some questions on how things are typically done in Golang. I am specifically talking about services and dao's/repositories.
In java, I would use dependency injection (probably as singleton/application-scoped), and have a Service injected into my rest endpoint / resource.
To give a bit more context. Imagine the following Golang code:
func main() {
http.ListenAndServe("localhost:8080", nil)
}
func init() {
r := httptreemux.New()
api := r.NewGroup("/api/v1")
api.GET("/blogs", GetAllBlogs)
http.Handle("/", r)
}
Copied this directly from my code, main and init are split because google app engine.
So for now I have one handler. In that handler, I expect to interact with a BlogService.
The question is, where, and in what scope should I instantiate a BlogService struct and a dao like datastructure?
Should I do it everytime the handler is triggered, or make it constant/global?
For completeness, here is the handler and blogService:
// GetAllBlogs Retrieves all blogs from GCloud datastore
func GetAllBlogs(w http.ResponseWriter, req *http.Request, params map[string]string) {
c := appengine.NewContext(req)
// need a reference to Blog Service at this point, where to instantiate?
}
type blogService struct{}
// Blog contains the content and meta data for a blog post.
type Blog struct {...}
// newBlogService constructs a new service to operate on Blogs.
func newBlogService() *blogService {
return &blogService{}
}
func (s *blogService) ListBlogs(ctx context.Context) ([]*Blog, error) {
// Do some dao-ey / repository things, where to instantiate BlogDao?
}
You can use context.Context to pass request scoped values into your handlers (available in Go 1.7) , if you build all your required dependencies during the request/response cycle (which you should to avoid race conditions, except for dependencies that manage concurrency on their own like sql.DB). Put all your services into a single container for instance, then query the context for that value :
container := request.Context.Value("container").(*Container)
blogs,err := container.GetBlogService().ListBlogs()
read the following material :
https://golang.org/pkg/context/
https://golang.org/pkg/net/http/#Request.Context