I have a Go function exposed in a .wasm file and accessible from JS:
app.computePrimes = js.FuncOf(func(this js.Value, args []js.Value) interface{} {
handler := js.FuncOf(func(this js.Value, args []js.Value) interface{} {
resolve := args[0]
// Commented out because this Promise never fails
//reject := args[1]
// Now that we have a way to return the response to JS, spawn a goroutine
// This way, we don't block the event loop and avoid a deadlock
go func() {
app.console.Call("log", "starting")
for i := 2; i < 100000; i++ {
if big.NewInt(int64(i)).ProbablyPrime(20) && i > 20000 {
app.console.Call("log", i)
}
}
app.console.Call("log", "finishing")
resolve.Invoke("Done")
}()
// The handler of a Promise doesn't return any value
return nil
})
return js.Global().Get("Promise").New(handler)
})
Despite the fact that it returns a Promise and executes the CPU-bound part in a goroutine, on the Web side it feels like everything is running on the main UI thread. I have read a bit on the state of development of WWebAssembly, and it seems like multi-threaded workloads are not yet commonplace.
Is a Web worker the only preferred way to execute such tasks?
Yes, I think you answered this one yourself. As long as WASM does not support something like lite-threads / concurrency itself (which would make Go's support for WASM a lot more appealing) you are kind of stuck doing this yourself with web-workers or packages based on web-workers.
You probably found those already:
https://pspdfkit.com/blog/2020/webassembly-in-a-web-worker/
https://www.sitepen.com/blog/using-webassembly-with-web-workers
Related
I am new to Golang and starting to use goroutines. I am curious if it is safe to read from a data structure that is being written to by another goroutines without using channel or mutex. In the example below, the events array is visible to main and goroutine, but main only reads from it and the goroutine is modifying it. Is this considered safe?
var events = []string{}
func main() {
go func() {
for {
if len(events) > 30 {
events = nil
}
event := "The time is now " + time.Now().String()
events = append(events, event)
time.Sleep(time.Millisecond * 200)
}
}()
for {
for i:=0; i < len(events); i++ {
fmt.Println(events[i])
}
time.Sleep(time.Millisecond * 100)
fmt.Println("--------------------------------------------------------")
}
}
Thanks.
No, this is absolutely not safe. You should use synchronization primitives around access to events, or redesign to use channels.
Your code may work fine today, on a specific architecture/OS, but it may break unexpectedly with small changes and/or on another architecture/OS.
Try to run your code with -race to verify. You'll probably see WARNING: DATA RACE.
See also this answer.
I have a server working with websocket connections and a database. Some users can connect by sockets, so I need to increment their "online" in db; and at the moment of their disconnection I also decrement their "online" field in db. But in case the server breaks down I use a local variable replica map[string]int of users online. So I need to postpone the server shutdown until it completes a database request that decrements all users "online" in accordance with my variable replica, because at this way socket connection doesnt send default "close" event.
I have found a package github.com/xlab/closer that handles some system calls and can do some action before program finished, but my database request doesnt work in this way (code below)
func main() {
...
// trying to handle program finish event
closer.Bind(cleanupSocketConnections(&pageHandler))
...
}
// function that handles program finish event
func cleanupSocketConnections(p *controllers.PageHandler) func() {
return func() {
p.PageService.ResetOnlineUsers()
}
}
// this map[string]int contains key=userId, value=count of socket connections
type PageService struct {
Users map[string]int
}
func (p *PageService) ResetOnlineUsers() {
for userId, count := range p.Users {
// decrease online of every user in program variable
InfoService{}.DecreaseInfoOnline(userId, count)
}
}
Maybe I use it incorrectly or may be there is a better way to prevent default program finish?
First of all executing tasks when the server "breaks down" as you said is quite complicated, because breaking down can mean a lot of things and nothing can guarantee clean up functions execution when something goes really bad in your server.
From an engineering point of view (if setting users offline on breakdown is so important), the best would be to have a secondary service, on another server, that receives user connection and disconnection events and ping event, if it receives no updates in a set timeout the service considers your server down and proceeds to set every user offline.
Back to your question, using defer and waiting for termination signals should cover 99% of cases. I commented the code to explain the logic.
// AllUsersOffline is called when the program is terminated, it takes a *sync.Once to make sure this function is performed only
// one time, since it might be called from different goroutines.
func AllUsersOffline(once *sync.Once) {
once.Do(func() {
fmt.Print("setting all users offline...")
// logic to set all users offline
})
}
// CatchSigs catches termination signals and executes f function at the end
func CatchSigs(f func()) {
cSig := make(chan os.Signal, 1)
// watch for these signals
signal.Notify(cSig, syscall.SIGKILL, syscall.SIGTERM, syscall.SIGINT, syscall.SIGQUIT, syscall.SIGHUP) // these are the termination signals in GNU => https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html
// wait for them
sig := <- cSig
fmt.Printf("received signal: %s", sig)
// execute f
f()
}
func main() {
/* code */
// the once is used to make sure AllUsersOffline is performed ONE TIME.
usersOfflineOnce := &sync.Once{}
// catch termination signals
go CatchSigs(func() {
// when a termination signal is caught execute AllUsersOffline function
AllUsersOffline(usersOfflineOnce)
})
// deferred functions are called even in case of panic events, although execution is not to take for granted (OOM errors etc)
defer AllUsersOffline(usersOfflineOnce)
/* code */
// run server
err := server.Run()
if err != nil {
// error logic here
}
// bla bla bla
}
I think that you need to look at go routines and channel.
Here something (maybe) useful:
https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/
I'm writing a package to control a Canon DSLR using their EDSDK DLL from Go.
This is a personal project for a photo booth to use at our wedding at my partners request, which I'll be happy to post on GitHub when complete :).
Looking at the examples of using the SDK elsewhere, it isn't threadsafe and uses thread-local resources, so I'll need to make sure I'm calling it from a single thread during usage. While not ideal, it looks like Go provides a "runtime.LockOSThread" function for doing just that, although this does get called by the core DLL interop code itself, so I'll have to wait and find out if that interferes or not.
I want the rest of the application to be able to call the SDK using a higher level interface without worrying about the threading, so I need a way to pass function call requests to the locked thread/Goroutine to execute there, then pass the results back to the calling function outside of that Goroutine.
So far, I've come up with this working example of using very broad function definitions using []interface{} arrays and passing back and forward via channels. This would take a lot of mangling of input/output data on every call to do type assertions back out of the interface{} array, even if we know what we should expect for each function ahead of time, but it looks like it'll work.
Before I invest a lot of time doing it this way for possibly the worst way to do it - does anyone have any better options?
package edsdk
import (
"fmt"
"runtime"
)
type CanonSDK struct {
FChan chan functionCall
}
type functionCall struct {
Function func([]interface{}) []interface{}
Arguments []interface{}
Return chan []interface{}
}
func NewCanonSDK() (*CanonSDK, error) {
c := &CanonSDK {
FChan: make(chan functionCall),
}
go c.BackgroundThread(c.FChan)
return c, nil
}
func (c *CanonSDK) BackgroundThread(fcalls <-chan functionCall) {
runtime.LockOSThread()
for f := range fcalls {
f.Return <- f.Function(f.Arguments)
}
runtime.UnlockOSThread()
}
func (c *CanonSDK) TestCall() {
ret := make(chan []interface{})
f := functionCall {
Function: c.DoTestCall,
Arguments: []interface{}{},
Return: ret,
}
c.FChan <- f
results := <- ret
close(ret)
fmt.Printf("%#v", results)
}
func (c *CanonSDK) DoTestCall([]interface{}) []interface{} {
return []interface{}{ "Test", nil }
}
For similar embedded projects I've played with, I tend to create a single goroutine worker that listens on a channel to perform all the work over that USB device. And any results sent back out on another channel.
Talk to the device with channels only in Go in a one-way exchange. LIsten for responses from the other channel.
Since USB is serial and polling, I had to setup a dedicated channel with another goroutine that justs picks items off the channel when they were pushed into it from the worker goroutine that just looped.
I am developing a simple go library for jsonrpc over http.
There is the following method:
rpcClient.Call("myMethod", myParam1, myParam2)
This method internally does a http.Get() and returns the result or an error (tuple).
This is of course synchron for the caller and returns when the Get() call returns.
Is this the way to provide libraries in go? Should I leave it to the user of my library to make it asynchron if she wants to?
Or should I provide a second function called:
rpcClient.CallAsync()
and return a channel here? Because channels cannot provide tuples I have to pack the (response, error) tuple in a struct and return that struct instead.
Does this make sense?
Otherwise the user would have to wrap every call in an ugly method like:
result := make(chan AsyncResponse)
go func() {
res, err := rpcClient.Call("myMethod", myParam1, myParam2)
result <- AsyncResponse{res, err}
}()
Is there a best practice for go libraries and asynchrony?
The whole point of go's execution model is to hide the asynchronous operations from the developer, and behave like a threaded model with blocking operations. Behind the scenes there are green-threads and asynchronous IO and a very sophisticated scheduler.
So no, you shouldn't provide an async API to your library. Networking in go is done in a pseudo-blocking way from the code's perspective, and you open as many goroutines as needed, as they are very cheap.
So your last example is the way to go, and I don't consider it ugly. Because it allows the developer to choose the concurrency model. In the context of an http server, where each command is handled in separate goroutine, I'd just call rpcClient.Call("myMethod", myParam1, myParam2).
Or if I want a fanout - I'll create fanout logic.
You can also create a convenience function for executing the call and returning on a channel:
func CallAsync(method, p1, p2) chan AsyncResponse {
result := make(chan AsyncResponse)
go func() {
res, err := rpcClient.Call(method, p1, p2)
result <- AsyncResponse{res, err}
}()
return result
}
I'm using Go with the gotk3 and webkit2 libraries to try and build a web crawler that can parse JavaScript in the context of a WebKitWebView.
Thinking of performance, I'm trying to figure out what would be the best way to have it crawl concurrently (if not in parallel, with multiple processors), using all available resources.
GTK and everything with threads and goroutines are pretty new to me. Reading from the gotk3 goroutines example, it states:
Native GTK is not thread safe, and thus, gotk3's GTK bindings may not be used from other goroutines. Instead, glib.IdleAdd() must be used to add a function to run in the GTK main loop when it is in an idle state.
Go will panic and show a stack trace when I try to run a function, which creates a new WebView, in a goroutine. I'm not exactly sure why this happens, but I think it has something to do with this comment. An example is shown below.
Current Code
Here's my current code, which has been adapted from the webkit2 example:
package main
import (
"fmt"
"github.com/gotk3/gotk3/glib"
"github.com/gotk3/gotk3/gtk"
"github.com/sourcegraph/go-webkit2/webkit2"
"github.com/sqs/gojs"
)
func crawlPage(url string) {
web := webkit2.NewWebView()
web.Connect("load-changed", func(_ *glib.Object, i int) {
loadEvent := webkit2.LoadEvent(i)
switch loadEvent {
case webkit2.LoadFinished:
fmt.Printf("Load finished for: %v\n", url)
web.RunJavaScript("window.location.hostname", func(val *gojs.Value, err error) {
if err != nil {
fmt.Println("JavaScript error.")
} else {
fmt.Printf("Hostname (from JavaScript): %q\n", val)
}
//gtk.MainQuit()
})
}
})
glib.IdleAdd(func() bool {
web.LoadURI(url)
return false
})
}
func main() {
gtk.Init(nil)
crawlPage("https://www.google.com")
crawlPage("https://www.yahoo.com")
crawlPage("https://github.com")
crawlPage("http://deelay.me/2000/http://deelay.me/img/1000ms.gif")
gtk.Main()
}
It seems that creating a new WebView for each URL allows them to load concurrently. Having glib.IdleAdd() running in a goroutine, as per the gotk3 example, doesn't seem to have any effect (although I'm only doing a visual benchmark):
go glib.IdleAdd(func() bool { // Works
web.LoadURI(url)
return false
})
However, trying to create a goroutine for each crawlPage() call ends in a panic:
go crawlPage("https://www.google.com") // Panics and shows stack trace
I can run web.RunJavaScript() in a goroutine without issue:
switch loadEvent {
case webkit2.LoadFinished:
fmt.Printf("Load finished for: %v\n", url)
go web.RunJavaScript("window.location.hostname", func(val *gojs.Value, err error) { // Works
if err != nil {
fmt.Println("JavaScript error.")
} else {
fmt.Printf("Hostname (from JavaScript): %q\n", val)
}
//gtk.MainQuit()
})
}
Best Method?
The current methods I can think of are:
Spawn new WebViews to crawl each page, as shown in the current code. Track how many WebViews are opened and either continually delete and create new ones, or reuse a set number created initially, to where all available resources on the machine are used. Would this be limited in terms of processor cores being used?
Basic idea of #1, but running the binary multiple times (instead of one gocrawler process running on the machine, have four) to utilize all cores/resources.
Run the GUI (gtk3) portion of the app in its own goroutine. I could then pass data to other goroutines which do their own heavy processing, such as searching through content.
What would actually be the best way to run this code concurrently, if possible, and max out performance?
Update
Method 1 and 2 are probably out of the picture, as I ran a test by spawning ~100 WebViews and they seem to load synchronously.