I'm trying to get recwatch to work. I'm confused by its interface, though. Yes, I can make a watcher and add folders to it, but there does not seem to be a way to start an event loop that will allow me to receive notifications.
In the original code, there was a Run receiver for just this purpose.
Am I missing something?
The watcher starts emitting events as soon as it is created. All that's required is to read them from RecursiveWatcher.Watcher.Events. Example:
package main
import (
"fmt"
"github.com/xyproto/recwatch"
)
func main() {
w, err := recwatch.NewRecursiveWatcher("sample_dir")
if err != nil {
panic(err)
}
for {
select {
case event := <-w.Events:
fmt.Printf("Event: %s\n", event)
case event := <-w.Errors:
fmt.Printf("Error: %s\n", event)
}
}
}
Related
I have the following code in Go using the semaphore library just as an example:
package main
import (
"fmt"
"context"
"time"
"golang.org/x/sync/semaphore"
)
// This protects the lockedVar variable
var lock *semaphore.Weighted
// Only one go routine should be able to access this at once
var lockedVar string
func acquireLock() {
err := lock.Acquire(context.TODO(), 1)
if err != nil {
panic(err)
}
}
func releaseLock() {
lock.Release(1)
}
func useLockedVar() {
acquireLock()
fmt.Printf("lockedVar used: %s\n", lockedVar)
releaseLock()
}
func causeDeadLock() {
acquireLock()
// calling this from a function that's already
// locked the lockedVar should cause a deadlock.
useLockedVar()
releaseLock()
}
func main() {
lock = semaphore.NewWeighted(1)
lockedVar = "this is the locked var"
// this is only on a separate goroutine so that the standard
// go "deadlock" message doesn't print out.
go causeDeadLock()
// Keep the primary goroutine active.
for true {
time.Sleep(time.Second)
}
}
Is there a way to get the acquireLock() function call to print a message after a timeout indicating that there is a potential deadlock but without unblocking the call? I would want the deadlock to persist, but a log message to be written in the event that a timeout is reached. So a TryAcquire isn't exactly what I want.
An example of what I want in psuedo code:
afterFiveSeconds := func() {
fmt.Printf("there is a potential deadlock\n")
}
lock.Acquire(context.TODO(), 1, afterFiveSeconds)
The lock.Acquire call in this example would call the afterFiveSeconds callback if the Acquire call blocked for more than 5 seconds, but it would not unblock the caller. It would continue to block.
I think I've found a solution to my problem.
func acquireLock() {
timeoutChan := make(chan bool)
go func() {
select {
case <-time.After(time.Second * time.Duration(5)):
fmt.Printf("potential deadlock while acquiring semaphore\n")
case <-timeoutChan:
break
}
}()
err := lock.Acquire(context.TODO(), 1)
close(timeoutChan)
if err != nil {
panic(err)
}
}
I register three events via SDL_RegisterEvents. However, SDL_PollEvent() only returns events of the first type and swallows the others.
This is my code (it's Go, but I don't think that's relevant here):
package main
import (
"github.com/veandco/go-sdl2/sdl"
"log"
)
func main() {
if err := sdl.Init(sdl.INIT_VIDEO | sdl.INIT_EVENTS); err != nil {
panic(err)
}
defer sdl.Quit()
evt1 := sdl.RegisterEvents(3)
evt2 := evt1 + 1
evt3 := evt1 + 2
sdl.PushEvent(&sdl.UserEvent{Type: evt1})
sdl.PushEvent(&sdl.UserEvent{Type: evt2})
sdl.PushEvent(&sdl.UserEvent{Type: evt3})
sdl.PushEvent(&sdl.UserEvent{Type: evt1})
event := sdl.WaitEvent()
for ; event != nil; event = sdl.PollEvent() {
userEvent, ok := event.(*sdl.UserEvent)
if ok {
switch (userEvent.Type) {
case evt1:
log.Println("got evt1")
case evt2:
log.Println("got evt2")
case evt3:
log.Println("got evt3")
}
}
}
}
This is the output:
2019/08/04 20:10:26 got evt1
2019/08/04 20:10:26 got evt1
So both evt1 events that I pushed got polled, but evt2 and evt3 I pushed in between vanished. I tried calling sdl.RegisterEvents(1) three times instead for registering the events, but the result is the same.
What am I doing wrong?
It only is because of go binding, specifically the line https://github.com/veandco/go-sdl2/blob/24851c1f2d98dcac2a68223a24e6f799fc921f1e/sdl/events.go#L1035 (type conversion only triggered on SDL_USEREVENT, which matches only with first registered event; anything else is reported as "CommonEvent", losing all extra data you may have passed) . You probably should report a bug to developer of said binding.
I have some trouble with go routines and channels regarding error handling.
Firstly I have a function that listen for messages (in a infinite for loop):
func main() {
messageChannel := make(chan messageHandler.MessageInfo)
for {
if token := client.Subscribe("#", 0, func(client MQTT.Client, msg MQTT.Message) {
go messageHandler.DecodeMessage(msg, messageChannel)
select {
case messageInfo := <-messageChannel:
//Handle
}
}); token.Wait() && token.Error() != nil {
fmt.Println(token.Error())
}
}
}
But in the DecodeMessage function, there could arise multiple errors.
func DecodeMessage(msg mqtt.Message, c1 chan MessageInfo) {
//do something, might result in error
//do another thing, might result in error
c1 <- MessageInfo{...}
}
Normally I would just return from the function. But seems a bit trickier with routines. I've looked at this post, but if both errors would occur, I would only see the last error message.
Example:
func DecodeMessage(msg mqtt.Message, c1 chan MessageInfo) {
var returnError error
if err != nil {
returnError = err
}
if err != nil {
returnError = err
}
c1 <- MessageInfo{
Error: returnError,
...
}
}
Should I have an array of some sort and append all errors? Is it bad practice to have multiple errors in one routine?
The best thing, for me, is that the routine would exit on an error and return that error like it would do "normally". Is that possible?
I'll start by saying that having to go through all the error checks for a function before returning even if they fail is a bit of a code smell. It might mean that you have something weird going on and there may be a better way to accomplish what you're trying to do.
However, assuming that you've boiled down your problem to this, then I see two options, depending on how the errors have to be handled.
If the errors can be handled one-by-one and don't really rely on each other, then you can just create an error channel and send them back one by one as you encounter the errors. See the following working example:
package main
import (
"errors"
"fmt"
"strings"
)
func main() {
errCh := make(chan error)
go HandleErrorsSerially("bad error", errCh)
for err := range errCh {
fmt.Printf("Found error serially: %v\n", err)
}
}
func HandleErrorsSerially(msg string, errCh chan<- error) {
if strings.Contains(msg, "error") {
errCh <- errors.New("message contained string 'error'")
}
if strings.Contains(msg, "bad") {
errCh <- errors.New("message contained string 'bad'")
}
close(errCh)
}
Alternatively, if you need to have a view of all the errors that occurred all at once (because two errors happening simultaneously may indicate some special circumstances) then you'd have to append them all to an array and then pass them through a channel. See the following working example:
package main
import (
"errors"
"fmt"
"strings"
)
func main() {
errArrCh := make(chan []error)
go HandleErrorsTogether("bad error", errArrCh)
errArr := <-errArrCh
fmt.Printf("Found the following errors together: %v\n", errArr)
}
func HandleErrorsTogether(msg string, errArrCh chan<- []error) {
errArr := make([]error, 0)
if strings.Contains(msg, "error") {
errArr = append(errArr, errors.New("message contained string 'error'"))
}
if strings.Contains(msg, "bad") {
errArr = append(errArr, errors.New("message contained string 'bad'"))
}
errArrCh <- errArr
close(errArrCh)
}
I can see cases where it's useful to return multiple errors, such as when you are parsing messages and there are multiple bad fields and you need to summarise these back to a client.
I think the best approach is to use a package like hashicorp's multierror which allows multiple errors to be collected with formatting in a structure type that implements error interface and so can still be sent on a chan error. The receive side can then either process as just a standard error or extract the information on each individual error.
The multierror documentation is pretty good, just read through the examples on the github page.
The best thing, for me, is that the routine would exit on an error and
return that error like it would do "normally".
Of course, you can, and get last error or all error are both pretty strange.
Get last error is unhelpful for debugging, compare with getting first error.
Get all error is same if the first error will cause the following failure, the following error message is unhelpful; Another situation is these errors do not have the association, I think this means you have to sperate they into different concurrency part for better controlling.
Ok, now back to original problem. Consider the code:
func foo(errTo chan error) {
defer close(errTo)
v, err := CouldFailOne()
if err != nil {
errTo <- err
return // Yp, just stop this routine, let it join back to Invoker
}
v2, err := CloudFailTwo()
if err != nil {
errTo <- err
return
}
// As the previous error handle until end of the function
}
If you want to return value from this kind of function. Just use a channel, and send the value into it only no error raise. I think this style will be more clear and like return style, just became using a channel to return the error.
I work currently on a micro service architecture.
Before I insert NATS into my project I wanted to test some simple scenarios with it.
In one scenario I have a simple publisher, which publishes 100.000 messages in a for loop over a basic Nats server running on localhost:4222.
The big problem with it, is the subscriber. When he receive between 30.000 - 40.000 messages my whole main.go program and all other go routines just stops and do nothing. I can just quit with ctrl + c. But the Publisher is still keep sending the messages. When I open a new terminal and start a new instance of the subscriber all again works well, till the Subscriber receive about 30000 messages. And the worst thing is that there appears not even one error and also no logs on the server so I have no idea whats going on.
After that I was trying replace the Subscribe-method with the QueueSubscribe-method and all works fine.
What is the main difference between Subscribe and QueueSubscribe?
Is NATS-Streaming a better opportunity? Or in which cases I should prefer Streaming and in which the standard NATS-Server
Here is my code:
Publisher:
package main
import (
"fmt"
"log"
"time"
"github.com/nats-io/go-nats"
)
func main() {
go createPublisher()
for {
}
}
func createPublisher() {
log.Println("pub started")
nc, err := nats.Connect(nats.DefaultURL)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
msg := make([]byte, 16)
for i := 0; i < 100000; i++ {
nc.Publish("alenSub", msg)
if (i % 100) == 0 {
fmt.Println("i", i)
}
time.Sleep(time.Millisecond)
}
log.Println("pub finish")
nc.Flush()
}
Subscriber:
package main
import (
"fmt"
"log"
"time"
"github.com/nats-io/go-nats"
)
var received int64
func main() {
received = 0
go createSubscriber()
go check()
for {
}
}
func createSubscriber() {
log.Println("sub started")
nc, err := nats.Connect(nats.DefaultURL)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
nc.Subscribe("alenSub", func(msg *nats.Msg) {
received++
})
nc.Flush()
for {
}
}
func check() {
for {
fmt.Println("-----------------------")
fmt.Println("still running")
fmt.Println("received", received)
fmt.Println("-----------------------")
time.Sleep(time.Second * 2)
}
}
The infinite for loops are likely starving the garbage collector: https://github.com/golang/go/issues/15442#issuecomment-214965471
I was able to reproduce the issue by just running the publisher. To resolve, I recommend using a sync.WaitGroup. Here's how I updated the code linked to in the comments to get it to complete:
package main
import (
"fmt"
"log"
"sync"
"time"
"github.com/nats-io/go-nats"
)
// create wait group
var wg sync.WaitGroup
func main() {
// add 1 waiter
wg.Add(1)
go createPublisher()
// wait for wait group to complete
wg.Wait()
}
func createPublisher() {
log.Println("pub started")
// mark wait group done after createPublisher completes
defer wg.Done()
nc, err := nats.Connect(nats.DefaultURL)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
msg := make([]byte, 16)
for i := 0; i < 100000; i++ {
if errPub := nc.Publish("alenSub", msg); errPub != nil {
panic(errPub)
}
if (i % 100) == 0 {
fmt.Println("i", i)
}
time.Sleep(time.Millisecond * 1)
}
log.Println("pub finish")
errFlush := nc.Flush()
if errFlush != nil {
panic(errFlush)
}
errLast := nc.LastError()
if errLast != nil {
panic(errLast)
}
}
I'd recommend updating the above subscriber code similarly.
The main difference between Subscribe and QueueSubscriber is that in Subscribe all subscribers are sent all messages from. While in QueueSubscribe only one subscriber in a QueueGroup is sent each message.
Some details on additional features for NATS Streaming are here:
https://nats.io/documentation/streaming/nats-streaming-intro/
We see both NATS and NATS Streaming used in a variety of use cases from data pipelines to control planes. Your choice should be driven by the needs of your use case.
As stated, remove the for{} loop. Replace with runtime.Goexit().
For subscriber you don't need to create the subscriber in a Go routine. Async subscribers already have their own Go routine for callbacks.
Also protected the received variable with atomic or a mutex.
See the examples here as well.
https://github.com/nats-io/go-nats/tree/master/examples
I'm creating a simple udp client that listens on multiple ports and saves the request to bigtable.
It's essential to listen on different ports before you ask.
Everything was working nicely until I included bigtable. After doing so, the listeners block completely.
My stripped down code, without bigtable, looks like this:
func flow(port string) {
protocol := "udp"
udpAddr, err := net.ResolveUDPAddr(protocol, "0.0.0.0:"+port)
if err != nil {
fmt.Println("Wrong Address")
return
}
udpConn, err := net.ListenUDP(protocol, udpAddr)
if err != nil {
fmt.Println(err)
}
defer udpConn.Close()
for {
Publish(udpConn, port)
}
}
func main() {
fmt.Print("Starting server.........")
for i := *Start; i <= *End; i++ {
x := strconv.Itoa(i)
go flow(x)
}
}
This works fine however, as soon as I add the following for bigtable, the whole thing blocks. If I remove the go routine that creates the listener (which means I can't listen on multiple ports) it works.
func createBigTable() {
ctx := context.Background()
client, err := bigtable.NewClient(ctx, *ProjectID, *Instance)
if err != nil {
log.Fatal("Bigtable NewClient:", err)
}
Table = client.Open("x")
}
I managed to get it working by adding a query in the createBigTable func but the program still blocks later on.
I have no idea if this is an issue with bigtable, grpc or just the way I'm doing it.
Would really appreciate some advise about how to fix.
--- UPDATE ---
I've discovered the issue isn't just with BigTable - I also have the same issue when I call gcloud pubsub.
--- UPDATE 2 ---
createBigtable is called in the init function (BEFORE THE MAIN FUNCTION):
func init() {
createBigTable
}
--- Update 3 ---
Output from sigquit can be found here:
https://pastebin.com/fzixqmiA
In your playground example, you're using for {} to keep the server running for forever.
This seems to deprive the goroutines from ever getting to run.
Try using e.g. a WaitGroup to yield control from the main() routine and let the flow() routines handle the incoming UDP packets.
import (
...
"sync"
...
)
...
func main() {
fmt.Print("Starting server.")
for i := *Start; i <= *End; i++ {
x := strconv.Itoa(i)
go flow(x)
}
var wg sync.WaitGroup
wg.Add(1)
wg.Wait()
}