Keep a MQTT Go client running - go

I think it is a silly question, I need a MQTT Client to keep running after connection and subscription. I never encountered the problem because my MQTT clients are always coupled with an HTTP server, and when launching a HTTP server, the code don't stop running.
But in the present use case I only need a MQTT Client to subscribe to some topic and stay alive.
Here is what I do (the function just connect to a broker and subcribe to one topic.)
func main() {
godotenv.Load("./.env")
_initMqttConnection()
}
I need the client to stay connected and not stop just after the subscription is done.
How to perform that simple thing ?
Edit 1 : Complete Code
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
"os"
"path/filepath"
"strings"
"github.com/yosssi/gmq/mqtt"
"github.com/yosssi/gmq/mqtt/client"
"github.com/joho/godotenv"
"github.com/skratchdot/open-golang/open"
)
var cli *client.Client
func _initMqttConnection() {
cli = client.New(&client.Options{
ErrorHandler: func(err error) {
fmt.Println(err)
},
})
defer cli.Terminate()
log.Println("Connecting to " + os.Getenv("mqtt_host"))
err := cli.Connect(&client.ConnectOptions{
Network: "tcp",
Address: os.Getenv("mqtt_host"),
UserName: []byte(os.Getenv("mqtt_user")),
Password: []byte(os.Getenv("mqtt_password")),
ClientID: []byte("mqtt_video_launcher"),
})
if err != nil {
log.Println("Error 1")
panic(err)
}
log.Println("Connected to MQTT")
topic_to_sub := []byte("/" + os.Getenv("video_topic"))
err = cli.Subscribe(&client.SubscribeOptions{
SubReqs: []*client.SubReq{
&client.SubReq{
TopicFilter: topic_to_sub,
QoS: mqtt.QoS0,
Handler: func(topicName, message []byte) {
//do struff with message
fmt.Println(string(topicName), string(message))
},
},
},
})
if err != nil {
panic(err)
}
log.Println("Subscription OK : " + string(topic_to_sub[:len(topic_to_sub)]))
}
func main() {
godotenv.Load("./.env")
_initMqttConnection()
}
The temporary solution I use is adding :
http.ListenAndServe(":", nil)
at the end.

You have to make the program run infinitely or unless you want to explicitly end it (Cntrl c). One good solution that worked for me is to wait for a channel before exiting the main function and that channel can keep listening for an interrupt.
Eg:
func main() {
keepAlive := make(chan os.Signal)
signal.Notify(keepAlive, os.Interrupt, syscall.SIGTERM)
// All your code
<-keepAlive
}

Related

How do you gracefully exit a go uber fx app

How do you stop an uber fx as in shutdown the entire program. There seems to be no other way other than ctrl+c
func main() {
fx.New(
fx.Invoke(register)
).Run
}
func register() {
time.Sleep(5*time.Seconds)
// shutdown somehow
}
The docs are not particularly clear, but there's a Shutdowner interface available to any Fx module with a Shutdown method that requests graceful application shutdown.
Here's a modified part of the example package that will have it simply shutdown upon receiving a request:
func NewHandler(logger *log.Logger, shutdowner fx.Shutdowner) (http.Handler, error) {
logger.Print("Executing NewHandler.")
return http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
logger.Print("Got a request. Requesting shutdown now that I've gotten one request.")
shutdowner.Shutdown()
}), nil
}
Edit: Here's how you could modify your solution:
func register(shutdowner fx.Shutdowner) {
time.Sleep(5*time.Seconds)
shutdowner.Shutdown()
}
You can wrap it inside a go routine and use context to gracefully exit.
import (
"context"
"log"
" go.uber.org/fx"
)
func main() {
f := fx.New(fx.Invoke(register))
go func() {
f.Run()
}()
stopCh := make(chan os.Signal)
signal.Notify(stopCh, syscall.SIGINT, syscall.SIGTERM)
<-stopCh
if err := f.Stop(context.Background()); err != nil {
log.Printf("error stopping gracefully")
}
}
func register() {
time.Sleep(5*time.Seconds)
// shutdown somehow
}

How to re-evaluate promhttp.Handler on database change?

I'm perhaps abusing promhttp.Handler() to realise the use case for my microservice to tell me the:
version
if it has database connectivity
If there is a better way to monitor my microservices, do let me know!
I'm not sure how to structure the handle in such a way that when /metrics are called, the db.Ping() is re-evaluated.
https://s.natalian.org/2019-06-02/msping.mp4
package main
import (
"log"
"net/http"
"os"
_ "github.com/go-sql-driver/mysql"
"github.com/gorilla/mux"
"github.com/jmoiron/sqlx"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
const version = "0.0.1"
type App struct {
Router *mux.Router
DB *sqlx.DB
}
func main() {
a := App{}
a.Initialize()
log.Fatal(http.ListenAndServe(":"+os.Getenv("PORT"), a.Router))
}
func (a *App) Initialize() {
connectionString := "root:secret#tcp(localhost:3306)/rest_api_example?multiStatements=true&sql_mode=TRADITIONAL&timeout=5s"
var err error
a.DB, err = sqlx.Open("mysql", connectionString)
if err != nil {
log.Fatal(err)
}
microservicecheck := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "mscheck",
Help: "Version with DB ping check",
},
[]string{
"commit",
},
)
if a.DB.Ping() == nil {
microservicecheck.WithLabelValues(version).Set(1)
} else {
microservicecheck.WithLabelValues(version).Set(0)
}
prometheus.MustRegister(microservicecheck)
a.Router = mux.NewRouter()
a.initializeRoutes()
}
func (a *App) initializeRoutes() {
a.Router.Handle("/metrics", promhttp.Handler()).Methods("GET")
}
https://play.golang.org/p/9DdXnz77S55
You could also add a middleware hook that does a preflight routine (i.e. your ping test) before calling promhttp.Handler(). However, at collection time, I think metrics should already have been tallied; and not generated at the instance of collection. So...
Try a separate go-routine that will poll the health of the DB connection at regular intervals. This avoids any messy hooks or custom collectors:
var pingPollingFreq = 5 * time.Second // this should probably match the Prometheus scrape interval
func (a *App) Initialize() {
// ...
prometheus.MustRegister(microservicecheck)
go func() {
for {
if a.DB.Ping() == nil {
microservicecheck.WithLabelValues(version).Set(1)
} else {
microservicecheck.WithLabelValues(version).Set(0)
}
time.Sleep(pingPollingFreq)
}
}()
// ...
}

ssh server in golang

I am trying the github.com/gliderlabs/ssh package in order to build a ssh server.
The example are working fine.
The code below listen and reply some text when I connect, then close the connection.
I would like to keep it open, listening to user input (lines) and echoing "you say : "... but I have no idea of what to do and it seems this is too simple to be explained somewhere.
Can someone give me an indication of what to do ?
package main
import (
"bufio"
"fmt"
"io"
"log"
"github.com/gliderlabs/ssh"
)
func main() {
ssh.Handle(func(s ssh.Session) {
io.WriteString(s, fmt.Sprintf("Hello %s\n", s.User()))
io.WriteString(s, fmt.Sprintf("Hello 2%s\n", s.User()))
io.WriteString(s, fmt.Sprintf("Hello 3%s\n", s.User()))
text,err:= bufio.NewReader(s).ReadString('\n')
if err != nil {
panic("GetLines: " + err.Error())
}
io.WriteString(s, fmt.Sprintf("ton texte %s\n", text))
})
log.Println("starting ssh server on port 2223...")
log.Fatal(ssh.ListenAndServe(":2223", nil))
}
If question is still actual, you shoud use for{} cycle inside handler, and always read input. BTW, it's more suitable to use "golang.org/x/crypto/ssh/terminal" package.
Here's little example:
func (s *Server) handler(sess ssh.Session) {
...
term := terminal.NewTerminal(sess, "> ")
for {
line, err := term.ReadLine()
if err != nil {
break
}
response := router(line)
log.Println(line)
if response != "" {
term.Write(append([]byte(response), '\n'))
}
}
log.Println("terminal closed")
}

Golang Nats subscribe issue

I work currently on a micro service architecture.
Before I insert NATS into my project I wanted to test some simple scenarios with it.
In one scenario I have a simple publisher, which publishes 100.000 messages in a for loop over a basic Nats server running on localhost:4222.
The big problem with it, is the subscriber. When he receive between 30.000 - 40.000 messages my whole main.go program and all other go routines just stops and do nothing. I can just quit with ctrl + c. But the Publisher is still keep sending the messages. When I open a new terminal and start a new instance of the subscriber all again works well, till the Subscriber receive about 30000 messages. And the worst thing is that there appears not even one error and also no logs on the server so I have no idea whats going on.
After that I was trying replace the Subscribe-method with the QueueSubscribe-method and all works fine.
What is the main difference between Subscribe and QueueSubscribe?
Is NATS-Streaming a better opportunity? Or in which cases I should prefer Streaming and in which the standard NATS-Server
Here is my code:
Publisher:
package main
import (
"fmt"
"log"
"time"
"github.com/nats-io/go-nats"
)
func main() {
go createPublisher()
for {
}
}
func createPublisher() {
log.Println("pub started")
nc, err := nats.Connect(nats.DefaultURL)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
msg := make([]byte, 16)
for i := 0; i < 100000; i++ {
nc.Publish("alenSub", msg)
if (i % 100) == 0 {
fmt.Println("i", i)
}
time.Sleep(time.Millisecond)
}
log.Println("pub finish")
nc.Flush()
}
Subscriber:
package main
import (
"fmt"
"log"
"time"
"github.com/nats-io/go-nats"
)
var received int64
func main() {
received = 0
go createSubscriber()
go check()
for {
}
}
func createSubscriber() {
log.Println("sub started")
nc, err := nats.Connect(nats.DefaultURL)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
nc.Subscribe("alenSub", func(msg *nats.Msg) {
received++
})
nc.Flush()
for {
}
}
func check() {
for {
fmt.Println("-----------------------")
fmt.Println("still running")
fmt.Println("received", received)
fmt.Println("-----------------------")
time.Sleep(time.Second * 2)
}
}
The infinite for loops are likely starving the garbage collector: https://github.com/golang/go/issues/15442#issuecomment-214965471
I was able to reproduce the issue by just running the publisher. To resolve, I recommend using a sync.WaitGroup. Here's how I updated the code linked to in the comments to get it to complete:
package main
import (
"fmt"
"log"
"sync"
"time"
"github.com/nats-io/go-nats"
)
// create wait group
var wg sync.WaitGroup
func main() {
// add 1 waiter
wg.Add(1)
go createPublisher()
// wait for wait group to complete
wg.Wait()
}
func createPublisher() {
log.Println("pub started")
// mark wait group done after createPublisher completes
defer wg.Done()
nc, err := nats.Connect(nats.DefaultURL)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
msg := make([]byte, 16)
for i := 0; i < 100000; i++ {
if errPub := nc.Publish("alenSub", msg); errPub != nil {
panic(errPub)
}
if (i % 100) == 0 {
fmt.Println("i", i)
}
time.Sleep(time.Millisecond * 1)
}
log.Println("pub finish")
errFlush := nc.Flush()
if errFlush != nil {
panic(errFlush)
}
errLast := nc.LastError()
if errLast != nil {
panic(errLast)
}
}
I'd recommend updating the above subscriber code similarly.
The main difference between Subscribe and QueueSubscriber is that in Subscribe all subscribers are sent all messages from. While in QueueSubscribe only one subscriber in a QueueGroup is sent each message.
Some details on additional features for NATS Streaming are here:
https://nats.io/documentation/streaming/nats-streaming-intro/
We see both NATS and NATS Streaming used in a variety of use cases from data pipelines to control planes. Your choice should be driven by the needs of your use case.
As stated, remove the for{} loop. Replace with runtime.Goexit().
For subscriber you don't need to create the subscriber in a Go routine. Async subscribers already have their own Go routine for callbacks.
Also protected the received variable with atomic or a mutex.
See the examples here as well.
https://github.com/nats-io/go-nats/tree/master/examples

Example of simple websocket connection from client

I would like to rewrite this JavaScript (run in Node) into Go.
let io = require('socket.io-client');
let socket = io('http://botws.generals.io');
socket.on('connect', function() {
console.log('Connected to server.');
});
(I've tried this library unsuccessfully; I can elaborate on my noob attempts if that helps)
Ok, got it working. There was a different URL I had to use.
package main
import (
"github.com/graarh/golang-socketio"
"github.com/graarh/golang-socketio/transport"
"log"
)
func main() {
transport := transport.GetDefaultWebsocketTransport()
ws_url := "ws://botws.generals.io/socket.io/?EIO=3&transport=websocket"
client, err := gosocketio.Dial(ws_url, transport)
if err != nil {
log.Fatal(err)
}
client.On(gosocketio.OnConnection, func(c *gosocketio.Channel, args interface{}) {
log.Println("Connected!")
})
// Block to give client time to connect
select {}
client.Close()
}

Resources