I'm trying to do a benchmark with go using rpc and exec.command, here are parts of my code.
I have a master to send rpc to worker to do some job.
func main() {
var wg sync.WaitGroup
var clients []*rpc.Client
client, err := rpc.DialHTTP("tcp", "addr"+":1234")
if err != nil {
log.Fatal("dialing:", err)
}
reply := &Reply{}
args := &Args{}
clients = append(clients, client)
fmt.Println(clients)
err = clients[0].Call("Worker.Init", args, reply)
if err != nil {
log.Fatal("init error:", err)
}
// call for server to init channel
// err = client.Call("Worker.Init", args, reply)
args.A = 1
wg.Add(200)
fmt.Println(time.Now().UnixNano())
for i := 0; i < 200; i++ {
go func() {
defer wg.Done()
err = client.Call("Worker.DoJob", args, reply)
if err != nil {
log.Fatal("dojob error:", err)
}
fmt.Println("Done")
}()
}
wg.Wait()
fmt.Println(time.Now().UnixNano())
}
and worker's code
func (w *Worker) DoJob(args *Args, reply *Reply) error {
// find a channel to do it
w.c <- 1
runtime.LockOSThread()
fmt.Println("exec")
// cmd := exec.Command("docker", "run", "--rm", "ubuntu:16.04", "/bin/bash", "-c", "date +%s%N")
cmd := exec.Command("echo", "hello")
err := cmd.Run()
fmt.Println("exec done")
if err != nil {
reply.Err = err
fmt.Println(err)
}
fmt.Println("done")
<-w.c
return nil
}
I use a chan of size 12 to simulate that the machine has only 12 threads, and after I find it would stuck at cmd.Run(), I changed the command from running a docker to just simply echo hello, but it got still stucked between fmt.Println("exec") and fmt.Println("exec done").
I don'k know why is this happening? Am I sending out too many rpcs so a lot of rpcs will be dropped?
Related
I have a go api project where I also run a worker (RabbitMQ). I just discovered a problem that my worker and my http listen and serve do not work together. The moment I run the worker, the port of api is not reached.
Here is what my code looks like.
app.go
func (a *App) StartWorker() {
connection, err := amqp091.Dial(os.Getenv("AMQP_URL"))
if err != nil {
panic(err)
}
defer connection.Close()
consumer, err := events.NewConsumer(connection, database.GetDatabase(a.Database))
if err != nil {
panic(err)
}
consumer.Listen(os.Args[1:])
}
func (a *App) Run(addr string) {
logs := log.New(os.Stdout, "my-service", log.LstdFlags)
server := &http.Server{
Addr: addr,
Handler: a.Router,
ErrorLog: logs,
IdleTimeout: 120 * time.Second, // max time for connections using TCP Keep-Alive
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
}
go func() {
if err := server.ListenAndServe(); err != nil {
logs.Fatal(err)
}
}()
// trap sigterm or interrupt and gracefully shutdown the server
c := make(chan os.Signal)
signal.Notify(c, os.Interrupt)
signal.Notify(c, os.Kill)
sig := <-c
logs.Println("Recieved terminate, graceful shutdown", sig)
tc, _ := context.WithTimeout(context.Background(), 30*time.Second)
server.Shutdown(tc)
}
here is my
consumer.go
// NewConsumer returns a new Consumer
func NewConsumer(conn *amqp.Connection, db *mongo.Database) (Consumer, error) {
consumer := Consumer{
conn: conn,
db: db,
}
err := consumer.setup()
if err != nil {
return Consumer{}, err
}
return consumer, nil
}
// Listen will listen for all new Queue publications
// and print them to the console.
func (consumer *Consumer) Listen(topics []string) error {
ch, err := consumer.conn.Channel()
if err != nil {
return err
}
defer ch.Close()
if err != nil {
return err
}
msgs, err := ch.Consume("update.package.rating", "", true, false, false, false, nil)
if err != nil {
return err
}
forever := make(chan bool)
go func() {
for msg := range msgs {
switch msg.RoutingKey {
case "update.package.rating":
worker.RatePackage(packageRepo.NewPackagesRepository(consumer.db), msg.Body)
}
// acknowledege received event
log.Printf("Received a message: %s", msg.Body)
}
}()
log.Printf("[*] Waiting for message [Exchange, Queue][%s, %s]. To exit press CTRL+C", getExchangeName(), "update.package.rating")
<-forever
return nil
}
main.go
func main() {
start := app.App{}
start.StartApp()
start.StartWorker()
start.Run(":3006")
}
the port 3006 is not reached.
I am using gin-gonic to serve my http request.
Any help is welcomed.
I had a similar problem while using gin framework.Solved the issue by running my consumer inside a go routine.I invoked my consumer like below.
go notificationCallback.ConsumeBankTransaction()
and both the server and the rabbitmq consumer run seamlessly.Still monitoring performance to see if it is robust and resilient enough.
i'm making screen record web based using command exec to run FFMPEG. here I created a startRecording function but I am still confused about stopping the command process in the stopRecording function, because the command is executed in the startRecording function. How to stop a process that is already running in the srartRecording function in the stopRecording function?
here my code
//Handler to create room/start record
func RoomCreate(c *fiber.Ctx) error {
fileName := "out.mp4"
fmt.Println(fileName)
if len(os.Args) > 1 {
fileName = os.Args[1]
}
errCh := make(chan error, 2)
ctx, cancelFn := context.WithCancel(context.Background())
// Call to function startRecording
go func() { errCh <- startRecording(ctx, fileName) }()
go func() {
errCh <- nil
}()
err := <-errCh
cancelFn()
if err != nil && err != context.Canceled {
log.Fatalf("Execution failed: %v", err)
}
return c.Redirect(fmt.Sprintf("/room/%s", guuid.New().String()))
}
//Function to run command FFMPEG
func startRecording(ctx context.Context, fileName string) error {
ctx, cancelFn := context.WithCancel(ctx)
defer cancelFn()
// Build ffmpeg
ffmpeg := exec.Command("ffmpeg",
"-f", "gdigrab",
"-framerate", "30",
"-i", "desktop",
"-f", "mp4",
fileName,
)
// Stdin for sending data
stdin, err := ffmpeg.StdinPipe()
if err != nil {
return err
}
//var buf bytes.Buffer
defer stdin.Close()
// Run it in the background
errCh := make(chan error, 1)
go func() {
fmt.Printf("Executing: %v\n", strings.Join(ffmpeg.Args, " "))
if err := ffmpeg.Run(); err != nil {
return
}
//fmt.Printf("FFMPEG output:\n%v\n", string(out))
errCh <- err
}()
// Just start sending a bunch of frames
for {
// Check if we're done, otherwise go again
select {
case <-ctx.Done():
return ctx.Err()
case err := <-errCh:
return err
default:
}
}
}
//Here function to stop Recording
func stopRecording(ctx context.Context) error {
//Code stop recording in here
}
Thanks for advance
As requested from comments.
The basic idea is to use global storage to store your active commands. It doesn't necessarily be global but you need to have bigger scope so that your functions can access it.
var commands = map[string]*exec.Cmd{}
func startRecording(fileName string) error {
ffmpeg := exec.Command("ffmpeg",
"-f", "gdigrab",
"-framerate", "30",
"-i", "desktop",
"-f", "mp4",
fileName,
)
commands[fileName] = ffmpeg
...
}
func stopRecording(fileName string) error {
cmd, ok := commands[fileName]
if !ok {
return errors.New("command not found")
}
defer func() {
delete(commands, fileName)
}()
return cmd.Process.Kill()
}
You probably want to use sync.Mutex or sync.RWMutex to avoid concurrent map writes.
So your commands cloud look like:
type Commands struct {
sync.RWMutex
items map[string]*exec.Cmd
}
// use Commands.Lock() for writing, Commands.RLock() for reading
I wrote a connection closure function. It sends a closing frame and expects the same in response.
func TryCloseNormally(wsConn *websocket.Conn) error {
closeNormalClosure := websocket.FormatCloseMessage(websocket.CloseNormalClosure, "")
defer wsConn.Close()
if err := wsConn.WriteControl(websocket.CloseMessage, closeNormalClosure, time.Now().Add(time.Second)); err != nil {
return err
}
if err := wsConn.SetReadDeadline(time.Now().Add(time.Second)); err != nil {
return err
}
_, _, err := wsConn.ReadMessage()
if websocket.IsCloseError(err, websocket.CloseNormalClosure) {
return nil
} else {
return errors.New("Websocket doesn't send a close frame in response")
}
}
I wrote a test for this function.
func TestTryCloseNormally(t *testing.T) {
done := make(chan struct{})
exit := make(chan struct{})
ctx := context.Background()
ln, err := net.Listen("tcp", "localhost:")
require.Nil(t, err)
handler := HandlerFunc(func(conn *websocket.Conn) {
for {
_, _, err := conn.ReadMessage()
if err != nil {
require.True(t, websocket.IsCloseError(err, websocket.CloseNormalClosure), err.Error())
return
}
}
})
s, err := makeServer(ctx, handler)
require.Nil(t, err)
go func() {
require.Nil(t, s.Run(ctx, exit, ln))
close(done)
}()
wsConn, _, err := websocket.DefaultDialer.Dial(addr+strconv.Itoa(ln.Addr().(*net.TCPAddr).Port), nil) //nolint:bodyclose
require.Nil(t, err)
require.Nil(t, wsConn.WriteMessage(websocket.BinaryMessage, []byte{'o', 'k'}))
require.Nil(t, TryCloseNormally(wsConn))
close(exit)
<-done
}
To my surprise, it works correctly. Readmessage() reads the closing frame. But in the test, I don't write anything.
Is this happening at the gorilla/websocket level?
Did I write the function correctly? Maybe reading the response frame also happens at the gorilla level.
The function is mostly correct.
Websocket endpoints echo close messages unless the endpoint has already send a close message on its own. See Closing Handshake in the Websocket RFC for more details.
In the normal close scenario, an application should expect to receive a close message after sending a close message.
To handle the case where the peer sent a data message before the sending the close message, read and discard data messages until an error is returned.
func TryCloseNormally(wsConn *websocket.Conn) error {
defer wsConn.Close()
closeNormalClosure := websocket.FormatCloseMessage(websocket.CloseNormalClosure, "")
if err := wsConn.WriteControl(websocket.CloseMessage, closeNormalClosure, time.Now().Add(time.Second)); err != nil {
return err
}
if err := wsConn.SetReadDeadline(time.Now().Add(time.Second)); err != nil {
return err
}
for {
_, _, err := wsConn.ReadMessage()
if websocket.IsCloseError(err, websocket.CloseNormalClosure) {
return nil
}
if err != nil {
return err
}
}
}
I'm writing a Go websocket server and I want to graceful stop the connections when my server goes down.
I have a map of active connections stored in the following variable:
var connections = make(map[string]*websocket.Conn)
My main function looks like this:
func main() {
// ... stuff ....
gracefulStop := make(chan os.Signal)
signal.Notify(gracefulStop, syscall.SIGTERM)
signal.Notify(gracefulStop, syscall.SIGINT)
signal.Notify(gracefulStop, syscall.SIGQUIT)
signal.Notify(gracefulStop, syscall.SIGKILL)
signal.Notify(gracefulStop, syscall.SIGHUP)
go func() {
sig := <-gracefulStop
log.Printf("Exiting from process due to %+v", sig)
log.Println("Closing all websocket connections")
for id, conn := range connections {
closeConnection(id, conn)
}
os.Exit(0)
}()
r := mux.NewRouter()
r.HandleFunc("/{id}", wsHandler)
err := http.ListenAndServe(fmt.Sprintf(":%d", *argPort), r)
if err != nil {
log.Println("Could not start http server")
log.Println(err)
}
}
closeConnection does 4 things:
conn.Close()
sets conn as nil
removes the id from the map
calls an AWS Lambda function
The same function is called as a defer function inside the wsHandler function, so if a client disconnects by its own, I execute function in the handler.
It's all working nicely, except that when I ctrl+c the server my closeConnection function is called two times per client, one in the graceful stop handler and the other in the wsHandler defer function.
I tried to check in my closeConnection function if the connection is still defined in connections, but it returns true both of the times.
I thought that it was due to the fact that they are called two times because they are in different goroutines, so I replaced the for loop above with just a time.Sleep(2 * time.Second), but in this case nothing happens (the closeConnection inside the wsHandler defer function is not even called).
This is what I mean:
go func() {
sig := <-gracefulStop
log.Printf("Exiting from process due to %+v", sig)
log.Println("Closing all websocket connections")
// for chargeboxIdentity, conn := range connections {
// chargeboxDisconnected(chargeboxIdentity, conn)
// }
time.Sleep(2 * time.Second)
os.Exit(0)
}()
EDIT: Here is the closeConnection function:
func closeConnection(id string, conn *websocket.Conn) {
_, ok := connections[id]
log.Println(ok)
log.Printf("%s (%s) disconnected", id, conn.RemoteAddr())
conn.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
time.Sleep(300 * time.Millisecond)
conn.Close()
conn = nil
delete(connections, id)
request := LambdaPayload{ID: id}
payload, err := json.Marshal(request)
if err != nil {
log.Println("Could not create payload for lambda call")
log.Println(err)
return
}
_, err = client.Invoke(&lambda.InvokeInput{FunctionName: aws.String(lambdaPrefix + "MainDisconnect"), Payload: payload})
if err != nil {
log.Println("Disconnect Lambda returned an error")
log.Println(err)
}
}
EDIT: Here's the wsHandler function:
func wsHandler(w http.ResponseWriter, r *http.Request) {
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println("Could not upgrade websocket connection")
log.Println(err)
return
}
vars := mux.Vars(r)
if !clientConnected(vars["id"], conn) {
return
}
defer closeConnection(vars["id"], conn)
for {
msgType, msg, err := conn.ReadMessage()
if err != nil {
break
}
log.Printf("%s sent: %s", vars["id"], string(msg))
// ... stuff ...
}
}
every one,I am new to golang.I wanna get the data from log file generated by my application.cuz roll-back mechanism, I met some problem.For instance,my target log file is chats.log,it will be renamed to chats.log.2018xxx and a new chats.log will be created.so my go routine that read log file will fail to work.
so I need detect the change and shutdown the previous go routine and then establish the new go routine.
I looked for modules that can help me,and I found
func ExampleNewWatcher(fn string, createnoti chan string, wg sync.WaitGroup) {
wg.Add(1)
defer wg.Done()
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
defer watcher.Close()
done := make(chan bool)
go func() {
for {
select {
case event := <-watcher.Events:
if event.Op == fsnotify.Create && event.Name==fn{
createnoti <- "has been created"
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}()
err = watcher.Add("./")
if err != nil {
log.Fatal(err)
}
<-done
}
I use fsnotify to detech the change,and make sure the event of file is my log file,and then send some message to a channel.
this is my worker go routine:
func tailer(fn string,isfollow bool, outchan chan string, done <-chan interface{},wg sync.WaitGroup) error {
wg.Add(1)
defer wg.Done()
_, err := os.Stat(fn)
if err != nil{
panic(err)
}
t, err := tail.TailFile(fn, tail.Config{Follow:isfollow})
if err != nil{
panic(err)
}
defer t.Stop()
for line := range t.Lines{
select{
case outchan <- line.Text:
case <- done:
return nil
}
}
return nil
}
I using tail module to read the log file,and I add a done channel to it to shutdown the cycle(I don't know whether I put it in the right way)
And I will send every log content to a channel to consuming it.
So here is the question:how should I put it together?
ps: Actually,I can use some tool to do this job.like apache-flume,but all of those tools need dependency.
Thank you a lot!
Here is a complete example that reloads and rereads the file as it changes or gets deleted and recreated:
package main
import (
"github.com/fsnotify/fsnotify"
"io/ioutil"
"log"
)
const filename = "myfile.txt"
func ReadFile(filename string) string {
data, err := ioutil.ReadFile(filename)
if err != nil {
log.Println(err)
}
return string(data)
}
func main() {
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
defer watcher.Close()
err = watcher.Add("./")
if err != nil {
log.Fatal(err)
}
for {
select {
case event := <-watcher.Events:
if event.Op == fsnotify.Create && event.Name == filename {
log.Println(ReadFile(filename))
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}
Note this doesn't require goroutines, channels or a WaitGroup. Better to keep things simple and reserve those for when they're actually needed.