I am using goroutines in a package where there is a tcp server. The response most of the time is very heavy, but when the routines end it is not cleared from the memory.
func Handle() {
service := ":7777"
tcpAddr, err := net.ResolveTCPAddr("tcp4", service)
checkError(err)
listener, err := net.ListenTCP("tcp", tcpAddr)
checkError(err)
defer listener.Close()
for {
conn, err := listener.Accept()
checkError(err)
go handleRequest(conn, db)
}
}
func handleRequest(conn net.Conn, db *sql.DB) {
message := make([]byte, 0, 4096)
tmp := make([]byte, 256)
n, err := conn.Read(tmp)
if err != nil {
if err != io.EOF {
fmt.Println("read error:", err)
}
}
message = append(message, tmp[:n]...)
fmt.Println("Message Received:", string(message))
// do something to get resp
conn.Write(append(resp, []byte("\n")...))
conn.Close()
debug.FreeOSMemory()
return
}
So in this case the response is big, and a goroutine using 10% of the memory, thats okay, because I'm getting 170.000 users from the database and parse the result to JSON. But when the handleRequest and it is still in the memory, if I'm not using debug.FreeOsMemory(). I have doubts it is a good way to do it because it is in the debug pacakge so my question is it a good way to empty the memory what the goroutines are using? I tested it so it is not affected to the system and working very well. If not, what is the good way? I can't wait the GC for clear it up?! I read this and this is why I started to use it, in the first answer there is the last suggestion.
The Go runtime does not release free memory back to the OS "immediately", it would be inefficient. Read more about it here: Golang - Cannot free memory once occupied by bytes.Buffer.
You should let the Go runtime handle this. If your app is unstable without calling debug.FreeOsMemory(), there are bigger problems which you shouldn't cover up with this even if it "seemingly" helps. It may even make things worse, as if serving a request does require a large amount of memory (which is properly freed by the GC when done with the request), calling FreeOsMemory() will just return it to the OS which the runtime will have to ask for / allocate again when serving another request. Should you have not handed it back to the OS, it would be available for the next request...
Try to decrease the memory requirement of the request handler. If it is not possible (questionable), then limit the number of requests requiring large memory that may be served concurrently.
See this question+answer how to do that: Process Management for the Go Webserver
Also: Is this an idiomatic worker thread pool in Go?
As an alternative to occasionally invoking debug.FreeOsMemory(), you can control the aggressiveness of the garbage collector with the GOGC environment variable or with the debug.SetGCPercent() method.
Empirically, I've seen that with a value as low as 10, it does its job.
See Package runtime
GO lang clean all memory that is not in use, but is not inmediate. You can read the other question:
Golang - Cannot free memory once occupied by bytes.Buffer
That is crazy, I think is a bug for golang,My code include:
go func() {
for {
debug.FreeOSMemory()
log4sys.Warn("NumGoroutine:",runtime.NumGoroutine())
time.Sleep(1 * time.Minute)
}
}()
to solution this question
Related
I have this code (more or less):
resp, err := http.Get(url)
if err != nil {
// handle error
}
if resp.StatusCode != http.StatusOK {
// handle error
}
out, err := os.Create(filepath)
if err != nil {
return err
}
// Write the body to file
_, err = io.Copy(out, resp.Body)
resp.Body.Close()
out.Close()
My issue is that if I immediately try to do something (e.g. take the hash of this file), then I see that it is still copying for a while.
At first I was deferring the out.Close(), and I though that I need to out.Close after the io.Copy, which will block until its done with it. or so I thought.
This didn't work and I still have the same issue.
How do I block or wait for the io.Copy operation to finish?
Thanks!
Likely you are hitting some disk buffer/cache, where your OS or disk device keeps some data in memory before actually persisting the write to the disk.
Calling
out.Sync()
forces a fsync syscall, which will instruct the OS to force a flush of the buffer and write the data to disk. I suggest calling out.Flush() after the io.Copy call returns.
Related docs you may find interesting:
https://pkg.go.dev/os#File.Sync
https://man7.org/linux/man-pages/man2/fdatasync.2.html
I'm having an issue regarding the disposing of kafka consumer in the end of program execution. Here is code responsible for closing the consumer
func(kc *KafkaConsumer) Dispose() {
Sugar.Info("Disposing of consumer")
kc.mu.Lock()
kc.Consumer.Close();
Sugar.Info("Disposed of consumer")
kc.mu.Unlock()
}
As you might have already noticed, i'm making use of sync.Mutex, inasmuch as consumer is accessed by multiple goroutines. Below is another snippet responsible for reading messages from kafka
func (kc *KafkaConsumer) Consume(signalChan chan os.Signal, ctx context.Context) {
for{
select{
case sig := <-signalChan:
Sugar.Info("Caught signal %v", sig)
break
case <-ctx.Done():
Sugar.Info("Got context done message. Closing consumer...")
kc.Dispose()
break
default:
for{
message, err := kc.Consumer.ReadMessage(-1); if err != nil{
Log.Error(err.Error())
return
}
Sugar.Infof("Got a new message %v",message)
resp := make(chan *KafkaResponseEntity)
go router.UseMessage(*message, resp, ctx)
//Potential deadlock
response := <-resp
/*
Explicit commit of an offset in order to ensure
that request has been successfully processed
*/
kc.Consumer.Commit()
Sugar.Info("Successfully commited an offset")
Sugar.Infof("Just got a response %v", response)
go producer.KP.Produce(response.PaymentId, response.Bytes, "some_random_topic")
}
}
}
}
The problem is that when closing the consumer, program execution simply stalls.
Are there any issues? Should i use cond along with mutex? I'd be very glad if you provide thorough explanation of what might went wrong in my code.
Thanks in advance.
I suspect this is hanging because of:
kc.Consumer.ReadMessage(-1)
Which the documentation states will block indefinitely, hence why it's not closed. The simplest approach would be to make that value a positive time duration (e.g. 1 * time.Second), but then you may get time out errors if messages are not consumed within the timeouts. The time out error is generally innocuous, but is something to account for, from the linked documentation:
Timeout is returned as (nil, err) where err is
`err.(kafka.Error).Code() == kafka.ErrTimedOut`
I'm not yet sure of a good way to utilize the indefinite blocking and allow it to be interrupted. If anyone does know please post the findings!
I'm trying to make a web scraper, which can run a decent number (many thousands) of http queries per minute. The actual querying is fine but to speed up the process. I'm trying to make it concurrent. Initially I spawned a goroutine for each request but I ran out of file descriptors so after some googling I decided to use a semaphore to limit the number of concurrent goroutines.
Only I can't get this to work.
I've tried moving bits of code around but I always have the same issue: I have roughly three times as many goroutines running as I want
This is the only method I have that spawns goroutines. I limited the goroutines to 80. In my benchmarks I run this against a slice of 10000 URLs and it tends to hover at about 242 concurrent goroutines in flight, but then it suddenly goes up to almost double this and then back down to 242.
I get the same behaviour if I change the concurrent value from 80 - it usually hovers at just over three times the number of goroutines and sometimes spikes to around double that and I have no idea why.
func (B BrandScraper) ScrapeUrls(URLs ...string) []scrapeResponse {
concurrent := 80
semaphoreChan := make(chan struct{}, concurrent)
scrapeResults := make([]scrapeResponse, len(URLs))
for _, URL := range URLs {
semaphoreChan <- struct{}{}
go func(URL string) {
defer func() {
<-semaphoreChan
}()
scrapeResults = append(scrapeResults,
B.getIndividualScrape(URL))
fmt.Printf("#goroutines: %d\n", runtime.NumGoroutine())
}(URL)
}
return scrapeResults
}
I'm expecting it to be constantly at 80 goroutines - or at least constant.
This happens when I run it from a benchmarking test or when i run it from the main function.
Thanks very much for any tips!
EDIT
getIndividualScrape
calls another function:
func (B BrandScraper) doGetRequest(URL string) io.Reader {
resp, err := http.Get(URL)
if err != nil {
log.Fatal(err)
}
body, _ := ioutil.ReadAll(resp.Body)
resp.Body.Close()
return bytes.NewReader(body)
}
which obviously does an HTTP request. Could this be leaking goroutines? I thought since I'd closed the resp.Body I'd have covered that but maybe not?
I wrote simply server program to received data form client. I little not understand what sometimes I get error read tcp4 IP:PORT i/o timeout from function
int, err := conn.Read([]byte) event time set in function SetDeadline() not was exceeded. I present some part of my code, but I think that this is will enough.
main loop where I receive data is below.
c := NewClient()
c.kickTime: time.Now()
func (c *Client) Listen(){
durationToClose := time.Minute*time.Duration(5),
c.conn.SetDeadline(c.kickTime.Add(c.durationToClose))
buffer := make([]byte, 1024)
for{
reqLen, err := c.conn.Read(buffer)
if err != nil || reqLen == 0 {
fmt.Printf(err)
break
}
if err = c.CheckData(buffer) ; err != nil{
fmt.Printf("something is bad")
}else{
result := c.PrepareDataToSendInOtherPlace(buffer)
go c.RecievedData(result)
}
c.conn.SetDeadline(c.kickTime.Add(c.durationToKick))
}
}
For me only suspicious can be additional function as PrepareDataToSendInOtherPlace() , CheckData() which may take some times CPU, and then new data was be send by client, and server at the time doing something else and rejects connect. This is only my supposition, but I'm not sure.
Syntax errors and undeclared variables aside, what you're showing us can't possibly be walking the Read/Write deadline forward indefinitely.
The longest this could run is until a fixed duration after the first time.Now() (c.kickTime.Add(c.durationToKick)). You probably want something like:
c.conn.SetDeadline(time.Now().Add(c.durationToKick))
http.Serve either returns an error as soon as it is called or blocks if successfully executing.
How can I make it so that if it blocks it does so in its own goroutine? I currently have the following code:
func serveOrErr(l net.Listener, handler http.Handler) error {
starting := make(chan struct{})
serveErr := make(chan error)
go func() {
starting <- struct{}{}
if err := http.Serve(l, handler); err != nil {
serveErr <- err
}
}()
<-starting
select {
case err := <-serveErr:
return err
default:
return nil
}
}
This seemed like a good start and works on my test machine but I believe that there are no guarantees that serveErr <- err would be called before case err := <-serveErr therefore leading to inconsistent results due to a data race if http.Serve were to produce an error.
http.Serve either returns an error as soon as it is called or blocks if successfully executing
This assumption is not correct. And I believe it rarely occurs. http.Serve calls net.Listener.Accept in the loop – an error can occur any time (socket closed, too many open file descriptors etc.). It's http.ListenAndServe, usually being used for running http servers, which often fails early while binding listening socket (no permissions, address already in use).
In my opinion what you're trying to do is wrong, unless really your net.Listener.Accept is failing on the first call for some reason. Is it? If you want to be 100% sure your server is working, you could try to connect to it (and maybe actually transmit something), but once you successfully bound the socket I don't see it really necessary.
You could use a timeout on your select statement, e.g.
timeout := time.After(5 * time.Millisecond) // TODO: ajust the value
select {
case err := <-serveErr:
return err
case _ := <- timeout:
return nil
}
This way your select will block until serveErr has a value or the specified timeout has elapsed. Note that the execution of your function will therefore block the calling goroutine for up to the duration of the specified timeout.
Rob Pike's excellent talk on go concurrency patterns might be helpful.