Related
when i try to run etcd from user account (Total RAM: 8GB, virtual memory of 1GB) in a 64 bit Linux environment (limited resources), I am getting the following error:
fatal error: failed to reserve page summary memory
runtime stack:
runtime.throw(
{0x108df50?, 0x7fffd8aa6850?}
)
/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7fffd8aa6800 sp=0x7fffd8aa67d0 pc=0x437edd
runtime.(*pageAlloc).sysInit(0x194f2f0)
/usr/local/go/src/runtime/mpagealloc_64bit.go:82 +0x195 fp=0x7fffd8aa6888 sp=0x7fffd8aa6800 pc=0x42c9b5
runtime.(*pageAlloc).init(0x194f2f0, 0x194f2e0, 0x0?)
/usr/local/go/src/runtime/mpagealloc.go:324 +0x70 fp=0x7fffd8aa68b0 sp=0x7fffd8aa6888 pc=0x42a610
runtime.(*mheap).init(0x194f2e0)
/usr/local/go/src/runtime/mheap.go:721 +0x13f fp=0x7fffd8aa68e8 sp=0x7fffd8aa68b0 pc=0x427a3f
runtime.mallocinit()
/usr/local/go/src/runtime/malloc.go:407 +0xb2 fp=0x7fffd8aa6910 sp=0x7fffd8aa68e8 pc=0x40c4f2
runtime.schedinit()
/usr/local/go/src/runtime/proc.go:693 +0xab fp=0x7fffd8aa6970 sp=0x7fffd8aa6910 pc=0x43b8ab
runtime.rt0_go()
/usr/local/go/src/runtime/asm_amd64.s:345 +0x11c fp=0x7fffd8aa6978 sp=0x7fffd8aa6970 pc=0x469c3c
But the same works fine from root account.
Can someone please support?
I was expecting this should give same output from root and user account. This seems like a virtual memory issue to me.
As your request isn't very clear, I'm going to share with you a working piece of code that can successfully interact with etcd:
package main
import (
"context"
"fmt"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
)
func main() {
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"localhost:2379"},
DialTimeout: 5 * time.Second,
})
if err != nil {
panic(err)
}
defer cli.Close()
// insert values
res, err := cli.Put(context.Background(), "key1", "val1")
if err != nil {
panic(err)
}
fmt.Printf("revision: %q\n", res.Header.Revision)
// delete all keys
defer cli.Delete(context.Background(), "key", clientv3.WithPrefix())
// get object from etcd
key, err := cli.Get(context.Background(), "key1")
if err != nil {
panic(err)
}
fmt.Printf("key: %q\tvalue: %q\n", key.Kvs[0].Key, key.Kvs[0].Value)
}
This code assumes that you've got a running instance of ETCD that listens on port 2379 for incoming requests. I run ETCD with Docker with the following script:
docker run -d -p 2379:2379 -p 2380:2380 --env ALLOW_NONE_AUTHENTICATION=yes --env ETCD_ADVERTISE_CLIENT_URLS=http://etcd-server:2379 bitnami/etcd:latest
Hope this helps in clarifying a little bit!
I am trying to read multiple files in parallel in such a way so that each go routine that is reading a file write its data to that channel, then have a single go-routine that listens to that channel and adds the data to the map. Here is my play.
Below is the example from the play:
package main
import (
"fmt"
"sync"
)
func main() {
var myFiles = []string{"file1", "file2", "file3"}
var myMap = make(map[string][]byte)
dataChan := make(chan fileData, len(myFiles))
wg := sync.WaitGroup{}
defer close(dataChan)
// we create a wait group of N
wg.Add(len(myFiles))
for _, file := range myFiles {
// we create N go-routines, one per file, each one will return a struct containing their filename and bytes from
// the file via the dataChan channel
go getBytesFromFile(file, dataChan, &wg)
}
// we wait until the wait group is decremented to zero by each instance of getBytesFromFile() calling waitGroup.Done()
wg.Wait()
for i := 0; i < len(myFiles); i++ {
// we can now read from the data channel N times.
file := <-dataChan
myMap[file.name] = file.bytes
}
fmt.Printf("%+v\n", myMap)
}
type fileData struct {
name string
bytes []byte
}
// how to handle error from this method if reading file got messed up?
func getBytesFromFile(file string, dataChan chan fileData, waitGroup *sync.WaitGroup) {
bytes := openFileAndGetBytes(file)
dataChan <- fileData{name: file, bytes: bytes}
waitGroup.Done()
}
func openFileAndGetBytes(file string) []byte {
return []byte(fmt.Sprintf("these are some bytes for file %s", file))
}
Problem Statement
How can I use golang.org/x/sync/errgroup to wait on and handle errors from goroutines or if there is any better way like using semaphore? For example if any one of my go routine fails to read data from file then I want to cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller). And it should automatically waits for all the supplied go routines to complete successfully for success case.
I also don't want to spawn 100 go-routines if total number of files is 100. I want to control the parallelism if possible if there is any way.
How can I use golang.org/x/sync/errgroup to wait on and handle errors from goroutines or if there is any better way like using semaphore? For example [...] I want to cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller). And it should automatically waits for all the supplied go routines to complete successfully for success case.
There are many ways to communicate error states across goroutines. errgroup does a bunch of heavy lifting though, and is appropriate for this case. Otherwise you're going to end up implementing the same thing.
To use errgroup we'll need to handle errors (and for your demo, generate some). In addition, to cancel existing goroutines, we'll use a context from errgroup.NewWithContext.
From the errgroup reference,
Package errgroup provides synchronization, error propagation, and Context cancelation for groups of goroutines working on subtasks of a common task.
Your play doesn't support any error handling. We can't collect and cancel on errors if we don't do any error handling. So I added some code to inject error handling:
func openFileAndGetBytes(file string) (string, error) {
if file == "file2" {
return "", fmt.Errorf("%s cannot be read", file)
}
return fmt.Sprintf("these are some bytes for file %s", file), nil
}
Then that error had to be passed back from getBytesFromFile as well:
func getBytesFromFile(file string, dataChan chan fileData) error {
bytes, err := openFileAndGetBytes(file)
if err == nil {
dataChan <- fileData{name: file, bytes: bytes}
}
return err
}
Now that we've done that, we can turn our attention to how we're going to start up a number of goroutines.
I also don't want to spawn 100 go-routines if total number of files is 100. I want to control the parallelism if possible if there is any way.
Written well, the number of tasks, channel size, and number of workers are typically independent values. The trick is to use channel closure - and in your case, context cancellation - to communicate state between the goroutines. We'll need an additional channel for the distribution of filenames, and an additional goroutine for the collection of the results.
To illustate this point, my code uses 3 workers, and adds a few more files. My channels are unbuffered. This allows us to see some of the files get processed, while others are aborted. If you buffer the channels, the example will still work, but it's more likely for additional work to be processed before the cancellation is handled. Experiment with buffer size along with worker count and number of files to process.
var myFiles = []string{"file1", "file2", "file3", "file4", "file5", "file6"}
fileChan := make(chan string)
dataChan := make(chan fileData)
To start up the workers, instead of starting one for each file, we start the number we desire - here, 3.
for i := 0; i < 3; i++ {
worker_num := i
g.Go(func() error {
for file := range fileChan {
if err := getBytesFromFile(file, dataChan); err != nil {
fmt.Println("worker", worker_num, "failed to process", file, ":", err.Error())
return err
} else if err := ctx.Err(); err != nil {
fmt.Println("worker", worker_num, "context error in worker:", err.Error())
return err
}
}
fmt.Println("worker", worker_num, "processed all work on channel")
return nil
})
}
The workers call your getBytesFromFile function. If it returns an err, we return an err. errgroup will cancel our context automatically in this case. However, the exact order of operations is not deterministic, so more files may or may not get processed before the context is cancelled. I'll show several possibilties below.
by rangeing over fileChan, the worker automatically picks up end of work from the channel closure. If we get an error, we can return it to errgroup immediately. Otherwise, if the context has been cancelled, we can return the cancellation error immediately.
You might think that g.Go would automatically cancel our function. But it cannot. There is no way to cancel a running function in Go other than process termination. errgroup.Group.Go's function argument must cancel itself when appropriate based on the state of its context.
Now we can turn our attention to the thing that puts the files on fileChan. We have 2 options here: we can use a buffered channel of the size of myFiles, like you did. We can fill the entire channel with pending jobs. This is only an option if you know the number of jobs when you create the channel. The other option is to use an additional "distribution" goroutine that can block on writes to fileChan so that our "main" goroutine can continue.
// dispatch files
g.Go(func() error {
defer close(fileChan)
done := ctx.Done()
for _, file := range myFiles {
select {
case fileChan <- file:
continue
case <-done:
break
}
}
return ctx.Err()
})
I'm not sure it's strictly necessary to put this in the same errgroup in this case, because we can't get an error in the distributor goroutine. But this general pattern, drawn from the Pipeline example from errgroup, works regardless of whether the work dispatcher might generate errors.
This functions pretty simple, but the magic is in select along with ctx.Done() channel. Either we write to the work channel, or we fail if our context is done. This allows us to stop distributing work when one worker has failed one file.
We defer close(fileChan) so that, regardless of why we have finished (either we distributed all work, or the context was cancelled), the workers know there will be no more work on the incoming work queue (ie fileChan).
We need one more synchronization mechanism: once all the work is distributed, and all the results are in or work was finished being cancelled, (eg, after our errgroup's Wait() returns), we need to close our results channel, dataChan. This signals the results collector that there are no more results to be collected.
var err error // we'll need this later!
go func() {
err = g.Wait()
close(dataChan)
}()
We can't - and don't need to - put this in the errgroup.Group. The function can't return an error, and it can't wait for itself to close(dataChan). So it goes into a regular old goroutine, sans errgroup.
Finally we can collect the results. With dedicated worker goroutines, a distributor goroutine, and a goroutine waiting on the work and notifying that there will be no more writes to the dataChan, we can collect all the results right in the "primary" goroutine in main.
for data := range dataChan {
myMap[data.name] = data.bytes
}
if err != nil { // this was set in our final goroutine, remember
fmt.Println("errgroup Error:", err.Error())
}
I made a few small changes so that it was easier to see the output. You may already have noticed I changed the file contents from []byte to string. This was purely so that the results were easy to read. Pursuant also to that end, I am using encoding/json to format the results so that it is easy to read them and paste them into SO. This is a common pattern that I often use to indent structured data:
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(myMap); err != nil {
panic(err)
}
Finally we're ready to run. Now we can see a number of different results depending on just what order the goroutines execute. But all of them are valid execution paths.
worker 2 failed to process file2 : file2 cannot be read
worker 0 context error in worker: context canceled
worker 1 context error in worker: context canceled
errgroup Error: file2 cannot be read
{
"file1": "these are some bytes for file file1",
"file3": "these are some bytes for file file3"
}
Program exited.
In this result, the remaining work (file4 and file5) were not added to the channel. Remember, an unbuffered channel stores no data. For those tasks to be written to the channel, a worker would have to be there to read them. Instead, the context was cancelled after file2 failed, and the distribution function followed the <-done path within its select. file1 and file3 were already processed.
Here's a different result (I just ran the playground share a few times to get different results).
worker 1 failed to process file2 : file2 cannot be read
worker 2 processed all work on channel
worker 0 processed all work on channel
errgroup Error: file2 cannot be read
{
"file1": "these are some bytes for file file1",
"file3": "these are some bytes for file file3",
"file4": "these are some bytes for file file4",
"file5": "these are some bytes for file file5",
"file6": "these are some bytes for file file6"
}
In this case, it looks a little like our cancellation failed. but what really happened is that the goroutines just "happened" to queue and finish the rest of the work before errorgroup picked upon worker `'s failure and cancelled the context.
what errorgroup does
When you use errorgroup, you're really getting 2 things out of it:
easily accessing the first error your workers returned;
getting a context that errorgroup will cancel for you when
Keep in mind that errorgroup does not cancel goroutines. This tripped me up a bit at first. Errorgroup cancels the context. It's your responsibility to apply the status of that context to your goroutines (remember, the goroutine must end itself, errorgroup can't end it).
A final aside about contexts with file operations, and failing outstanding work
Most of your file operations, eg io.Copy or os.ReadFile, are actually a loop of subsequent Read operations. But io and os don't support contexts directly. so if you have a worker reading a file, and you don't implement the Read loop yourself, you won't have an opportunity to cancel based on context. That's probably okay in your case - sure, you may have read some more files than you really needed to, but only because you were already reading them when the error occurred. I would personally accept this state of affairs and not implement my own read loop.
The code
https://go.dev/play/p/9qfESp_eB-C
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"golang.org/x/sync/errgroup"
)
func main() {
var myFiles = []string{"file1", "file2", "file3", "file4", "file5", "file6"}
fileChan := make(chan string)
dataChan := make(chan fileData)
g, ctx := errgroup.WithContext(context.Background())
for i := 0; i < 3; i++ {
worker_num := i
g.Go(func() error {
for file := range fileChan {
if err := getBytesFromFile(file, dataChan); err != nil {
fmt.Println("worker", worker_num, "failed to process", file, ":", err.Error())
return err
} else if err := ctx.Err(); err != nil {
fmt.Println("worker", worker_num, "context error in worker:", err.Error())
return err
}
}
fmt.Println("worker", worker_num, "processed all work on channel")
return nil
})
}
// dispatch files
g.Go(func() error {
defer close(fileChan)
done := ctx.Done()
for _, file := range myFiles {
if err := ctx.Err(); err != nil {
return err
}
select {
case fileChan <- file:
continue
case <-done:
break
}
}
return ctx.Err()
})
var err error
go func() {
err = g.Wait()
close(dataChan)
}()
var myMap = make(map[string]string)
for data := range dataChan {
myMap[data.name] = data.bytes
}
if err != nil {
fmt.Println("errgroup Error:", err.Error())
}
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(myMap); err != nil {
panic(err)
}
}
type fileData struct {
name,
bytes string
}
func getBytesFromFile(file string, dataChan chan fileData) error {
bytes, err := openFileAndGetBytes(file)
if err == nil {
dataChan <- fileData{name: file, bytes: bytes}
}
return err
}
func openFileAndGetBytes(file string) (string, error) {
if file == "file2" {
return "", fmt.Errorf("%s cannot be read", file)
}
return fmt.Sprintf("these are some bytes for file %s", file), nil
}
Do I need a mutex in this case? I am refreshing the token with a goroutine, the token is used in another goroutine. In other words, will my token be empty at some point so that the response will be a 401?
If yes, is it part of the structure c *threatq or is it a simple variable, I mean, a "standalone" one inside my code.
// IndicatorChannelIterator returns indicators from ThreatQ into a channel.
func (c *threatq) IndicatorChannelIterator() (<-chan *models.Indicator, error) {
// Authenticate
token, err := c.authenticate(c.clientID, c.email, c.password)
if err != nil {
return nil, fmt.Errorf("Error while authenticating to TQ : %s", err)
}
// Periodically refresh the token
ticker := time.NewTicker(30 * time.Minute)
go func() {
for range ticker.C {
token, err = c.authenticate(c.clientID, c.email, c.password)
if err != nil {
logrus.Errorf("Error while authenticating to TQ : %s", err)
}
}
}()
// Prepare the query
query := &Query{}
// Get the first page
firstTQResponse, err := c.advancedSearch(query, token, 0)
if err != nil {
return nil, fmt.Errorf("Error while getting the first page from TQ : %s", err)
}
// Create the channel
indicators := make(chan *models.Indicator)
// Request the others
go func() {
req := 1
total := firstTQResponse.Total
for offset := 0; offset < total; offset += c.perPage {
// Search the indicators
tqResponse, err := c.advancedSearch(query, token, offset)
if err != nil {
logrus.Errorf("Error while getting the indicators from TQ : %s", err)
continue
}
...
The rule is simple: if a variable is accessed from multiple goroutines and at least one of them is a write, explicit synchronization is needed.
This is true in your case: one of your goroutine writes the token variable (and also the err variable!), and another reads it, so you must synchronize access.
Since token is not a field of the threatq structure, putting the mutex that protects it would not be wise. Always put the mutex close to the data it ought to protect.
Some notes: as mentioned earlier, you also write and read the local err variable from multiple goroutines. You should not do this, instead create another local variable to hold the error from other goroutines (unless you want to "translfer" the error between goroutines, but this is not the case here).
See related questions:
Immutability of string and concurrency
Should we synchronize variable assignment in goroutine?
golang struct concurrent read and write without Lock is also running ok?
Reading values from a different thread
Why does this code cause data race?
Yes, you could also try to run this test with -race flag enabled. Go's race detector will probably tell you that token is a shared variable across multiple goroutines. Thus, it must be protected with a Mutex or RWMutex.
In your case I think that RWMutex is more appropriate because there is one goroutine that changes (i.e. write) the state of token every 30 mins and another goroutine that reads its value.
If you don't protect the shared variable with a lock, the second goroutine might read an old value of token, that could be expired.
Believe I am either misunderstanding how go routines work, how buffered readers work or both.
Expect an asynchronous execution of the goroutine ( a buffered reader with a for loop reading the buffer, waiting for a message from the server )
Try METHOD A to call go xyz() before the client dials the server; so xyz() creates the buffer and starts reading in the background. Then, the client dials the server; server sends message back; the client is reading the buffer so, it gets the message and prints to console
What Actually Happens the client to send the message to the server, but does not pick up anything on the buffer while reading for possible reply from server; so it is running concurrently because I know the for loop has not stopped, but it lets the next line of code execute ( client sending message to server ).
But When METHOD B I call xyz() NOT concurrently and after the client dials the server, all things work as expected. The client gets the message back from the server and prints to console.
METHOD A, we have the order :
///////////// steps 1 and 2 are in the goroutine called by go xyz()
creates the buffered reader
for loop -- reading the buffer for message from the server -- print out
client dials the server
go xyz(conn, p)
fmt.Fprintf(conn, "Give me a hash to work on ...")
METHOD B, we have the order :
///////////// steps 2 and 3 are in the goroutine called by xyz()
client dials the server
creates buffered reader
for loop -- reading the buffer for message from the server -- print out
fmt.Fprintf(conn, "Give me a hash to work on ...")
xyz(conn, p)
client.go
package main
import (
"fmt"
"net"
"bufio"
)
func xyz(conn net.Conn, p []byte) {
rd := bufio.NewReader(conn)
for {
_, err := rd.Read(p)
if err == nil {
fmt.Printf("SERVER : %s\n", p)
} else {
fmt.Printf("Some error %v\n", err)
}
}
}
func main() {
p := make([]byte, 2048)
conn, err := net.Dial("udp", "127.0.0.1:1234")
if err != nil {
fmt.Printf("Some error %v", err)
return
}
go xyz(conn, p)
fmt.Fprintf(conn, "Give me a hash to work on ...")
}
server.go
package main
import (
"fmt"
"net"
)
func sendResponse(conn *net.UDPConn, addr *net.UDPAddr, hash string) {
_,err := conn.WriteToUDP([]byte("Hello, here is the hash - " + hash), addr)
if err != nil {
fmt.Printf("Couldn't send response %v", err)
}
}
func main() {
hash := "36";
p := make([]byte, 2048)
addr := net.UDPAddr{
Port: 1234,
IP: net.ParseIP("127.0.0.1"),
}
ser, err := net.ListenUDP("udp", &addr)
if err != nil {
fmt.Printf("Some error %v\n", err)
return
}
for {
_, remoteaddr, err := ser.ReadFromUDP(p)
fmt.Printf("CLIENT : %v : %s\n", remoteaddr, p)
if err != nil {
fmt.Printf("Some error %v", err)
continue
}
go sendResponse(ser, remoteaddr, hash)
}
}
The Go Programming Language Specification
Go statements
A "go" statement starts the execution of a function call as an
independent concurrent thread of control, or goroutine, within the
same address space.
... unlike with a regular call, program execution does not wait for
the invoked function to complete.
client.go starts the goroutine xyz and then keeps going to the end of the main function which terminates the program. The program doesn't wait for the xyz goroutine to run or finish.
I wrote simply server program to received data form client. I little not understand what sometimes I get error read tcp4 IP:PORT i/o timeout from function
int, err := conn.Read([]byte) event time set in function SetDeadline() not was exceeded. I present some part of my code, but I think that this is will enough.
main loop where I receive data is below.
c := NewClient()
c.kickTime: time.Now()
func (c *Client) Listen(){
durationToClose := time.Minute*time.Duration(5),
c.conn.SetDeadline(c.kickTime.Add(c.durationToClose))
buffer := make([]byte, 1024)
for{
reqLen, err := c.conn.Read(buffer)
if err != nil || reqLen == 0 {
fmt.Printf(err)
break
}
if err = c.CheckData(buffer) ; err != nil{
fmt.Printf("something is bad")
}else{
result := c.PrepareDataToSendInOtherPlace(buffer)
go c.RecievedData(result)
}
c.conn.SetDeadline(c.kickTime.Add(c.durationToKick))
}
}
For me only suspicious can be additional function as PrepareDataToSendInOtherPlace() , CheckData() which may take some times CPU, and then new data was be send by client, and server at the time doing something else and rejects connect. This is only my supposition, but I'm not sure.
Syntax errors and undeclared variables aside, what you're showing us can't possibly be walking the Read/Write deadline forward indefinitely.
The longest this could run is until a fixed duration after the first time.Now() (c.kickTime.Add(c.durationToKick)). You probably want something like:
c.conn.SetDeadline(time.Now().Add(c.durationToKick))