func CheckKafkaReadPartitions(kafkabroker string, topic string, conf config.Config) bool {
var conn *kafka.Conn
if conf.TlsEnabled {
d := &kafka.Dialer{
TLS: &tls.Config{},
}
conn, err := d.Dial("tcp", kafkabroker)
log.Info("conn is: ", conn)
log.Info("Using TLS connection")
if err != nil {
log.WithError(err).Warn("Kafka broker connection error")
return false
}
defer conn.Close()
} else {
conn, err := kafka.Dial("tcp", kafkabroker)
log.Info("conn is: ", conn)
log.Info("Using Plaintext connection")
if err != nil {
log.WithError(err).Warn("Kafka broker connection error")
return false
}
defer conn.Close()
}
log.Info("conn is: ", conn)
log.Info("Reading Partitions")
partitions, err := conn.ReadPartitions()
// SOME OTHER WORK
}
I noticed that when calling ReadPartitions() method, conn is empty even after affecting values to it either in conn, err := kafka.Dial("tcp", kafkabroker) or conn, err := d.Dial("tcp", kafkabroker)
What am I missing? Is there any way I could take conn var out of that if/else block without emptying its content?
So basically what happens here is a variable shadowing.
Go has variable scopes, you can have a variable in a global scope by defining it outside of a function. Then you'll be able to use this variable anywhere in the same package (or if it is exported anywhere in your code).
Then you have the variables that are defined in a block of code. Similar to var conn *kafka.Conn you can access this variable from everywhere in the block it was defined (and all the sub-blocks).
Think the blocks as the code that is enclosed by the curly brackets {}
This means that the if/else are separate blocks under the func block.
Now what you need to understand is the difference between the = and :=
= is used to assign a value to a variable while := is a shorthand that is used to declare and assign a variable.
So by using conn, err := d.Dial("tcp", kafkabroker) code what you essentially do is declare new variables in the if block and assign the values to them from the returned values of the d.Dial func cal.
There are some cases that you may want to do that. The most common case is when you have a for loop that launches goroutines that uses a variable from an outer block.
Related
I have many of net.Conn and i want to implement multi writer. so i can send data to every available net.Conn.
My current approach is using io.MultiWriter.
func route(conn, dst []net.Conn) {
targets := io.MultiWriter(dst[0], dst[1])
_, err := io.Copy(targets, conn)
if err != nil {
log.Println(err)
}
}
but the problem is, i must specify every net.Conn index in io.MultiWriter and it will be a problem, because the slice size is dynamic.
when i try another approach by pass the []net.Conn to io.MultiWriter, like code below
func route(conn, dst []net.Conn) {
targets := io.MultiWriter(dst...)
_, err := io.Copy(targets, conn)
if err != nil {
log.Println(err)
}
}
there is an error "cannot use mirrors (variable of type []net.Conn) as []io.Writer value in argument to io.MultiWriter"
Is there proper way to handle this case? so i can pass the net.Conn slice to io.MultiWriter.
Thank you.
io.MultiWriter() has a param of type ...io.Writer, so you may only pass a slice of type []io.Writer.
So first create a slice of the proper type, copy the net.Conn values to it, then pass it like this:
ws := make([]io.Writer, len(dst))
for i, c := range dst {
ws[i] = c
}
targets := io.MultiWriter(ws...)
I'm new to GO, and I want to achieve something basic. passing a couple of variables (ordersPubsub, ordersWriter) to a function and get them back afterward.
main.go
var rdb *redis.Client
var ordersPubsub redis.PubSub
var ordersWriter csv.Writer
func main() {
rdb = redis.NewClient(&redis.Options{
Network: "tcp",
Addr: "addr",
Password: "password",
})
go utils.SaveData("orders", rdb, &ordersPubsub, &ordersWriter)
time.Sleep(2 * time.Second)
ordersWriter.Flush()
}
utils/subscriber.go
func SaveData(channelSub string, rdb *redis.Client, pubSub *redis.PubSub, writer *csv.Writer) {
csvfile, err := os.Create("data/" + channelSub + ".csv")
if err != nil {
log.Fatalf("failed creating file: %s", err)
}
writer = csv.NewWriter(csvfile)
ctx := context.TODO()
pubSub = rdb.Subscribe(ctx, channelSub)
for msg := range channel {
order := Order{}
json.Unmarshal([]byte(msg.Payload), &order)
var row []string
row = append(row, order.OrderID)
writer.Write(row)
}
}
I tested another type of the variable and it works as expected, but for this, I'm getting the following errors :
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0xf5788d]
You are passing pointers to a function, and then setting those pointers in the function. That only assigns the copy of the pointer passed in, not the actual variable you passed to the function in the first place.
For your specific problem, you don't need to pass those variables to the goroutine because the goroutine can already access and set them. In the general case, you have to set the contents of those pointers:
*writer = *csv.NewWriter(csvfile)
ctx := context.TODO()
*pubSub = *rdb.Subscribe(ctx, channelSub)
Or, you have to declare them as pointers, pass the address of them:
var ordersPubsub *redis.PubSub
var ordersWriter *csv.Writer
...
*writer = csv.NewWriter(csvfile)
ctx := context.TODO()
*pubSub = rdb.Subscribe(ctx, channelSub)
Also note that you will be using ordersWriter from two goroutines without synchronization, so it will be a data race.
I have around 220'000 image files (.png) to create. I run into this error message when trying to create the 1'081th file:
panic: open /media/Snaps/pics/image1081_0.png: too many open files
I've added the defer w.Close() line but it did not change the error.
i := 1
for i <= 223129 {
(some other code to prepare the data and create the chart)
img := vgimg.New(450, 600)
dc := draw.New(img)
canvases := table.Align(plots, dc)
plots[0][0].Draw(canvases[0][0])
plots[1][0].Draw(canvases[1][0])
plots[2][0].Draw(canvases[2][0])
testFile := "/media/Snaps/pics/image"+strconv.Itoa(i+60)+"_"+gain_loss+".png"
w, err := os.Create(testFile)
if err != nil {
panic(err)
}
defer w.Close()
png := vgimg.PngCanvas{Canvas: img}
if _, err := png.WriteTo(w); err != nil {
panic(err)
}
//move to next image
i = i + 1
}
Surely this limit can be worked around ? Maybe I'm not closing the files properly ?
The Go Programming Language Specification
Defer statements
A "defer" statement invokes a function whose execution is deferred to
the moment the surrounding function returns, either because the
surrounding function executed a return statement, reached the end of
its function body, or because the corresponding goroutine is
panicking.
DeferStmt = "defer" Expression .
The expression must be a function or method call; it cannot be
parenthesized. Calls of built-in functions are restricted as for
expression statements.
Each time a "defer" statement executes, the function value and
parameters to the call are evaluated as usual and saved anew but the
actual function is not invoked. Instead, deferred functions are
invoked immediately before the surrounding function returns, in the
reverse order they were deferred. If a deferred function value
evaluates to nil, execution panics when the function is invoked, not
when the "defer" statement is executed.
In other words, if you are processing files in a loop, put the processing for a single file in a separate function to pair the Open with the defer Close(). This avoids the "too many open files" error.
For example, use a file processing structure like this to guarantee each file is closed immediately after use.
package main
import (
"fmt"
"io/ioutil"
"os"
)
// process single file
func processFile(name string) error {
f, err := os.Open(name)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
fmt.Println(fi.Name(), fi.Size())
return nil
}
func main() {
wd, err := os.Getwd()
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
fis, err := ioutil.ReadDir(wd)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
// process all files
for _, fi := range fis {
processFile(fi.Name())
if err != nil {
fmt.Fprintln(os.Stderr, err)
}
}
}
Playground: https://play.golang.org/p/FrBWqlMOzaS
Output:
dev 1644
etc 1644
tmp 548
usr 822
Deferred statements are not executed until the surrounding function returns, that is why your files stay open until after the for-loop.
To fix this you can simply insert an anonymous function call inside the loop:
for ... {
func() {
w, err := os.Create(testFile)
if err != nil {
panic(err)
}
defer w.Close()
...
}()
}
That way, after each iteration of the loop, the current file is closed.
ok, i got it, changed defer w.Close() to w.Close() and moved it after
png := vgimg.PngCanvas{Canvas: img}
if _, err := png.WriteTo(w); err != nil {
panic(err)
}
I'm now above 10'000 images and running...
I'm confusing about the reassignment of the err variable for errors in Go.
For example, I tend to be doing this:
err1 := Something()
checkErr(err1)
str, err2 := SomethingElse()
checkErr(err2)
err3 := SomethingAgain()
checkErr(err3)
But I'm always losing track of this and have millions of useless err variables floating around that I don't need, and it makes the code messy and confusing.
But then if I do this:
err := Something()
checkErr(err)
str, err := SomethingElse()
checkErr(err)
err := SomethingAgain()
checkErr(err)
...it gets angry and says err is already assigned.
But if I do this:
var err error
err = Something()
checkErr(err)
str, err = SomethingElse()
checkErr(err)
err = SomethingAgain()
checkErr(err)
...it doesn't work because str needs to be assigned with :=
Am I missing something?
you're almost there... at the left side of := there needs to be at least one newly create variable. But if you don't declare err in advance, the compiler tries to create it on each instance of :=, which is why you get the first error. so this would work, for example:
package main
import "fmt"
func foo() (string, error) {
return "Bar", nil
}
func main() {
var err error
s1, err := foo()
s2, err := foo()
fmt.Println(s1,s2,err)
}
or in your case:
//we declare it first
var err error
//this is a normal assignment
err = Something()
checkErr(err)
// here, the compiler knows that only str is a newly declared variable
str, err := SomethingElse()
checkErr(err)
// and again...
err = SomethingAgain()
checkErr(err)
I am a newbee to golang, and I write a program to test io package:
func main() {
readers := []io.Reader{
strings.NewReader("from string reader"),
bytes.NewBufferString("from bytes reader"),
}
reader := io.MultiReader(readers...)
data := make([]byte, 1024)
var err error
//var n int
for err != io.EOF {
n, err := reader.Read(data)
fmt.Printf("%s\n", data[:n])
}
os.Exit(0)
}
The compile error is "err declared and not used". But I think I have used err in for statement. Why does the compiler outputs this error?
The err inside the for is shadowing the err outside the for, and it's not being used (the one inside the for). This happens because you are using the short variable declaration (with the := operator) which declares a new err variable that shadows the one declared outside the for.