Goroutine behaves differently on Windows and Linux - go

I`m new to the GO. I have a following legacy code.
var db *sql.DB
func init() {
go feedChan()
connString := os.Getenv("DB_CONN")
var err error
db, err = sql.Open("postgres", connString)
if err != nil {
log.Fatalf("Failed to connect to database at %q: %q\n", connString, err)
}
// confirm connection
if err = db.Ping(); err != nil {
log.Fatalf("Unable to ping database at %q: %q\n", connString, err)
}
}
func feedChan() {
selectQuery, err := db.Prepare(`
SELECT id, proxy
FROM proxy
WHERE fail_count < 2
ORDER BY date_added DESC, last_used ASC, fail_count ASC
LIMIT 5
`)
....
Following code works on linux. But it fails on windows with nil error on
selectQuery, err := db.Prepare(`
Which make sense for me, since db initialized after a launch of feedChan goroutine. What doesnt make sense for me is why it work on linux.
So the question is why this code work at linux without errors?

That's probably a race condition. Import "time", put this line after go feedChan(), and see if it still works on Linux:
time.Sleep(3 * time.Second)
In order to avoid this situation, you could either initialize db before you spawn the routine (which uses db) or use some sort of barrier:
func init() {
barrier := make(chan int)
go feedChan(barrier)
connString := os.Getenv("DB_CONN")
var err error
db, err = sql.Open("postgres", connString)
if err != nil {
log.Fatalf("Failed to connect to database at %q: %q\n", connString, err)
// Retry.
} else {
barrier <- 1 // Opens barrier.
}
// ...
}
func feedChan(barrier chan int) {
<-barrier // Blocks until db is ready.
selectQuery, err := db.Prepare(`
SELECT id, proxy
FROM proxy
WHERE fail_count < 2
ORDER BY date_added DESC, last_used ASC, fail_count ASC
LIMIT 5
`)
// ...
}

After reading the first lines of your functions I just can say you that your legacy code has a huge bug and it can be easily fixed just moving this line go feedChan() to the end of the init() function.
Also note the main reason is not a race condition, just a matter of wait for the correct initialization of db variable.

Related

Cannot identify an error in sync.Once usage

I'm doing an online course on Golang. The following piece of code is presented in the course material as an example of misuse of sync.Once:
var (
once sync.Once
db *sql.DB
)
func DbOnce() (*sql.DB, error) {
var err error
once.Do(func() {
fmt.Println("Am called")
db, err = sql.Open("mysql", "root:test#tcp(127.0.0.1:3306)/test")
if err != nil {
return
}
err = db.Ping()
})
if err != nil {
return nil, err
}
return db, nil
}
Supposedly, the above is a faulty implementation of an SQL connection manager. We, the students, are to find the error ourselves, which I struggle with. The code runs fine even in parallel. This is how I used it:
func main() {
wg := sync.WaitGroup{}
wg.Add(10)
for i := 0; i < 10; i++ {
go (func() {
db, err := DbOnce()
if err != nil {
panic(err)
}
var v int
r := db.QueryRow("SELECT 1")
err = r.Scan(&v)
fmt.Println(v, err)
wg.Done()
})()
}
wg.Wait()
}
I understand that homework questions are discouraged here, so I'm not asking for a complete solution, just a hint would be fine. Is the error related to concurrency (i.e. I need to run it in a specific concurrent context)? Is it usage of sql.Open specifically?
Initialization of the db variable is OK. The problem is with the returned error.
If you call DbOnce() for the first time and opening a DB connection fails, that error will be returned properly. But what about subsequent calls? The db initialization code will not be run again, so nil db may be returned, and since the initialization code is not run, the default value of the err variable is returned, which will be nil. To sum it up, the initialization error is lost and will not be reported anymore.
One solution is to stop the app if connection fails (at the first call). Another option is to store the initialization error too in a package level variable along with db, and return that from DbOnce() (and not use a local variable for that). The former has the advantage that you don't have to handle errors returned from DbOnce(), as it doesn't even have to return an error (if there's an error, your app will terminate).
The latter could look like this:
var (
once sync.Once
db *sql.DB
dbErr error
)
func DbOnce() (*sql.DB, error) {
once.Do(func() {
fmt.Println("Am called")
db, dbErr = sql.Open("mysql", "root:test#tcp(127.0.0.1:3306)/test")
if dbErr != nil {
return
}
dbErr = db.Ping()
})
return db, dbErr
}

How to dump huge csv data (4GB) into mysql

If anyone had tried this before using Go, please get the idea with code, that would be really appreciated.
I wrote few line which is slow
// This is to read the csv file
func usersFileLoader(filename string, channel chan User) {
defer close(channel)
file, err := os.Open(filename)
if err != nil {
panic(err)
}
defer file.Close()
var user User
reader := csv.NewReader(file)
for {
err := Unmarshal(reader, &user)
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
channel <- user
}
}
// This is to insert csv file
func saveUser(channel <-chan User, db *sql.DB) {
stmt, err := db.Prepare(`
INSERT INTO Users( id, name, address) values ( ?, ?, ?)`)
if err != nil {
log.Fatal(err)
}
for usr := range channel {
_, err := stmt.Exec(
usr.ID,
usr.Name,
usr.Address,
)
if err != nil {
log.Fatal(err)
}
}
}
// here is the struct of the user
type User struct {
ID int `csv:"id"`
Name int `csv:"name"`
Address int `csv:"address"`
}
// here is my main func
func main() {
db := DBconnect(ConnectionString(dbConfig()))
channel := make(chan User)
go usersFileLoader("../user.csv", channel)
saveUser(channel, db)
defer db.Close()
}
// This code is working but slow for me.
Share your thought and ideas
I wouldn't attempt to use Go's built in standard library functions for loading a very large CSV file into MySQL (unless, of course, you are simply trying to learn how they work).
For best performance I would simply use MySQL's built in LOAD DATA INFILE functionality.
For example:
result, err := db.Exec("LOAD DATA INFILE ?", filename)
if err != nil {
log.Fatal(err)
}
log.Printf("%d rows inserted\n", result.RowsAffected())
If you haven't used LOAD DATA INFILE before, note carefully the documentation regarding LOCAL. Depending on your server configuration and permissions, you might need to use LOAD DATA LOCAL INFILE instead. (If you intend to use Docker containers, for instance, you will absolutely need to use LOCAL.)

jmoiron/sqlx, ...interface{}, and abstracting some boilerplate

I thought I'd try to be a little bit "clever" and abstract some of my boilerplate SQL code (using sqlx -- https://github.com/jmoiron/sqlx). The idea is to feed a code a function pointer to process the result, along with the sql string and args that produce the rows. As it happens, the code works fine provided I strip out the "sqlArgs interface" stuff, but in the "cleverer" format errors with the statement
sql: converting Exec argument $1 type: unsupported type []interface {}, a slice of interface
Here are two versions, the first one that errors, the second that works but without parameterization:
//GetRows (doesn't work)
func GetRows(parseRows func(*sqlx.Rows), sql string, sqlArgs ...interface{}) {
db := sqlx.MustConnect("mysql", ConnString)
defer db.Close()
rows, err := db.Queryx(sql, sqlArgs)
defer rows.Close()
if err != nil {
panic(err)
}
parseRows(rows)
}
//GetRows ... (works, but doesn't allow parameterization)
func GetRows(fp func(*sqlx.Rows), sql string) {
db := sqlx.MustConnect("mysql", ConnString)
defer db.Close()
rows, err := db.Queryx(sql)
defer rows.Close()
if err != nil {
panic(err)
}
fp(rows)
}
The idea is to call the code something like this:
func getUser(userID string) User {
var users []*User
parseRows := func(rows *sqlx.Rows) {
for rows.Next() {
var u User
err := rows.StructScan(&u)
if err != nil {
panic(err)
}
users = append(users, u)
}
}
sql := "SELECT * FROM users WHERE userid = ?;"
sqlutils.GetRows(parseRows, sql, userID)
if len(users) == 1{
return users[0]
}
return User{}
}
I guess my code doesn't actually pass through the userID from call to call, but instead it passes an []interface{}, which the sql package can't handle. I'm not sure about that, however. In any case, is there any way to accomplish this idea? Thanks.

SQL Query finishes at server but program never resumes

I am connecting my go scripts to redshift using go-lang postgres driver. When query takes 5+ minutes to complete, my program never gets its control back. After checking the query at redshift-server I do see that query completed in ~7 minutes.
Not sure why is this happening.
My code
func truncate_and_populate_set_1(db *sql.DB, parameter string){
insert_q := `...`
db := GetDB()
util.ExeQ(db, insert_q)
log.Println("Done adding records to table")
}
func GetDB() *sql.DB {
connection_string := "postgres://%s:%s#host"
db, err := sql.Open("postgres", connection_string)
if err != nil {
fmt.Println(err)
}
return db
}
func ExeQ(db *sql.DB, query string) {
_, err := db.Exec(query)
if err != nil {
log.Fatal(err)
}
}
You need to alter the keep alive behavior of the library that's managing the Redshift connection. Unfortuantely I can't advise you on how to do that in Go.
For a JDBC URL you could append the options:
jdbc:redshift://my-cluster … :5439/user?tcpKeepAlive=true&TCPKeepAliveMinutes=2
See the documentation here for more options: http://docs.aws.amazon.com/redshift/latest/mgmt/troubleshooting-connections.html

'go test' returns error "gocql: no response received from cassandra within timeout period."

So i want to test out an API that interacts with Cassandra on my local machine. In my func TestMain(m *testing.M) function, i want to clear the tables before running the tests. The TestMain function looks like this...
func TestMain(m *testing.M) {
keyspace = "staging"
cassandra.SetKeyspace(keyspace)
client = http.DefaultClient
// Ensure all tables are empty before tests run
err := ClearAllTables()
if err != nil {
logrus.Errorf("Failed to clear all tables: %v.", err)
os.Exit(1)
}
// Run tests
statusCode := m.Run()
os.Exit(statusCode)
}
The ClearAllTables function looks like this...
func ClearAllTables() (err error) {
// Create a DB session
session := cassandra.CreateSession()
defer session.Close()
// Get names of all existing tables
tableNameList := []string{"cliques", "users"}
// Remove all rows from each table
var count int
for _, tableName := range tableNameList {
if err := session.Query(`TRUNCATE TABLE ` + tableName).Exec(); err != nil {
return err
}
}
return nil
}
For some reason when i try to TRUNCATE the table, Cassandra times out, and i get the error...
level=error msg="Failed to clear all tables: gocql: no response received from cassandra within timeout period."
This only seems to happen when i test the program. I wrote a snippet of code in the main function that works fine.
func main () {
session := cassandra.CreateSession()
defer session.Close()
if err := session.Query(`TRUNCATE TABLE cliques`).Exec(); err != nil {
return
}
fmt.Println("Table truncated") //works
}
I also wrote a snippet of code that returns some rows from the database, and that also works fine in the main function.
Here is how i create my cassandra session...
// CreateSession connect to a cassandra instance
func CreateSession() *gocql.Session {
// Connect to the cluster
cluster := gocql.NewCluster("127.0.0.1")
cluster.Keyspace = "staging"
cluster.ProtoVersion = 4
cluster.CQLVersion = "3.0.0"
cluster.Consistency = gocql.One
session, err := cluster.CreateSession()
if err != nil {
logrus.Errorf("Failed to connect to Cassandra: %v.", err)
}
return session
}
I'm i missing anything, Why would Cassandra work fine with go run main.go but doesn't work with go test?

Resources