how to read mysql data using goroutine and channel in bulk - go

Question

I'm a newbee for golang, now need to read a big amount data in mysql, so I wanna use goroutine and channel to get data in high performance, but don't know how to avoid data duplication for each goroutine and make whole process stable. for instance, table schema is as below, I wanna get all records which create_time is smaller than 1000000000000000000, I wanna create 10 goroutines and read data concurrently, each goroutine do some business logic, how to design codes? thank u
id content last_id create_time

Answer 1

I would suggest you to create a goroutine that would publish the data to your channel. Then you can add listener go routines to handle the published data.
This can be done as following:
Main :
const GoroutineCount = 10
type SomeData []int
func main() {
ch := make(chan SomeData, 1)
go PublishData(ch)
for i := 0; i < GoroutineCount; i++ {
go ProcessData(ch)
}
}
For assumptions, I have used a simple slice of int as data. This can be slice of any struct as required.
Publish data to channel:
const ChunkSize = 1000
func PublishData(ch chan SomeData) {
// Assume having 10000 records in result set
// This has to come from db transaction
res := make([]int, 10000)
// split into chunks of 1000
chunks := GetChunk(res)
// write chunk data to channel
for i := range chunks {
ch <- chunks[i]
}
}
func GetChunk(input SomeData) []SomeData {
var result []SomeData
boundary := len(input)
index := 0
for index = 0; boundary >= ChunkSize; index+=ChunkSize {
boundary -= ChunkSize
lastIndex := index+ChunkSize
result = append(result, input[index:lastIndex])
}
boundary = len(input) % ChunkSize
if boundary > 0 {
lastIndex := index+ boundary
result = append(result, input[index:lastIndex])
}
return result
}
Process individual chunks as :
func ProcessData(ch chan SomeData) {
// Read single chunk
res := <-ch
// Process chunk data
fmt.Printf("len %d\n", len(res))
}
Code on go playground: https://play.golang.org/p/X9Q991h6ru_n