I am using segmentio/kafka-go to connect to Kafka.
// to produce messages
topic := "my-topic"
partition := 0
conn, _ := kafka.DialLeader(context.Background(), "tcp", "localhost:9092", topic, partition)
conn.SetWriteDeadline(time.Now().Add(10*time.Second))
conn.WriteMessages(
kafka.Message{Value: []byte("one!")},
kafka.Message{Value: []byte("two!")},
kafka.Message{Value: []byte("three!")},
)
conn.Close()
I am able to produce into my Kafka server using this code.
// to consume messages
topic := "my-topic"
partition := 0
conn, _ := kafka.DialLeader(context.Background(), "tcp", "localhost:9092", topic, partition)
conn.SetReadDeadline(time.Now().Add(10*time.Second))
batch := conn.ReadBatch(10e3, 1e6) // fetch 10KB min, 1MB max
b := make([]byte, 10e3) // 10KB max per message
for {
_, err := batch.Read(b)
if err != nil {
// err -> "invalid codec"
break
}
fmt.Println(string(b))
}
batch.Close()
conn.Close()
But I am unable to consume using the above code. I am getting the error invalid codec. What can be the reason?
In case relevant, I tweaked the minimum batch size to 1 so that it tries to consume something.
Just a guess:
try adding an import to load compression codecs, in case your topics use compression.
import _ "github.com/segmentio/kafka-go/snappy"
Related
I run the example from the package documentation segmentio/kafka-go, but in it I get 1 message at a time.
Is there any way to read all the messages that have accumulated in Kafka at a time and parse them immediately into []MyType?
func main() {
// to consume messages
kafkaBrokerUrl := "localhost:9092"
topic := "test"
conn, err := kafka.DialLeader(context.Background(), "tcp", kafkaBrokerUrl, topic, 0)
if err != nil {
log.Fatal("failed to dial leader:", err)
}
batch := conn.ReadBatch(10e3, 1e6) // fetch 10KB min, 1MB max
b := make([]byte, 10e3) // 10KB max per message
for {
n, err := batch.Read(b)
if err != nil {
break
}
fmt.Println(string(b[:n]))
}
}
Kafka doesn't help in batch processing , if you mean something like sliding window mechanism to take a batch of streams for a particular time ,size better to use a apache kafka and apache flink connector , flink provides sliding window mechanism , hdfs storage and moreover flink gives a much better fault tolerance by checkpoints . Unfortunately the flink go sdk available is unstable.
Problem
I have written a TCP echo server in Go and I am trying to write/read as often as I can in 10s to measure how much data got transfered in this time. Weirdly, the value is way too high and does not depend on the length of the bytearray which I am transfering (but it should!). It is always around 600k connections in this 10 seconds (The length of the "result" Array depicts how much connections were made in the 10s). As soon as I add let's say a print statement to the server and the values get processed, I get more realistic values that depend on the length of the bytearray as a result.
Why doesn't the length of the bytearray matter in the first case?
Code
Server
package main
import (
"fmt"
"log"
"net"
)
func main() {
tcpAddr, err := net.ResolveTCPAddr("tcp", fmt.Sprintf("127.0.0.1:8888"))
checkError(err)
ln, err := net.ListenTCP("tcp", tcpAddr)
checkError(err)
for {
conn, err := ln.Accept()
checkError(err)
go handleConnection(conn)
}
}
func checkError(err error) {
if err != nil {
log.Fatal(err)
}
}
func handleConnection(conn net.Conn) {
var input [1000000]byte
for {
n, err := conn.Read(input[0:])
checkError(err)
//fmt.Println(input[0:n])
_, err = conn.Write(input[0:n])
checkError(err)
}
}
Client
package main
import (
"fmt"
"log"
"net"
"time"
)
var (
result []int
elapsed time.Duration
)
func main() {
input := make([]byte, 1000)
tcpAddr, err := net.ResolveTCPAddr("tcp", "127.0.0.1:8888")
checkError(err)
conn, err := net.DialTCP("tcp", nil, tcpAddr)
checkError(err)
for start := time.Now(); time.Since(start) < time.Second*time.Duration(10); {
startTimer := time.Now()
_, err = conn.Write(input)
checkError(err)
_, err := conn.Read(input[0:])
checkError(err)
elapsed = time.Since(startTimer)
result = append(result, int(elapsed))
}
fmt.Println(fmt.Sprintf("result: %v", len(result)))
}
func checkError(err error) {
if err != nil {
log.Fatal(err)
}
}
Read in the client loop is not guaranteed to read all of the data sent in the previous call to Write.
When input is small enough to be transmitted in a single packet on the network, Read in the client returns all of the data in the previous call to Write in the client. In this mode, the application measures the time to execute request/response pairs.
For larger sizes of input, read on the client can fall behind what the client is writing. When this happens, the calls to Read complete faster because the calls return data from an earlier call to Write. The application is pipelining in this mode. The throughput for pipelining is higher than the throughput for request/response pairs. The client will not read all data in this mode, but the timing impact of that is not significant.
Use the following code to time request/response pairs for arbitrary sizes of input.
for start := time.Now(); time.Since(start) < time.Second*time.Duration(10); {
startTimer := time.Now()
_, err = conn.Write(input)
checkError(err)
_, err := io.ReadFull(conn, input) // <-- read all of the data
checkError(err)
elapsed = time.Since(startTimer)
result = append(result, int(elapsed))
}
To measure full-on pipelining, modify the client to read and write from different goroutines. An example follows.
go func() {
for start := time.Now(); time.Since(start) < time.Second*time.Duration(10); {
_, err = conn.Write(input)
checkError(err)
}
conn.CloseWrite() // tell server that we are done sending data
}()
start := time.Now()
output := make([]byte, 4096)
for {
_, err := conn.Read(output)
if err != nil {
if err == io.EOF {
break
}
checkError(err)
}
}
fmt.Println(time.Since(start))
I was sending some data using *ipconn.Write method in go , but it seems *ipconn.Read() can only read 20 bytes at a time
here is server sending data
ln, err := net.Listen("tcp", "localhost:8888")
conn, err := ln.Accept()
tmp := make([]byte,10000)
tmp = []byte("abcdefghijklmnopqrstuvwxyz")
conn.Write(tmp)
here is the client receiving data
conn, err := net.Dial("tcp", "localhost:8888")
data := make([]byte, 100000)
conn.Read(data)
fmt.Println(string(data)) // prints only first 20 chars
If i again call conn.Read(data) I get another 20 characters
Is there any way to read all the data or loop until the connection is closed ?
ioutil.ReadAll will read all the data from the tcp stream. Be careful as it doesnt return until the connection has been closed.
When I run this code an incoming UDP packet gets read in, however no packet gets sent back out. Why is this? (I verified this fact with wireshark). I want to be able to communicate two ways over a UDP connection, how do I achieve this with golang?
//Node 1
func main() {
addr := net.UDPAddr{
Port: 7000,
IP: net.ParseIP("127.0.0.1"),
}
conn, err := net.ListenUDP("udp", &addr)
defer conn.Close()
if err != nil {
panic(err)
}
for {
b := make([]byte, 10)
conn.Read(b)
fmt.Println(string(b[:]))
conn.Write([]byte("sending back"))
}
}
func main() {
sock, _ := net.Dial("udp", "127.0.0.1:7000")
buf := make([]byte, 10)
sock.Write([]byte("first send"))
sock.Read(buf)
fmt.Println(string(buf[:]))
}
Remember, UDP is connection-less. When you call conn.Write, your listener doesn't know where to send the packet. In your server code, you should be using UDPConn.ReadFromUDP and UDPConn.WriteToUDP to obtain and specify the client address, as mentioned in the documentation:
The returned connection's ReadFrom and WriteTo methods can be used to receive and send UDP packets with per-packet addressing.
Your modified Node 1 loop could then look something like the following:
for {
b := make([]byte, 10)
n, clientAddr, _ := conn.ReadFromUDP(b) // TODO: error check
fmt.Println(string(b[:n]))
conn.WriteToUDP([]byte("sending back"), clientAddr)
}
I'm using gob to send messages from client to serve and this is working, however when the server response to client this don't read from connection.
func servidor(porta int){
var site string
addr := net.UDPAddr{
Port: porta,
IP: net.ParseIP("localhost"),
}
conn, erro := net.ListenUDP("udp", &addr)
verificaErro(erro)
enc := gob.NewEncoder(conn)
dec := gob.NewDecoder(conn)
dec.Decode(&site)
enc.Encode(site)
}
func cliente(site string){
porta := "1200"
conn := ConnectToSocket("localhost:"+porta)
enc := gob.NewEncoder(conn)
dec := gob.NewDecoder(conn)
enc.Encode("Test")
dec.Decode(&site)
fmt.Println(site)
}
How can I solve this problem?
There are two issues:
UDP is a packet based protocol, but gob is expecting a stream. The encoder can call the connection Write method multiple times when encoding a value. Each write sends a packet. You probably want one packet per encoded value.
Gob streams have state. The encoder sends information about any type once and expects the decoder to remember the information. Even if the gob encoder calls Write exactly once per encoded value, packets sent by subsequent writes will not include the type information.
The fix is to encode to a buffer and send that buffer:
var buf Bytes.Buffer
if err := gob.NewEncoder(&buf).Encode(value); err != nil {
// handle error
}
_, err := c.WriteTo(buf.Bytes(), addr)
Receive to a buffer and decode from that buffer:
buf := make([]byte, 1024)
n, addr, err := c.ReadFrom(buf)
if err != nil {
// handle error
}
if err := gob.NewDecoder(bytes.NewReader(buf[:n])).Decode(&v); err != nil {
// handle error
}