I want to extend existing code that writes data to a file in an unbuffered way.
The code expects a writer that implements the io.WriteCloser interface. Therefore just wrapping the existing file with a bufio.Writer does not work, as it does not implement this interface.
How can I make a bufio.Writer to implement and pass the necessary close call to the underlaying file?
io.WriteCloser is the interface:
type WriteCloser interface {
Writer
Closer
}
Which ultimately "prescribes" these 2 methods:
Write(p []byte) (n int, err error)
Close() error
bufio.Writer already has a Write() method, so to make it a WriteCloser, only a Close() method is needed.
Extending bufio.Writer with a noop Close() method:
type MyWriteCloser struct {
*bufio.Writer
}
func (mwc *MyWriteCloser) Close() error {
// Noop
return nil
}
A value of type *MyWriteCloser is now a WriteCloser. This is the easiest extension. Using it:
bw := bufio.NewWriter(w)
mwc := &MyWriteCloser{bw}
Although we can –and we should– add a more meaningful Close() method. As bufio.Write does buffered writes, we should flush its internal buffer before we declare it closed:
func (mwc *MyWriteCloser) Close() error {
return mwc.Flush()
}
Also note that since bufio.Write cannot be closed (does not provide a Close() method), this will not close its underlying io.Writer, this is just to conform to the io.Closer and io.WriteCloser interfaces.
If you also want to close the underlying file, you also have to store it, and after calling bufio.Flush() (to make sure everything is written out), given it's not returning any errors, you may proceed to also close the file.
This is how it could look like:
type MyWriteCloser struct {
f *os.File
*bufio.Writer
}
func (mwc *MyWriteCloser) Close() error {
if err := mwc.Flush(); err != nil {
return err
}
return mwc.f.Close()
}
Using it:
// Open a file:
f, err := os.Open("myfile.txt")
if err != nil {
panic(err) // Handle error
}
mwc := &MyWriteCloser{f, bufio.NewWriter(f)}
defer mwc.Close()
// use mwc
Related
I cannot find an API to flush Write in a direct connection with a peer in libp2p (it works fine in pubsub). I create a direct libp2p connection in Go using libp2p-go using host.NewStream. The stream is used by the wrapper below. The problem is that when I write to this stream, the peer at the other end does not receive the message. The underlying network.Stream seems to be internally buffered in libp2p's networking. But I cannot find a way to flush writes anywhere in network.Stream or any of the other libp2p subpackages in go-libp2p-core. Actually I can't find it anywhere. Changing my wrapper to use bufio readers and writers does not have an effect, because it doesn't flush libp2p's internal buffering. I know the messages are sent because when I close the stream, the remote peer receives them.
// NetReaderWriter is a wrapper for a libp2p network stream to an individual peer.
type NetReaderWriter struct {
stream network.Stream
}
// NewNetReaderWriter creates a new ReadWriteCloser based on a ReadCloser and a WriteCloser.
func NewNetReaderWriter(stream network.Stream) *NetReaderWriter {
return &NetReaderWriter{
stream: stream,
}
}
// Read like in io.Reader
func (rwc *NetReaderWriter) Read(p []byte) (int, error) {
return rwc.stream.Read(p)
}
// Write like in io.Writer
func (rwc *NetReaderWriter) Write(p []byte) (int, error) {
k, err := rwc.stream.Write(p)
// <== flush here, but how???
return k, err
}
// Close like in io.Closer - this closes both the reader and writer.
func (rwc *NetReaderWriter) Close() error {
err := rwc.stream.Close()
if err != nil {
return fmt.Errorf("unable to close network stream: %v", err)
}
return nil
}
bufio.ReadWriter has a Flush function as documented here
Here is the code you can refer.
func writeData(rw *bufio.ReadWriter) {
stdReader := bufio.NewReader(os.Stdin)
for {
fmt.Print("> ")
sendData, err := stdReader.ReadString('\n')
if err != nil {
log.Println(err)
return
}
rw.WriteString(fmt.Sprintf("%s\n", sendData))
rw.Flush()
}
}
taken from the chat example.
type Product struct {
productName string
}
func (p *Product) GetProductName() string {
return p.productName
}
In Go, how should one typically handle a scenario where the receiver on a method is nil and the method logic itself yields no error (e.g a getter)?
Don't handle it, let it panic
Check for nil and return zero value if true
Check for nil and panic with meaningful message
Check for nil and enhance the method to return error on nil
Other
It really depends
I lean towards #1, but figure that while #3 is a bit verbose it could make debugging easier.
My thoughts are the calling code should be testing for nil and know what to do in such a scenario. That returning an error on a simple getter method is too verbose.
Don't handle it, let it panic
You can see examples in the go standard library. For example, in the net/http package, there is the following:
func (c *Client) Do(req *Request) (*Response, error) {
return c.do(req)
}
And, another example from encoding/json:
// Buffered returns a reader of the data remaining in the Decoder's
// buffer. The reader is valid until the next call to Decode.
func (dec *Decoder) Buffered() io.Reader {
return bytes.NewReader(dec.buf[dec.scanp:])
}
I'm trying to create a layer on top of a third party library, in this case libchan. Here's an interface I've defined:
type ReceiverStream interface {
Receive(msg interface{}) error
}
type InboundTransport interface {
WaitReceiveChannel() (ReceiverStream, error)
}
The InboundTransport is meant to be a stand-in for a type Transport:
// libchan.go
type Transport interface {
// NewSendChannel creates and returns a new send channel. The receive
// end will get picked up on the remote end of the transport through
// the remote calling WaitReceiveChannel.
NewSendChannel() (Sender, error)
// WaitReceiveChannel waits for a new channel be created by the
// remote end of the transport calling NewSendChannel.
WaitReceiveChannel() (Receiver, error)
}
Just for context, this is the libchan.Receiver definition (please note that it matches my ReceiverStream:
// libchan.go
type Receiver interface {
// Receive receives a message sent across the channel from
// a sender on the other side of the underlying transport.
// Receive is expected to receive the same object that was
// sent by the Sender, any differences between the
// receive and send type should be handled carefully. It is
// up to the application to determine type compatibility, if
// the receive object is incompatible, Receiver will
// throw an error.
Receive(message interface{}) error
}
The Transport is returned by the libchan library here:
// libchan/session.go:62
func NewTransport(provider StreamProvider) libchan.Transport {
...
}
Since libchan.Transport and InboundTransport share a WaitReceiveChannel() (ReceiverStream, error) method, I figured I should be able to sub one for the other, like so:
func (ln SpdyListener) Accept(addr string) InboundTransport {
var listener net.Listener
var err error
listener, err = net.Listen("tcp", addr)
if err != nil {
log.Fatal(err)
}
c, err := listener.Accept()
if err != nil {
log.Fatal(err)
}
p, err := spdy.NewSpdyStreamProvider(c, true)
if err != nil {
log.Fatal(err)
}
return spdy.NewTransport(p)
}
But I get an error:
cannot use spdy.NewTransport(p) (type libchan.Transport) as type InboundTransport in return argument:
libchan.Transport does not implement InboundTransport (wrong type for WaitReceiveChannel method)
have WaitReceiveChannel() (libchan.Receiver, error)
want WaitReceiveChannel() (ReceiverStream, error)
I assume that what this error means is that a type of ReceiverStream does not match libchan.Receiver, but I thought that golang interfaces were implicit, meaning that as long as the return type implements the same methods as the expected interface, it would pass compilation. Is there anything I can change so that I can superimpose a self-defined interface onto one returned by a third part library?
TLDR: A third party lib is returning an object of interface Transport. The Transport interface specifies a method WaitReceiveChannel(). I have a self-defined interface InboundTransport that also specifies WaitReceiveChannel(). The third-party method I'm calling returns an object that implements Transport by way of method WaitReceiveChannel(). I assumed that it would also implement InboundTransport since the latter also specifies a WaitReceiveChannel() of the same type. This isn't working. Why not?
As you already know interfaces in Go are satisfied implicitly.
But, as the error states,
WaitReceiveChannel() (libchan.Receiver, error)
and
WaitReceiveChannel() (ReceiverStream, error)
are two different method types, resulting in libchan.Transport not implicitly implementing InboundTransport.
To work around this you have to write a thin wrapper around libchan.Transport that implements the InboundTransport properly.
type TransportWrapper struct {
t *libchan.Transport
}
func (w *TransportWrapper) WaitReceiveChannel() (Receiver, error) {
return w.t.WaitReceiveChannel()
}
// ...
func (ln SpdyListener) Accept(addr string) InboundTransport {
var listener net.Listener
var err error
listener, err = net.Listen("tcp", addr)
if err != nil {
log.Fatal(err)
}
c, err := listener.Accept()
if err != nil {
log.Fatal(err)
}
p, err := spdy.NewSpdyStreamProvider(c, true)
if err != nil {
log.Fatal(err)
}
return &TransportWrapper{spdy.NewTransport(p)}
}
In the effort of learning Go a bit better, I am trying to refactor a series of functions which accept a DB connection as the first argument into struct methods and something a bit more "idiomatically" Go.
Right now my "data store" methods are something like this:
func CreateA(db orm.DB, a *A) error {
db.Exec("INSERT...")
}
func CreateB(db orm.DB, b *B) error {
db.Exec("INSERT...")
}
These the functions work perfectly fine. orm.DB is the DB interface of go-pg.
Since the two functions accept a db connection I can either pass an actual connection or a transaction (which implements the same interface). I can be sure that both functions issuing SQL INSERTs run in the same transaction, avoiding having inconsistent state in the DB in case either one of them fails.
The trouble started when I decided to read more about how to structure the code a little better and to make it "mockable" in case I need to.
So I googled a bit, read the article Practical Persistence in Go: Organising Database Access and tried to refactor the code to use proper interfaces.
The result is something like this:
type Store {
CreateA(a *A) error
CreateB(a *A) error
}
type DB struct {
orm.DB
}
func NewDBConnection(p *ConnParams) (*DB, error) {
.... create db connection ...
return &DB{db}, nil
}
func (db *DB) CreateA(a *A) error {
...
}
func (db *DB) CreateB(b *B) error {
...
}
which allows me to write code like:
db := NewDBConnection()
DB.CreateA(a)
DB.CreateB(b)
instead of:
db := NewDBConnection()
CreateA(db, a)
CreateB(db, b)
The actual issue is that I lost the ability to run the two functions in the same transaction. Before I could do:
pgDB := DB.DB.(*pg.DB) // convert the interface to an actual connection
pgDB.RunInTransaction(func(tx *pg.Tx) error {
CreateA(tx, a)
CreateB(tx, b)
})
or something like:
tx := db.DB.Begin()
err = CreateA(tx, a)
err = CreateB(tx, b)
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
which is more or less the same thing.
Since the functions were accepting the common interface between a connection and a transaction I could abstract from my model layer the transaction logic sending down either a full connection or a transaction. This allowed me to decide in the "HTTP handler" when to create a trasaction and when I didn't need to.
Keep in mind that the connection is a global object representing a pool of connections handled automatically by go, so the hack I tried:
pgDB := DB.DB.(*pg.DB) // convert the interface to an actual connection
err = pgDB.RunInTransaction(func(tx *pg.Tx) error {
DB.DB = tx // replace the connection with a transaction
DB.CreateA(a)
DB.CreateB(a)
})
it's clearly a bad idea, because although it works, it works only once because we replace the global connection with a transaction. The following request breaks the server.
Any ideas? I can't find information about this around, probably because I don't know the right keywords being a noob.
I've done something like this in the past (using the standard sql package, you may need to adapt it to your needs):
var ErrNestedTransaction = errors.New("nested transactions are not supported")
// abstraction over sql.TX and sql.DB
// a similar interface seems to be already defined in go-pg. So you may not need this.
type executor interface {
Exec(query string, args ...interface{}) (sql.Result, error)
Query(query string, args ...interface{}) (*sql.Rows, error)
QueryRow(query string, args ...interface{}) *sql.Row
}
type Store struct {
// this is the actual connection(pool) to the db which has the Begin() method
db *sql.DB
executor executor
}
func NewStore(dsn string) (*Store, error) {
db, err := sql.Open("sqlite3", dsn)
if err != nil {
return nil, err
}
// the initial store contains just the connection(pool)
return &Store{db, db}, nil
}
func (s *Store) RunInTransaction(f func(store *Store) error) error {
if _, ok := s.executor.(*sql.Tx); ok {
// nested transactions are not supported!
return ErrNestedTransaction
}
tx, err := s.db.Begin()
if err != nil {
return err
}
transactedStore := &Store{
s.db,
tx,
}
err = f(transactedStore)
if err != nil {
tx.Rollback()
return err
}
return tx.Commit()
}
func (s *Store) CreateA(thing A) error {
// your implementation
_, err := s.executor.Exec("INSERT INTO ...", ...)
return err
}
And then you use it like
// store is a global object
store.RunInTransaction(func(store *Store) error {
// this instance of Store uses a transaction to execute the methods
err := store.CreateA(a)
if err != nil {
return err
}
return store.CreateB(b)
})
The trick is to use the executor instead of the *sql.DB in your CreateX methods, which allows you to dynamically change the underlying implementation (tx vs. db). However, since there is very little information out there on how to deal with this issue, I can't assure you that this is the "best" solution. Other suggestions are welcome!
Is it possible to have my function definition below accept any type of struct?
I've tried to refactor like so:
// This method should accept any type of struct
// Once I receive my response from the database,
// I scan the rows to create a slice of type struct.
func generateResponse(rows *sqlx.Rows, structSlice []struct{}, structBody struct{}) ([]struct{}, error) {
for rows.Next() {
err := rows.StructScan(&structBody)
if err != nil {
return nil, err
}
structSlice = append(structSlice, structBody)
}
err := rows.Err()
if err != nil {
return nil, err
}
return structSlice, nil
}
Assume my struct is of type OrderRevenue.
When I call the function above:
structSlice, err := generateResponse(rows, []OrderRevenue{}, OrderRevenue{})
The error I get is:
cannot use []OrderRevenue literal as type []struct{} in argument...
Am I going about this the wrong way?
This is considered the cornerstone (or more of a limitation) of Go's type system. struct{} is an unnamed type that is different from struct{ field1 int } and of course is not the same as OrderRevenue{}.
Go emphasizes abstraction through interfaces, and perhaps you should try that. Here is the first take:
type OrderRevenue interface {
MarshalMyself() ([]byte, error)
}
type Anonymous struct {}
func (a Anonymous) MarshalMyself() ([]byte, error) {
// implementation's up to you
return []byte{}, nil
}
// the function signature
generateResponse(rows *sqlx.Rows, structSlice []OrderRevenue, structBody Body) ([]Body, error) {
// ...
}
In this case you can also use empty interface interface{}, which all types implement, but you'll have to recursively go through the structure to do manual type assertion. The best approach in Go is to know the shape of your data in advance, at least partially.