I needed to create an equivalent of ioutil.Discard that can satisfy a "WriteCloser" interface. With a bit of Googling I came up with
package main
import (
"io"
"io/ioutil"
"strings"
"fmt"
)
type discardCloser struct {
io.Writer
}
func (discardCloser) Close() error {
return nil
}
func main() {
src := strings.NewReader("Hello world")
dst := discardCloser{Writer: ioutil.Discard}
count, err := io.Copy(dst, src)
fmt.Println(count, err)
err = dst.Close()
fmt.Println(err)
}
Go playground link here
Is there a more idiomatic way of doing this?
Background: some standard library methods return a WriteCloser, such as net/smtp.Data. When implementing automated tests, it's nice to be able to exercise functions like this, while sending their output to Discard.
I took bereal's tip and looked at NopCloser. The approach works nicely, and is useful in test functions built around the libraries that require a WriteCloser.
I renamed the type myWriteCloser as it can be used to promote "real" writers such as as &bytes.Buffer, as well as the special system discard writer.
type myWriteCloser struct {
io.Writer
}
func (myWriteCloser) Close() error {
return nil
}
Related
I have following code that uses generics. I know that one can't use generics with methods, but can with types. Technically, my code complies with both restrictions, but still I get en error
./main.go:12:9: cannot use generic type GenericCacheWrapper[T any] without instantiation
The instantiation is on the first line of main function.
Is there any way to achieve this? May this be considered a Golang bug?
import (
"encoding/json"
"fmt"
)
type GenericCacheWrapper[T any] struct {
Container T
}
func (c GenericCacheWrapper) MarshalBinary() (data []byte, err error) {
return json.Marshal(c.Container)
}
func (c GenericCacheWrapper) UnmarshalBinary(data []byte) error {
return json.Unmarshal(data, &c.Container)
}
func main() {
wrapper := GenericCacheWrapper[int]{Container: 4}
data, err := wrapper.MarshalBinary()
if err != nil {
panic(err)
}
fmt.Println(data)
}
https://go.dev/play/p/9sWxXYmAcUH
You just have to add [T] to the end of GenericCacheWrapper or [_] if you want to make it clear that you are not actually using T in the functions.
package main
import (
"encoding/json"
"fmt"
)
type GenericCacheWrapper[T any] struct {
Container T
}
func (c GenericCacheWrapper[T]) MarshalBinary() (data []byte, err error) {
return json.Marshal(c.Container)
}
func (c GenericCacheWrapper[T]) UnmarshalBinary(data []byte) error {
return json.Unmarshal(data, &c.Container)
}
func main() {
wrapper := GenericCacheWrapper[int]{Container: 4}
data, err := wrapper.MarshalBinary()
if err != nil {
panic(err)
}
fmt.Println(data)
}
This rule is defined in the language spec:
A generic type may also have methods associated with it. In this case, the method receivers must declare the same number of type parameters as present in the generic type definition.
But the reason behind this isn't very clear, perhaps to make implementation of the compiler/type checking easier.
Related: Go error: cannot use generic type without instantiation
I'm starting to write an own prometheus exporter using golang.
I think I got the basics but I don't know what to do exactly to get the value of the metric up to date. Using Set only does it once.
It is not changing on runtime.
What I have so far:
package main
import (
"log"
"net/http"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"time"
"io/ioutil"
"github.com/tidwall/gjson"
"strconv"
)
var (
sidekiqProcessed = setGaugeMetric("sidekiq_processed", "Sidekiq Processed", "lable", "lablevalue")
)
func setGaugeMetric(name string, help string, label string, labelvalue string) (prometheusGauge prometheus.Gauge) {
var (
gaugeMetric = prometheus.NewGauge(prometheus.GaugeOpts{
Name: name,
Help: help,
ConstLabels: prometheus.Labels{label: labelvalue},
})
)
return gaugeMetric
}
func getSidekiqProcessed() (sidekiq float64) {
body := getContent("http://example.com/sidekiq/stats")
processed := gjson.Get(body, "sidekiq.processed")
conv, err := strconv.ParseFloat(processed.String(), 64)
if err != nil {
log.Fatal(err)
}
return conv
}
func getContent(url string) (body string) {
httpClient := &http.Client{Timeout: 10 * time.Second}
res, err := httpClient.Get(url)
if err != nil {
log.Fatal(err)
}
content, err := ioutil.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
return string(content)
}
func init() {
prometheus.MustRegister(sidekiqProcessed)
}
func main() {
sidekiqProcessed.Set(getSidekiqProcessed())
// The Handler function provides a default handler to expose metrics
// via an HTTP server. "/metrics" is the usual endpoint for that.
http.Handle("/metrics", promhttp.Handler())
log.Fatal(http.ListenAndServe(":8080", nil))
}
Read something about Collector but have no clue how to implement it.
Can somebody help me to complete/correct my code so that the value of the metric also updates at runtime?
Here is an example of custom collector (from https://www.robustperception.io/setting-a-prometheus-counter):
package main
import "github.com/prometheus/client_golang/prometheus"
type MyCollector struct {
counterDesc *prometheus.Desc
}
func (c *MyCollector) Describe(ch chan<- *prometheus.Desc) {
ch <- c.counterDesc
}
func (c *MyCollector) Collect(ch chan<- prometheus.Metric) {
value := 1.0 // Your code to fetch the counter value goes here.
ch <- prometheus.MustNewConstMetric(
c.counterDesc,
prometheus.CounterValue,
value,
)
}
func NewMyCollector() *MyCollector {
return &MyCollector{
counterDesc: prometheus.NewDesc("my_counter_total", "Help string", nil, nil),
}
}
// To hook in the collector: prometheus.MustRegister(NewMyCollector())
You probably want to implement a collector instead, and run the http request when the Prometheus server scrapes. See the best practices.
When implementing the collector for your exporter, you should never use the usual direct instrumentation approach and then update the metrics on each scrape.
Rather create new metrics each time. In Go this is done with MustNewConstMetric in your Update() method. For Python see https://github.com/prometheus/client_python#custom-collectors and for Java generate a List in your collect method, see StandardExports.java for an example.
The github.com/prometheus/client_golang library may be non-trivial to use when writing Prometheus exporters in Go. Try https://pkg.go.dev/github.com/VictoriaMetrics/metrics library instead. It is much easier to use in general. See the following code as an example, which allows dynamically updating sidekiq_processed metric with the given label:
import (
"fmt"
"github.com/VictoriaMetrics/metrics"
)
// UpdateSidekiqProcessed updates `sidekiq_processed{label="<labelValue>"}` metric to the given value
func UpdateSidekiqProcessed(labelValue string, value float64) {
metricName := fmt.Sprintf("sidekiq_processed{label=%q}", labelValue)
metrics.GetOrCreateFloatCounter(metricName).Set(value)
}
I am attempting to implement an interface based message queue where jobs are pushed as bytes to a redis queue. But I keep receiving an EOF error when attempting to decode the byte stream.
https://play.golang.org/p/l9TBvcn9qg
Could someone point me in the right direction?
Thank you!
In your Go Playground example, you're trying to encode an interface and interfaces don't have a concrete implementation. If you remove the interface from your A struct, that should work. Like the following:
package main
import "fmt"
import "encoding/gob"
import "bytes"
type testInterface interface{}
type A struct {
Name string
Interface *B // note this change here
}
type B struct {
Value string
}
func main() {
var err error
test := &A {
Name: "wut",
Interface: &B{Value: "BVALUE"},
}
buf := bytes.NewBuffer([]byte{})
enc := gob.NewEncoder(buf)
dec := gob.NewDecoder(buf)
// added error checking as per Mark's comment
err = enc.Encode(test)
if err != nil {
panic(err.Error())
}
result := &A{}
err := dec.Decode(result)
fmt.Printf("%+v\n", result)
fmt.Println("Error is:", err)
fmt.Println("Hello, playground")
}
Also, just as a side note you will see some sort of output like the following: &{Name:wut Interface:0x1040a5a0} because A is referencing a reference to a B struct. To clean that up further:
type A struct{
Name string
Interface B // no longer a pointer
}
func main() {
// ...
test := &A{Name: "wut", Interface: B{Value: "BVALUE"}}
// ...
}
Found the answer to the problem from Mark above. I have forgotten to do a gob.Register(B{})
https://play.golang.org/p/7rQDHvMhD7
I want to be able to unmarshal yaml files less rigidly. That is, my library has a predefined number of options the yaml file must have. Then, the user should be able to extend this to include any custom options.
Here is what I have
package main
import (
"net/http"
"yamlcms"
"github.com/julienschmidt/httprouter"
)
type Page struct {
*yamlcms.Page
Title string
Date string
}
func getBlogRoutes() {
pages := []*Page{}
yamlcms.ReadDir("html", pages)
}
// This section is a work in progress, I only include it for loose context
func main() {
router := httprouter.New()
//blogRoutes := getBlogRoutes()
//for _, blogRoute := range *blogRoutes {
// router.Handle(blogRoute.Method, blogRoute.Pattern,
// func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {})
//}
http.ListenAndServe(":8080", router)
}
Here is the yamlcms package:
package yamlcms
import (
"io/ioutil"
"os"
"strings"
"gopkg.in/yaml.v2"
)
type Page struct {
Slug string `yaml:"slug"`
File string `yaml:"file"`
}
func (page *Page) ReadFile(file string) (err error) {
fileContents, err := ioutil.ReadFile(file)
if err != nil {
return
}
err = yaml.Unmarshal(fileContents, &page)
return
}
func isYamlFile(fileInfo os.FileInfo) bool {
return !fileInfo.IsDir() && strings.HasSuffix(fileInfo.Name(), ".yaml")
}
func ReadDir(dir string, pages []*Page) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
pages[i].ReadFile(fileInfo.Name())
}
}
return
}
There is a compiler issue here:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.Page in argument to yamlcms.ReadDir
My main intent in this question is to learn the idiomatic way of doing this kind of thing in Go. Other 3rd-party solutions may exist but I am not immediately interested in them because I have problems like this frequently in Go having to do with inheritance, etc. So along the lines of what I've presented, how can I best (idiomatically) accomplish what I am going for?
EDIT:
So I've made some changes as suggested. Now I have this:
type FileReader interface {
ReadFile(file string) error
}
func ReadDir(dir string, pages []*FileReader) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
(*pages[i]).ReadFile(fileInfo.Name())
}
}
return
}
However, I still get a similar compiler error:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.FileReader in argument to yamlcms.ReadDir
Even though main.Page should be a FileReader because it embeds yamlcms.Page.
EDIT: I forgot that slices of interfaces don't work like that. You'd need to allocate a new slice, convert all pages to FileReaders, call the function, and convert them back.
Another possible solution is refactoring yamlcms.ReadDir to return the contents of the files, so that they could be unmarshaled later:
// In yamlcms.
func ReadYAMLFilesInDir(dir string) ([][]byte, error) { ... }
// In client code.
files := yamlcms.ReadYAMLFilesInDir("dir")
for i := range pages {
if err := yaml.Unmarshal(files[i], &pages[i]); err != nil { return err }
}
The original answer:
There are no such things as inheritance or casting in Go. Prefer composition and interfaces in your designs. In your case, you can redefine your yamlcms.ReadDir to accept an interface, FileReader.
type FileReader interface {
ReadFile(file string) error
}
Both yamlcms.Page and main.Page will implement this, as the latter embeds the former.
I'm still quite new to Go and I was surprised to not be able to use the subtype of an embedded interface.
Here is a small example to explain what I mean:
func test(sl bufio.ReadWriter){
// cannot use sl(type bufio.ReadWriter) as type bufio.Reader in function argument
readStuff(sl)
[...]
writeStuff(sl) // same kind of error
}
func readStuff(sl bufio.Reader){
[...]
}
As every interface have the same memory layout and ReadWriter is a Reader and a Writer, I was expecting this code to work.
I did try to convert the interface type with:
readStuff(sl.(buffio.Reader))
But it doesn't work either. So I've got two questions:
Why doesn't it work?
What's the go philosophy about that problem?
They're different types. However, a bufio.ReadWriter contains a pointer to both a bufio.Reader type and a bufio.Writer type as elements of its struct. So passing the correct one should be easy enough. Try this:
func test(sl bufio.ReadWriter){
readStuff(sl.Reader)
[...]
writeStuff(sl.Writer)
}
// Changed this bufio.Reader to a pointer receiver
func readStuff(sl *bufio.Reader) {
[...]
}
bufio.ReadWriter is a concrete type, not an interface. However, it does satisfy an interface (io.ReadWriter) so it can be assigned to a variable/function argument of an appropriate interface type. Then it works the way you may have anticipated (your code actually doesn't use any interfaces):
package main
import (
"bufio"
"bytes"
"fmt"
"io"
"log"
)
func readStuff(r io.Reader) {
b := make([]byte, 10)
n, err := r.Read(b)
if err != nil && err != io.EOF {
log.Fatal(err)
}
fmt.Printf("readStuff: %q\n", b[:n])
}
func writeStuff(w io.Writer) {
b := []byte("written")
n, err := w.Write(b)
if n != len(b) {
log.Fatal("Short write")
}
if err != nil {
log.Fatal(err)
}
}
func test(rw io.ReadWriter) {
readStuff(rw)
writeStuff(rw)
}
func main() {
r := io.Reader(bytes.NewBufferString("source"))
var uw bytes.Buffer
w := io.Writer(&uw)
rw := bufio.NewReadWriter(bufio.NewReader(r), bufio.NewWriter(w))
test(rw)
rw.Flush()
fmt.Printf("The underlying bytes.Buffer writer contains %q\n", uw.Bytes())
}
(Also here)
Output:
readStuff: "source"
The underlying bytes.Buffer writer contains "written"
This way test can consume any io.ReadWriter, not only a specific one. Which is a hint towards your question about go "philosophy".