I am trying over the Bud full-stack framework and I wanted to implement a simple RSS parser to get data at each request.
I have been following pretty closely the tutorial made by Bud's creator, Matt Mueller https://www.youtube.com/watch?v=LoypcRqn-xA&t=601s and then I modified some of the code of the client he uses in the video to get hackernews json. I just want to be able to call for the data at each request and print it to the screen. However, While it compiles, I encounter an error at runtime
conjure: generate "bud/.app/controller/controller.go" > controller: unable to load definition for field Extensions in sto
ries > parser: unable to find declaration for "Extensions" in "ext"
|
I have tried using feed instead of stories as an object name, creating a struct with a pointer to gofeed.Feed but nothing worked. I hypothize this has something to do with the way I use gofeed parser but it seems I can't manage to find the issue. Do you have any suggestion on what I am doing wrong?
This is my client.go file
// Package feedparser is a simple RSS reader client that leverages gofeed
package feedparser
import (
"context"
"fmt"
"github.com/mmcdole/gofeed"
)
const baseURL = `https://ec.europa.eu/info/news/feed_en`
// Feed Client, it uses gofeed parser to open a connection with the rss/atom/json data feed
func New() *Client {
feedParser := gofeed.NewParser()
return &Client{feedParser}
}
// Feed Client
type Client struct {
*gofeed.Parser
}
// FrontPage gets the feeds and prints them as a list
func (c *Client) FrontPage(ctx context.Context) (*gofeed.Feed, error) {
feedData := `<rss version="2.0">
<channel>
<webMaster>example#site.com (Example Name)</webMaster>
</channel>
</rss>`
fd, err := c.RSSTranslator.Translate(feedData)
if err != nil {
return fd, err
}
fmt.Println(fd)
return fd, nil
}
This is my controller.go file
package controller
import (
context "context"
"github.com/AlessioNar/eunews/feedparser"
"github.com/mmcdole/gofeed"
)
type Controller struct {
// Dependencies...
FP *feedparser.Client
}
// Index of stories
// GET /
func (c *Controller) Index(ctx context.Context) (stories *gofeed.Feed, err error) {
return c.FP.FrontPage(ctx)
}
This is the original repository holding the hackernews client
https://github.com/matthewmueller/hackernews
Related
I've looked into various different tools that can be used for mock testing in golang, but I'm trying to accomplish this task using httptest. In particular, I have a function as such:
type contact struct {
username string
number int
}
func getResponse(c contact) string {
url := fmt.Sprintf("https://mywebsite/%s", c.username)
req, err := http.NewRequest(http.MethodGet, url, nil)
// error checking
resp, err := http.DefaultClient.Do(req)
// error checking
return response
}
A lot of the documentation I've read seems to require creating a client interface or a custom transport. Is there no way to mock a response in a test file without changing this main code at all? I want to keep my client, response, and all the related details within the getResponse function. I could have the wrong idea, but I'm trying to find a way to intercept the http.DefaultClient.Do(req) call and return a custom response, is that possible?
https://pkg.go.dev/net/http/httptest#example-Server is a good example for your use case with a small refactoring of your code.
You just have to change the getResponse() by getResponse(url string) to be able to give the server mock url.
I've read seems to require creating a client interface
without changing this main code at all
Keeping your code clean is a good practice and you'll finally get used to it, a testable code is cleaner and a cleaner code is more testable, so don't worry to change your code (using interfaces) so it can accept mock objects.
Your code in its simplest form can be like this:
package main
import (
"fmt"
"net/http"
)
type contact struct {
username string
number int
}
type Client interface {
Do(req *http.Request) (*http.Response, error)
}
func main() {
getResponse(http.DefaultClient, contact{})
}
func getResponse(client Client, c contact) string {
url := fmt.Sprintf("https://mywebsite/%s", c.username)
req, _ := http.NewRequest(http.MethodGet, url, nil)
// error checking
resp, _ := http.DefaultClient.Do(req)
// error checking and response processing
return response
}
And your test can be like this:
package main
import (
"net/http"
"testing"
)
type mockClient struct {
}
// Do function will cause mockClient to implement the Client interface
func (tc mockClient) Do(req *http.Request) (*http.Response, error) {
return &http.Response{}, nil
}
func TestGetResponse(t *testing.T) {
client := new(mockClient)
getResponse(client, contact{})
}
But if you prefer to use httptest:
package main
import (
"fmt"
"io"
"net/http"
"net/http/httptest"
)
type contact struct {
username string
number int
}
func main() {
fmt.Println(getResponse(contact{}))
}
func getResponse(c contact) string {
// Make a test server
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "your response")
}))
defer ts.Close()
// You should still set your base url
base_url := ts.URL
url := fmt.Sprintf("%s/%s", base_url, c.username)
req, _ := http.NewRequest(http.MethodGet, url, nil)
// Use ts.Client() instead of http.DefaultClient in your tests.
resp, _ := ts.Client().Do(req)
// Processing the response
response, _ := io.ReadAll(resp.Body)
resp.Body.Close()
return string(response)
}
Is there a way to generate OpenAPI v3 specification from go source code? Let's say I have a go
API like the one below and I'd like to generate the OpenAPI specification (yaml file) from it. Something similar to Python's Flask RESTX. I know there are tools that generate go source code from the specs, however, I'd like to do it the other way around.
package main
import "net/http"
func main() {
http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("world\n"))
})
http.ListenAndServe(":5050", nil)
}
You can employ github.com/swaggest/rest to build a self-documenting HTTP REST API. This library establishes a convention to declare handlers in a way that can be used to reflect documentation and schema and maintain a single source of truth about it.
In my personal opinion code first approach has advantages comparing to spec first approach. It can lower the entry bar by not requiring to be an expert in spec language syntax. And it may help to come up with a spec that is well balanced with implementation details.
With code first approach it is not necessary to implement a full service to get the spec. You only need to define the structures and interfaces and may postpone actual logic implementation.
Please check a brief usage example.
package main
import (
"context"
"errors"
"fmt"
"log"
"net/http"
"time"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
"github.com/swaggest/rest"
"github.com/swaggest/rest/chirouter"
"github.com/swaggest/rest/jsonschema"
"github.com/swaggest/rest/nethttp"
"github.com/swaggest/rest/openapi"
"github.com/swaggest/rest/request"
"github.com/swaggest/rest/response"
"github.com/swaggest/rest/response/gzip"
"github.com/swaggest/swgui/v3cdn"
"github.com/swaggest/usecase"
"github.com/swaggest/usecase/status"
)
func main() {
// Init API documentation schema.
apiSchema := &openapi.Collector{}
apiSchema.Reflector().SpecEns().Info.Title = "Basic Example"
apiSchema.Reflector().SpecEns().Info.WithDescription("This app showcases a trivial REST API.")
apiSchema.Reflector().SpecEns().Info.Version = "v1.2.3"
// Setup request decoder and validator.
validatorFactory := jsonschema.NewFactory(apiSchema, apiSchema)
decoderFactory := request.NewDecoderFactory()
decoderFactory.ApplyDefaults = true
decoderFactory.SetDecoderFunc(rest.ParamInPath, chirouter.PathToURLValues)
// Create router.
r := chirouter.NewWrapper(chi.NewRouter())
// Setup middlewares.
r.Use(
middleware.Recoverer, // Panic recovery.
nethttp.OpenAPIMiddleware(apiSchema), // Documentation collector.
request.DecoderMiddleware(decoderFactory), // Request decoder setup.
request.ValidatorMiddleware(validatorFactory), // Request validator setup.
response.EncoderMiddleware, // Response encoder setup.
gzip.Middleware, // Response compression with support for direct gzip pass through.
)
// Create use case interactor.
u := usecase.IOInteractor{}
// Describe use case interactor.
u.SetTitle("Greeter")
u.SetDescription("Greeter greets you.")
// Declare input port type.
type helloInput struct {
Locale string `query:"locale" default:"en-US" pattern:"^[a-z]{2}-[A-Z]{2}$" enum:"ru-RU,en-US"`
Name string `path:"name" minLength:"3"` // Field tags define parameter location and JSON schema constraints.
}
u.Input = new(helloInput)
// Declare output port type.
type helloOutput struct {
Now time.Time `header:"X-Now" json:"-"`
Message string `json:"message"`
}
u.Output = new(helloOutput)
u.SetExpectedErrors(status.InvalidArgument)
messages := map[string]string{
"en-US": "Hello, %s!",
"ru-RU": "Привет, %s!",
}
u.Interactor = usecase.Interact(func(ctx context.Context, input, output interface{}) error {
var (
in = input.(*helloInput)
out = output.(*helloOutput)
)
msg, available := messages[in.Locale]
if !available {
return status.Wrap(errors.New("unknown locale"), status.InvalidArgument)
}
out.Message = fmt.Sprintf(msg, in.Name)
out.Now = time.Now()
return nil
})
// Add use case handler to router.
r.Method(http.MethodGet, "/hello/{name}", nethttp.NewHandler(u))
// Swagger UI endpoint at /docs.
r.Method(http.MethodGet, "/docs/openapi.json", apiSchema)
r.Mount("/docs", v3cdn.NewHandler(apiSchema.Reflector().Spec.Info.Title,
"/docs/openapi.json", "/docs"))
// Start server.
log.Println("http://localhost:8011/docs")
if err := http.ListenAndServe(":8011", r); err != nil {
log.Fatal(err)
}
}
I needed to create an equivalent of ioutil.Discard that can satisfy a "WriteCloser" interface. With a bit of Googling I came up with
package main
import (
"io"
"io/ioutil"
"strings"
"fmt"
)
type discardCloser struct {
io.Writer
}
func (discardCloser) Close() error {
return nil
}
func main() {
src := strings.NewReader("Hello world")
dst := discardCloser{Writer: ioutil.Discard}
count, err := io.Copy(dst, src)
fmt.Println(count, err)
err = dst.Close()
fmt.Println(err)
}
Go playground link here
Is there a more idiomatic way of doing this?
Background: some standard library methods return a WriteCloser, such as net/smtp.Data. When implementing automated tests, it's nice to be able to exercise functions like this, while sending their output to Discard.
I took bereal's tip and looked at NopCloser. The approach works nicely, and is useful in test functions built around the libraries that require a WriteCloser.
I renamed the type myWriteCloser as it can be used to promote "real" writers such as as &bytes.Buffer, as well as the special system discard writer.
type myWriteCloser struct {
io.Writer
}
func (myWriteCloser) Close() error {
return nil
}
I have a dummy GO plugin, using that, i want send data to telegraf. But, i'm not able to find any way to send data from the plugin to telegraf.
this external Go plugin looks like below
package main
import (
"fmt"
"time"
)
type greeting string
type n int
func (g greeting) Greet() {
for i := 0; i < 10; i++ {
timer1 := time.NewTimer(2 * time.Second)
<-timer1.C
fmt.Println("value ", i)
sum := 0
sum += i * 100
fmt.Println(sum)
}
}
// exported
var Greeter greeting
And the main file looks like
import (
"fmt"
"os"
"plugin"
)
type Greeter interface {
Greet()
}
func main() {
var mod string
mod = "./eng/eng.so"
// load module
// 1. open the so file to load the symbols
plug, err := plugin.Open(mod)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// 2. look up a symbol (an exported function or variable)
// in this case, variable Greeter
symGreeter, err := plug.Lookup("Greeter")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// 3. Assert that loaded symbol is of a desired type
// in this case interface type Greeter (defined above)
var greeter Greeter
greeter, ok := symGreeter.(Greeter)
if !ok {
fmt.Println("unexpected type from module symbol")
os.Exit(1)
}
// 4. use the module
greeter.Greet()
}
Can anyone help me find a way or a direction on how to make interaction between GO plugin and telegraf work. Any heads up is appreciated.
Currently, Telegraf does not support the use of external plugins.
An issue was made and closed regarding this requested feature.
If you were planning on writing code in Golang I would suggest creating a custom Telegraf plugin. Telegraf plugins do not use go external plugin package instead they are built with Telegraf itself. Your plugin must satisfy the specific interface based on the type of plugin you want to create. Based on your question I will assume that you desire to create an input plugin, Telegraf has examples for creating each type of plugin in their source code.
I am using a golang function to print logs to fluentd. Fluent Post() method accepts only a map and not a string. I need a function to print raw text rather than forcing me to create a map.
Is that possible?
Below is my code:
package main
import (
"fmt"
"os"
"time"
"github.com/fluent/fluent-logger-golang/fluent"
)
var Flogger *fluent.Fluent
func init() {
initFluent()
}
func initFluent() {
var err error
Flogger, err = fluent.New(fluent.Config{FluentPort: 24224, FluentHost: "127.0.0.1"})
if err != nil {
fmt.Printf("Could not connect to Fluent at %s Error : %v", os.Getenv("FLUENTHOST"), err)
}
}
// DebugLog (msg string) Logs the string to Fluent server
func DebugLog(msg string) {
//Flogger.Post("debug.access", map[string]string{"data": msg})
Flogger.EncodeAndPostData("debug.access", time.Now(), map[string]string{"data": msg}) // both methods give same result
}
func main() {
msg := `{"mykey":"myval"}`
fmt.Println(msg) // prints {"mykey":"myval"}
DebugLog(msg) // prints {"data":"{\"mykey\":\"myval\"}"} with extra \
}
The function signatures as per the Fluent GoDoc as as follows:
func (f *Fluent) EncodeAndPostData(tag string, tm time.Time, message interface{}) error
func (f *Fluent) Post(tag string, message interface{}) error
Both functions accept the empty interface as the message parameter. All Go types satisfy the interface, so you can pass in anything that you'd like.
However, Fluentd is a structured logging library that primarily relies on JSON encoded data. Usually when using a third-party library, it serves well to use it the way it was intended.
Fluentd tries to structure data as JSON as much as possible: this
allows Fluentd to unify all facets of processing log data: collecting,
filtering, buffering, and outputting logs across multiple sources and
destinations
Source: Fluentd Website