I am using buf to generate grpc server and client code for several languages (go, python, js, java, c#), while using grpc-ecosystem/plugins/openapiv2 plugin to generate swagger documentation from the same proto files.
In some files I'm using custom
option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_tag) = {description: "Manage datasets and examples used for training."}; to add additional metadata to the documentation. This requires me to import annotations.proto from grpc-gateway project which causes the imports to also appear in generated source files. Now languages like go for example can handle this by using import for side effects
import (
_ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2/options"
_ "google.golang.org/genproto/googleapis/api/annotations"
)
but in java and c# there are some lines that are being generated which look like this
registry.add(com.google.api.AnnotationsProto.http);
registry.add(grpc.gateway.protoc_gen_openapiv2.options.Annotations.openapiv2Tag);
which causes compilation errors, because package grpc.gateway... does not exist (I was able to import the googleapis package via Maven and nuget). When I remove the options from .proto files there are no issues and I can compile the source files to a package for distribution. Is there any way to exclude these imports from generated code?
I have tried separating the documentation to its own files, but it's impossible to do with metadata which are part of Service or Message definitions as I'm getting duplicate definition errors.
Since there is no official Java library that corresponds to that annotations.proto file, you need to generate your own or do some shenanigans to modify the compiled descriptors before generating the Java code. I'll explain both.
You can generate Java code for annotations.proto, not just your own proto that imports it.
If you use Buf, you can actually tell it to generate source code for your imports using a --include-imports flag to buf generate (though this generates sources for all imports, not just ones that don't otherwise have a corresponding Java library).
These annotations are available in Buf's Schema Registry, so you could also separately generate these files and compile them into a separate JAR using buf generate buf.build/grpc-ecosystem/grpc-gateway.
One issue with this file is that it does not declare a Java package option in the file. That's why the Java package in the generated code doesn't have a proper reverse domain name. With Buf, you could use managed mode to actually inject a Java package option, to generate these files into whatever package you want (so you could generate them as if they were in a "shaded" package in your own JAR).
This second route is much less advised and not for the faint of heart, but it lets you omit the import in the generated code. You need to first compile your sources to a file descriptor set. (Buf can produce this via buf build; With protoc, use a -o option instead of --java_out.) This file is a binary encoded FileDescriptorSet.
You could write something that reads this file (unmarshalling its contents into a FileDescriptorSet) and then modifies it. You'd modify it by examining the dependency field of every file in the set and removing the entries like "protoc-gen-openapiv2/options/annotations.proto".
You can then re-marshal this to a file and feed that back in for code generation. So instead of generating Java code from sources, you'd generate them from the modified descriptor set (buf generate <file>#format=bin if using Buf or protoc --descriptor_set_in=<file> --java_out=<output-dir> if using protoc).
Note that this approach can only work if the only things that use the import being removed are custom options. That's because custom options can safely be represented as unrecognized fields in the descriptor (and effectively ignored). If you remove an import that has type definitions that are referenced in the file, the compiler will not accept the modified file descriptor set.
If that last bullet looks like Greek, it's because that is quite advanced descriptor-fiddling. Realistically, I think the first bullet is your best approach.
I am trying to use the promql package here
package main
import (
"fmt"
"github.com/prometheus/prometheus/promql/parser"
)
func main() {
fmt.Println("Hello")
parser.ParseExpr("foobar")
}
Having trouble importing. This is the error:
no required module provides package
github.com/prometheus/prometheus/promql/parser; to add it:
go get github.com/prometheus/prometheus/promql/parser (compile)
I tried to run go get github.com/prometheus/prometheus/promql/parser as suggested but it fails.
go get: module github.com/prometheus/prometheus#upgrade found
(v2.5.0+incompatible), but does not contain package
github.com/prometheus/prometheus/promql/parser
Here is my go.mod currently:
module foo.com/bar/parser
go 1.17
require github.com/prometheus/prometheus v2.5.0+incompatible // indirect
Use go get github.com/prometheus/prometheus#83032011a5d3e6102624fe58241a374a7201fee8 (that commit is the latest release at this point in time, v2.33.4)
The reason this is needed is that
This is a known issue with Go Modules. The semantic versioning of Prometheus versions the behavior of Prometheus as a server, not its code as a library. By changing the module path to v2, we would suggest that Prometheus obeys the contract of Go Modules as a library, but it doesn't, i.e. there are many breaking changes to expect even in a minor release.
and:
Prometheus was not intended to be used as a library. Now that has changed, and it is intended to be used as such, even if we do not accept all general-purpose contributions.
The error you are seeing is because go get is grabbing an old release v2.5.0 by default which was released back in 2018 and does not include the parser package. This happens because the versioning scheme used by Prometheus does not align with that assumed by Go.
See this issue for additional info.
I'm starting a new project and considering gb as my build tool but it doesn't appear to be integrating very well with vscode...
I've referenced 3rd party dependencies no problem using gb vendor fetch but as for creating local packages, this is proving a little trickier! Am I missing something obvious?
Here's my local src directory:
src
/cmd
/model
calc.go
/server
server.go
The following code compiles and creates a bin\server.exe file successfully but the import path isn't picked up, nor does gocode recognise it
Here's the server code:
package main
import (
"cmd/model" // not a happy reference...
"fmt"
)
func main() {
fmt.Println(model.Add(1, 2))
}
Here's the model code:
package model
func Add(a int, b int) int {
return a + b
}
I've found what appears to be a similar issue on Github (https://github.com/joefitzgerald/go-plus/issues/325) and while nsf's solution sorts out auto-complete (post import), the import statement itself still claims to be searching in the GOROOT and GOPATHs.
Any ideas?
Thanks to an answer from lukehoban here https://github.com/Microsoft/vscode-go/issues/249 I was able to get my environment working.
I simply created a settings.json file under the .vscode directory (which will now have to be checked in) into which I've configured:
{
"go.gopath": "${workspaceRoot}"
}
This makes me feel unclean and it still doesn't provide a way to reference both 3rd party dependencies and local packages together...
Do not try to work against Go, work with Go.
First of all give all your packages fully qualified import paths. Go is designed around global import paths, do not try to force Go into using flat hierarchies or even relative paths.
You can point to your import path repository endpoints either directly or by using Go's remote import path mechanism. BTW, if you happen to run a self-hosted GitLab instance, it supports remote import path meta tags out of the box.
I prefer glide, but maybe the following is possible with gb, too. Certainly something simililar will be possible with the upcoming go dep: You can point to ssh+git endpoints and others using glide's repo stanza. Frankly I have no idea if gb supports an equivalent mechanism, but if it doesn't this is a good reason to reconsider.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I want to build an API based application using GO and MongoDB. I'm from Asp.net MVC background. Probably if I make an architecture with MVC web application things to be consider are
Separation of concerns(SoC)
DataModel
BusinessEntities
BusinessServices
Controllers
Dependeny Injection and Unity of Work
Unit Testing
MoQ or nUnit
Integration with UI frameworks
Angularjs or others
RESTful urls that enables SEO
Below architecture could be a solution for my need in MVC based appications
There are resources around the web to build Asp.Net or Java based applications, but I have not find solution to Golang application architecture.
Yes GO is different to C# or Java, but still there are Structs, Interfaces to create reusable code and a generic application architecture.
Consider above points in mind, how we can make a clean and reusable project structure in GO applications and a generic repositories for DB(Mongodb) transactions. Any web resources also a great point to start.
It depends on your own style and rules, in my company, we develop our projects this way:
Configuration is determined by environment variables, so we have a company/envs/project.sh file which has to be evaluated before service (outside the project in the image).
We add a zscripts folder that contains all extra scripts, like adding users or publishing a post. Intended to be used only for debug proposes.
Data models (entities) are in a package called project/models.
All controllers and views (HTML templates) are categorized into "apps" or "modules". We use the REST path as main group delimiter, so path /dogs goes to package project/apps/dogs and /cats to project/apps/cats.
Managers are in their separated package at project's root project/manager.
Static files (.css, .png, .js, etc.) are located at project/static/[app/]. Sometimes is required to have the optional [app/] folder, but it only happens when two apps have dashboards or conflicting file names. Most of cases you won't need to use [app/] for static resources.
Managers
We call a manager, a package that contains pure functions which helps apps to perform its task, for example, databases, cache, S3 storage, etc. We initialize each manager calling package.Startup() before we start to listen, and finalize calling package.Finalize() when program is interrupted.
An example of a manager could be project/cache/cache.go:
type Config struct {
RedisURL string `envconfig:"redis_url"`
}
var config Config
var client *redis.Client
func Startup(c Config) error {
config = c
client, err := redis.Dial(c.RedisURL)
return err
}
func Set(k,v string) error {
return client.Set(k, v)
}
in main.go (or your_thing_test.go):
var spec cache.Config
envconfig.Process("project", &spec)
cache.Startup(spec)
And in a app (or module):
func SetCacheHandler(_ http.ResponseWriter, _ *http.Request){
cache.Set("this", "rocks")
}
Modules
A module is a container of views and controllers that are isolated from other modules, using our configuration I would recommend to not create dependencies between modules. Modules are also called apps.
Each module configures its routes using a router, sub-router or what your framework provides, for example (file project/apps/dogs/configure.go):
func Configure(e *echo.Echo) {
e.Get("/dogs", List)
}
Then, all handlers live in project/apps/dogs/handlers.go:
// List outputs a dog list of all stored specimen.
func List(c *echo.Context) error {
// Note the use of models.Xyz
var res := make([]models.Dog, 0) // A little trick to not return nil.
err := store.FindAll("dogs", nil, &res) // Call manager to find all dogs.
// handle error ...
return c.JSON(200, res) // Output the dogs.
}
Finally you configure the app in main (or in a test):
e := echo.New()
dogs.Configure(e)
// more apps
e.Run(":8080")
Note: for views, you can add them to project/apps/<name>/views folder and configure them the using the same function.
Other
Sometimes we also add a project/constants and a project/utils package.
Here is what it looks like:
Note that in above sample, templates are separated from apps, thats because its a placeholder, directory is empty.
Hope it was useful. Greetings from México :D.
I've also struggled about how to structure my Go web APIs in the past and don't know any web resources that tell you exactly how to write a Go web API.
What I did was just check out other projects on Github and try out how they structured their code, for example, the Docker repo has very idomatic Go code on it's API.
Also, Beego is a RESTful framework that generates the project structure for you in a MVC way and according to their docs, it can also be used for APIs.
I've been building a web APIs in golang for a little while now.
You'll have to do some research but I can give you some starting points:
Building Web Apps with Go -- ebook
github.com/julienschmidt/httprouter -- for routing addresses
github.com/unrolled/render/ -- for rendering various forms of responses(JSON, HTML, etc..)
github.com/dgrijalva/jwt-go -- JSON Web Tokens
www.gorillatoolkit.org/pkg/sessions -- Session Management
And for reference on how some things work together in the end:
Go Web API Repo -- personal project
1. Separation of concerns (SoC)
I haven't worked with SoC directly, but I have my own pattern. You can adapt to whatever pattern (MVC, your own, etc.).
In my code, I separate my code into different packages:
myprojectname (package main) — Holds the very basic setup and configuration/project consts
* handlers (package handlers) — Holds the code that does the raw HTTP work
* models (package models) — Holds the models
* apis (NOT a package)
- redis (package redis) — Holds the code that wraps a `sync.Pool`
- twilio (package twilio) — Example of layer to deal with external API
Notes:
In each package other than main, I have a Setup() function (with relevant arguments) that is called by the main package.
For the packages under the apis folder, they are often just to initialize external Go libraries. You also can directly import existing libraries into the handlers/models without an apis package.
I setup my mux as an exported global in the handlers package like this...
Router := mux.NewRouter()
...and then create a file for each URL (different methods with the same URL are in the same file). In each file, I use Go's init() function, which is ran after global variables are initialized (so it's safe to use the router) but before main() is ran (so it's safe for main to assume everything has been setup). The great thing about init() is that you can have as many of those methods as you want in a single package, so they automatically get ran when the package is imported.
Main then imports myprojectname/handlers and then serves the handlers.Router in main.
2. Dependency Injection and Unity of Work
I haven't worked with Unity of Work, so I have no idea of possible Go implementations.
For DI, I build an interface that both the real object and the mock objects will implement.
In the package, I add this to the root:
var DatabaseController DatabaseControllerInterface = DefaultController
Then, the first line of each test I can change DatabaseController to whatever that test needs. When not testing, the unit tests should not be ran, and it defaults to DefaultController.
3. Unit Testing
Go provides a built in testing with the go test package command. You can use go test --cover to also emit a coverage percent. You can even have coverage displayed in your browser, highlighting the parts that are/aren't covered.
I use the testify/assert package to help me with testing where the standard library falls short:
// something_test.go
//
// The _test signifies it should only be compiled into a test
// Name the file whatever you want, but if it's testing code
// in a single file, I like to do filename_test.go.
package main
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestMath(t *testing.T) {
assert.Equal(t, 3+1, 4)
}
4. Integration with UI frameworks
I haven't seen any for Angular. Although I haven't used it, Go has a good template engine built into the standard lib.
5. RESTful URLs that enables SEO
Again, I can't help you here. It's up to you: send proper status codes (don't send a 200 with a 404 page as you'll get docked for duplicate pages), don't duplicate pages (pay attention to google.com/something vs google.com/something/; hopefully your framework will not mess this up), don't try to trick the search engine, and so on.
To my mind Go webapp project folder on production server can looks like on your picture just much simpler. Nothing special in assets structure - Static, Templates, Content, Styles, Img, JSlibs, DBscripts etc. usual folders. Nothing special in WebAPI - as usual you design which URI will respond required functionality and route requests to handlers accordingly. Some specifics - many gophers don't believe in MVC architecture, it's up to you surely. And you deploy one staticaly linked executable without dependencies. In your development environment you structure yours and imported/vendored sorce files in $GOPATH as in stdlib done but deploy only one executable in production environment, sure with static assets needed. You can see how to orginize Go source packages just in stdlib. Having just one executable what would you structure on production?
I have a small web application written in Go. It is created a base for a larger system and I would like it to be extendable where components can be added/removed without needing this base to be modified in any way.
The structure is currently:
App
Modules
Core
... Core Files Here ...
app.go
main.go
app.go will contain a routing method which should take a web request and based on the request path know which module is responsible for handling the request. Each module/component having its on controller.
Each component will have its own package name so i think this is going to be impossible since go forces an explicit import.
For example i may add a new module/component called say blog such as:
App
Modules
Core
... Core Files Here ...
controller.go
Blog
... Blog Files Here ...
controller.go
app.go
main.go
There are several ways to achieve your goal. Since Go does not support dynamically loaded libraries at the moment, you need to probably recompile your application whenever you add/remove any components. The simplest way therefore would be a yourapp/core package with the following:
an Application type with an ServeHTTP method that acts as your main application
a Component interface that all your components have to implement. Your might want include a BaseUrl() string and a ServeHTTP method there.
a Register method to your Application type, that can be used to add new components to your app.
Your components can then be implemented in separate packages (e.g. yourapp/blog) and they will probably depend on your yourapp/core package.
The only thing that still needs to be "user-editable" is the main.go file, which might look like this:
func main() {
app := core.NewApplication()
app.Register(blog.Blog{
Title: "My Personal Blog",
})
app.Register(...)
app.Run()
}
Another approach might be to define an RPC interface for your components (which might include functions like RegisterComponent, UnregisterComponent and a GetGlobalConfig).
Then, you can run those components in separate processes which has the advantage that you can start/stop/reload those components dynamically and they can not break your main app. Take a look at the net/rpc package and maybe even httputil.NewSingleHostReverseProxy if you want to use this approach instead.