I like splitting my modules/packages into many files (e.g. a bigger function file and a coupled helpers file). Now a module can have many bigger functions as it grows. Let's say my modules/packages each consist of around 30 files.
When I started working with Go a few days ago, I was a bit confused about the module pattern. I have a JavaScript (ES6+) background. After some research, I found you could build packages consisting of multiple files by running the following command go build ./.... This worked out pretty well, until my package consisted of about 10 files. All of a sudden I saw errors about functions being undefined. Meanwhile those functions where clearly defined in the src of the module/package (and properly capitalized).
My gut feeling says that this is because of the build order. Say we have 2 files 'a.go' and 'b.go'. file 'a.go' got a reference to a function in file 'b.go'. When file 'a.go' is build before 'b.go', an undefined error appears. My second gut feeling is that file 'a.go' contains a reference to a function inside file 'b.go' and the same vice-versa. So file 'b.go' also has reference to a function in file 'a.go'.
Is my first of my second gut feeling correct?
How do people commonly tackle this problem, or even better, what is the best practice?
What is the proper way creating a single module which consists out of many files?
You can have as many .go files in a single directory as you'd like.
A Go package consists of all of the .go files in a single directory. Each .go file in a single directory should have the same package foo declaration as the other .go files in the same directory.
Your source code imports code from another package (another directory) via an import path supplied in an import statement such as import "github.com/my/repo/pkg1".
To build a package, you can cd to the directory and issue go build . (where . means build all the files in the current directory, or you can also just issue go build in that directory without any arguments because the default is .).
Code within the same package (the same directory) can reference code in other files in the same package (same directory) without needing to import the other files, and without needing the symbols to be exported (uppercase first character).
It is well worth the time to read "How to Write Go Code", which includes an overview of these concepts, including:
Go programmers typically keep all their Go code in a single workspace.
A workspace contains many version control repositories (managed by Git, for example).
Each repository contains one or more packages.
Each package consists of one or more Go source files in a single directory.
The path to a package's directory determines its import path.
and a bit later it describes the relationship between "workspace" and GOPATH:
The GOPATH environment variable specifies the location of your workspace. It defaults to a directory named go inside your home directory, so $HOME/go on Unix, $home/go on Plan 9, and %USERPROFILE%\go (usually C:\Users\YourName\go) on Windows.
Regarding your more specific concerns:
Splitting a single package into multiple (many) files gives undefined error on go build ./
Splitting a package across multiple individual .go files in the same directory (same package) should not cause a problem in and of itself, but of course it needs to be done properly (e.g., all .go files in the same directory (same package) need to have the same package foo declaration near the top; files in the same directory should not attempt to use import statements to import other files in the same directory; etc.).
My gut feeling says that this is because of the build order.
Build order shouldn't matter here.
My second gut feeling is that file 'a.go' contains a reference to a function inside file 'b.go' and the same vice-versa. So file 'b.go' also has reference to a function in file 'a.go'.
This is fine if a.go and b.go are in the same package (the same directory).
On the other hand, this is not allowed if a.go and b.go are in the different packages (different directories). This is because Go does not allow circular dependencies at the package level. If this is the problem, the simplest solution is to move a.go and b.go into the same directory, or refactor to introduce a third package to break the cycle so that a and b can both depend on the third package, or use interfaces, or other possible solutions, but given you say you are new to Go, the simplest solution might be best while you are learning.
Related
I wrote a function (not! main) and prompted go install. This command generated a path and a package in my pkg-directory. I tested the function by using it in a main function, generated the .exe and everything worked just fine.
After that, I wanted to see if I understood the concept of packages in go correctly and deleted the source file of the function in the src-directory and deleted the main .exe. I did not remove the package file in my pkg-directory. Then I tried to go install the main .exe again, but it didn't work: "package can not be found". I obviously missunderstood the whole concept because I thought I could use the packages in pkg without the source files in src. If my conclusion was correct, why do I need the "pkg" directory at all?
For more explanation take a look at this picture please:
In /bin is the binary code of the main function "hello". This main function also contains the function "reverse" of the "stringutil" package.
By generating the "hello.exe", Go also generates the package "stringutil" into pkg.
My question is: Should I not be able to delete "reverse.go" in src and still be able to use the same function because it was already put into pkg?
Is it just the way the AST works now that they've rewritten the compiler in go? It checks for GOPATH/src/**/.go when it parses the "imports", then when the linker goes to build the final binary, it'll go check pkg. So the compiler errors out first when trying to feed the ast to the assembler because of the incomplete source tree.
Thank you very much!!!
It is true that the pkg dir is usually used as a cache directory, but also It is possible to use packages without having the source code available, with a feature named binary only packages AFAIK It was implemented since Go 1.7.
However, there's a caveat for this approach: Versions of the compiler used to build the package and the compiler to 'use' the package to generate a new library/executable must match. Also, the files must match the pair of os/architecture to build against. If you want cross-compiling, you'll need to distribute your package to every pair of os/architecture you'll want to build.
This project has a demo for the aforementioned feature.
I hope my explanation was detailed enough :)
The pkg dir is just a local cache for compiling, it's not something you can depend upon and it doesn't replace .go files, it's not a static lib dir but a temp build products dir. It speeds up compilation, so you don't need the pkg dir, but compiles might be slower if it is empty, and it is for the compiler's use, not yours.
As you've discovered, you do need the src dir.
To link a static library to your project, you could copy out that .a file and use ldflags to link but then you lose all the nice things like cross-compiling and having the entire source for your app, so unlike c for example people don't typically do that.
My organization uses Rails to develop its app but I'm attempting to re-write one of our back-end processes in Golang because it's much quicker.
I've structure my application with our company a namespace for my app (example.co), and a subfolder for each of the packages within my app.
Each library that I've included (e.g. sqlx, etc...) also has it's own folder.
src/
github.com/
jmoiron/
(sqlx package files)
example.co
my_app/
(my app package files)
model/
(model package files...)
However looking at other packages like sqlx, it appears they scrap this directory structure entirely and put all their files in the root directory
Is this because I'm writing an application and sqlx is a package that's meant to be included in other applications? Or is it just a difference in preference since there's no real accepted "standard"
I did this too on my first project. I have since learned:
the $GOPATH/bin/ pkg/ src/ layout is constructed by go get and similar commands
you can organize your .go files as a single flat project dir or with subfolders (caveat: all .go files in the same folder must have the same package name)
put other people's code in a /vendor directory inside your project root, if it is code your app needs to work (google this, it's the worst part of go imo)
put your own project under your gopath, symlink to it if you want it more accessible
So I would imagine your code could look something like:
/Users/user2490003/MyGoPath/
▾ src/github.com/user2490003/myproject/
▾ model/
user.go
▾ myapp/
myapp.go
▾ vendor/github.com/jmoiron/sqlx/
sqlx.go
main.go
Import the full package references, like this:
// main.go
package main
import (
github.com/jmoiron/sqlx
github.com/user2490003/myproject/myapp
github.com/user2490003/myproject/model
)
I would recommend to start with a layout that seems logical and works at the present moment and then refactor/restructure as needed when your applications grows and evolves.
Using your company namespace is reasonable - I would consider creating a directory for your app underneath it (such as company.co/my_app) and inside of it, subdirectories for library packages (for example, company.co/my_app/db etc.) as well as the cmd one that would contain directories for the actual executables (package main programs) that you want to produce: cmd/exe1, cmd/exe2 etc. This will allow you to have multiple executables as well as multiple library "subpackages" inside of the my_app which can be included independently with the corresponding import path.
Each library that I've included (e.g. sqlx, etc...) also has it's own folder.
If you are ok with using the latest version of dependencies from Github, you don't have to include the dependencies' code into your repositories but instead install them with go get into the build area. If you want to build from a local copy - and for corporate usage, it may be preferable for stability and audit track - you should put them in a vendor subdirectory, for example company.co/my_app/vendor/github.com/jmoiron/sqlx. This way you have control on when you upgrade to a newer version of the dependencies and assurance that the upstream changes will not break your build or otherwise affect your programs without your knowledge and until you have a chance to do thorough testing.
I am building a web application in Go and as part of it I have several middelware functions defined. Right now they all live in "my/middleware" package. That namespace is becoming very cluttered by all the different functions I've defined so I decided to put them all in their own subdirectories, e.g. "my/middleware/gzip". When I do this I get the error:
no buildable Go source files my/middleware
I don't want all of these functions in the same namespace, but it seems my only option is to create a placeholder .go file in the my/middleware directory with an empty init function or something. That sounds terrible so I'd like suggestions on how to achieve my goal to group a similar class of packages when there isn't any shared/common code to live in the parent package.
You are actually taking the right decision by splitting the files into different subfolders. It is not different than what is done here
https://golang.org/pkg/compress/
This allows for the clients of your framework to take only what they need. The idea is to avoid dependency bloating. Go is all about being lean.
The error you receive is because you try to build a package that doesn't exist. Think of that folder as a logical grouping mechanism, you need to build the packages given by the child folders individually.
I'm playing with golang and made a tool for password generation. Initially it was intended to be used as a command line tool. I later separated the generation logic into a separate package (still same github repository) and left the main function in the root of the project. Now I also want to add a simple web frontend (nothing fancy), but I don't know how to structure the packages.
Am I supposed to put both the command line entry point as well as the web UI into their own packages in the same project (which leaves the root empty). Or maybe I should move the actual generation library to the root and the UIs in separate packages. I guess the other option is to have the UIs in separate projects on github, but they are only going to be used for this library, so it does not seem like a good idea.
I remember seeing in some projects packages named cmd for example, but never have I encountered one, with multiple front ends. Is there a go(-gettable-)way for doing this?
I agree that there's not much point in making separate projects/repositories if they're only going to be used for this library. I would just have a cmd directory with a subdirectory for each executable you're building.
Something like this:
github.com/user/project
generation
cmd
cmdline
main.go
web
main.go
The main.go files can use the functionality that you've broken out into your "generation" package.
The name of the executables produced by go build will be the name of the parent directory, so cmdline and web in this example (you would want to choose better names).
Note: you don't actually have a package cmdline or web. The files in those directories would all be in [their own separate] package main.
I would like to use signatures for a program that I am writing in Go, but I can't figure out the documentation, which is here. In particular, I would like to use the SignPKCS1v15 and VerifyPKCS1v15 functions, but I'm not sure exactly what I have to pass as arguments. I would greatly benefit from some example code of these two functions. Thanks.
Note: The message that I would like to send is a struct that I defined.
I think the src\pkg\crypto\rsa\pkcs1v15_test.go file in the Go source tree should be a good start.
An update striving provide more context… Go source contains many tests for the code in its standard library (and the crypto/rsa package is a part of it), so whenever you have no idea how to use a standard package (or, actually, any other Go package), a good place to start is to look at the tests involving that package as testing code naturally uses the package! Tests are kept in files ending in _test.go, usually have meaningful names and are located in the same directories actual code implementing a particular package is kept.
So in your particular case you could do this:
Download the Go source package of the version matching your compiler (what go version shows) and unpack it somewhere.
Navigate to the directory matching the package of interest. Code for standard Go packages is located in the "pkg" directory under the "src" top-level directory, so if you're interested in the crypto/rsa package, you need the src/pkg/crypto/rsa directory.