I'm learning golang and decided to give it a go (pun mildly intended) with a side project that I had in mind for some time (though question is probably language agnostic).
I decided to learn more about the clean architecture along the way because I learnt the hard way about having a bad architecture in the current project I'm dealing with in my day job.
Here is a simplified layout of relevant part of my project:
.
└── app
├── error_codes
│ └── error_codes.go
├── interfaces
│ └── interfaces.go
├── models
│ └── repo
│ └── repo.go
├── managers
│ └── vcs_managers
│ └── git_manager
│ └── git_manager.go
└── helpers
└── helpers.go
Now error_codes, interfaces and models are all in the innermost circle of the architecture.
Here is where I'm getting confused. On my Repo model, I have some functions that does stuff and doing that stuff requires some helper methods in helpers package. But as per the requirements of clean architecture, I cannot refer to helpers package from Repo model, because of the direction of the dependency.
I feel like I'm missing something really fundamental. What am I missing in this picture?
Related
I'm working on automating deployment of Lambda functions using Terraform. The goal is to be able to deploy either a single Lambda function or all of the functions in the repo. I'm able to deploy a number of functions from a structure that looks like:
├── README.md
├── js
│ ├── README.md
│ ├── jsFunction1/
│ │ └── main.tf
│ └── jsFunction2/
│ └── main.tf
├── py
│ ├── README.md
│ ├── pyFunction1/
│ │ └── main.tf
│ └── pyFunction2/
│ └── main.tf
└── terraform
├── README.md
├── common/
├── global/
├── main.tf
├── modules/
├── prod/
└── stage/
The goal is to be able to deploy js/jsFunction1 independently (without building packages for any other functions) while also having terraform/main.tf able to deploy all the lambda functions in the repository (after they've been built). This is so a developer can update the stage deployment with changes to an individual function without concern that they might have an improper version of a function that the developer isn't working on.
What I was hoping to do is create a back-end for each Lambda function so that the developer can use terraform apply from within the source folder. I don't see how to import the state of the Lambda functions that were deployed from the terraform/ module. Can you tell me if importing the state of the resources is a reasonable approach or recommend a better way to achieve the ability to deploy one of many Lambda functions?
Here is the main.tf from js/jsFunction1
module "jsFunction1" {
source = "../../terraform/modules/lambda"
source_path = "${path.module}/dist"
lambda_function_name = "jsFunction1"
lambda_handler = "lambdaAdapter.handler"
}
There is a similar main.tf in each of the folders under js and py. This is main.tf from the terraform folder
module "jsFunction1" {
source = "./../js/jsFunction1"
}
module "jsFunction2" {
source = "./../js/jsFunction2"
}
module "pyFunction1" {
source = "./../py/pyFunction1"
}
module "pyFunction2" {
source = "./../py/pyFunction2"
}
You can make use lambda package module to zip seperate packages for specific lambda's and deploy them.
The package module has the ability to build the package only if change is happened in the respective folders.
So you can have a single main.tf and the reference the output values from package module.
So I created a simple test project to demonstrate the issue I'm having in a much larger code base here https://github.com/cleblanc189/test.go
.
├── go.mod
├── main.exe
├── main.go
├── server
│ ├── v1beta1
│ │ ├── foo.go
│ │ └── go.mod
│ └── v1ga
│ ├── foo.go
│ └── go.mod
└── service
├── v1beta1
│ ├── foo.go
│ └── go.mod
└── v1ga
├── foo.go
└── go.mod
I'd like to be able to reference service/v1beta1 from server/v1beta1 but I'm getting this error
go: test.go/src/server/v1beta1#v0.0.0 requires
test.go/src/service/v1beta1#v0.0.0: unrecognized import path "test.go/src/service/v1beta1" (https fetch: Get https://test.go/src/ser
vice/v1beta1?go-get=1: dial tcp: lookup test.go: no such host)
In foo.go from server/v1beta1 I import like this
svc "test.go/src/service/v1beta1"
and have a go.mod file like below;
require test.go/src/service/v1beta1 v0.0.0
replace test.go/src/service/v1beta1 => ../../service/v1beta1
When I try to cd src && go mod tidy I get that error above. Also if anyone knows should I even have a go.mod above the src directory?
Appreciate any help anyone can offer. This is maddening.
Your go.mod has to declare the module path/name, typically like this:
module test.go
Then all the imports in your code will be taken relative to that. When you import something relative to your declared module name, the go tool will automatically pick up the code from the right place locally.
Not sure where your src comes into play though... is everything in folder src? See this post and accompanying code for a complete example you could start from.
I have some confusion regarding Golang directory structure.
Base on the book The Way to Go by Ivo Balbaert, project code should be placed into src, and recommends the following directory structure.
├──src/
| ├──main.go
| ├──say/
| | ├──say.go
| | ├──say_test.go
├──bin/
| ├──say
└──pkg/
└──linux_amd64/
└──say.a
but I found that many packages in github.com, have no src directory.
For example:
https://github.com/facebookgo/grace
https://github.com/astaxie/beego
So, I don't know whether src directory is needed.
I have some project, their have inter-dependency.
They are managed in a private GitLab repository.
How can I organized them?
This article by Ben Johnson has guided me on this when I was starting with Go.
It's generally good to start with something like this (assuming you are inside your project directory like $GOPATH/src/myproject:
├──cmd/ -- this is where you compose several packages in to main package
| ├──foo -- an example would be `foo`
| | ├──main.go
├──pkg/ -- this is where put your reusable packages
| ├──pkg1 -- reusable package 1
| ├──pkg2 -- reusable package 2
├──otherpackage1
| ├── ...
├──otherpackage2
| ├── ...
You can have a look at this example from go-kit for this kind of project structure.
Sometimes it will depend on your needs. On our workflow, we are using a hot code reload tool called fresh, so we needed to put the main.go on the project root so that the tool can detect all the file changes and rebuild the source code.
├──app/
| ├──app.go
├──model/ --
| ├──model.go
├──store
| ├──store.go
├──main.go -- this is where the app starts
├──...
On the app.go package, I have something like func Run() error which starts the application. On the main.go, I am just calling the function:
...
func main(){
log.Fatal(app.Run())
}
Now there are new ways to organize Go projects.
The GitHub golang-standards repository says:
This is a basic layout for Go application projects. It represents the
most common directory structure with a number of small enhancements
along with several supporting directories common to any real world
application.
This project layout is intentionally generic and it doesn't try to
impose a specific Go package structure.
Or you can follow this slides:
$GOPATH/
src/
github.com/user/repo/
mypkg/
mysrc1.go
mysrc2.go
cmd/mycmd/
main.go
bin/
mycmd
Here is another project layout sample Simple Go project layout with modules
├── LICENSE
├── README.md
├── config.go
├── go.mod
├── go.sum
├── clientlib
│ ├── lib.go
│ └── lib_test.go
├── cmd
│ ├── modlib-client
│ │ └── main.go
│ └── modlib-server
│ └── main.go
├── internal
│ └── auth
│ ├── auth.go
│ └── auth_test.go
└── serverlib
└── lib.go
Some answer points out the go standard layout, however, there is one issue this is not a standard Go project layout by Russ Cox
There are two problems with this GitHub repo:
it claims to host Go standards and does not, in the sense that these are in no way official standards
the project-layout standard it puts forth is far too complex and not a standard
Regarding "why not tell us the standard Go project layout and we'll update the doc?", that only addresses point 2. If there really were standards, they would be in the main Go project doc tree. The standard for project layout would also be a lot shorter. I appreciate your trying to provide a useful resource, but calling it 'golang-standards' is claiming more than it is.
But for the record, the minimal standard layout for an importable Go repo is really:
Put a LICENSE file in your root
Put a go.mod file in your root
Put Go code in your repo, in the root or organized into a directory tree as you see fit
That's it. That's the "standard".
Update at 11/30/2021
Here are a summary from How to structure Go code
Before we start
doc.go file puts the general description of the package
Readme file a general overview of this project
When you have more documentation to provide put them into the docs folder
For linting use golangci-lint. Enable all linters that seem to be reasonable for your project
The flat structure (single package)
courses/
main.go
server.go
user_profile.go
lesson.go
course.go
When to create a new package?
When you have more than one way of starting your application
When you want to extract more detailed implementation
When you started to add a common prefix to closely related things
Modularisation
Organising by kind
.
├── handlers
│ ├── course.go
│ ├── lecture.go
│ ├── profile.go
│ └── user.go
├── main.go
├── models
│ ├── course.go
│ ├── lecture.go
│ └── user.go
├── repositories
│ ├── course.go
│ ├── lecture.go
│ └── user.go
├── services
│ ├── course.go
│ └── user.go
└── utils
└── stings.go
Organising by components
.
├── course
│ ├── httphandler.go
│ ├── model.go
│ ├── repository.go
│ └── service.go
├── main.go
└── profile
├── httphandler.go
├── model.go
├── repository.go
└── service.go
Clean Architecture
You have 4 layers of your application or module (depending on how big your codebase is): Domain, Application, Ports, Adapters. In some sources, names may differ.
The src directory is not needed and in fact a lot of public repositories do not use this structure.
There is a few different way of organizing your project. If you plan on having your project used by an other repository, like lib. I would recommend using a cmd struct something like this. This would also be the recommended way of doing it if there would be more then one way of starting the application. (multipliable main.go files)
├──cmd/
| ├──(application name)
| | ├──main.go
└──say/
├──say.go
└──say_test.go
Otherwise for example if it is a standalone application. You can place the main.go in the root of the repository.
bin and pkg you can keep in the root and add this to .gitignore. (assuming you are using git)
The book describes the directory structure after checkout. It would have been helpful if the book included the .git directory.
$GOPATH/src is required for imports to work.
├──src/
| ├──.git
| | ├──...
| ├──main.go
| ├──say/
| | ├──say.go
| | ├──say_test.go
├──bin/
| ├──say
└──pkg/
└──linux_amd64/
└──say.a
In practice, main.go would actually be in a path that reflects the remote git repository, for instance
.
├── bin
│ └── say
├── pkg
│ └── linux_amd64
│ └── github.com
│ └── pschultz
│ └── hello-world
│ └── say.a
└── src
└── github.com
└── pschultz
└── hello-world
├── .git
│ └── ...
├── main.go
└── say
├── say.go
└── say_test.go
I'm coming from a Puppet background using Vagrant and have some trouble making sense of Ansible and its differences.
My Puppet structure looked like this:
puppet
├── servers
│ └── Backend
│ └── Vagrantfile
└── src
├── manifests
│ └── nodes
│ └── development
│ └── backend.pp
└── modules
└── mysql
Setup was simple, cd to the Vagrantfile and fire up the VM with Vagrant.
Now this is my first draft of an Ansible folder structure:
ansible
├── servers
│ └── Backend
│ ├── Vagrantfile
│ └── ansible.cfg
└── sources
├── backend.yml
├── site.yml
├── inventories
│ └── development
│ ├── group_vars
│ │ ├── all
│ │ └── backend
│ └── hosts
├── playbooks
└── roles
└── mysql
Following questions now:
Is this best practise for Ansible or too close to Puppet?
Is it correct to treat backend.yml like a Puppet node manifest?
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
Sources:
http://leucos.github.io/ansible-files-layout/
https://github.com/ansible/ansible-examples
https://github.com/enginyoyen/ansible-best-practises
In my case, I use following structures according environment complexity (check directory-layout) :
Simple environment
I use this structure when there is one environment or when I use playbooks in provision mode
ansible
├── inventory
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── roles
│ └── mysql
├── ansible.cfg
├── README.md
├── playbook1.yml
└── playbook2.yml
In ansible.cfg, I use variable inventory = ./inventory in [default] in order to avoid setting inventory path with commands ansible-*.
Medium/Complex environment
I use this structure when there are more than one environment
ansible
├── inventories
│ ├── production
│ │ ├── hosts
│ │ └── group_vars
│ │ └── my_group.yml
│ └── development
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── playbooks
│ ├── playbook1
│ │ ├── group_vars
│ │ │ └── my_group.yml
│ │ ├── roles
│ │ │ └── mysql
│ │ ├── README.md
│ │ └── site.yml
│ ...
├── README.md
└── ansible.cfg
In this case, there is a folder for each environments in ./inventories.
I prefer also to use a specific folder for each playbooks in order to be able to use easily a folder group_vars at playbook level as defined in variable precedence section. As environment become more complex, there are much more variables. A group_vars (and host_vars) in playbooks allows to defines common variables for all environment which makes there are less inventories variables.
Ultimate level environment
I already used Ansible to adresse systems with more than 5000 servers, here under some tips for adressing more complex environments :
Split inventory file
Use multiple files to define your inventory servers instead of a single hosts file. In this case hosts file contains only server's names, and other files contains groups with different perspectives :
└── production
├── hosts
├── middleware
└── trigram
middleware: Groups with mapping to used middlewares or other stuf. I use this file to map, for example, servers to tomcat, java, postgresql, etc. And I use it, for example, with playbooks that deploys monitoring agents : How to retrieve metrics, logs from tomcat, java, postgresql, etc.
trigram: On my project, I usually use codes with a fixed length (3 or 4) to identify my business components (ex. : 'STK' for stock management) then I create a group file to map a business component to my servers (which servers are used to deploy 'STK')
When you create a new playbook, choose your perspective to address different environments.
Caution : I think ansible load files with alphabetical name order, you can't define a group that refers a group not yet loaded.
Use folders for group_vars
In group_vars, instead of using files, you can use a folder with subfiles :
└── production
└── group_vars
└── my_group
├── vars1.yml
└── vars2.yml
This is usefull to split huge files or if you have tools that generate variables, in this case you have vars1.yml under git and vars2.yml is generated
Split git repo
When you are using ansible for huge system, there is a lot of commits and a question comes up often : How to split my huge git repo ?
I my case, I use one git repo for each folder in ./inventories with differents access rules. And a git repo for each folder in ./playbooks also with different access rules.
You found several examples of the recommended layout, but not the official one. That should hopefully answer many of your questions, but I'll try to address them here as well.
Is this best practise for Ansible or too close to Puppet?
It's definitely not the recommend layout for Ansible. In the best practices layout, there is no servers or sources - the things inside those all live at the top level (what does "servers" mean, anyways?).
Is it correct to treat backend.yml like a Puppet node manifest?
I'm not familiar with Puppet, and so can't answer this question.
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
The official recommendation has all playbooks splattered across the root directory. However, this gets a bit messy, so some people put them in a subdirectory (in your examples, plays). This works reasonably well, but you'll need to adjust the paths in your playbooks accordingly.
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
You shouldn't put them in a subdirectory of the inventory folder, as they're not part of inventory. There are a number of places to define variables, and you should read the documentation to learn what they are and when you should use them, but group_vars should be at the root of the directory with everything else.
Are there any examples of how Qunit can be implemented into a full scale development cycle. The existing examples seems to require hardcoding test scripts into production source code. Is there currently a way to separate unit tests and source code? I just want unit test code in dev code, not production.
Thanks
P
Take a look at existing projects using QUnit, e.g. jQuery UI. The folder tests/unit contains all the QUnit-based unit tests. Is that separate enough?
1) Keep your tests on the dev. environment in a dedicated folder. E.g. like that
app-wwwroot/
├── js/
│ └── moduleA.js
└── tests/
├── moduleA/
│ ├── dummies.js
│ ├── stubs.js
│ ├── mocks.js
│ ├── fixtures.js
│ └── unit-tests.js
└── unit-tests.html
2) Exclude the folder from your deployment script or production branch in your version control system