I'm working on automating deployment of Lambda functions using Terraform. The goal is to be able to deploy either a single Lambda function or all of the functions in the repo. I'm able to deploy a number of functions from a structure that looks like:
├── README.md
├── js
│ ├── README.md
│ ├── jsFunction1/
│ │ └── main.tf
│ └── jsFunction2/
│ └── main.tf
├── py
│ ├── README.md
│ ├── pyFunction1/
│ │ └── main.tf
│ └── pyFunction2/
│ └── main.tf
└── terraform
├── README.md
├── common/
├── global/
├── main.tf
├── modules/
├── prod/
└── stage/
The goal is to be able to deploy js/jsFunction1 independently (without building packages for any other functions) while also having terraform/main.tf able to deploy all the lambda functions in the repository (after they've been built). This is so a developer can update the stage deployment with changes to an individual function without concern that they might have an improper version of a function that the developer isn't working on.
What I was hoping to do is create a back-end for each Lambda function so that the developer can use terraform apply from within the source folder. I don't see how to import the state of the Lambda functions that were deployed from the terraform/ module. Can you tell me if importing the state of the resources is a reasonable approach or recommend a better way to achieve the ability to deploy one of many Lambda functions?
Here is the main.tf from js/jsFunction1
module "jsFunction1" {
source = "../../terraform/modules/lambda"
source_path = "${path.module}/dist"
lambda_function_name = "jsFunction1"
lambda_handler = "lambdaAdapter.handler"
}
There is a similar main.tf in each of the folders under js and py. This is main.tf from the terraform folder
module "jsFunction1" {
source = "./../js/jsFunction1"
}
module "jsFunction2" {
source = "./../js/jsFunction2"
}
module "pyFunction1" {
source = "./../py/pyFunction1"
}
module "pyFunction2" {
source = "./../py/pyFunction2"
}
You can make use lambda package module to zip seperate packages for specific lambda's and deploy them.
The package module has the ability to build the package only if change is happened in the respective folders.
So you can have a single main.tf and the reference the output values from package module.
Related
I have written some infrastructure code with AWS CDK (Go). My code is structured like so:
.
├── api
│ └── aj
│ ├── lambda
│ │ └── main.go
│ ├── email.go
│ ├── emailService.go
│ ├── handler.go
│ └── handler_test.go
├── cdk
│ ├── cdk.go
│ ├── cdk.json
│ ├── cdk_test.go
│ └── README.md
├── 00-test-image.jpg
├── go.mod
└── go.sum
5 directories, 12 files
My CDK code simply creates an AWS Lambda HTTP Endpoint that will handle a form submission and send an email.
I am using the algnhsa Go adapter package to make things easier to deploy.
At the top of my emailService.go I have the following:
var (
host = os.Getenv("EMAIL_HOST")
username = os.Getenv("EMAIL_USERNAME")
password = os.Getenv("EMAIL_PASSWORD")
portNumber = os.Getenv("EMAIL_PORT")
)
My question is, when I run cdk deploy, how do I set those environment variables so that they're available and set within the code?
You configure the Lambda Function's environment variables with its Environment prop as a *map[string]*string.
You'd typically pass the values down as stack props from its parent stack to the Lambda construct. Or simply hardcode the values in the Function construct code. The CDK best practice is to have such input values fixed at synth-time, so that deploys are deterministic and under source control.
Envs
GO111MODULE=on
golang:1.15.2-alpine
What I want to do
I implemented some features in a local project and I want to confirm the behavior before pushing
changes to the remote repo.
But my project can't access the new package.
Error text
When docker-compose run 'go build' I got his error.
Both 'github.com/Asuha-a/URLShortener/api/pb/url' and 'github.com/Asuha-a/URLShortener/api/utility' don't exist in the remote repo because it's not pushed yet.
main.go:10:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a), but does not contain package github.com/Asuha-a/URLShortener/api/pb/url
main.go:12:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a), but does not contain package github.com/Asuha-a/URLShortener/api/utility
Project architecture
I deleted some of the files look like not related to this issue.
$ tree
.
├── api
│ ├── Dockerfile
│ ├── go.mod
│ ├── go.sum
│ ├── main.go
│ ├── pb
│ │ ├── url // new package
│ │ ├── url_grpc.pb.go
│ │ ├── url.pb.go
│ │ └── url.proto
│ ├── services
│ │ ├── url
│ │ │ ├── db
│ │ │ │ ├── settings.go
│ │ │ │ └── url.go
│ │ │ ├── Dockerfile
│ │ │ ├── go.mod
│ │ │ ├── go.sum
│ │ │ └── main.go // failed building
│ └── utility // new package
│ ├── jwt.go
│ └── url.go
├── docker-compose.yml
What I tried
I found this issue.
Importing local changes of a package without pushing code in Go
That says I can access local packages by adding 'replace MODULE_URL => /PATH/TO/MODULE/DIR'.
I edited the 'api/services/url/go.mod' like this.
module github.com/Asuha-a/URLShortener/api/services/url
go 1.15
replace github.com/Asuha-a/URLShortener/api => ../../.
require (
github.com/jackc/pgx/v4 v4.9.2 // indirect
github.com/satori/go.uuid v1.2.0
golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9 // indirect
golang.org/x/text v0.3.4 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/grpc v1.33.2
gorm.io/datatypes v1.0.0
gorm.io/driver/postgres v1.0.5
gorm.io/gorm v1.20.6
)
But I got this error:
main.go:10:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a, replaced by ../../.), but does not contain package github.com/Asuha-a/URLShortener/api/pb/url
main.go:12:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a, replaced by ../../.), but does not contain package github.com/Asuha-a/URLShortener/api/utility
Then I tried the full path '/home/asuha/go/src/github.com/Asuha-a/URLShortener/api'.
But I got this error:
main.go:10:2: github.com/Asuha-a/URLShortener/api#v0.0.0-00010101000000-000000000000: replacement directory /home/asuha/go/src/github.com/Asuha-a/URLShortener/api does not exist
main.go:12:2: github.com/Asuha-a/URLShortener/api#v0.0.0-00010101000000-000000000000: replacement directory /home/asuha/go/src/github.com/Asuha-a/URLShortener/api does not exist
I want to know
How to fix this error?
Some basics first:
A module is a set of packages versioned together.
A module has a name declared in the go.mod, e.g. github.com/anyone/someproject
A package is imported by its import path (the import path basically is package identity).
A package belonging to a module must have an import path beginning with the module name. In the example above any package belonging to the module github.com/anyone/someproject must have an import path like e.g. github.com/anyone/someproject/whatever/hierarchy/pkgname
You never need to replace packages from the same module. replace is for replacing other modules.
You declared the module
module github.com/Asuha-a/URLShortener/api/services/url
That makes the name of your module "github.com/Asuha-a/URLShortener/api/services/url" and only packages with import path like module "github.com/Asuha-a/URLShortener/api/services/url/I/really/like/deeply/nested/stuff" would belong to that package.
Clearly you utility package (btw. don't do such packages, this is fundamental code smell!) doesn't belong to your module: You probably try to import it as "github.com/Asuha-a/URLShortener/api/utiliyt" but this is not part of the module "github.com/Asuha-a/URLShortener/api/services/url".
The fact that the folder utility is "below" the go.mod file in the filesystem doesn't overrule the fact that your module declares on its first line: "All my packages have import paths starting with github.com/Asuha-a/URLShortener/api/services/url! Anything else does not belong to me!"
You probably should name your module at least module github.com/Asuha-a/URLShortener/api (if not module github.com/Asuha-a/URLShortener).
I urge you to (re-)read "How to Write Go Code" and stick to it.
Takeaway:
Modules are really just a set of packages with the same import path prefix (the module name) versioned together.
I have some problems with how Go importing and modules works.
I have a project with this structure:
My-Project
|- Program 1
|- main.go
|- go.mod
|- Program 2
|- main.go
|- go.mod
|- Support
|- go_file.go
The two programs have different module "definition". I need to include the "Support" folder, which is shared between the two programs. I tried
import "My-Project/Support"
inside Program1's main.go, but this is not working, because Support can’t be resolved. How can I do it?
As given, your Support/go_file.go is not in any module so it's not importable using Go modules.
Either use a single module for all three packages or use three separate modules (i.e. put Support in it's own module). You could also use sub-modules.
Single module:
For a single module you'd do something like:
└── My-Project
├── go.mod
├── Program\ 1
│ └── main.go
├── Program\ 2
│ └── main.go
└── Support
└── go_file.go
And the project level go.mod would be something like:
module github.com/My-Name/My-Project
(substitute with appropriate module name) and then both main packages would import the support package with:
import "github.com/My-Name/My-Project/Support"
Note Go packages normally do not start with capitals like this.
More idiomatic for such a setup with a single repository which mixes packages and commands is a layout that puts commands into a cmd subdirectory (e.g. see the golang.org/x/tools layout). This would look something like this:
└── My-Project
├── cmd
│ ├── program1
│ │ └── main.go
│ └── program2
│ └── main.go
├── go.mod
├── internal
│ └── support
│ └── support.go
└── other-package
└── some-other-package.go
Here the support package is in an internal sub-directory which makes it non-importable from outside My-Project where-as the other-package is importable by anyone.
Again, go.mod would define the module name which sets the prefix to be used for all import statements.
Separate modules:
To use separate modules you'd do something like:
└── My-Project
├── Program\ 1
│ ├── go.mod
│ └── main.go
├── Program\ 2
│ ├── go.mod
│ └── main.go
└── Support
├── go_file.go
└── go.mod
Here My-Project/Program 1/go.mod would be something like:
module "github.com/My-Name/My-Project/Program 1"
require (
github.com/My-Name/My-Project/Support
)
The imports would be the same as for the single module example.
Similarly for Program 2. For Support/go.mod it'd look something like:
module github.com/My-Name/My-Project/Support
For local development you'll probably also want/need a replace directive in each program's go.mod to use your local working copy of the support module (go mod edit -replace github.com/My-Name/My-Project/Support=../Support should add this).
WHAT I TRIED
I am using JEST for testing resolvers and schema but I am having trouble in creating folder and structure.Currently I import resolver functions and call functions and compare result or check if fields are defined.But it does not sometimes satisfy complex scenarios.
WHAT I AM LOOKING FOR
The best practices to test graphql schema and resolver functions, and what testing tool is recommended or mostly used?
Also, you can try this npm package that will test your Schema, Queries and Mutations... there is an example of it using mocha & chai.. here is the link
What you need to do, is import the schema and pass to the easygraphql-tester and then you can create UT.
There are multiple frameworks out there for integration testing your API using for example YAML files. There you can specify request and response. A simpler approach can be to use Jest snapshots and simply execute test queries using the graphql function from graphql-js. It will return a promise with the result and you can then await it and expect it to match the snapshot.
import { graphql } from 'graphql';
import schema from './schema';
import createContext from './createContext';
describe('GraphQL Schema', () => {
let context;
before(() => {
context = createContext();
database.setUp();
});
after(() => {
database.tearDown();
});
it('should resolve simple query', async () => {
const query = '{ hello }';
const result = await graphql(schema, query, null, context);
expect(result).toMatchSnapshot();
});
});
Tipp: You can also create dynamic tests for example by reading queries from files in a directory and then iterating over them creating a new test for each file. An example for that (not GraphQL though) can be found on my Github.
There is not a recomendable way to do it. Specially for documents and folder structure.
In my case, I am working on this repo. And this is my folder structure in the first level:
src/
├── App.js
├── configs
├── helpers
├── index.js
├── middlewares
├── models
├── resolvers
├── routes
├── schema
├── seeds
├── templates
├── tests
└── utils
In the root, I have the test folder, mainly to checkout the App basic behavior and some utils functions. In the other hand, inside the resolvers i have the main test for the GraphQl, queries and mutations.
src/resolvers/
├── camping
│ ├── camping.mutations.js
│ ├── camping.query.js
│ ├── camping.query.test.js
│ └── camping.resolver.js
├── clientes.resolver.js
├── guest
│ ├── guest.mutation.js
│ ├── guest.mutation.test.js
│ ├── guest.query.js
│ ├── guest.query.test.js
│ └── guest.resolver.js
├── index.js
├── turno
│ ├── turno.mutations.js
│ ├── turno.query.js
│ ├── turno.query.test.js
│ └── turno.resolver.js
└── users
├── user.mutations.js
├── user.mutations.test.js
├── user.queries.js
├── user.query.test.js
└── user.resolver.js
Every single resolver has his test, you can check there if the basic endpoints are working as expected.
I am planning to do some workflow tests, they will be in the test root folder later.
I have some confusion regarding Golang directory structure.
Base on the book The Way to Go by Ivo Balbaert, project code should be placed into src, and recommends the following directory structure.
├──src/
| ├──main.go
| ├──say/
| | ├──say.go
| | ├──say_test.go
├──bin/
| ├──say
└──pkg/
└──linux_amd64/
└──say.a
but I found that many packages in github.com, have no src directory.
For example:
https://github.com/facebookgo/grace
https://github.com/astaxie/beego
So, I don't know whether src directory is needed.
I have some project, their have inter-dependency.
They are managed in a private GitLab repository.
How can I organized them?
This article by Ben Johnson has guided me on this when I was starting with Go.
It's generally good to start with something like this (assuming you are inside your project directory like $GOPATH/src/myproject:
├──cmd/ -- this is where you compose several packages in to main package
| ├──foo -- an example would be `foo`
| | ├──main.go
├──pkg/ -- this is where put your reusable packages
| ├──pkg1 -- reusable package 1
| ├──pkg2 -- reusable package 2
├──otherpackage1
| ├── ...
├──otherpackage2
| ├── ...
You can have a look at this example from go-kit for this kind of project structure.
Sometimes it will depend on your needs. On our workflow, we are using a hot code reload tool called fresh, so we needed to put the main.go on the project root so that the tool can detect all the file changes and rebuild the source code.
├──app/
| ├──app.go
├──model/ --
| ├──model.go
├──store
| ├──store.go
├──main.go -- this is where the app starts
├──...
On the app.go package, I have something like func Run() error which starts the application. On the main.go, I am just calling the function:
...
func main(){
log.Fatal(app.Run())
}
Now there are new ways to organize Go projects.
The GitHub golang-standards repository says:
This is a basic layout for Go application projects. It represents the
most common directory structure with a number of small enhancements
along with several supporting directories common to any real world
application.
This project layout is intentionally generic and it doesn't try to
impose a specific Go package structure.
Or you can follow this slides:
$GOPATH/
src/
github.com/user/repo/
mypkg/
mysrc1.go
mysrc2.go
cmd/mycmd/
main.go
bin/
mycmd
Here is another project layout sample Simple Go project layout with modules
├── LICENSE
├── README.md
├── config.go
├── go.mod
├── go.sum
├── clientlib
│ ├── lib.go
│ └── lib_test.go
├── cmd
│ ├── modlib-client
│ │ └── main.go
│ └── modlib-server
│ └── main.go
├── internal
│ └── auth
│ ├── auth.go
│ └── auth_test.go
└── serverlib
└── lib.go
Some answer points out the go standard layout, however, there is one issue this is not a standard Go project layout by Russ Cox
There are two problems with this GitHub repo:
it claims to host Go standards and does not, in the sense that these are in no way official standards
the project-layout standard it puts forth is far too complex and not a standard
Regarding "why not tell us the standard Go project layout and we'll update the doc?", that only addresses point 2. If there really were standards, they would be in the main Go project doc tree. The standard for project layout would also be a lot shorter. I appreciate your trying to provide a useful resource, but calling it 'golang-standards' is claiming more than it is.
But for the record, the minimal standard layout for an importable Go repo is really:
Put a LICENSE file in your root
Put a go.mod file in your root
Put Go code in your repo, in the root or organized into a directory tree as you see fit
That's it. That's the "standard".
Update at 11/30/2021
Here are a summary from How to structure Go code
Before we start
doc.go file puts the general description of the package
Readme file a general overview of this project
When you have more documentation to provide put them into the docs folder
For linting use golangci-lint. Enable all linters that seem to be reasonable for your project
The flat structure (single package)
courses/
main.go
server.go
user_profile.go
lesson.go
course.go
When to create a new package?
When you have more than one way of starting your application
When you want to extract more detailed implementation
When you started to add a common prefix to closely related things
Modularisation
Organising by kind
.
├── handlers
│ ├── course.go
│ ├── lecture.go
│ ├── profile.go
│ └── user.go
├── main.go
├── models
│ ├── course.go
│ ├── lecture.go
│ └── user.go
├── repositories
│ ├── course.go
│ ├── lecture.go
│ └── user.go
├── services
│ ├── course.go
│ └── user.go
└── utils
└── stings.go
Organising by components
.
├── course
│ ├── httphandler.go
│ ├── model.go
│ ├── repository.go
│ └── service.go
├── main.go
└── profile
├── httphandler.go
├── model.go
├── repository.go
└── service.go
Clean Architecture
You have 4 layers of your application or module (depending on how big your codebase is): Domain, Application, Ports, Adapters. In some sources, names may differ.
The src directory is not needed and in fact a lot of public repositories do not use this structure.
There is a few different way of organizing your project. If you plan on having your project used by an other repository, like lib. I would recommend using a cmd struct something like this. This would also be the recommended way of doing it if there would be more then one way of starting the application. (multipliable main.go files)
├──cmd/
| ├──(application name)
| | ├──main.go
└──say/
├──say.go
└──say_test.go
Otherwise for example if it is a standalone application. You can place the main.go in the root of the repository.
bin and pkg you can keep in the root and add this to .gitignore. (assuming you are using git)
The book describes the directory structure after checkout. It would have been helpful if the book included the .git directory.
$GOPATH/src is required for imports to work.
├──src/
| ├──.git
| | ├──...
| ├──main.go
| ├──say/
| | ├──say.go
| | ├──say_test.go
├──bin/
| ├──say
└──pkg/
└──linux_amd64/
└──say.a
In practice, main.go would actually be in a path that reflects the remote git repository, for instance
.
├── bin
│ └── say
├── pkg
│ └── linux_amd64
│ └── github.com
│ └── pschultz
│ └── hello-world
│ └── say.a
└── src
└── github.com
└── pschultz
└── hello-world
├── .git
│ └── ...
├── main.go
└── say
├── say.go
└── say_test.go