If I use bazel to build my protobuf dependent Go serverless functions, bazel will make the protobuf generated go code available at the import path that I specify.
Google cloud functions for go requires one to use go modules.
How can I add the dummy import path created by bazel to my go.mod file? The function deploy to google cloud fails because the dummy import can not be resolved. (G cloud requires me to upload my go source, AWS lambdas would allow me to upload a binary, which would work fine.)
I'm guessing I'll have to either go with AWS lambdas, use serverless containers, or write a genrule that copies the outputs of the proto generated code into my source directory but I'd like to avoid that ugliness.
I work at Google on Go and Google Cloud Functions.
I see a few options for using Cloud Functions:
Publish the generated code publicly. You may not want to do this for a variety of reasons.
Copy the generated code into your source directory. This is the easiest. When you deploy your function, the current directory gets zipped up and sent to be built. We don't copy any dependencies from outside your current directory. If you do this, you can import the generated code by having the package path be prefixed by the module path of your directory.
Use vendoring. If you run go mod vendor and have that grab your generated code (at whatever path you choose), it will create a vendor directory with all of your dependencies. The Cloud Functions builder prefers go.mod over vendor, though. So you would have to .gcloudignore the go.mod and go.sum file to make sure they don't get uploaded when you deploy your code. https://cloud.google.com/functions/docs/writing/specifying-dependencies-go has more information.
Related
I am trying to create my first monorepo in Go. The project structure looks as follows:
As you can see on the picture, the monoplay folder is the root.
The pb folder contains the generated gRPC code that I would like to consume in the srv_boo/main.go and srv_foo/main.go files.
The question is, how to consume the generated gRPC code from folder pb in the srv_boo/main.go and srv_foo/main.go files?
Is the folder structure correct?
Would like also to deploy the services individually.
Is maybe https://bazel.build/ the solution?
Having the entire repository as one go module will help with this, i.e only one go.mod file in the "Monoplay" root folder.
Then the services can reference the generated go files using "github.com/*/monoplay/pb/*" imports.
This will also centralize dependency management for all the entire repository, since there is only one go.mod file, if that is something you want.
Other alternatives:
Using "go mod edit":
https://go.dev/ref/mod#go-mod-edit
Or, as DazWilkin suggests, use "go_package" in proto files together with "go-grpc_opt" and "go_opt".
I use the single module approach and recommend it.
If the repository will contain a lot of code and building everything (including container images) is cumbersome and takes to long then look into bazel.
Example Scenario
I have an AWS S3 bucket with lots of object which a couple of my application needs to access. The applications use some info available to them to form the S3 object name, get the object and run a transformation on the object data before using it for further processing specific to the application.
I would like to create a module which will hold the logic for forming the object name, obtain it from S3 and then run the transformation on the object data so that I wont be duplicating these functions in multiple places.
In this scenario should I add AWS SDK as a dependency in the module? Keep in mind that the applications might have to use AWS SDK for something else specific to that application or they might not require it at all.
In general what is the best way to solve problems like this i.e where to add the dependency? And how to manage different versions?
If your code has to access packages from the AWS SDK then yes, you have no choice but to add it as dependency. If it doesn't and the logic is generic, or you can abstract it away from the AWS SDK then you don't need the dependency (and in fact the go tooling like go mod tidy will remove the dependency from go.mod if you add it)
In this scenario should I add AWS SDK as a dependency in the module?
Keep in mind that the applications might have to use AWS SDK for
something else specific to that application or they might not require
it at all.
Yes, if any package from your module depends on AWS SDK, Go Modules system is going to add AWS SDK as a dependency for your module. There is nothing special you are supposed to do with your module.
Try this script with Go 1.11 or higher (and make sure to work out of GOPATH):
Write your module like this:
Tree:
moduledir/packagedir1
moduledir/packagedir2
Initialize the module:
Recipe:
cd moduledir
go mod init moduledir ;# or go mod init github.com/user/moduledir
Build module packages:
Recipe:
go install ./packagedir1
go install ./packagedir2
Module things are supposed to automagically work!
In general what is the best way to solve problems like this i.e where to add the dependency? And how to manage different versions?
The Modules system is going to automatically manage dependencies for your module and record them in the files go.mod and go.sum. If you need to override some dependency, you should use the 'go get' command for that. For instance, see this question: How to point Go module dependency in go.mod to a latest commit in a repo?
You can also find much information on Modules here: https://github.com/golang/go/wiki/Modules
Fabric chaincode requires separate folders for each chaincode for deployment.
for e.g., chaincode_1 will need to be in chaincode_1 folder with all the dependencies (vendor), util/library functions + chaincode_1.go and same for chaincode_2.
My question is how to organize util/library folder if it has functions that i want to use it across chaincodes. Fabric chaincode deployment does not allow it.. i think. And the util folder is replicated/redundant in each chaincode folder
You could put all of your shared dependencies, including your own util package, in a separate repository and then vendor them via dep. They would still be replicated per chaincode, but it might be easier to manage them via dep ensure rather than having to manually copy them.
I have some local packages defined within my application, for example, I have a crud model located at model/crud/crud.go
Within my application I am calling upon them using import("model/crud"), for all of my local dependencies.
This resolves perfectly fine within the context of my application on my local machine, however when I try to push to openshift I get the following error:
imports model/crud: unrecognized import path "model/crud"
It looks like when openshift runs the build tool, it is attempting to run go get on these imports in order to include them in the build path when compiling.
Is there a better way to resolve vendor specific dependencies without having to create a separate repo for them? I don't want to have to manage two separate repositories if I don't have to.
To find the import path you are supposed to use, take "$GOPATH/src/[...]/model/crud" and just remove the "$GOPATH/src/" part.
You should use the full import path. For example "github.com/user/project/model/crud"
Revel models are defined under the models package; so in order to import them one must use the full repo path relative to the %GOPATH/src folder which in this case project/app/models thus results in
import PROJECTNAME/app/models
so far, so good i'f you'r using your app name as the folder name of your local dev machine and have dev+prod environments only.
Heroku's docs recommends using multiple apps for different environment (i.e. for staging). with the same repository with distinct origins;
This is where problem starts, now, since the staging enviromnent resides on alternative appname(let's say PROJECTNAME_STAGING), it's sources are stored under PROJECTNAME_STAGING but the actual code still import PROJECTNAME/app/models instead of import PROJECTNAME_STAGING/app/models; so compile fails, etc.
Is there any possibility to manage multiple environments with a single local repo and multiple origins with revel's heroku buildpack? or a feature is needed in the buildpack that is yet to be implemented?
In addition, there is this possible issue with the .godir file that is required to be versioned and contain the git path to the app, so what about the multi-environment duality regarding this file?
Solution was simple enougth;
The buildpack uses the string in .godir both for the argument for revel run as well as the directory name under GOPATH/src. My .godir file had a git.heroku.com/<APPNAME>.git format; Instead I just used APPNAME format.