Golang google cloud functions using go modules fail build when importing subdirectories - go

I am trying to deploy a google cloud function using go modules and I am having problem with it picking up code in subdirectories when using go modules.
It appears that inside the gcloud functions deploy build it is duplicating the package name. So for instance locally (and what i expect is this import)
import "github.com/lwaddicor/gofunctionmodules/demoa"
however it will only work in the gcloud functions deploy if set to
import "github.com/lwaddicor/gofunctionmodules/gofunctionmodules/demoa"
If anyone has any ideas what is happening here it would be much appreciated. Thanks
Structure
demoa/demo.go
func.go
go.mod
go.mod
module github.com/lwaddicor/gofunctionmodules
go 1.13
func.go
package gofunctionmodules
import (
"net/http"
// This is what i would expect, and it builds locally but it wont deploy
"github.com/lwaddicor/gofunctionmodules/demoa"
// With this one it deploys but locally wont build
//"github.com/lwaddicor/gofunctionmodules/gocloudfunctions/demoa"
)
// Demo demonstrates the issue with package management in cloud functions
func Demo(w http.ResponseWriter, r *http.Request) {
w.Write(demo.Demo())
}
package demoa
// Demo returns a demo string
func Demo() []byte {
return []byte("demo")
}
Then deploying a new cloud function using
gcloud functions deploy demofunction --entry-point Demo --project my-demo-project --runtime go113 --trigger-http --allow-unauthenticated
Deploying function (may take a while - up to 2 minutes)...failed.
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Build failed: {"error":{"errorType":2,"canonicalCode":2,"errorId":"32633c29","errorMessage":"
serverless_function_app imports
github.com/lwaddicor/gofunctionmodules/gofunctionmodules imports
github.com/lwaddicor/gofunctionmodules/gocloudfunctions/demo: git ls-remote -q origin in /layers/google.go.gomod/gopath/pkg/mod/cache/vcs/48848540be122d75f1d2e6549925c34394ce402803530dc7962ab1442de3b180: exit status 128:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information."}}
Edit: removed the github link as it just downloaded the dependency from the public github repo. Changed demo package to demoa to prevent using cached github copy

Related

Why does terraformer not find plugin?

I am new to the world of terraform. I am trying to use terraformer on a GCP project, but keep getting plugin not found:
terraformer import google --resources=gcs,forwardingRules,httpHealthChecks --connect=true --
regions=europe-west1,europe-west4 --projects=myproj
2021/02/12 08:17:03 google importing project myproj region europe-west1
2021/02/12 08:17:04 google importing... gcs
2021/02/12 08:17:05 open \.terraform.d/plugins/windows_amd64: The system cannot find the path specified.
Prior to the above, installed terraformer thus: choco install terraformer I downloaded the Google provider, terraform-provider-google_v3.56.0_x5.exe, placed it in my terraform dir, and did terraform init. I checked %AppData\Roaming\terraform.d\plugins\windows_amd64 and the plugins\windows_amd64 did not exist, so I created it, dropped in the Google exe and redid the init.
My main.tf was:
provider "google" {
project = "{myproj"
region = "europe-west2"
zone = "europe-west2-a"
}
Could it be the chocolatey install causing this issue? I have never used it before but the alternative install looks daunting!
If you install from Chocolatey, it seems it looks for the provider files at C:\.terraform.d\plugins\windows_amd64. If you create those directories and drop the terraform-provider-xxx.exe file in there it will be picked up.
I would also suggest running it with the --verbose flag.
Remember to execute "terraform init"
Once you install the terraformer, check this
Create the versions.tf file
versions.tf example
terraform {
required_providers {
google = {
source = "hashicorp/aws"
}
}
required_version = ">= 0.13"
}
Execute terraform init
Start the import process, something like:
terraformer import aws --profile [my-aws-profile] --resources="*" --regions=eu-west-1
Where --resources="*" will get everything from aws, but you could specify route53 or other resources
terraformer import aws --profile [my-aws-profile] --resources="route53" --regions=eu-west-1
The daunting instructions worked!
Run git clone <terraformer repo>
Run go mod download
Run go build -v for all providers OR build with one provider go run build/main.go {google,aws,azure,kubernetes and etc}
Run terraform init against an versions.tf file to install the plugins required for your platform. For example, if you need plugins for the google provider, versions.tf should contain:
terraform {
required_providers {
google = {
source = "hashicorp/google"
}
}
required_version = ">= 0.13"
}
making sure to run that version and not the chocolatey one.
go mod download the flag -x will give output while that command runs, otherwise you might think nothing is happening.

Go get is pulling the wrong repository

My module is gitlab.com/getsote/utilities/slogger
My repository is gitlab.com/getsote/utilities/slogger.git
When I run go get gitlab.com/getsote/utilities/slogger, I get the message below.
Scotts-Mac-mini:seeding syacko$ go get gitlab.com/getsote/utilities/slogger
go get gitlab.com/getsote/utilities/slogger: module gitlab.com/getsote/utilities/slogger: git ls-remote -q origin in /Users/syacko/workspace/sotesoft/golang/pkg/mod/cache/vcs/80b3644beae1b986f1c659355360479e2463820660aa328d2edb1e571aba259b: exit status 128:
remote: The project you were looking for could not be found.
fatal: repository 'https://gitlab.com/getsote/utilities.git/' not found
Scotts-Mac-mini:seeding syacko$
The gitlab.com/getsote/utilities.git is a sub-directory and not a repository. I don't understand why go get is going to the utilities as a repository?
==========================
PREVIOUS Updates
Directory Structure:
GOPATH/src/slogger
|----go.mod
|----slogger.go
|----slogger_test.go
go.mod file
module slogger or gitlab.com/getsote/utilities/slogger -> still gets the error below
go 1.14
gitlab.com/getsote/utilities contains repository slogger.git
I have run a test to see if the issue is the number of nodes in the path. So, I create a new repository with no sub-directory and pushed the slogger code. Then ran go get gitlab.com/getsote/slogger which generate a different error message.
GOPATH/gitlab.com/getsote/test-go-mod -> create new directory and added slogger files listed above
gitblab.com/getsote/test-go-mod -> new repository with one less level
Scotts-Mac-mini:test-go-mod syacko$ go get gitlab.com/getsote/test-go-mod
go: downloading gitlab.com/getsote/test-go-mod v0.0.0-20200409023538-794310bf7cf9
go get gitlab.com/getsote/test-go-mod: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: verifying module: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: reading https://sum.golang.org/lookup/gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: 410 Gone
server response:
not found: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/7753c92c9bd1419156d8120684b7f3707fd207e01a2947ba89e2acfd2ecfb4d0: exit status 128:
fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled
Scotts-Mac-mini:test-go-mod syacko$
This is still getting the status error of 128 for the missing version. Additionally, it is looking in the right location for the code. If this is true, then I just need help with the version missing. Moving to a shorted directory structure is doable.
========================
Newest Update
#praveent > The solution at https://medium.com/cloud-native-the-gathering/go-modules-with-private-git-repositories-dfe795068db4 didn't work for me. So I started from scratch to see how to resolve the issue.
The reason is because for a git repository it assumes that utilities is the repo and not utilities/slogger
There is a way to override this behavior by implementing go get API. But, gitlab is yet to implement the same due to security concerns. You can read more here. Gitlab issue
Update: Add reference to gitlab issue tracking this problem.
So, here is how I got this to work using gitlab.com. I'm not saying other ways will not work, they just didn't for me and my setup. First, since I don't care if the code is available to the public, I created a new group at gitlab.com. This new group is public from the start, so no need to adjust permissions. Then I create a repository called packages and cloned the repository to my local machine with the same directory structure that is in gitlab.com, gitlab.com/soteapps/packages with ~/workspace/soteapps/packages on my machine. Both of these are out side the GOPATH. I'm not sure this matters, but it is working this way, so I'm putting it here.
Under packages, I copied the slogger directory and code.
cp -R slogger ~/workspace/soteapps/packages/.
Edited the go.mod file to match the repository structure, which is in the packages directory. There is no go.mod file in the slogger directory.
module gitlab.com/soteapps/packages
go 1.14
Edited the hello.go import to match the package.
package main
import (
"fmt"
"rsc.io/quote"
"gitlab.com/soteapps/packages/slogger"
)
func main() {
fmt.Println(quote.Hello())
slogger.Info("Test message")
}
Built the program using go build -o hello and then ran it hello with the following results:
Scotts-Mac-mini:hello syacko$ hello
Hello, world.
INFO:2020/04/10 21:11:33 Test message
Scotts-Mac-mini:hello syacko$
Worked! Thank you all that helped. This wouldn't of gotten solved without your help.
Note: This only works for public repositories.

How to deploy Go apps with internal package to Heroku using `dep`

I am using Heroku's Golang buildpack to deploy a simple web app with the following structure
my-app/
handler/
user.go
session.go
vendor/
github.com/
golang.org/
main.go
Gopkg.toml
Gopkg.lock
Inside my main file, I imported my own handler package
import (
"fmt"
"net/http"
"my-app/handler"
)
Heroku was not able to run go install on my project due the following error:
----> Using go1.9.3
----> Running: go install -v -tags heroku .
----> cannot find package "my-app/handler" in any of: ...
I can run go install and my-app without any problem locally. It seems to me that heroku does not recognize my internal project package.
I am using dep and I have the following configurations in my Gopkg.toml:
[metadata.heroku]
root-package = "github.com/mygithub/my-app"
go-version = "go1.9.3"
install = ["."]
ensure = "false"
What else do I need to do to deploy a Go app with internal package? Thanks.
Try to set the root package to just the name of the project without the domain
[metadata.heroku]
root-package = "my-app"
go-version = "go1.9.2"
install = [ "./..." ]

K8s Go client library fails to find package on go get

We wrote some Go code to talk to our Kubernetes cluster and fetch the IP of a Service exposed. We do it like so:
(import "gopkg.in/kubernetes/kubernetes.v1/pkg/client/restclient")
(import kubectl "gopkg.in/kubernetes/kubernetes.v1/pkg/client/unversioned")
svc, err := c.Services(k8sNS).Get(svcName)
if err != nil {
panic(l.Errorf("Could not retrieve svc details. %s", err.Error()))
}
svcIP := svc.Status.LoadBalancer.Ingress[0].IP
go get works fine, and our script executes when we do go run ... and everybody is happy. Now, as of yesterday (from the time this question is posted) on the same script - go get fails. The error is like so:
[09.07.2016 10:56 AM]$ go get
package k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install: cannot find package "k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install" in any of:
/usr/local/go/src/k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install (from $GOROOT)
/home/ckotha/godir/src/k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install (from $GOPATH)
We have not specifically used authentication package in our code. Are we importing kubernetes libraries correctly? is there another way to do this ?
ls on $GOPATH/k8s.io/kubernetes/pkg/apis/ and found this:
:~/godir/src/k8s.io/kubernetes/pkg/apis
[09.07.2016 10:53 AM]$ ls
abac apps authentication authorization autoscaling batch certificates componentconfig extensions imagepolicy OWNERS policy rbac storage
It looks like a package you imported has changed.
You can update existing repositories:
go get -u
The -u flag instructs get to use the network to update the named
packages and their dependencies. By default, get uses the network to
check out missing packages but does not use it to look for updates to
existing packages.
You do use gopkg.io to pin the version to v1, but I think you want to be more specific, eg, v1.3.6 (EDIT: this won't work because gopkg.in doesn't permit package selectors more specific than the major version.).
Alternatively, a good way to ensure code stays the same is to compile your binary and execute that, instead of using go run.

Writing an openshift extension

Trying to understand how one builds a tool consuming the Openshift Origin and Kubernetes APIs (I'm new to Go). It seems that I have a version of Kubernetes installed from Openshift and another version installed by godep and I'm getting all sorts of exciting version mismatches.
Here's my code so far:
package main
import (
kclient "k8s.io/kubernetes/pkg/client/unversioned"
client "github.com/openshift/origin/pkg/client"
)
func main() {
config := kclient.Config{
Host: "...",
}
client.SetOpenShiftDefaults(&config)
client := client.NewOrDie(&config)
}
I then did a go get github.com/openshift/origin which I can see has a copy of Kubernetes in it's Godeps/_workspace/src directory.
However when I do a godep save I get:
godep: Package (k8s.io/kubernetes/pkg/client/unversioned) not found
If I do a go get to install Kubernetes I get a version that doesn't match with the version used by Openshift. What's the correct way to do this?

Resources