I have the following bazel BUILD configured to
gazelle(name = "gazelle")
go_embed_data(
name = "static_files",
srcs = glob(["static/**/*"]),
package = "main",
var = "staticFS",
)
go_library(
name = "kmdr_lib",
srcs = ["main.go"],
importpath = "github.com/myorg/myrepo",
visibility = ["//visibility:private"],
deps = [
"//api",
"//cmd",
],
)
With the following embed tag
package main
import (
"embed"
"github.com/myorg/myrepo/api"
"github.com/myorg/myrepo/cmd"
)
//go:embed static/*
var staticFS embed.FS
func main() {
api.StaticFS = staticFS
cmd.Execute()
}
However, when running...
bazel run //:gazelle
bazel build //...
I receive the following error that the static files tagged in the go main package could not be matched.
ERROR: GoCompilePkg kmdr_osx_amd64.a failed: (Exit 1): builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix darwin_amd64 -src main.go -arc ... (remaining 17 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
compilepkg: baf0dff8fcdeaf74ff5ba5ff8921e77f/sandbox/linux-sandbox/425/execroot/__main__/main.go:22:12: could not embed static/*: no matching files found
INFO: Elapsed time: 1.249s, Critical Path: 0.15s
INFO: 5 processes: 5 internal.
FAILED: Build did NOT complete successfully
The go_embed_data documentation doesn't give much details on how to use the library. I've also tried referencing the :static_files in the go_library srcs however, gazelle rewrites that.
bazel will rewrite the go_library if I reference go_emebed_data in the library srcs
go_embed_data generates a .go file that contains data from a file or a list of files. It should be consumed in the srcs list of one of the core go rules.
go_library(
name = "kmdr_lib",
srcs = ["main.go", ":static_files"],
importpath = "github.com/myorg/myrepo",
visibility = ["//visibility:private"],
deps = [
"//api",
"//cmd",
],
)
EDIT:
go build will parse the tag and embed the data as expected
It looks like go_embed_data is not the right method.
There was a PR made to address this issue https://github.com/bazelbuild/rules_go/pull/2806#issuecomment-784690934
Adding embedsrcs to your go_library will respect the go:embed directive
Related
I am working on a C++ library where I am following the structure below:
ROOTDIR = ~/path/to/proj
BLDDIR = $(ROOTDIR)/build
INCDIR = $(ROOTDIR)/include
SRCDIR = $(ROOTDIR)/src
LIBSRCDIR = $(SRCDIR)/lib
TESTSRCDIR = $(SRCDIR)/test
OBJDIR = $(ROOTDIR)/obj
LIBOBJDIR = $(OBJDIR)/lib
TESTOBJDIR = $(OBJDIR)/test
LIBDIR = $(ROOTDIR)/lib
BINDIR = $(ROOTDIR)/bin
And the file definitions are:
MKFILE = $(BLDDIR)/Makefile
HDRFILES = $(wildcard $(INCDIR)/*.hpp)
LIBSRCFILES := $(wildcard $(LIBSRCDIR)/*.cpp)
TESTSRC = $(TESTSCRCDIR)/test.cpp
LIBOBJFILES := $(patsubst $(LIBSRCDIR)/%.cpp, $(LIBOBJDIR)/%.o, $(LIBSRCFILES))
TESTOBJ := $(TESTOBJDIR)/test.o
LIBFILE = $(LIBDIR)/$(NAME)-$(VERSION).a
TESTFILE = $(BINDIR)/test
ZIPFILE = $(ROOTDIR)/$(NAME).$(VERSION).zip
I have tested & confirmed that everything works, but for some reason $(TESTSRC) is printing empty even though Make is able to set $(TESTOBJ). I tried setting up the variable definitions in the hierarchy to ensure everything is generated in level, but that one variable remains empty.
EDIT 1:
Make is finding all of the directories properly
I have made sure that the file extension is correct
The variable is not being overwritten
EDIT 2:
The specific error I"mg getting is:
make: *** No rule to make target /test.cpp', needed by ..../test.o
..../test.o is in the right directory & defined separately without being dependent on a patsubst like the libsrc.
Read carefully:
TESTSRCDIR = $(SRCDIR)/test
^^^
and:
TESTSRC = $(TESTSCRCDIR)/test.cpp
^^^
I need to include some generated files (for example output from go-swagger) in a binary. Without bazel you can do that using go:generate in combination with go:embed:
package demo
//go:generate swagger generate spec -w . -o ./openapi.json
import "embed"
// Content contains openapi.json file generated via go-swagger from spec
//go:embed openapi.json
var Content embed.FS
I am trying to do the same thing with bazel. As a simple test I have have this BUILD.bazel file:
genrule(
name = "hellodata",
srcs = ["hello.go"],
outs = ["hello.txt"],
cmd = "cat $(SRCS) | tr A-Za-z N-ZA-Mn-za-m > $#",
)
go_library(
name = "hello",
srcs = ["hello.go"],
importpath = "wiggy.net/hello",
visibility = ["//visibility:public"],
embedsrcs = [":hellodata"],
)
with hello.go looking like this:
package hello
import (
_ "embed"
"io"
)
//go:embed hello.txt
var greeting []byte
func Hello(out io.Writer) error {
_, err := out.Write(greeting)
return err
}
The intention here is to have Hello output the rot13 of it's own source. When I try to compile this it successfully generates hello.txt (located in bazel-out/darwin_arm64-fastbuild/bin/hello.txt), but the compiler can not find it:
❯ bazel build //...
INFO: Analyzed 5 targets (0 packages loaded, 0 targets configured).
INFO: Found 5 targets...
ERROR: /Users/wichert/Hack/bzl/BUILD.bazel:14:11: GoCompilePkg hello.a failed: (Exit 1): builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix darwin_arm64 -src hello.go -embedsrc bazel-out/darwin_arm64-fastbuild/bin/hello.txt -importpath wiggy.net/hello ... (remaining 12 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
compilepkg: /private/var/tmp/_bazel_wichert/e7573342ee9452df4c3dfa671d399a16/sandbox/darwin-sandbox/76/execroot/__main__/hello.go:8:12: could not embed hello.txt: no matching files found
INFO: Elapsed time: 0,112s, Critical Path: 0,04s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
I noticed -embedsrc bazel-out/darwin_arm64-fastbuild/bin/hello.txt in the command line for the failing action in your question, so as a hunch I tried the equivalent on my machine:
//go:embed bazel-out/k8-fastbuild/bin/hello.txt
var greeting []byte
And that seemed to work.
This is not so great because configuration information gets embedded in the source files (indeed, on your mac machine it's darwin_arm64-fastbuild and on my linux machine it's k8-fastbuild), so now your source code is artificially platform dependent.
It looks like this feature is relatively new (https://github.com/bazelbuild/rules_go/issues/2775, https://github.com/bazelbuild/rules_go/issues/2986). I would file an issue with rules_go about this.
There also appears to be go_embed_data which might behave differently:
https://github.com/bazelbuild/rules_go/blob/master/docs/go/extras/extras.md#go_embed_data
To answer my own question: the trick is to use genrule to generate the file(s) to be embedded, and then use go_embed_data to embed them. A work BUILD.bazel looks like this:
genrule(
name = "hellodata",
srcs = ["hello.go"],
outs = ["hello.txt"],
cmd = "cat $(SRCS) | tr A-Za-z N-ZA-Mn-za-m > $#",
)
go_embed_data(
name = "hello_embed",
src = ":hellodata",
package = "hello",
var = "greeting",
)
go_library(
name = "hello",
srcs = [
"hello.go",
":hello_embed",
],
importpath = "wiggy.net/hello",
visibility = ["//visibility:public"],
)
Starting with micro and everything works fine. I dont have VisualStudio, but instead I have make through chocolatey(dont know if could be the issue)
I bootstrap the service with micro new like this.
λ micro new my.test
Creating service my.test
.
├── micro.mu
├── main.go
├── generate.go
├── handler
│ └── my.test.go
├── proto
│ └── my.test.proto
├── Dockerfile
├── Makefile
├── README.md
├── .gitignore
└── go.mod
download protoc zip packages (protoc-$VERSION-$PLATFORM.zip) and install:
visit https://github.com/protocolbuffers/protobuf/releases
download protobuf for micro:
go get -u github.com/golang/protobuf/proto
go get -u github.com/golang/protobuf/protoc-gen-go
go get github.com/micro/micro/v3/cmd/protoc-gen-micro
compile the proto file my.test.proto:
cd my.test
make proto
Installed dependencies and everything was fine. Then I moved to my.test and after make proto this error shows up
protoc --proto_path=. --micro_out=. --go_out=:. proto/proto.test.proto
proto/proto.test.proto:7:14: Expected "{".
make: *** [Makefile:16: proto] Error 1
I have all dependencies and my PATH is fine, but I dnt know what the problem is.
Edit: This is my proto, conveniently generated by micro
syntax = "proto3";
package proto.test;
option go_package = "./proto;proto.test";
service Proto.Test {
rpc Call(Request) returns (Response) {}
rpc Stream(StreamingRequest) returns (stream StreamingResponse) {}
rpc PingPong(stream Ping) returns (stream Pong) {}
}
message Message {
string say = 1;
}
message Request {
string name = 1;
}
message Response {
string msg = 1;
}
message StreamingRequest {
int64 count = 1;
}
message StreamingResponse {
int64 count = 1;
}
message Ping {
int64 stroke = 1;
}
message Pong {
int64 stroke = 1;
}
Go package names cannot contain dots - it will just not compile:
$ cat pkg.go
package dot.test
$ go build
./pkg.go:1:12: syntax error: unexpected .
so one has to ensure generated code produces a valid Go package name.
From the Go spec, the package clause is defined as:
A package clause begins each source file and defines the package to
which the file belongs.
PackageClause = "package" PackageName .
PackageName = identifier .
and an identifier as:
... a sequence of one or more letters and digits. The first
character in an identifier must be a letter.
identifier = letter { letter | unicode_digit } .
After some reading of the basics I found the problem. Even if the proto file is generated by micro this code service Proto.Test at line seven is the problem, because of the dot. I mean, replacing that for service Test solved the issue. Now I dont know why. Any explanation would be preciated. I am in windows by the way
I want to write a yocto recipe for a cross-compiled golang application with Yocto 2.4.1 but I cannot get external dependencies to work.
Can anyone help me?
current RECIPE_FILE: hello-world_%.bb
LICENSE = "CLOSED"
LIC_FILES_CHKSUM = ""
DESCRIPTION = "Hello world test with golang."
inherit go
COMPATIBLE_MACHINE = "(<machine>)"
DEPENDS = "go-cross-${TARGET_ARCH}"
GO_IMPORT = "hello-world"
SRC_URI = "<git_url>/${GO_IMPORT}.git;branch=${SRCBRANCH};tag=${PV}"
SRCBRANCH = "master"
PV = "0.01"
S = "${WORKDIR}/git"
do_compile() {
export GOPATH="${WORKDIR}/build"
export GOARCH="<machine_arch>"
export GOOS="linux"
export CGO_ENABLED="0"
go build src/${GO_IMPORT}/hello-world.go
}
do_install() {
install -d "${D}/${bindir}"
install -m 0755 "${WORKDIR}/build/hello-world" "${D}/${bindir}/hello-world"
}
RDEPENDS_${PN}-dev += "bash"
This recipe works fine for internal dependencies only. But how do I integrate external dependencies like "github.com/golang/protobuf/ptypes"?
PROJECT_FILE: hello-world.go
package main
import (
"fmt"
"github.com/golang/protobuf/ptypes"
)
func main() {
timestamp := ptypes.TimestampNow()
fmt.Println(timestamp)
}
Does anyone knows a solution for this use case?
Or does anyone know how "go-dep" could handle this?
Best regards
I used go dep for the deps, here's an example. Most trouble was about the proxy which is also solved in the recipe:
inherit go
LICENSE = "CLOSED"
LIC_FILES_CHKSUM = ""
DESCRIPTION = "Hello world test with golang."
COMPATIBLE_MACHINE = "(<machine>)"
DEPENDS += "go-dep-native"
GO_LINKSHARED = ""
GO_IMPORT = "<git_url>/hello-world.git"
SRC_URI = "<git_url>/${GO_IMPORT}.git;branch=${SRCBRANCH};tag=${PV}"
SRCBRANCH = "master"
do_compile() {
export SSH_AUTH_SOCK="${SSH_AUTH_SOCK}"
export HTTP_PROXY="${HTTP_PROXY}"
( cd ${WORKDIR}/build/src/${GO_IMPORT} && dep ensure -v )
}
do_compile[vardepsexclude] += "SSH_AUTH_SOCK HTTP_PROXY"
do_install() {
install -d "${D}/${bindir}"
install -m 0755 "${WORKDIR}/bin/<arch>/hello-world" "${D}/${bindir}/hello-world"
}
I believe there is only two types of dependencies
1. host dependencies (dependencies when the app compiling time in yocto)
in yocto recipe(.bb file) keep DEPENDS = "some lib"
target dependencies (dependencies when the app running time on board)
your yocto recipe RDEPENDS = "some lib"
hello.bb
DESCRIPTION =
LIC =
SRC_URI =
DEPENDS ="sqlite3"
inherit autools
you can add external go dependencies with SRC_URI:
SRC_URI = "\
<git_url>/${GO_IMPORT}.git;branch=${SRCBRANCH};tag=${PV} \
git://github.com/golang/protobuf/ptypes;protocol=https;name=ptype;destsuffix=${PN}-${PV}/src/github.com/golang/protobuf/ptypes \
"
SRCREV_ptype = "v0.1.0" <-- whatever revision you need (branch, tag, sha)
Is there a good way to integrate protobuf compiles with the go build command?
goprotobuf "provides Go support, in the form of a library and protocol compiler plugin, for Google's protocol buffers".
The README at the goprotobuf library has some good info.
From https://code.google.com/p/goprotobuf/source/browse/README#106 :
Consider file test.proto, containing
package example;
enum FOO { X = 17; };
message Test {
required string label = 1;
optional int32 type = 2 [default=77];
repeated int64 reps = 3;
optional group OptionalGroup = 4 {
required string RequiredField = 5;
}
}
To build a package from test.proto and some other Go files, write a Makefile like this:
include $(GOROOT)/src/Make.$(GOARCH)
TARG=path/to/example
GOFILES=\
test.pb.go\
other.go
include $(GOROOT)/src/Make.pkg
include $(GOROOT)/src/pkg/code.google.com/p/goprotobuf/Make.protobuf