I'm new to bazel. I am trying out migrating some parts of my company's vast build to bazel. We have some protos zipped up in nexus. I need to grab several archives and compile all the contained proto files together. I have tried all sorts and just can't get it to work.
I'm using an http_archive with a filegroup to get the files.
http_archive(
name = "protos_1",
url = "..."
build_file_content = """
filegroup(
name = "files",
srcs = glob(["**/*.proto"]),
visibility = ["//visibility:public"]
)
""",
)
Now I figured I would just do something like:
proto_library(
name = "combined_protos",
srcs = [
"#protos_1//:files",
"#protos_2//:files",
],
)
but no dice. I've tried using the filegroups as deps and data and everything else I can think of. Any tips?
thanks.
Related
There is a protobuf that my project needs. The protobuf itself is in another repo altogether owned by a completely different team. However they do publish their artifacts on my company's internal Artifactory.
How can I use that (non Bazel) protobuf in my Bazel project?
This is the direction I was trying to go before I hit a deadend.
WORKSPACE file:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file")
http_archive(
name = "rules_proto",
sha256 = "80d3a4ec17354cccc898bfe32118edd934f851b03029d63ef3fc7c8663a7415c",
strip_prefix = "rules_proto-5.3.0-21.5",
urls = [
"https://github.com/bazelbuild/rules_proto/archive/refs/tags/5.3.0-21.5.tar.gz",
],
)
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()
http_file(
name = "some-service-protobuf",
url = "https://artifactory.xyz.com/artifactory/x/y/some-service-protobuf.dbgrel.zip",
)
BUILD file:
load("#rules_proto//proto:defs.bzl", "proto_library")
proto_library(
name = "SomeService.proto",
)
The resources I'm trying to use:
https://bazel.build/reference/be/java#java_proto_library
https://github.com/bazelbuild/rules_proto
Is fetching proto files from remote URL possible? How show I do it after copying those files in my repo.
You might try using http_archive instead:
https://bazel.build/rules/lib/repo/http#http_archive
And then use the build_file or build_file_content attributes to add the build file with the proto_library definition to the external workspace created from the zip file:
http_archive(
name = "some-service-protobuf",
url = "https://artifactory.xyz.com/artifactory/x/y/some-service-protobuf.dbgrel.zip",
build_file = "BUILD.some_service",
)
BUILD.some_service:
load("#rules_proto//proto:defs.bzl", "proto_library")
proto_library(
name = "some_service_proto",
srcs = ["SomeService.proto"],
)
Then in another build file you can do something like:
java_proto_library(
name = "some_service_java_protobuf",
deps = ["#some-service-protobuf//:some_service_proto"],
)
I am using a package graphviz as part of a service and to use it I begin the bazel WORKSPACE file like this
new_local_repository(
name = "graphviz",
path = "/usr/local/Cellar/graphviz/2.49.1",
build_file_content = """
package(default_visibility = ["//visibility:public"])
cc_library(
name = "headers",
srcs = glob(["**/*.dylib"]),
hdrs = glob(["**/*.h"])
)
"""
)
...
The problem with it is its depends upon graphviz being downloaded, pre-installed and present in the path /usr/local/Cellar/graphviz/2.49.1. Is there a way to make it part of the bazel build process such that if its not present it will be fetched and put in the right place?
You can use http_archive to download one of graphviz's release archives:
https://docs.bazel.build/versions/main/repo/http.html#http_archive
From https://graphviz.org/download/source/ the 2.49.1 release is available at https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.1/graphviz-2.49.1.tar.gz
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "graphviz",
url = "https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.1/graphviz-2.49.1.tar.gz",
strip_prefix = "graphviz-2.49.1",
sha256 = "ba1aa7a209025cb3fc5aca1f2c0114e18ea3ad29c481d75e4d445ad44e0fb0f7",
build_file_content = """
package(default_visibility = ["//visibility:public"])
cc_library(
name = "headers",
srcs = glob(["**/*.dylib"]),
hdrs = glob(["**/*.h"])
)
""",
)
To answer the "if its not present" part of the question, I'm not aware of a straight forward way to accomplish automatically switching between something that's locally installed and downloading something. http_archive will always download the archive, and new_local_repository will always use something local.
There's the --override_repository flag, which replaces a repository with a local one, e.g. --override_repository=graphviz=/usr/local/Cellar/graphviz/2.49.1 would effectively replace the http_archive with a local_repository pointing to that path. However, bazel would then expect there to be a WORKSPACE file and BUILD file already at that location (i.e., there's no way to specify build_file_content)
You could specify both repository rules in the WORKSPACE file, and then use some indirection, a Starlark flag, and a select() to switch between the repositories using a command-line flag. It's a little involved though, and also not automatic. Something like this:
WORKSPACE:
http_archive(
name = "graphviz-download",
...,
)
new_local_repository(
name = "graphviz-installed",
...,
)
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
urls = [
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.1.1/bazel-skylib-1.1.1.tar.gz",
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.1.1/bazel-skylib-1.1.1.tar.gz",
],
sha256 = "c6966ec828da198c5d9adbaa94c05e3a1c7f21bd012a0b29ba8ddbccb2c93b0d",
)
load("#bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
bazel_skylib_workspace()
BUILD (e.g. in //third_party/graphviz):
load("#bazel_skylib//rules:common_settings.bzl", "bool_flag")
bool_flag(
name = "use-installed-graphviz",
build_setting_default = False,
)
config_setting(
name = "installed",
flag_values = {
":use-installed-graphviz": "True",
}
)
alias(
name = "headers",
actual = select({
":installed": "#graphviz-installed//:headers",
"//conditions:default": "#graphviz-download//:headers",
})
)
Then your code depends on //third_party/graphviz:headers, by default the alias will point to the downloaded version, and the flag --//third_party/graphviz:use-installed-graphviz will switch it to the installed version:
$ bazel cquery --output build //third_party/graphviz:headers
alias(
name = "headers",
actual = "#graphviz-download//:headers",
)
$ bazel cquery --output build //third_party/graphviz:headers --//third_party/graphviz:use-installed-graphviz
alias(
name = "headers",
actual = "#graphviz-installed//:headers",
)
Another option is to write (or find) a custom repository rule that combines the functionality of http_archive and local_repository, but that would probably be a fair bit of work.
Generally I think most people just use an http_archive and download the dependencies, and if there are specific needs for being offline or caching, there's --distdir for using already-downloaded artifacts for remote repository rules: https://docs.bazel.build/versions/main/guide.html#distribution-files-directories
Edit: An example of using rules_foreign_cc with graphviz:
WORKSPACE:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_foreign_cc",
sha256 = "69023642d5781c68911beda769f91fcbc8ca48711db935a75da7f6536b65047f",
strip_prefix = "rules_foreign_cc-0.6.0",
url = "https://github.com/bazelbuild/rules_foreign_cc/archive/0.6.0.tar.gz",
)
load("#rules_foreign_cc//foreign_cc:repositories.bzl", "rules_foreign_cc_dependencies")
# This sets up some common toolchains for building targets. For more details, please see
# https://bazelbuild.github.io/rules_foreign_cc/0.6.0/flatten.html#rules_foreign_cc_dependencies
rules_foreign_cc_dependencies()
http_archive(
name = "graphviz",
url = "https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.1/graphviz-2.49.1.tar.gz",
strip_prefix = "graphviz-2.49.1",
sha256 = "ba1aa7a209025cb3fc5aca1f2c0114e18ea3ad29c481d75e4d445ad44e0fb0f7",
build_file_content = """\
filegroup(
name = "all_srcs",
srcs = glob(["**"]),
visibility = ["//visibility:public"],
)
""",
)
BUILD:
load("#rules_foreign_cc//foreign_cc:defs.bzl", "configure_make")
# see https://bazelbuild.github.io/rules_foreign_cc/0.6.0/configure_make.html
configure_make(
name = "graphviz",
lib_source = "#graphviz//:all_srcs",
out_shared_libs = ["libcgraph.so"], # or other graphviz libs
)
cc_binary(
name = "foo",
srcs = ["foo.c"],
deps = [":graphviz"],
)
foo.c:
#include "graphviz/cgraph.h"
int main() {
Agraph_t *g;
g = agopen("G", Agdirected, NULL);
agclose(g);
return 0;
}
Usage:
$ bazel build foo
INFO: Analyzed target //:foo (0 packages loaded, 2 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
bazel-bin/foo
INFO: Elapsed time: 0.229s, Critical Path: 0.06s
INFO: 7 processes: 5 internal, 2 linux-sandbox.
INFO: Build completed successfully, 7 total actions
I've use the below bazel rule to build static libraries with bazel:
def _cc_static_library_impl(ctx):
cc_deps = [dep[CcInfo] for dep in ctx.attr.deps]
libraries = []
for cc_dep in cc_deps:
for link_input in cc_dep.linking_context.linker_inputs.to_list():
for library in link_input.libraries:
libraries += library.pic_objects
args = ["r", ctx.outputs.out.path] + [f.path for f in libraries]
ctx.actions.run(
inputs = libraries,
outputs = [ctx.outputs.out],
executable = "/usr/bin/ar",
arguments = args,
)
return [DefaultInfo()]
cc_static_library = rule(
implementation = _cc_static_library_impl,
attrs = {
"deps": attr.label_list(providers = [CcInfo]),
},
outputs = {"out": "lib%{name}.a"},
)
How can I extract the command to use from the current toolchain instead of using the hardcoded /usr/bin/ar? I've based the rule on what I've found on the internet and I have very limited knowledge about this. This example seems to do something related:
https://github.com/bazelbuild/rules_cc/blob/main/examples/my_c_archive/my_c_archive.bzl
This is the relevant part of my_c_archive:
archiver_path = cc_common.get_tool_for_action(
feature_configuration = feature_configuration,
action_name = CPP_LINK_STATIC_LIBRARY_ACTION_NAME,
)
That gets you the path, and then you need to add cc_toolchain.all_files to your ctx.actions.run call, so it ends up looking like this:
ctx.actions.run(
inputs = depset(
direct = [libraries],
transitive = [
cc_toolchain.all_files,
],
),
outputs = [ctx.outputs.out],
executable = archiver_path,
arguments = args,
)
However, you'll also notice that my_c_archive builds up a command line and environment variables to call the archiver with. A simple toolchain won't have anything to pass in either of those, but a more complex one may not work correctly without adding them (for example, passing -m32 or setting PATH).
Part of the starlark implementation of cc_import in _create_archive_action in cc_import.bzl is a good place to start for handling all the intricacies. It creates an action to produce a static library from a set of object files, with flexibility to work with many toolchains.
I am currently using bazel for my GoLang app.
container_image(
name = "my-golang-app",
base = "#ubuntu_base//image",
cmd = ["/bin/my-golang-app"],
directory = "/bin/",
files = [":my-golang-app"],
tags = [
"manual",
VERSION,
],
visibility = ["//visibility:public"],
)
Except my-golang-app folder I need to copy, while building this image, another folder named my-new-folder and everything in it.
How does one do that with bazel? I can't seem to find the solution in the bazel docs.
dockerfile {
run 'mkdir -p /usr/local/bin'
// add the lines below
add {
from 'docker/my-new-folder/'
into '/'
}
Use pkg_tar like the container_image docs reference. Something like this:
pkg_tar(
name = "my-files",
srcs = glob(["my-new-folder/**"]),
strip_prefix = ".",
)
container_image(
name = "my-golang-app",
files = [":my-golang-app"],
<everything else you already have>,
tars = [":my-files"],
)
pkg_tar has various options to control where your files end up, if you want something beyond just taking an entire directory. For more complicated arrangements, I find multiple pkg_tar rules for various directories linked together via deps a helpful pattern.
What is the best way to build C++ code that uses the dlib library using Bazel? I.e., what would the BUILD rules look like?
I tried following the answer for OpenCV as follows, but had no luck:
cc_library(
name = "dlib",
srcs = glob(["build/dlib/*.so*"]),
hdrs = glob(["dlib/*.h"]),
includes = ["include"],
visibility = ["//visibility:public"],
linkstatic = 1,
)
I think I figured it out. Assuming dlib was unzipped to /opt/dlib-19.2 and built in /opt/dlib-19.2/build.
In your WORKSPACE file:
new_local_repository(
name = "dlib",
path = "/opt/dlib-19.2",
build_file = "dlib.BUILD",
)
In dlib.BUILD:
cc_library(
name = "dlib",
srcs = glob(["build/dlib/*.so*"]),
hdrs = glob(["dlib/**/*.h"]),
includes = ["."],
visibility = ["//visibility:public"],
linkstatic = 1,
)