I have some projects using Bazel, C++ and protobuf. I also use gitlab CI/CD to build, test, check coverage, etc.
The problem is that when the project compiles first time it also compiles a protobuf compiler, which adds about 15 minutes to each step (the step itself takes 1-5 min).
I was using a setup example from this documentation:
https://blog.bazel.build/2017/02/27/protocol-buffers.html
Here I created a simple hello world example with protobuf.
When I use protoc to generate *.pb.cc, *.pb.h files it takes about 5 seconds.
When I use bazel build ... it takes 15 minutes, because it builds protobuf compiler.
Build log: https://gitlab.com/mvfwd/issue-bazel-protobuf-compile/-/jobs/1532045913
Main Question
Is there any other way to setup Bazel to use already precompiled protoc and to skip 15 min on every step?
Update 2021-08-27
Added overwriting proto_compiler and proto_toolchain_for_cc as described in Implicit Dependencies and Proto Toolchains
Building :person_proto now works fine
$ bazel build :person_proto
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
INFO: Analyzed target //:person_proto (19 packages loaded, 61 targets configured).
INFO: Found 1 target...
Target //:person_proto up-to-date:
bazel-bin/person_proto-descriptor-set.proto.bin
INFO: Elapsed time: 0.428s, Critical Path: 0.08s
INFO: 5 processes: 4 internal, 1 linux-sandbox.
INFO: Build completed successfully, 5 total actions
but building :person_cc_proto fails
$ bazel build :person_cc_proto
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
ERROR: /home/m/Synology/drive/prog/2021/b/issue-bazel-protobuf-compile/BUILD:2:14: in :aspect_cc_proto_toolchain attribute of BazelCcProtoAspect aspect on proto_library rule //:person_proto: '#local_config_cc//:toolchain' does not have mandatory providers: ProtoLangToolchainProvider
ERROR: Analysis of target '//:person_cc_proto' failed; build aborted: Analysis of target '//:person_proto' failed
INFO: Elapsed time: 0.124s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 25 targets configured)
From https://blog.bazel.build/2017/02/27/protocol-buffers.html#implicit-dependencies-and-proto-toolchains
The proto_library rule implicitly depends on #com_google_protobuf//:protoc, which is the protocol buffer compiler. It must be a binary rule (in protobuf, it's a cc_binary). The rule can be overridden using the --proto_compiler command-line flag.
Hence, you could (not tested)
add the precompiled binary in your workspace,
define a cc_import for this target
pass the --proto_compiler command-line flag
Related
After making a GitHub repo and adding my code to my account my code now throws me an error, previous resolutions to the error include changing the sequence of the build phases in Xcode but I seem to have no luck, I also don't have a header phase in my build phases which rules out the older method, error below.
Showing Recent Messages
Prepare build
note: Using new build system
note: Planning
note: Build preparation complete
note: Building targets in dependency order
error: Cycle inside Target.swift; building could produce unreliable results.
Cycle details:
→ Target 'Target': CodeSign /Users/macbook/Desktop/target/Build/Products/Debug-iphonesimulator/target.app
Raw dependency cycle trace:
target: ->
node: ->
command: ->
CYCLE POINT ->
node: /Users/name/Desktop/Target/Build/Products/Debug-iphonesimulator/Target.app/_CodeSignature ->
Build failed 4/10/22, 10:12 PM 0.7 seconds
I have a simple Gradle project that has org.gradle.caching=true set in gradle.properties in order to enable the local build cache.
When I run the build directly (./gradlew clean build) I can see that the local build cache is being used: https://scans.gradle.com/s/ykywrv3lzik3s/performance/build-cache
However, when I run the build with Coverity (bin/cov-build --dir cov-int ./gradlew clean build) I see the build cache is disabled for the same build: https://scans.gradle.com/s/j2pvoyhgzvvxk/performance/build-cache
How is Coverity causing the build cache to be disabled, and is there a way to run a build with Coverity and the Gradle Build Cache?
You can't use the build cache with Coverity, or at least you don't want to.
The Gradle Build Cache causes compilation to be skipped:
The Gradle build cache is a cache mechanism that aims to save time by reusing outputs produced by other builds. The build cache works by storing (locally or remotely) build outputs and allowing builds to fetch these outputs from the cache when it is determined that inputs have not changed, avoiding the expensive work of regenerating them.
Were that mechanism to be used with Coverity, it would prevent cov-build from seeing the compilation steps, and hence it would be unable to perform its own compilation of the source code, which is a necessary prerequisite to performing its static analysis.
I don't know precisely how Coverity is disabling the cache (or if that is even intentional on Coverity's part), but if it didn't do so, then you would have to yourself, as described in the Synopsys article Cov-build using gradle shows "No files were emitted" error message, the key step of which is:
Use "clean" and "cleanBuildCache" task to remove all saved cache data which prevent full compilation.
before running cov-build.
I'd like to use Gazelle to manage my Go dependencies (and their dependencies) in Bazel. Running bazel run //:gazelle update-repos firebase.google.com/go adds a properly configured go_repository to my WORKSPACE file:
go_repository(
name = "com_google_firebase_go",
importpath = "firebase.google.com/go",
sum = "h1:3TdYC3DDi6aHn20qoRkxwGqNgdjtblwVAyRLQwGn/+4=",
version = "v3.13.0+incompatible",
)
However, this does not work out of the box. Running bazel build #com_google_firebase_go//:go_default_library returns an error:
ERROR: /private/var/tmp/_bazel_spencerconnaughton/9b09d78e8f2190e9af61aa37bcab571e/external/com_google_firebase_go/BUILD.bazel:3:11: no such package '#org_golang_google_api//option': The repository '#org_golang_google_api' could not be resolved and referenced by '#com_google_firebase_go//:go'
ERROR: Analysis of target '#com_google_firebase_go//:go_default_library' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.596s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (23 packages loaded, 133 targets configured)
Is there a way to tell gazelle to load the #org_golang_google_api transitive dependency and others without needing to run update-repos for each one?
I have been struggling with this as well, but it seems that you can simply solve it by using go mod :)
If you use the go mod and go get commands to generate go.mod files, the transative dependencies are included automatically. You can then use this go.mod file in your bazel-gazelle command ;)
let's say your project is called github.com/somesampleproject:
go mod init github.com/somesampleproject
Then, use go get to add your dependency to the go.mod file:
go get firebase.google.com/go
go mod actually handles the transitive dependencies as well, as described in the documentation here. So your go.mod file should contain all of your required dependencies now :)
I quickly tried this out with gazelle in one of our projects (as described in the gazelle documentation)
bazel run //:gazelle -- update-repos -from_file=\<insert-subfolder-here>/go.mod
now our WORKSPACE file includes the firebase dependency, but also the dependencies that firebase has itself.
For bonus points you could use tools to validate your go.mod file to make sure you have no dependencies with known security bugs, a quick google search returned snyke, nancy and built-in support in ide's such as vs code apparently ;)
After updating to Xcode 10 there are some issues with initial project configurations. The steps look like this:
Generating some files using Sourcery
Linting with SwiftLint
Build
And configuration works like this:
And this was working fine in Xcode 9, but apparently, it's not working under Xcode 10 build system. The issue is if I download repository (.generated files are not a part of the repository). And hit build it will show me results like:
...
Using configuration file at '.sourcery.yml'
Scanning sources...
Found 239 types.
Loading templates...
Loaded 9 templates.
Generating code...
Finished.
Processing time 0.491948962211609 seconds
...
So sourcery works ok, then linting:
Linting 'FromResponse.generated.swift' (1/186)
Works fine as well as a process, but in the end, build will fail with:
error: Build input files cannot be found:
'/path/Generated/FromResponse.generated.swift',
...
So it's strange since a files are physically there and according to build settings file should be compiled later than generating them. How should I approach sourcery in build process then?
We are able to "fix" this in a similar way as this Github issue:
https://github.com/mac-cain13/R.swift/issues/438#issuecomment-416982928
We have to add each of the generated files by Sourcery to the output folder and it will get picked up by the Xcode correctly.
This relates to changes in the New Xcode Build System that has been enabled by default in Xcode 10. If a build phase creates files which are needed as an input to a later build phase then it needs to specify them explicitly in the Output Files or Output File List.
In your example the Sourcery build phase is generating Swift source file(s) which are needed as input to the Compile Sources phase.
However, the issue at this point is that after you specify the output files for Sourcery, the build phase doesn't re-run every time as it sees the output file is already there. So far I haven't worked out a useable solution to this part, beyond running tools like Sourcery and SwiftGen manually or keeping the generated files in Git so that they are always present.
I was having a similar problem with a buildphase calling mogenerator to build my data model classes. The approach given by #Yuchen should work.
#Andrew: To force always (re-)running this build phase, I add the line
touch .alwaysRun
as the last line in the shell command to run and mark the build phase "Input Files" to have $(SRCROOT)/.alwaysRun. That seems to do the trick.
Let's get this out of the way first: This question is not a duplicate of this earlier question. I'll explain why below.
I'm running the command xcodebuild test -scheme 'ISO8601ForCocoa' SYMROOT=../build, and here's the output:
Executed 16 tests, with 0 failures (0 unexpected) in 0.047 (0.051) seconds
profiling: invalid magic number (0x656d6954)
profiling: invalid magic number (0x00000000)
** TEST SUCCEEDED **
The person who asked that earlier question got the same error message, but they fixed it by cleaning their build folder.
In my case, cleaning will not help, because I don't have a build folder yet. I still get this error message even on a completely fresh build. In fact, the .gcda and .gcno files are not even created, so there is nothing to clean.
xcodebuild puts build products in $SYMROOT, but stores profile products in $OBJROOT.
If you want to completely confine a build, including all intermediate and ultimate products, to a single directory, you need to set three build settings:
SYMROOT: Build Products Path (ultimate products, such as apps and test bundles)
OBJROOT: Intermediate Build Files Path (intermediate products, such as per-module object files—generated by the compiler, read by the linker—and .gcno/.gcda files)
SHARED_PRECOMPS_DIR: Precompiled Headers Cache Path (guess)