How does protoc find definitions not on IMPORT_PATH? - protocol-buffers

In one of my .proto definitions, I import Google's timestamp definition:
import "google/protobuf/timestamp.proto";
From the documentation,
IMPORT_PATH specifies a directory in which to look for .proto files
when resolving import directives.
I am pretty sure timestamp.proto is not on my IMPORT_PATH, which happens to be the current working directory.
Are google protobuf definitions special somehow? Are these google definitions installed locally, when protoc was installed?

They are indeed special - they are referred to as well-known types.
The compiler looks to resolve these imports relative to where the protoc executable is installed; various locations are checked. If a location is found that contains proto definitions for the well-known types, this is location is looked at as a fall-back - in other words, a path relative to where the protoc executable is run is checked first, before checking the location with the well-known type definitions.
You can see this when executing the compiler with strace or dtruss, e.g. on OSX:
stat64("google/protobuf/timestamp.proto\0", 0x7FFEE4219B98, 0x0) = -1 2
open("google/protobuf/timestamp.proto\0", 0x0, 0xFFFFFFFFFFFFFFFF) = -1 2
stat64("/usr/local/Cellar/protobuf/21.8/include/google/protobuf/timestamp.proto\0", 0x7FFEE4219B98, 0x0) = 0 0
open("/usr/local/Cellar/protobuf/21.8/include/google/protobuf/timestamp.proto\0", 0x0, 0x0) = 3 0

Related

How do I include mkl_vsl modules for random number generators e.g. 'vsrnggaussian'? (Fortran)

I'm new to Fortran (gfortran on windows) and want to use the random number generator vsrnggaussian.
vsrnggaussian needs the Intel MKL VSL modules 'mkl_vsl_type' and 'mkl_vsl' which are provided in the mkl_vsl.f90 file. This file needs to be included to generate the module files 'mkl_vsl_type.mod' and 'mkl_vsl.mod' which are used to process the Fortran use clauses referencing to the VSL interface:
use mkl_vsl_type
use mkl_vsl
I tried multiple things to include the mkl_vsl.f90 file but none of them work:
I inserted the file into the project directory --> 793 errors occured (can't list all of them here in detail) such as:
Unexpected data declaration statement at (1)
[...] at (1) has no implicit type
expecting [...] statement at (1)
unexpected [...] statement at (1)
and finally Fatal Error: Cannot open module file 'mkl_vsl_type.mod' for reading at (1): No such file or directory. compilation terminated. "gfortran -g -o incl_mkl.exe ../incl_mkl.f90" terminated with exit code 1. Build might be incomplete.
I used the full path in the include statement: INCLUDE 'C:/Program Files (x86)/IntelSWTools/compilers_and_libraries_2016.1.146/windows/mkl/include/mkl_vsl.f90' --> Error: Can't open included file
I tried the -Idir compiler command: gfortran -I/Program Files (x86)/IntelSWTools/compilers_and_libraries_2016.1.146/windows/mkl/include/ -g -o incl_mkl.exe --> gfortran: error: Files: no such file or directory
I put this into the makefile: INCLUDES=-I. -I/Program Files (x86)/IntelSWTools/compilers_and_libraries_2016.1.146/windows/mkl/include/ --> Error: Can't open included file
How do I include the mkl_vsl.f90 file??
edit: my simple code to include the mkl_vsl.f90 file
(this code will be compiled first before i compile my main code with vsrnggaussian):
Program INCL
IMPLICIT NONE
INCLUDE 'mkl_vsl.f90'
!INCLUDE 'C:/Program Files (x86)/IntelSWTools/compilers_and_libraries_2016.1.146/windows/mkl/include/mkl_vsl.f90'
END Program
You'd need to USE both vsl_type and vsl modules. The reason is that the derived types in vsl are defined in vsl_type. MWE
Program INCL
use mkl_vsl_type
use mkl
IMPLICIT NONE
write(6, *) 'hello world'
END Program
You would need to pass the address to file mkl_vsl.f90 to ifort and the mkl flag.
ifort -mkl foo.f90 <address>/mkl_vsl.f90
For using any specific subroutine of MKL consult their website. There is also extensive examples that come with MKL. You may find it in C:/Program Files (x86)/IntelSWTools/compilers_and_libraries_2016.1.146/windows/mkl/examples or something like that.

Every "go" command leads to a panic from a certain main.go

I've noticed that every go command has stopped working, due to a panic from a main.go in a particular module:
> go env
panic: required key FOO missing value
goroutine 1 [running]:
github.com/kelseyhightower/envconfig.MustProcess(...)
/Users/kurtpeek/go/pkg/mod/github.com/kelseyhightower/envconfig#v1.4.0/envconfig.go:233
main.main()
/Users/kurtpeek/go/pkg/mod/github.com/myorg/mymodule/go#v0.0.0-20210129234103-92f90e2df5c0/main.go:13 +0x314
where the 'offending' main go is similar to
package main
import (
"github.com/kelseyhightower/envconfig"
"github.pie.apple.com/someorg/somemodule/config"
)
func main() {
cfg := &config.Config{}
envconfig.MustProcess("", cfg)
}
I have no idea why a go env command should fail for this reason?
Your module is called github.com/myorg/mymodule/go, which means that the installed binary is called "go" (after the last path segment). This binary likely shadows the go tool depending on how your PATH is configured.
I suggest you change the module path to avoid this problem.
You might try to reinstall go as it seems that somehow the binary for go env command is replaced by a binary you were potentially trying to compile, possible reason could be you built the program in the directory containing the go tools. I recommend reinstalling go

Unexpected path to libprotocd.lib after add gRPC::grpc++ library dependence in CMake

I have a gRPC 1.23.0 compiled by conan(and all dependencies also resolved by conan).
And I have a CMake project, that use grpc library:
set (LIB_DEPS
protobuf::libprotobuf
gRPC::grpc++
)
target_link_libraries(${PROJECT_NAME} PUBLIC ${LIB_DEPS})
When I generate a VS project from this CMakeLists.txt in Linker - Input - Addition Dependencies property, I have there next value:
C:\.conan\data\protobuf\3.9.1\kpa_conan\stable\package\b786e9ece960c3a76378ca4d5b0d0e922f4cedc1\lib\libprotobufd.lib <-- (1)
C:\.conan\data\grpc\1.23.0\kpa_conan\stable\package\d85cccdf40588ac852bd1445d45838487543194f\lib\grpc++.lib
libprotocd.lib <-- (2)
libprotobufd.lib <-- (3)
C:\.conan\data\grpc\1.23.0\kpa_conan\stable\package\d85cccdf40588ac852bd1445d45838487543194f\lib\grpc.lib
C:\.conan\data\grpc\1.23.0\kpa_conan\stable\package\d85cccdf40588ac852bd1445d45838487543194f\lib\gpr.lib
C:\.conan\data\c-ares\1.15.0\kpa_conan\stable\package\b786e9ece960c3a76378ca4d5b0d0e922f4cedc1\lib\cares.lib
C:\.conan\data\grpc\1.23.0\kpa_conan\stable\package\d85cccdf40588ac852bd1445d45838487543194f\lib\address_sorting.lib
wsock32.lib
kernel32.lib
(1) - expected valid path, that I add in CMakeLists.txt
(2),(3) - unexpected and invalid path, that added by gRPC::grpc++ in CMakeLists.txt.
If I change gRPC::grpc++ to gRPC::grpc, lines (2) and (3) will disappear, but path to grpc++.lib will disappear too, but I need it.
How to avoid this strange invalid path to libprotocd.lib and libprotobufd.lib?
To properly link against grpc targets you can use:
target_link_libraries(${PROJECT_NAME} PUBLIC CONAN_PKG::grpc)
This should contain everything required.
More details on this approach here: https://docs.conan.io/en/latest/integrations/build_system/cmake/cmake_generator.html#targets-approach
I have found workaround solution:
find_library(GRPC_GRPC++_LIBRARY NAMES grpc++)
set (LIB_DEPS
protobuf::libprotobuf
gRPC::grpc
${GRPC_GRPC++_LIBRARY NAMES}
)
target_link_libraries(${PROJECT_NAME} PUBLIC ${LIB_DEPS})
And opened the issue about it in gRPC repository: https://github.com/grpc/grpc/issues/20578

Buiding CockroachDB for RPi yields 'segmentation violation error'

I am trying to build CockroachDB from sources on a RPi 3, following a couple of posts (this and this).
My RaspberryPi setup:
OS: 2018-06-27-raspbian-stretch
CPU: ARMv7 rev 4(v71)
Mem: 1G + SWAP: 1.5G
Go version: go1.11 linux/arm
I fixed a couple of errors regarding 3rd party libraries while 'make'ing it but now I got stuck on this:
pi#raspberrypi:~/work/go/src/github.com/cockroachdb/cockroach $ make build TAGS='stdmalloc' -j2
GOPATH set to /home/pi/work/go
go build -o cockroach -v -tags 'stdmalloc make arm_linux_gnueabihf_stdmalloc' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v2.2.0-alpha.00000000-668-gf76d921f42-dirty" -X "github.com/cockroachdb/cockroach/pkg/build.rev=f76d921f4262356fde83312d0f471262d995f1e0" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=arm-linux-gnueabihf" -X "github.com/cockroachdb/cockroach/pkg/build.utcTime=2018/09/11 12:47:32"' ./pkg/cmd/cockroach
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2372b8]
goroutine 1 [running]:
runtime/internal/atomic.goLoad64(0x3d65bdc, 0x0, 0x5607a20)
/usr/local/go/src/runtime/internal/atomic/atomic_arm.go:124 +0x1c
github.com/cockroachdb/cockroach/pkg/util/humanizeutil.(*BytesValue).String(0x55fb578, 0x5607a01, 0x56d9ea0)
/home/pi/work/go/src/github.com/cockroachdb/cockroach/pkg/util/humanizeutil/humanize.go:109 +0x28
github.com/cockroachdb/cockroach/pkg/cli.(*bytesOrPercentageValue).String(0x57b4920, 0x56d9ea0, 0x7)
/home/pi/work/go/src/github.com/cockroachdb/cockroach/pkg/cli/flags_util.go:492 +0x20
github.com/cockroachdb/cockroach/vendor/github.com/spf13/pflag.(*FlagSet).VarPF(0x54d3a00, 0x25cf778, 0x57b4920, 0x1ef3864, 0xe, 0x0, 0x0, 0x56d2900, 0x169, 0x56d9ea0)
/home/pi/work/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/pflag/flag.go:778 +0x24
github.com/cockroachdb/cockroach/vendor/github.com/spf13/pflag.(*FlagSet).VarP(0x54d3a00, 0x25cf778, 0x57b4920, 0x1ef3864, 0xe, 0x0, 0x0, 0x56d2900, 0x169)
/home/pi/work/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/pflag/flag.go:786 +0x5c
github.com/cockroachdb/cockroach/pkg/cli.VarFlag(0x54d3a00, 0x25cf778, 0x57b4920, 0x1ef3864, 0xe, 0x0, 0x0, 0x0, 0x0, 0x1fa4462, ...)
/home/pi/work/go/src/github.com/cockroachdb/cockroach/pkg/cli/flags.go:129 +0x6c
github.com/cockroachdb/cockroach/pkg/cli.init.5()
/home/pi/work/go/src/github.com/cockroachdb/cockroach/pkg/cli/flags.go:324 +0x6c0
Makefile:1288: recipe for target 'docs/generated/settings/settings.html' failed
make: *** [docs/generated/settings/settings.html] Error 2
make: *** Deleting file 'docs/generated/settings/settings.html'
Any ideas what might be wrong?
thanks for the question. I was the original author of that blog post. It looks like you're running into an issue related to https://github.com/golang/go/issues/9959. It looks like golang requires 64-bit alignment of 64-bit words when atomically accessed. This is somewhat tricky to enforce across an entire project. Still, we can try to get this working by removing the atomic access on pkg/util/humanizeutil/humanize.go:109 (which doesn't look necessary) and replacing it with return IBytes(*b.val). That should at least get you past this roadblock. There may be other issues you run into after this though. Unfortunately, CockroachDB has never set up CI around a 32-bit ARMv7 build because 32-bit architectures have never been officially supported by the project.
However, a 64-bit ARMv8 build is more actively maintained. In fact, an entire cross-compilation toolchain has been introduced to build CockroachDB for ARMv8 on an x86_64 host. Compilation using this toolchain can be kicked off by running build/builder.sh mkrelease arm64-linux-gnueabi. The resulting binary can be dropped onto a RaspberryPi running a 64-bit OS like https://wiki.debian.org/RaspberryPi3. I actually tried this out last week and ran into no problems.

Stringer tool complains about wrong archive header

I am trying to use go generate/stringer (golang.org/x/tools/cmd/stringer) to generate String() methods on enums. I have problems, which I believe, are because of slightly different format of .a packages on different systems. I have this file:
package main
import (
"math/rand"
)
//go:generate stringer -type=Foo
type Foo int;
const (
FooPrime Foo = iota
FooBis
)
func main() {
//Just use rand anywhere, otherwise we get a compiler error
rand.Seed(1)
}
Now if I run go generate example.go on my machine everything is all right: foo_string.go is created. However, on a test machine I get:
stringer: checking package: example.go:4:2: could not import math/rand (reading export data: /usr/lib64/go/pkg/linux_amd64/math/rand.a: go archive is missing __.PKGDEF)
Now, after some digging in the code I think that I get this error, because on my machine rand.a has the following header:
!<arch>
__.PKGDEF 0 0 0 644 2051
`
while on test machine it has the following header:
!<arch>
__.PKGDEF/ 0 399 399 100644 2051
`
I think that the crucial difference is slash after PKGDEFF. gcimporter refuses to process .a file, if it doesn't have __.PKGDEF header.
To check this, I edited manually gcimporter/exportdata.go and changed one of the line from this:
if name != "__.PKGDEF"
to this:
if name != "__.PKGDEF" && name != "__.PKGDEF\"
After this change (and compiling and installing everything) I was able to run go generate on example.go.
My questions are: why do I get this problem and how do I get rid of it (other then manually editing external library)?
What I can see from the spec for openSUSE's packaging they are disabling reinstallation of the standard library at updates. __.PKGDEF is a Go specific informational section, and some linker OpenSUSE has used has simply produced incompatible output.
There's nothing you can do except install a healthy Go from the official source.

Resources