I want to make some changes to the Go crypto/tls standard library.
Is making a copy of crypto/tls in the vendor folder a good way to do this?
It almost works, it seems the vendored is copy used when I compile the application (Caddy webserver). Apart from one error I get:
go/src/github.com/user/caddy/caddytls/httphandler.go:40: cannot use "vendor/crypto/tls".Config literal (type *"vendor/crypto/tls".Config) as type *"crypto/tls".Config in field value
Is there a way of casting to get around this one error? Doesn't sound like good practice to me though.
I would have thought that the vendored copy would always be used, but it seems something is still using the standard crypto/tls library? (I think "net/http" is. Do I have to vendor this too?)
I actually had to do that. The most practical way is to copy and modify the package (as well as its internal dependencies) - this includes some import paths. And it is not really vendored (vendoring is basically using unmodified packages, otherwise vendoring tools will not work), it is a fork, under a different name. I would guess that caddy does not need the modified version - if it does, you need to fork and modify caddy as well.
It goes without saying, but be extremely careful when modifying crypto/tls package - I for example has to make a minimal change that does not really modify TLS operation (I needed to be able to derive key material from the session master secret and randoms).
Also, you have to fully realize that this has significant cost - when a new version of go is out, one that potentially has updates for the crypto/tls package or its dependencies, you will have to apply your changes once again, manually. Committing a diff between the vanilla version and your version helps. I don't think this is at all practical for non-trivial changes (mine were quite limited - a new public field in Config, few lines of code in handshakes, a new interface).
I need the same functionality. It seems like the crypto/tls package does not allow the reading of custom TLS extensions added from the client side in the ClientHello payload. It would be great to be able to check for any custom extensions and then marshal them out accordingly.
It is a pity that this is not a separate package as we could then in our go.mod file be able to use replace to specify a custom TLS package.
e.g
replace golang.org/x/crypto/tls => ./tls
Then running go mod vendor.
Where the ./tls is my local version with the changes applied.
Related
Consider two software projects, proj_a and proj_b, with the latter depending on the former; and with both using CMake.
When reading about modern CMake, one gets the message that the "appropriate" way to express dependencies is via target dependencies; and one should arrange it so that dependent projects are represented as (imported) targets you can depend on. More specifically, in our example, proj_b will idiomatically have:
find_package(proj_a)
# etc etc.
target_link_library(bar proj_a::foo)
and proj_a will need to have been installed, utilizing the CMake installation-and-export-related commands, someplace where proj_b's CMake invocation will search for proj_a-config.cmake.
I like this approach and encourage others to adapt to it. It offers flexibility in the choice of your own version of proj_a vs the system version; and also allows for non-CMake proj_a's via a Findproj_a.cmake script (which again, can be system-level or part of proj_b).
So far so good, right? However, there are people who want to "take matters into their own hands" in terms of dependencies - and CMake officially condones this, with commands such as ExternalProject and more recently, FetchContent: This allows proj_b's configuration stage to actually download a (built, or in our case source-form) version of proj_a.
The puzzling part to me is that, after proj_a is downloaded, say to an external/proj_a directory, CMake's default behavior will be to
add_subdirectory(external/proj_a)
that is, to use proj_a as a subproject of proj_b and build them together. This, while the idiomatic use above allows the maintainer of proj_a to "do their own thing" in my CMakeFile, and only keep things neat and tidy for others via what I export/install.
My questions:
Why does it make sense to add_subdirectory(), rather than to build, install, and perform the equivalent of find_package() to meet the dependency? Or rather, why should the former, rather than the latter, be the default?
Should I really have to write my project-level CMakeLists.txt to be compatible with being add_subdirectory()'ed?
Note: Just to give some concrete examples of how this use constrains proj_a:
Must use unique option names which can't possibly clash with super-project names. So no more WITH_TESTS, BUILD_STATIC_LIB - it has to be: WITH_PROJ_A_TESTS and BUILD_PROJ_A_STATIC_LIB.
You have to account for the parent project having searched for other dependencies already, and perhaps differently than how you would like to search for them.
Following the discussion in comments, I decided to post a bug report about this:
#22904: Support FetchContent_MakeAvailable performing build+install+find_package rather than add_subdirectory
So maybe this will change and the question becomes moot.
Why does it make sense to add_subdirectory(), rather than to build, install, and perform the equivalent of find_package() to meet the dependency? Or rather, why should the former, rather than the latter, be the default?
FetchContent doesn't just have to be for project() dependencies. It can be used for fetching utility scripts too. I'm guessing it was designed with that kind of consideration in mind. If your utility script is just one file, you can just file(DOWNLOAD) and add_subdirectory() directly, but the utilities could be multiple files, such as is the case with aminaya/project_options. FetchContent() uses a lot of the same machinery as ExternalProject, so it can do a lot of the useful things that ExternalProject does. For example, you can use FetchContent to fetch aminaya/project_options as a remote git repo, or as its archive artifacts- ex. v0.20.0.zip
Should I really have to write my project-level CMakeLists.txt to be compatible with being add_subdirectory()'ed?
It's your choice! The reasoning here can be highly objective, or subjective. It's up to you. Some people just like to put in a lot of effort to support whatever their users might want. Some people have a lot of historical configuration baggage and are still catching up to newer CMake. And as you mentioned at the end of your question post, there are certain adjustments that need to be made to accomodate for cleanly allowing people to add_subdirectory() you as a dependency. One example of a project which chose "no" is glew (see issue #314 for explanation).
Just to give another reference to some related work mentioned in responses to the KitWare/CMake ticket your raised, here's the ticket which tracked work on "FetchContent and find_package() integration".
Suppose I have a Go project which act as shared library in another Go project. How do I markup Go symbols (consts, structs, vars) that they are used outside this project?
I guess the underlying problem is that I have a hard time knowing which code uses said symbols.
Please note: This is not about semantic versioning, of which I am very well aware and which I use. I know Semver can help to identify breaking changes.
Instead, this is about finding out if I actually break one of my own projects (compared to: This symbol should be unexported or used outside the package). I am thinking of some sort of annotation which don't exist in Go.
As an aside, IntelliJ doesn't know either and marks those symbols as "Unnecessarily exported". Maybe an IntelliJ-centric solution could suffice.
To illustrate my problem:
package sharedlib
import "time"
// MyFavoriteTimeFormat is a blablabla...
const MyFavoriteTimeFormat = Time.RFC3339
package dependingproject
import "github.com/thething/sharedlib"
import "time"
func convertToString(timestamp time.Time) string {
return timestamp.Format(sharedlib.MyFavoriteTimeFormat)
}
When I happily rename MyFavoriteTimeFormate and release it, the code will break in the depending project when it updates the dependency.
Don't export anything until some other package needs it. If another package needs something, then do export, and then you'll know that if something is exported, it is because it is used outside of the package. And do not do breaking changes on exported identifiers. If you really must, then increment major version. Using go modules, that won't break existing other packages, they will continue to use the old version.
If your module is broken down into multiple packages (because it is "big"), and you wish to export something solely for the other packages of your module, then use the internal package concept, so it will still be "unexported" (unimportable) to other modules. For details, see Can I develop a go package in multiple source directories?
I'm working on a library with multiple layers of functionality. I want developers to be able to import only the parts they need, ie mylib-core, mylib-feature1, mylib-feature2, etc. Each lives in its own git repo. I'd also like to provide a simple mylib package which exposes a default set of functionality which is useful for developers new to the library. See d3.js version 4+ for something very similar to what I'm trying to accomplish.
The problems I've run into are
Apparently you can't share a package name between packages. This is a problem because it would be nice to import all the desired repos and then simply have everything available under the mylib name.
I don't see an obvious way to re-export functionality, in order to build the default mylib package.
Are there good solutions or a more idomatic go way to accomplish what I'm shooting for?
Answering your question, there is no idiomatic way of doing what you want. It is common in JavaScript to import a library and export its members without interference. This is not the case in Golang.
I advise you to host your whole library under a single repo, and split the functionality to packages. The Go compiler will only compile and use the imported packages.
And a tip for the future, Go is very different than almost any other language you previously know 😅
In Go, if you reference another package, e.g. something on GitHub, then Go always gets the latest version from the master branch. While this is great for development, I guess it's a problem in production: This way a build is not reproducible.
So, what is the correct way in Go to fix a version of a dependency, and how to handle this efficiently?
A friend pointed me to godep, and this seems fine, but I wanted to know what alternatives are there, and what's good / bad about godep?
Update 2018 with Go 1.11
Dependencies should now be referenced with modules (derived from the vgo project):
Go 1.11 adds preliminary support for a new concept called “modules,” an alternative to GOPATH with integrated support for versioning and package distribution.
Using modules, developers are no longer confined to working inside GOPATH, version dependency information is explicit yet lightweight, and builds are more reliable and reproducible.
See Defining a module. (and the original design proposal)
Update June 2015: first support for vendoring is making its way in Go 1.5!
See c/10923/:
When GO15VENDOREXPERIMENT=1 is in the environment, this CL changes the resolution of import paths according to the Go 1.5 vendor proposal:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Update March 2015: the go team is thinking about defining a go dependency management system integrated to the language: the debate is in this thread.
We think it’s time to start addressing the dependency & vendoring issue, especially before too many conflicting tools arise and fragment best practices in the Go ecosystem, unnecessarily complicating tooling. It would be nice if the community could converge on a standard way to vendor.
Our proposal is that the Go project,
officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.
defines a common config file format for dependencies & vendoring
makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.
One possible downside of godep is that you can no longer use "go build" or "go test" directly.
You need to precede those commands with godep (or type godep save).
An alternative is glide, which remains compatible with classic go commands.
Manage project-specific GOPATHs
Ease dependency management
Support versioning packages
Support aliasing packages (e.g. for working with github forks)
Remove the need for "vendoring" or munging import statements
Work with all of the go tools
More generally, the article "Know your guarantees, Go edition" is interesting:
It’s also a deliberate choice, where the Go authors chose not to implement a feature when they felt that the trade-offs were no good.
One low-level reason they made this choice is to avoid slow compilation and bloated binaries (which are two sides of the same coin).
Remember, packages depend on other packages. So Foo might depend on Bar 2.1. Foo might also depend on Baz which in turn depends on Bar 1.9, and on down the tree. So that would mean compiling and linking several copies of nearly identical code.
Depending on several versions of the same package also means knowing which version one is calling, whereby the dependency mess seeps into your source code.
Which leads us to the high-level reasoning behind the Go platform punting on this feature: they did not have a logical solution they considered acceptable. It’s not that they don’t understand the problem; it’s that, at the moment, there is not a solution they like. So they choose no feature over over a regressive one.
You handle dependencies like you do handle dependencies in other languages too: You vendor. For Go there is no Nexus which does the vendoring so most just copy external libraries into a "vendor" folder, there are tools helping here. Personally I found all this "fix version" panic a bit exaggerated as it works pretty well without.
You might wanna take a look at http://labix.org/gopkg.in and search golang-nuts for dependency management. I think there is even a whole mailing list devoted to this.
i would like to import a package rather than require bundle in a manifest and have all bundles that require the former package inherit the imported package. I am assuming that it is possible to set visibility: reexport, but Eclipse does not complain if i had this option...
It's not possible and it's not necessary either. A bundle using Import-Package can simply obtain the package from the original bundle that exports it; there is no need to "route" the dependency through an intermediate bundle.
This is one of the biggest advantages of Import-Package: an importing bundle neither knows nor cares which other bundle(s) it's getting the packages from.
You can only reexport when requiring bundles.
Import-Package should be preferred over Require-Bundle or DynamicImport-Package. The former is only really necessary if you must deal with split packages whereas the latter was only intended to address situations where you didn't know the class name in advance (e.g., SPI-like situations), although it can also be used safely as an optional import-like facility. In general, you should be avoiding things that hide dependencies (e.g., broad dynamic imports) or obscure them (e.g., requiring bundles). The fact that Require-Bundle supports reexporting is a decision that should have never been made and there is no reasonable use case for this feature, all it does it further obscure dependencies and create a tangled mess.
No, it's not possible to reexport a package. For that you will need to use a bundle, or a different class loading strategy.
I'm apparently missing the "reply to comment" link, so forgive me that I'm replying to the question in the comments here...
Whether you are in a "hierarchy" or not doesn't really make a difference. First bundles are used in different contexts, so they won't always be used in a container that knows that they want. Second, a bundle still doesn't use everything available to it in a "hierarchy", so acting it does just leads to dependency fan out, since the real dependencies are hidden.
I am not sure what your problem is. If you do not want to require a bundle, you have to import all packages you need. To generate this list you can use the maven-bundle-plugin (for maven projects). If you do not want to generate the concrete list of needed imports, than you can try
DynamicImport-Package: *
which should import the needed packages on demand.