Why do individual module versions sometimes need 2 lines in go.sum?
one line is just for the module version (v0.1.1 in the example below)
one line also has /go.mod tacked onto the version (v0.1.1/go/mod in the example below).
For example:
github.com/foo/bar v0.1.1 h1:kDgnGXZpvZUi7ym6Rm23yVn3gRqBag+vU6M/wytZR9c=
github.com/foo/bar v0.1.1/go.mod h1:MZcarCLffCxoj/EF1yhRb4HvOSmCkm5Z8FPmzWrMG+g=
The reason I ask is because sometimes when I go get a package, an indirect dependency will be generated in go.sum with only the second line from the example above, and then the build will fail with 410 gone for that package#version. However if I manually go get the indirect dependency, the build no longer fails with 410 gone.
I believe this only happens with private repositories, so I understand it will not play well with sum.golang.org. However, I'd like to figure out if it's possible to avoid getting the 410 in the first place, especially with regards to automated module updates, etc.
The v0.1.1/go.mod entry contains the checksum for the go.mod file in isolation. That is needed to ensure consistency any time you are loading or changing dependencies.
The v0.1.1 entry (without the /go.mod suffix) contains the checksum for the full source code of the module, including all of the .go source files for the packages within it.
The two parts are downloaded separately so that you don't need to download the full source code for dependencies that you don't intend to build or test (a fairly common situation for projects with casual contributors). But because the go command downloads them separately, it needs separate checksums for them.
Related
I'm trying to understand how the source code for third-party dependencies is or is not compiled into my Go binary. I'm building in a Docker container, so I can see precisely what's fetched for my build without interference from other builds.
After my go build completes I see source code files for several dependencies under go/pkg/mod/$module#$version directories. The Module cache documentation tells me that these directories contain "extracted contents of a module .zip file. This serves as a module root directory for a downloaded module." My best guess is that the presence of extracted source code for these dependencies indicates that "yes, these dependencies are definitely compiled into your binary."
I also see many more dependencies pulled into go/pkg/mod/cache/download/$module directories. The Module cache documentation tells me that this directory contains "files downloaded from module proxies and files derived from version control systems," which I don't fully understand. As far as I can see, these files do not include any extracted source code, though there are several .zip files that I assume contain the source. For the most part these files seem to be .mod files that just contain text representing some sort of dependency graph.
My question is: if a third-party dependency has module files under go/pkg/mod/cache/download but no source code under go/pkg/mod/$module#$version, does that mean that dependency's code was NOT compiled into my Go binary?
I don't understand why the Go build pulls in all these module files but only has extracted source code for some of the third-party modules. Perhaps Go preemptively parses and pulls module information for the full transitive set of modules referenced from the modules my first-party code imports, but perhaps many of those modules don't end up being needed for my binary's compile + build process and therefore don't get extracted. If that's not true and the answer to my question is no, then I don't understand how or why my binary can link in those dependencies without go build fetching their source code.
As mentioned in "Compile and install packages and dependencies"
Compiled packages are cached automatically.
GOPATH and Module includes:
When using modules, GOPATH is no longer used for resolving imports.
However, it is still used to store downloaded source code (in GOPATH/pkg/mod) and compiled commands (in GOPATH/bin).
So if you see sources in pkg/mod which are not in pkg/mod/cache, try a go mod tidy
add missing and remove unused modules
From there, you should have the same modules between sources (pkg/mod) and compiled modules (pkg/mod/cache)
Based on the OP's comment
I need to know exactly what's included in the binary for compliance reasons.
I would recommend a completely different approach: dumping the list of symbols contained in the binary and then correlating it with whatever information is needed.
The command
go tool nm -type /path/to/the/executable/image/file
would dump the symbols — names of the functions — whose code was taken from both the standard library packages, 3rd-party and/or vendored packages and internal packages, compiled and linked into the binary, and print to its standard output stream a sequence of lines
address type name
which you can then process programmatically.
Another approach you might employ is to use go list with various flags to query the program's source code about the packages and/or modules which will be used when building: whatever that command outputs describing the full dependency graph of the source code is whatever go build will use when building — provided the source code is not changed between these calls.
Yet another possibility is to build the program using go build -x, save the debug trace it produces on its standard error stream and parsing it for exact module names the command reported as used during building.
I am rewriting in Go a home-grade program of mine that was previously in Python, learning on the way. As I reconsider whole sections of the code, it would be great if I could ignore some .go files I wrote (they throw compilation errors which is OK as they are not synchronized with my latest approach to the problem).
I could move them out of the directory, or rename them but I was wondering whether there would be a more idiomatic way to ignore some .go files in a module directory?
Build tags fit perfectly to your use case.
A Build Constraint or also known as Build Tag is used to include or exclude files in a package during a build process. With this, we can build different types of builds from the same source code.
So if you want to exclude a file from the build process, give it a different Build Tag. For example if you use //go:build exclude at the top of the file when you perform go build it won't be included. Note that exclude can be any word.
For more details on the Build Tags check out the following article by Dave Cheney here
TLDR
Add
//go:build exclude for go v1.17 and above
// +build exclude
at the top of the file you wish to ignore.
I am beginner in go and facing difficulty in understanding go/pkg folder.As suggested by documentation it contains pkg/mod and pkg/windowsamd_64. pkg/windowsamd_64 for storing compiled files. What happens if I have a file importing some external github modules and do go build on that.
Will it go first to pkg/mod (but modules are compiled in
pkg/windowsamd_64) to search for external modules
Will it go first to pkg/windowsamd_64 (then what will be use of
pkg/mod) to search for modules
Will it go to {gopath}/src and do something from there
pkg/mod is just a folder ,why do we call it cache as it will keep on
filling or better when does it populate?
The go command has two different modes of locating packages: module mode (introduced in Go 1.11) and GOPATH mode (much older). Module mode is the default as of Go 1.16, and if you are new to Go you will probably want to work exclusively in that mode. (There isn't much point to working in GOPATH mode unless you have a large legacy codebase that requires it.)
pkg/mod stores cached source code for use in module mode. The source code for a given version of a module is downloaded automatically when you build a package from that module (for example, as a dependency of some other package).
GOPATH/src stores source code for use in GOPATH mode. You can also choose to work in that directory in module mode, but that's a completely optional/aesthetic choice and shouldn't change the behavior of anything in module mode.
pkg/windows_amd64 stores installed packages in GOPATH mode. However, installed packages aren't very useful anyway because Go has a separate build cache (even in GOPATH mode). So you can mostly ignore pkg/windows_amd64 completely.
coming from a java and grails background, (and having written millions of lines of C 30 years ago), I cant see how go can be usable with a fixed gopath on windows.
installing go creates this structure
c:\users\me\go\scr
\pkg
\bin
As you will want to have many go projects it seems they have to be mixed together in the same src/kpg/bin dirs, polluting each other. e.g.
/src/project1/hello.go
/project2/hello.go
/pkg/xx
/bin/hello.exe
Which hello.go will hello.exe run?
Unless I am missing something fundamental, this seems crazy - all completely separate projects are expected to share the same package and bin dirs. It means you dont know which versions of which packages, and which exe files belong to which project, and there is presumably plenty of scope for conflicts. I would have expected a /src, /pkg and /bin for each separate go app (like java or grails or even C), and go path is completely redundant, it can be relative to the current project root (like in grails).
To make matters works, for work, we have to use a different directry, e.g.
d:\work\project3
\project4
\package5
\go_utility6
\go_utility7
So now we have a total of 6 separate directories where go progams live. It is not feasible to change the path every time you switch to working on a different project. I guess the other option is to add the 6 paths to the GOPATH. Presumably, all 7 go projects write to the same pkg and bin dir, which is going to be a disaster.
Is there a tenable solution to this situation (on windows at least)?
If we need to add a PATH to GOPATH for every project, what should the file structure be under each project?
E.g. uner xxx\go_utility6, which is a stand alone command line go app, what should the structure be? does there need to be a src dir in there somewhere? does this dir need gopath to point to it? does it need its own pkg, or should it use the c:\users\me\pkg dir?
UPDATE: When I posted this Go did not have modules support and we built and used a tool called vg. These days the recommended way to go is to use go modules.
I use vg for that, it takes care of keeping separate GOPATH paths per project and it switches automatically when you cd a project.
Your example "which hello.exe" should be used honestly makes not much sense. Two tools with the same name?
Even if both are, let's say, an api, your devops will be happier with more meaningful names.
The bin folder is used for 3rd party tools you install, you so not have to install your project binaries. Except they are tools, but then the name should be meaningful again.
You can get more information about the project structure here: https://golang.org/doc/code.html
Since go 1.8 supports a vendor folder below project folders, it is possible to break the original structure. (imho vendors were not maintainable before 1.8, yes that was crazy)
You might want to use a tool like direnv, which would support your desire to change GOPATH per project.
https://github.com/direnv/direnv
It also has some built in function for adding the current path to the GOPATH.
https://github.com/direnv/direnv/blob/master/stdlib.sh#L355:1
For example GoLang also supports handling multiple GOPATHs and per project GOPATHs. So direnv should also work properly.
In my company we have one go folder right next to our other projects.
Under go/src are our projects. No problem so far, since vendors are in the projects' vendor folders and committed.
The so far best dependency manager I would recommend for go is:
https://github.com/golang/dep
I hope that input helps.
With Go 1.11 Go Modules were introduced. You can use Go Modules to have Go projects outside the GOPATH directory.
Here is an example of how to configure a project using GoModules.
So, I am setting up a new site and my project's folder structure looks like this now.
foo.com/
index.php
assets/
css/
img/
js/
vendor/
I have added vendor/ for js/css libraries that I must install to keep them separate, since I want anyone who installs my project to install those in vendor from package.json - most libraries contain too many files 99% which I don't want to push to production.
Now once the project is finished, I would like to push the code to production with only the necessary js/css files.
This is where the problem comes. For example, if I install bulma css using:
yarn add bulma --modules-folder ./assets/vendor
It will dump all bulma-related files which are almost 70 into /vendor/bulma/ but I will only be needing one or two css files afterwards, since I will compiles the sass file to css as:
sass vendors/bulmna/style.scss assets/css/style.css
So my questions is: I am assuming this is how every developer does it and there are no documentations I can find that suggest how to do it. Is it safe to ignore the /vendor directory? What if I install vue, font-awesome, bootstrap .. how can I only fetch the files I need but not everything in /vendors folder?
Your question is actually quite broad - however, I'll try to list as much as possible.
Lets say you're building a project from scratch and needed to include vuejs, jquery, fontawesome but only need to include a couple of files.
The issue you're hitting here is module dependency with respect to npm modules. (and there are many different tools that you can use to manage versions on your library dependencies as well as ensuring they are included into your project, i.e. package managers).
Ok - now from here, you may say to yourself
but I only need say, one icon from fontawesome in your final build (dist) and I don't want to commit all my modules into source control
Again, this is where you omit node_modules and other dependent libraries from source control (i.e. include node_modules your .gitignore)
To reiterate
You can install the required library,
add node_modules to .gitignore ,
bundle those libraries into a vendor single file to be consumed by your users (can be via browserify/webpack/rollup/gulp/grunt/yarn etc).
generate bundle within npm script
Now you may ask further -
Why would I use any of those tools? - they're distracting me from simply copy/pasting vendor libaries into my source control.
Build tools were created to
streamline the developer pipeline so that you DONT have to copy/paste vendor libaries into a vendor folder.
ensures that all files are bundled to the client automatically
allows you to keep track/restrict library version updates/ when required via package.json
allows you to add other build steps (such as minification, md5hash versioning, compression, code splitting, asset management to name a few).
Now lets break down the original question here:
How to ensure other developers get everything they need when cloning the repository
how do I ensure I can provide only the necessary files to the end user (if I only use specific parts of vendor libaries?)
1. How to ensure developers get what they need
Again, to reiterate above, adding devDependancies and .gitignoring allows you to only add the necessary files to your project.
2. How can I ensure clients get what they need without bloating request files?
This is where build tools such as webpack, browserify, gulp, grunt, rollup, attempt to achieve. This is because with some libraries that exceed in file size of 200kb minified, you may need to separate these files into different client requests (and as such, not demand the user to request one large file, which was symtomatic of browserify projects).
The second technique you will need to be aware of, is with specific libraries, you can use import mdn link where you can require one function/class from a dependant library (which further reduces file size).
Another technique is using less import statements (which can extract less functions/styles similar to above, but this isn't currently supported in SCSS). for SCSS, you're basically left with copy/pasting the necessary styles into your base scss and that'll save you space as well.
EDIT
How to create a bundle from npm install libaries
From the comments you've mentioned above (about not wanting to include a tool into your workflow, there's no point outlining any one particular strategy - you can find answers/tutorials online about how to setup gulp/browserify/webpack for your particular needs).
However, As you are currently using yarn - I'll go into details about that.
Firstly, yarn is a package manager (like npm). All it does with the --modules-folder is install the package into the specified folder, that's all. So we don't really care about that (since it's doing the same thing as npm). (i.e. your vendor folder is the same as node_modules in many respects).
We could use
webpack
gulp
grunt
rollup
browserify
brunch
(All build tools that essentially allow you to bundle all those packages into a single entry point to be packaged to the client).
I won't go into how, because that is a process you can find online, and from the above comments, I can tell you don't particularly care either.
What you are looking for is a zero-config javascript build tool. (Extremely out of the scope of your original question, and i'll only answer that in a separate Q&A).
I'd try Googling on "tree shaking CSS" to see if that gives you something.
Maybe something like: https://github.com/jacobp100/es-css-modules
Rollup plugin may be useful. It works for js, with postcss, the link says it works with css also.
https://code.lengstorf.com/learn-rollup-css
Have a look at Pancake. It has been built specifically for the purpose of moving only those files out of the node_modules folder that you need. I wrote an article about it here: https://medium.com/dailyjs/npm-and-the-front-end-950c79fc22ce
(probably not an answer but a good tip)
PS: I am the author of both, the article and the tool so with clear bias :)