What is the recommended way to include all the files for your project?
I have seen many examples that use a similar structure: an ordered list of include statements in a single top-level file (the file that defines a Module, or the "main" file in an application).
This also seemed to be the conclusion in the following thread:
https://discourse.julialang.org/t/best-practices-for-structuring-larger-projects/2652?u=nhdaly
However, Lint.jl seems to be very unhappy with this sort of project structure, complaining about the persistent use of undeclared symbols.
Consider the following example:
# --- src/lib1.jl ---
struct MyStruct
x::Int32
end
# --- src/lib2.jl ---
foo() = MyStruct(3) # Lint.jl says: E321 MyStruct: use of undeclared symbol
# --- src/MyPackage.jl ---
module MyPackage
include("lib1.jl")
include("lib2.jl")
foo()
end
Of course, this code will work correctly, because MyStruct will be available before lib2.jl is compiled. But Lint.jl cannot know that. Even when using the Atom-integrated linter, it shows the same error.
So is it preferable to have lib2.jl include("lib1.jl") instead of all the includes in the top file? Or is it best to put them in both places, as you might for C++ headers? I haven't seen a definitive recommendation for this anywhere, and I'd love if someone could provide one! :)
EDIT: changed file names to reflect a more typical julia directory structure.
I usually do a:
PkgDev.generate("MyPackage", "MIT")
This generates the whole folder structure. I put my Julia(.jl) files in the src folder and use 'using' to import functions in other Julia files in the same folder.
Related
I successfully used rules_go to build a gRPC service:
go_proto_library(
name = "processor_go_proto",
compilers = ["#io_bazel_rules_go//proto:go_grpc"],
importpath = "/path/to/proto/package",
proto = ":processor_proto",
deps = ["//services/shared/proto/common:common_go_proto"],
)
However, I'm not sure how to import the resulting file in VSCode. The generated file is nested under bazel_bin and under the original proto file path; so to import this, it seems like I would need to write out the entire path (including the bazel_bin part) to the generated Go file. To my understanding, there doesn't seem to be a way to instruct VSCode to look under certain folders that only contain Go packages/files; everything seems to need a go.mod file. This makes it quite difficult to develop in.
For clarity, my directory structure looks something like this:
WORKSPACE
bazel-bin
- path
- to
- generated_Go_file.go
go.mod
go.sum
proto
- path
- to
- gRPC_proto.proto
main.go
main.go should use the generated_Go_file.go.
Is there a way around this?
I don't use Bazel and so cannot help with the Bazel configuration. It's likely there is a way to specify the generated code location so that you can revise this to reflect you preference.
The outline you provide of the generated code, is workable though and a common pattern. Often the generated proto|gRPC code is placed in a module's gen subdirectory.
This is somewhat similar to vendoring where your code incorporates what may often be a 3rd-party's stubs (client|server) into your code. The stubs must reflect the proto(s) package(s) and, when these are 3rd-party, using gen or bazel-bin provide a way to keep potentially multiple namespaces discrete.
You're correct that the import for main.go, could (!) be prefixed with the module name from go.mod (first line) followed by the folder path to the generated code. This is standard go packaging and treats the generated code in a similar way to vendored modules.
Another approach is to use|place the generated code in a different module.
For code generated from 3rd-party protos, this may be preferable and the generated code may be provided by the 3rd-party in a module that you can go get or add to your go.mod.
An example of this approach is Google Well-Known Types. The proto (sources) are bundled with protoc (lib directory) and, when protoc compiles sources that references any of these, the Go code that is generated includes imports that reference a Google-hosted location of the generated code (!) for these types (google.golang.org/protobuf/types/known).
Alternatively, you can replicate this behavior without having to use an external repo. The bazel-bin folder must be outside of the current module. Each distinct module in bazel-bin, would need its own go.mod file. You would include in a require block in your code's go.mod file references to the modules' (one or more) locations. You don't need to publish the module to a external repo but can simply require ( name => path/to/module ) to provide a local reference.
Every Go file starts with package <something>.
As far as I understand - and this is probably where I am missing some information - there are only two possible values for <something>: The name of the directory it is in*, or main. If it is main, all other files in that directory can only have main, too. If it is something else, the project is inconsistent/violating convention.
Now if it is the name of the directory, it's redundant, because the same information is, well, in the name of the directory.
If it is main, it's kind of useless, because as far as I can see there is no way to tell go build to "please build all main packages".
* Because, in other words, one directory is one package.
The name of the package does not have to coincide with the directory name. It is possible to have package foobar in the directory xyz/go-foobar. In this case, xyz/go-foobar becomes an import path, but the package name that you use to quality the identifiers (functions, types etc.) would be foobar.
Here's an example to make it more concrete: I created a test package http://godoc.org/github.com/dmitris/go-foobar (source in https://github.com/dmitris/go-foobar) - you can see from the documentation page, that the import path is "github.com/dmitris/go-foobar" but the package name is foobar, so you would call the function it provides as foobar.Demo() (not go-foobar.Demo()).
A similar real-life example - the import path for the NSQ Messaging platform is "github.com/nsqio/go-nsq" while the package name is "nsq": http://godoc.org/github.com/nsqio/go-nsq. However, for the sake of user-friendliness and simplicity, the standard and recommended practice is to keep the last portions of the import path and the package name being the same whenever possible.
package main is not useless - it tells the Go compiler to create an executable as opposed to a .a library file (with go install or go get; go build discards the compilation result). The executable is named after the directory name in which the package main file or files are placed. Again a concrete example - I made a test program https://github.com/dmitris/go-foobar-client, you install it with go get github.com/dmitris/go-foobar-client and you should get a go-foobar-client executable placed in your $GOPATH/bin directory. It is from the the directory name where the package main file is placed that the Go compiler takes the name of the executable from. The filename of the .go file that contains the main() function is not important - in the example above, we can rename main.go to client.go or something else, but as long as the enclosing directory is called go-foobar-client, that's how the resulting executable will be named.
For an additional accessible and practically oriented reading about Go packages, I recommend Dave Cheney's article "Five suggestions for setting up a Go project" http://dave.cheney.net/2014/12/01/five-suggestions-for-setting-up-a-go-project.
The missing information you "have" is that the package name does not need to be the same as the directory name.
It is perfectly fine to use a package name other than the folder name. If you do so, you still have to import the package based on the directory structure, but after the import you have to refer to it by the name you used in the package clause.
For example, if you have a folder $GOPATH/src/mypck, and in it you have a file a.go:
package apple
const Pi = 3.14
Using this package:
package main
import (
"mypck"
"fmt"
)
func main() {
fmt.Println(apple.Pi)
}
Just like you are allowed to use relative imports but is not advisable, you may use package names other that their containing folder, but this is not advisable also to avoid further misunderstanding.
Note that the specification doesn't even require all files belonging to the same package to be in the same folder (but it may be an implementation requirement). Spec: Package clause:
A set of files sharing the same PackageName form the implementation of a package. An implementation may require that all source files for a package inhabit the same directory.
What's the use of this?
Simple. A package name is a Go identifier:
identifier = letter { letter | unicode_digit } .
Which allows unicode letters to be used in identifiers, e.g. αβ is a valid identifier in Go. Folder and file names are not handled by Go but by the Operating System, and different file systems have different restrictions. There are actually many file systems which would not allow all valid Go identifiers as folder names, so you would not be able to name your packages what otherwise the language spec would allow.
So in one hand not all valid Go identifiers may be valid folder names. And on the other hand, not all valid folder names are valid Go identifiers, for example go-math is a valid folder name in most (all?) file systems, but it's not a valid Go identifier (as identifiers cannot contain the dash - character).
Having the option to use package names different than their containing folders, you have the option to really name your packages what the language spec allows, regardless of the underlying operating and file system, and put it in a folder named anything that the underlying OS and file system allows - regardless of the package name.
Source file from Golang's stdlib
File's base directory: ast
Package specified in the file: ast_test ???
Package specified in all other files inside the same directory: ast
From golang.org:
src contains Go source files organized into packages (one package per directory) ...
By convention, packages are given lower case, single-word names; there should be no need for underscores or mixedCaps
... Another convention is that the package name is the base name of its source directory
How is it possible to have multiple packages (here 2) in one folder?
You find another example in src/pkg/go/ast/commentmap_test.go, with the comment:
// To avoid a cyclic dependency with go/parser, this file is in a separate package.
I suppose it allows for an othogonal command like:
go test
That will test parser features while avoiding for that test to be part of the same parser features (since it has been put in a separate package)
From go command man page:
Test files that declare a package with the suffix "_test" will be compiled as a separate package, and then linked and run with the main test binary.
This thread asked the question:
Now that the go tool requires each directory to be one package and doesn't allow to have files with different package names inside the same folder, how is the package keyword useful? It seems like a unnecessary repetition.
Is it required by the compiler or is there any plan to remove it?
The answers hinted at the fact that you can have more than one package in a folder:
The package declaration declares the name of the package.
The language Go doesn't know what a file or a directory is and the import path itself doesn't effect the actual name of the package that is being imported. So the only way the compiler knows what to call the package is the package declaration.
The language doesn't require separate packages to be in separate directories; it is a requirement of the go tool.
Another hypothetical implementation may not have this requirement.
Even this go tool requirement can be bypassed thanks to the "// +build" build tags.
For example, read misc/cgo/gmp or misc/cgo/stdio (some files include // +build ignore)
This repo has 3 go files all begin with "package lumber".
To use this package, I'm supposed to put this in my GOROOT and simply
import lumber
in my program. How do variables and types in this package connect with each other across multiple files? How does the go compiler know which file to begin reading first?
In case I want to read the package, where should I begin reading to understand the package? What exactly is the flow of things here?
To elaborate on jnml's answer:
When you use import "foo/bar" in your code, you are not referring to the source files (which will be located in $GOPATH/src/foo/bar/).
Instead, you are referring to a compiled package file at $GOPATH/pkg/$GOOS_$GOARCH/foo/bar.a. When you build your own code, and the compiler finds that the foo/bar package has not yet been compiled (or is out of date), it will do this for you automatically.
It does this by collating* all the relevant source files in the $GOPATH/src/foo/bar directory and building them into a single bar.a file, which it installs in the pkg directory. Compilation then resumes with your own program.
This process is repeated for all imported packages, and packages imported by those as well, all the way down the dependency chain.
*) How the files are collated, depends on how the file itself is named and what kind of build tags are present inside it.
For a deeper understanding of how this works, refer to the build docs.
No, you're not "supposed to put this in my GOROOT". You're supposed to execute
$ go get github.com/jcelliott/lumber
which will clone the repository into $GOPATH/src/github.com/jcelliott/lumber. Then you can use the package by importing it in your code as
import "github.com/jcelliott/lumber"
About the scoping rules: Declarations and scope
I recently decided to organize the files in my project directory. I moved the parsers I had for a few different file types into their own directory and also decided to use ocamlbuild (the as the project was getting more complicated and the simple shell script was not sufficient any longer).
I was able to successfully include external projects by modifying myocamlbuild with some basic rules (calling ocaml_lib, I'll use ocamlfind some other time), but I am stuck on how to include the folder as a module into the project properly. I created a parser.mlpack file and filled it with the proper modules to be included (eg, "parser/Date", et cetera), wrote a parser.mli in the root of the directory for their implementations, and modified the _tags file (see below).
During the compilation, the parser directory is traversed properly, and parser.cmi, parser.mli.depends were both created in the _build directory; as well as all *.cm[xio] files in the parsers subdirectory.
I feel I might be doing something redundant, but regardless, the project still cannot find the Parser module when I compile!
Thanks!
_tags
debug : true
<*.ml> : annot
"parser" : include
<parser/*.cmx>: for-pack(Parser)
<curlIO.*> : use_curl
<mySQL.*> : use_mysql
<**/*.native> or <**/*.byte> : use_str,use_unix,use_curl,use_mysql
compilation error
/usr/local/bin/ocamlopt.opt unix.cmxa str.cmxa -g -I /usr/local/lib/ocaml/site-lib/mysql mysql.cmxa -I /usr/local/lib/ocaml/curl curl.cmxa curlIO.cmx utilities.cmx date.cmx fraction.cmx logger.cmx mySQL.cmx data.cmx project.cmx -o project.native
File "\_none\_", line 1, characters 0-1:
Error: **No implementations provided for the following modules:**
Parser referenced from project.cmx
Command exited with code 2.
You'll notice -I parser is not included in the linking phase above; actually none of the parser related files are included!
edit: Added new details from comments and answer below.
You need to "include" the parser directory in the search path. You can do this in _tags:
"parser": include
Then ocamlbuild can search the parser directory for interesting files.
I wonder if parser.mli is somehow interfering with the dependencies in processing the mlpack file. parser.cmi will be generated from the pack operation when parser.mlpack is processed and compiled. Try building with the parser.mli file removed. If that works, then this can be re-processed into a real answer.
Also, you don't need parser/ as a prefix to your modules in parser.mlpack if parser.mlpack is in the parser directory and you have the include tag set. But that shouldn't make a difference for this.
Update: this worked around the problem, but wasn't the root cause. Root cause, per comment below, was a file mentioned in the .mlpack that had been relocated.