So, I am setting up a new site and my project's folder structure looks like this now.
foo.com/
index.php
assets/
css/
img/
js/
vendor/
I have added vendor/ for js/css libraries that I must install to keep them separate, since I want anyone who installs my project to install those in vendor from package.json - most libraries contain too many files 99% which I don't want to push to production.
Now once the project is finished, I would like to push the code to production with only the necessary js/css files.
This is where the problem comes. For example, if I install bulma css using:
yarn add bulma --modules-folder ./assets/vendor
It will dump all bulma-related files which are almost 70 into /vendor/bulma/ but I will only be needing one or two css files afterwards, since I will compiles the sass file to css as:
sass vendors/bulmna/style.scss assets/css/style.css
So my questions is: I am assuming this is how every developer does it and there are no documentations I can find that suggest how to do it. Is it safe to ignore the /vendor directory? What if I install vue, font-awesome, bootstrap .. how can I only fetch the files I need but not everything in /vendors folder?
Your question is actually quite broad - however, I'll try to list as much as possible.
Lets say you're building a project from scratch and needed to include vuejs, jquery, fontawesome but only need to include a couple of files.
The issue you're hitting here is module dependency with respect to npm modules. (and there are many different tools that you can use to manage versions on your library dependencies as well as ensuring they are included into your project, i.e. package managers).
Ok - now from here, you may say to yourself
but I only need say, one icon from fontawesome in your final build (dist) and I don't want to commit all my modules into source control
Again, this is where you omit node_modules and other dependent libraries from source control (i.e. include node_modules your .gitignore)
To reiterate
You can install the required library,
add node_modules to .gitignore ,
bundle those libraries into a vendor single file to be consumed by your users (can be via browserify/webpack/rollup/gulp/grunt/yarn etc).
generate bundle within npm script
Now you may ask further -
Why would I use any of those tools? - they're distracting me from simply copy/pasting vendor libaries into my source control.
Build tools were created to
streamline the developer pipeline so that you DONT have to copy/paste vendor libaries into a vendor folder.
ensures that all files are bundled to the client automatically
allows you to keep track/restrict library version updates/ when required via package.json
allows you to add other build steps (such as minification, md5hash versioning, compression, code splitting, asset management to name a few).
Now lets break down the original question here:
How to ensure other developers get everything they need when cloning the repository
how do I ensure I can provide only the necessary files to the end user (if I only use specific parts of vendor libaries?)
1. How to ensure developers get what they need
Again, to reiterate above, adding devDependancies and .gitignoring allows you to only add the necessary files to your project.
2. How can I ensure clients get what they need without bloating request files?
This is where build tools such as webpack, browserify, gulp, grunt, rollup, attempt to achieve. This is because with some libraries that exceed in file size of 200kb minified, you may need to separate these files into different client requests (and as such, not demand the user to request one large file, which was symtomatic of browserify projects).
The second technique you will need to be aware of, is with specific libraries, you can use import mdn link where you can require one function/class from a dependant library (which further reduces file size).
Another technique is using less import statements (which can extract less functions/styles similar to above, but this isn't currently supported in SCSS). for SCSS, you're basically left with copy/pasting the necessary styles into your base scss and that'll save you space as well.
EDIT
How to create a bundle from npm install libaries
From the comments you've mentioned above (about not wanting to include a tool into your workflow, there's no point outlining any one particular strategy - you can find answers/tutorials online about how to setup gulp/browserify/webpack for your particular needs).
However, As you are currently using yarn - I'll go into details about that.
Firstly, yarn is a package manager (like npm). All it does with the --modules-folder is install the package into the specified folder, that's all. So we don't really care about that (since it's doing the same thing as npm). (i.e. your vendor folder is the same as node_modules in many respects).
We could use
webpack
gulp
grunt
rollup
browserify
brunch
(All build tools that essentially allow you to bundle all those packages into a single entry point to be packaged to the client).
I won't go into how, because that is a process you can find online, and from the above comments, I can tell you don't particularly care either.
What you are looking for is a zero-config javascript build tool. (Extremely out of the scope of your original question, and i'll only answer that in a separate Q&A).
I'd try Googling on "tree shaking CSS" to see if that gives you something.
Maybe something like: https://github.com/jacobp100/es-css-modules
Rollup plugin may be useful. It works for js, with postcss, the link says it works with css also.
https://code.lengstorf.com/learn-rollup-css
Have a look at Pancake. It has been built specifically for the purpose of moving only those files out of the node_modules folder that you need. I wrote an article about it here: https://medium.com/dailyjs/npm-and-the-front-end-950c79fc22ce
(probably not an answer but a good tip)
PS: I am the author of both, the article and the tool so with clear bias :)
Related
I am using 'animate.css' as a npm package. to use it I specify import 'animate.css' during development. But in production it says to include 'animate.min.css'. How to specify to parceljs to pick the animate.min.css during production build.
or would import animate.css would be able to resolve to right animate.css in development build and animate.min.css in production build. how does the resolver know to pick right file as the main file is specified as animate.css in the package.json
But the there is "Files" property in package.json ...not sure if it can help.
The high-level answer is that parcel automatically minifies and optimizes css files, so you don't need to do anything special to make sure that your production build is as small as possible.
If you want to understand in detail how parcel handles this particular package, read on.
Parcel generally follows node resolution conventions to figure out which files to actually import when you reference a package name.
In this case, the import "animate.css" will tell parcel to go looking in the nearest node_modules folder for a for a subfolder called animate.css. It finds it, and then looks for the package.json file at the root. In this case, the package.json has a main field that points to a file called animate.css (e.g. the un-minified version).
Parcel will use that file as the basis for development and production builds. When you run parcel in development, it leaves this file untouched. When you run parcel build it will process this (unminified) file through cssnano so the build output is minified.
So the interesting fact is that even though there is another file called node_modules/animate/animate.min.css living there, Parcel is able to achieve the desired behavior without touching it. The publishers of animate.css included it for other folks that aren't using a bundler that's as awesome as parcel - you can safely ignore it.
There is this new shining Yarn feature called Plug'n'Play.
I would like to know what it does exactly?
I know it's creating a .pnp folder and a .pnp.js file, but does it change anything else on the machine, like a config file somewhere?
Thank you.
I designed and implemented PnP, so I can talk hours about it 🙂
tl;dr: We only write the .pnp.js and .pnp folders (on top of the regular Yarn cache). We don't store configuration anywhere else.
Without Plug'n'Play
When you run yarn install (even without PnP), a few things happen:
If you use the offline mirror feature, we download the tarballs from the registry and store them within the offline mirror folder
Regardless of whether or not you use the offline mirror, we unpack all the tarballs downloaded and store their files in the Yarn cache
We then figure out which files from the cache should be copied into which location in the node_modules
We apply the computed changes (a bunch of rsync operations, basically)
With Plug'n'Play
With PnP, the workflow becomes like this:
No changes, we download the tarballs from the registry in the offline mirror (if enabled)
No changes, we still unpack them into the Yarn cache
We generate a .pnp.js fileÂą
And that's it. There is no other generated file than the .pnp.js file (and the cache, but it already was there before).
Âą As you mentioned, we also generate a .pnp folder (.yarn as of Yarn 2) in the project. This folder is meant to contain two types of data:
Unplugged packages are packages that must be local to the project. Typically, those are the packages with postinstall scripts (we cannot store them into the cache, as the generated artifacts might be different from a project to another).
Virtual packages, which are symlinks created for each package in your dependency tree that lists peer dependencies. Without going into the details, they are a necessary part of the design, and are required to make require.resolve work as before. Those files don't exist anymore as of Yarn 2 🎉
How does it work?
The .pnp.js file contains information similar to the following:
webpack#1.0.0 -> /cache/webpack-1.0.0/
-> it depends on lodash#1.0.0
lodash#1.0.0 -> /cache/lodash-1.0.0/
-> no dependencies
By having those information, the resolution can correctly infer that when a file within /cache/webpack-1.0.0 makes a require call to lodash, then the required files must be loaded from /cache/lodash-1.0.0. It's a bit more complex in practice (we keep an inverse map for improved perfs, we use relative paths to ensure portability, etc), but the general concept is there.
Bonus round: With Plug'n'Play+Zip loading (Yarn 2)
Bonus: With Yarn 2, we're about to improve this workflow even more. This is what it will look like:
We download the tarballs from the registry and store then into the cache (no more distinction between offline mirror and cache - they are the same)
We generate the same .pnp.js file as before
And that's it! As you can see we don't unpack the packages anymore (instead, we use a Node loader to read them from the package archives at runtime).
Doing this has a very interesting property: if both your cache and .pnp.js files are there, you don't need to run yarn install for your application to work! And to ensure you have those files, you just have to add them to your repository and version them as you would with everything else.²
It's very useful, as you don't need to remember to run yarn install after git rebase, git pull, or git checkout, and your CI systems become faster and stabler as they don't need special setup - just clone your application and it'll just work.
² Before someone mentions it - checking-in binary files within a repository is perfectly fine. The reason why node_modules were a very bad thing to check-in within your repository was because of the exponential number of text files, which was putting a huge strain on Git - technically, but also philosophically as code reviews were made impossible.
In the case I described we don't suffer from the same problem, because the number of files is constrained (exactly one file for each package), and reviewing them is very easy - in fact, it's better in that you can clearly see how many new packages are added to your project by a PR!
It imports only the parts of a package you are going to use, making the bloated node_modules folder much, much leaner.
Think about for example having relative big libraries like lodash or ramda when you use only 4-5 functions from them - how much you could save getting only the actually used minimum.
I believe it is not yet 100% fully stable, but still a nice option to keep on your radar :)
coming from a java and grails background, (and having written millions of lines of C 30 years ago), I cant see how go can be usable with a fixed gopath on windows.
installing go creates this structure
c:\users\me\go\scr
\pkg
\bin
As you will want to have many go projects it seems they have to be mixed together in the same src/kpg/bin dirs, polluting each other. e.g.
/src/project1/hello.go
/project2/hello.go
/pkg/xx
/bin/hello.exe
Which hello.go will hello.exe run?
Unless I am missing something fundamental, this seems crazy - all completely separate projects are expected to share the same package and bin dirs. It means you dont know which versions of which packages, and which exe files belong to which project, and there is presumably plenty of scope for conflicts. I would have expected a /src, /pkg and /bin for each separate go app (like java or grails or even C), and go path is completely redundant, it can be relative to the current project root (like in grails).
To make matters works, for work, we have to use a different directry, e.g.
d:\work\project3
\project4
\package5
\go_utility6
\go_utility7
So now we have a total of 6 separate directories where go progams live. It is not feasible to change the path every time you switch to working on a different project. I guess the other option is to add the 6 paths to the GOPATH. Presumably, all 7 go projects write to the same pkg and bin dir, which is going to be a disaster.
Is there a tenable solution to this situation (on windows at least)?
If we need to add a PATH to GOPATH for every project, what should the file structure be under each project?
E.g. uner xxx\go_utility6, which is a stand alone command line go app, what should the structure be? does there need to be a src dir in there somewhere? does this dir need gopath to point to it? does it need its own pkg, or should it use the c:\users\me\pkg dir?
UPDATE: When I posted this Go did not have modules support and we built and used a tool called vg. These days the recommended way to go is to use go modules.
I use vg for that, it takes care of keeping separate GOPATH paths per project and it switches automatically when you cd a project.
Your example "which hello.exe" should be used honestly makes not much sense. Two tools with the same name?
Even if both are, let's say, an api, your devops will be happier with more meaningful names.
The bin folder is used for 3rd party tools you install, you so not have to install your project binaries. Except they are tools, but then the name should be meaningful again.
You can get more information about the project structure here: https://golang.org/doc/code.html
Since go 1.8 supports a vendor folder below project folders, it is possible to break the original structure. (imho vendors were not maintainable before 1.8, yes that was crazy)
You might want to use a tool like direnv, which would support your desire to change GOPATH per project.
https://github.com/direnv/direnv
It also has some built in function for adding the current path to the GOPATH.
https://github.com/direnv/direnv/blob/master/stdlib.sh#L355:1
For example GoLang also supports handling multiple GOPATHs and per project GOPATHs. So direnv should also work properly.
In my company we have one go folder right next to our other projects.
Under go/src are our projects. No problem so far, since vendors are in the projects' vendor folders and committed.
The so far best dependency manager I would recommend for go is:
https://github.com/golang/dep
I hope that input helps.
With Go 1.11 Go Modules were introduced. You can use Go Modules to have Go projects outside the GOPATH directory.
Here is an example of how to configure a project using GoModules.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have a go project that is starting to become more complex, and want to lay the filesystem out in such a way to reduce pain.
Are there some good examples out there of what makes sense?
Update May 2013: the official documentation is in the section "Code organization"
Go code must be kept inside a workspace.
A workspace is a directory hierarchy with three directories at its root:
src contains Go source files organized into packages (one package per directory),
pkg contains package objects, and
bin contains executable commands.
The go tool builds source packages and installs the resulting binaries to the pkg and bin directories.
The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages.
bin/
streak # command executable
todo # command executable
pkg/
linux_amd64/
code.google.com/p/goauth2/
oauth.a # package object
github.com/nf/todo/
task.a # package object
src/
code.google.com/p/goauth2/
.hg/ # mercurial repository metadata
oauth/
oauth.go # package source
oauth_test.go # test source
Update July 2014: see "Structuring Applications in Go" from Ben Johnson
That article include tips like:
Separate your binary from your application
combining the main.go file and my application logic in the same package has two consequences:
It makes my application unusable as a library.
I can only have one application binary.
The best way I’ve found to fix this is to simply use a “cmd” directory in my project where each of its subdirectories is an application binary.
camlistore/
cmd/
camget/
main.go
cammount/
main.go
camput/
main.go
camtool/
main.go
Library driven development
Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library.
Sometimes you might want users to interact in multiple ways so you create multiple binaries.
For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version.
You can easily do this by organizing your project like this:
adder/
adder.go
cmd/
adder/
main.go
adder-server/
main.go
Users can install your “adder” application binaries with “go get” using an ellipsis:
$ go get github.com/benbjohnson/adder/...
And voila, your user has “adder” and “adder-server” installed!
Don’t go crazy with subpackages
Usually my project’s types are all very related so it fits better from a usability and API standpoint.
These types can also take advantage of calling unexported between them which keeps the API small and clear.
Group related types and code together in each file. If your types and functions are well organized then I find that files tend to be between 200 and 500 SLOC. This might sound like a lot but I find it easy to navigate. 1000 SLOC is usually my upper limit for a single file.
Organize the most important type at the top of the file and add types in decreasing importance towards the bottom of the file.
Once your application starts getting above 10,000 SLOC you should seriously evaluate whether it can be broken into smaller projects.
Note: that last practice isn't always good:
Sorry I just cant agree with this practice.
Separating type to files helps code management, readability, maintenancability, testability.
It may also ensure single responsibility and the follow of open/closed principle…
The rule for not allowing circular dependency is to force we have a clear structure of the packages.
(Alternative February 2013, regarding src only)
You can find the classic layout illustrated in "GitHub Code Layout":
The app and both libraries live on Github, each in its own repository.
$GOPATH is the root of the project - each of your Github repos will be checked out several folders below $GOPATH.
Your code layout would look like this:
$GOPATH/
src/
github.com/
jmcvetta/
useless/
.git/
useless.go
useless_test.go
README.md
uselessd/
.git/
uselessd.go
uselessd_test.go
README.md
Each folder under src/github.com/jmcvetta/ is the root of a separate git checkout.
That attracted some criticisms though, in this reddit page:
I highly recommend not structuring the repo the way you have, it'll break "go get", which is one of the most useful things about Go.
It's far better to write your code for people who do know Go, since they're most likely to be the ones compiling it.
And for people who don't, they'll at least get a feel for the language.
Put the main package in the root of the repo.
Have the assets in a subdirectory (to keep things neat).
Keep the meat of the code in a subpackage (in case anyone wants to reuse it outside your binary).
Include a setup script in the root of the repo so it's easy to find.
It's still only a two step process to download, build, install, and setup.:
"go get <your repo path>": downloads and installs the go code, with a subdir for the assets
$GOPATH/<your repo path>/setup.sh: distributes the assets to the right place and installs the service
I assume that with 'project' you don't mean a Go package but a software you develop.
Otherwise you can get help here and here.
However it's not so much different to writing packages for Go: Use packages, create a folder for each package and combine those packages in your application.
To build yourself an opinion, you can look at trending Go repositories on github: https://github.com/trending/go. Notable examples are cayley
and zeus.
The most popular scheme is probably to have a main Go file and many modules and submodules in their own directories. In case you have many meta files (doc, license, templates, ...) you may want to put the source code into a subdirectory. That's what I did so far.
There is a recommended approach from the authors of Golang that defines how to layout your code to work best with the go tools and to support source control systems
You should probably also have a look into this repo. It shows a lot of ideas how to structure go applications: https://github.com/golang-standards/project-layout
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have a go project that is starting to become more complex, and want to lay the filesystem out in such a way to reduce pain.
Are there some good examples out there of what makes sense?
Update May 2013: the official documentation is in the section "Code organization"
Go code must be kept inside a workspace.
A workspace is a directory hierarchy with three directories at its root:
src contains Go source files organized into packages (one package per directory),
pkg contains package objects, and
bin contains executable commands.
The go tool builds source packages and installs the resulting binaries to the pkg and bin directories.
The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages.
bin/
streak # command executable
todo # command executable
pkg/
linux_amd64/
code.google.com/p/goauth2/
oauth.a # package object
github.com/nf/todo/
task.a # package object
src/
code.google.com/p/goauth2/
.hg/ # mercurial repository metadata
oauth/
oauth.go # package source
oauth_test.go # test source
Update July 2014: see "Structuring Applications in Go" from Ben Johnson
That article include tips like:
Separate your binary from your application
combining the main.go file and my application logic in the same package has two consequences:
It makes my application unusable as a library.
I can only have one application binary.
The best way I’ve found to fix this is to simply use a “cmd” directory in my project where each of its subdirectories is an application binary.
camlistore/
cmd/
camget/
main.go
cammount/
main.go
camput/
main.go
camtool/
main.go
Library driven development
Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library.
Sometimes you might want users to interact in multiple ways so you create multiple binaries.
For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version.
You can easily do this by organizing your project like this:
adder/
adder.go
cmd/
adder/
main.go
adder-server/
main.go
Users can install your “adder” application binaries with “go get” using an ellipsis:
$ go get github.com/benbjohnson/adder/...
And voila, your user has “adder” and “adder-server” installed!
Don’t go crazy with subpackages
Usually my project’s types are all very related so it fits better from a usability and API standpoint.
These types can also take advantage of calling unexported between them which keeps the API small and clear.
Group related types and code together in each file. If your types and functions are well organized then I find that files tend to be between 200 and 500 SLOC. This might sound like a lot but I find it easy to navigate. 1000 SLOC is usually my upper limit for a single file.
Organize the most important type at the top of the file and add types in decreasing importance towards the bottom of the file.
Once your application starts getting above 10,000 SLOC you should seriously evaluate whether it can be broken into smaller projects.
Note: that last practice isn't always good:
Sorry I just cant agree with this practice.
Separating type to files helps code management, readability, maintenancability, testability.
It may also ensure single responsibility and the follow of open/closed principle…
The rule for not allowing circular dependency is to force we have a clear structure of the packages.
(Alternative February 2013, regarding src only)
You can find the classic layout illustrated in "GitHub Code Layout":
The app and both libraries live on Github, each in its own repository.
$GOPATH is the root of the project - each of your Github repos will be checked out several folders below $GOPATH.
Your code layout would look like this:
$GOPATH/
src/
github.com/
jmcvetta/
useless/
.git/
useless.go
useless_test.go
README.md
uselessd/
.git/
uselessd.go
uselessd_test.go
README.md
Each folder under src/github.com/jmcvetta/ is the root of a separate git checkout.
That attracted some criticisms though, in this reddit page:
I highly recommend not structuring the repo the way you have, it'll break "go get", which is one of the most useful things about Go.
It's far better to write your code for people who do know Go, since they're most likely to be the ones compiling it.
And for people who don't, they'll at least get a feel for the language.
Put the main package in the root of the repo.
Have the assets in a subdirectory (to keep things neat).
Keep the meat of the code in a subpackage (in case anyone wants to reuse it outside your binary).
Include a setup script in the root of the repo so it's easy to find.
It's still only a two step process to download, build, install, and setup.:
"go get <your repo path>": downloads and installs the go code, with a subdir for the assets
$GOPATH/<your repo path>/setup.sh: distributes the assets to the right place and installs the service
I assume that with 'project' you don't mean a Go package but a software you develop.
Otherwise you can get help here and here.
However it's not so much different to writing packages for Go: Use packages, create a folder for each package and combine those packages in your application.
To build yourself an opinion, you can look at trending Go repositories on github: https://github.com/trending/go. Notable examples are cayley
and zeus.
The most popular scheme is probably to have a main Go file and many modules and submodules in their own directories. In case you have many meta files (doc, license, templates, ...) you may want to put the source code into a subdirectory. That's what I did so far.
There is a recommended approach from the authors of Golang that defines how to layout your code to work best with the go tools and to support source control systems
You should probably also have a look into this repo. It shows a lot of ideas how to structure go applications: https://github.com/golang-standards/project-layout