Related
I always build with -i flag to install packages and .a files are installed in GOPATH/pkg directory.
GOCACHE directory shown by go env GOCACHE seems to store cache files as well.
What's difference between them?
And what I wanna know is both of them should be saved if I want to make build time faster?
TLDR; The cache folder is internal to the go tool and its working should be opaque to the user, and its purpose is to speed up builds and tests. For example if you use a version control system (such as git), switching between branches or versions, the GOPATH/pkg may only contain package files of one version. The go cache folder may contain (partially) compiled packages of multiple branches and versions, speeding up future builds when you switch between branches and versions.
The cache folder was introduced in Go 1.10:
The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. For more details, see go help cache.
So you don't need to use -i anymore for fast builds.
Some quotes from the output of go help cache:
The go command caches build outputs for reuse in future builds.
The default location for cache data is a subdirectory named go-build
in the standard user cache directory for the current operating system.
Setting the GOCACHE environment variable overrides this default,
and running 'go env GOCACHE' prints the current cache directory.
The go command periodically deletes cached data that has not been
used recently. Running 'go clean -cache' deletes all cached data.
The build cache correctly accounts for changes to Go source files,
compilers, compiler options, and so on: cleaning the cache explicitly
should not be necessary in typical use. However, the build cache
does not detect changes to C libraries imported with cgo.
If you have made changes to the C libraries on your system, you
will need to clean the cache explicitly or else use the -a build flag
(see 'go help build') to force rebuilding of packages that
depend on the updated C libraries.
The go command also caches successful package test results.
See 'go help test' for details. Running 'go clean -testcache' removes
all cached test results (but not cached build results).
The cache folder is also used to store test results, so in some circumstances, the cached results may be presented without running the tests again.
Your questions is self-answering:
$ ls $(go env GOCACHE)
$ cat $(go env GOCACHE)/README
and
$ ls $(go env GOPATH)/pkg
As you can see - there is nothing similar between them:
GOPATH/pkg - compiled packages that remains static between builds. Those files are not "cashe" files actually
GOCACHE - collections of build artefacts that constantly changes between builds
More elaborate answer could be done by examining sources of go build
I have been using npm for a personal project and just recently stumbled across yarn. Would there be any harm or "intended side effects" to switching to yarn's package manager in the same project where I had been using npm?
Although a few commenters here say its ok to mix both yarn and npm on the same project, after using yarn and npm and then yarn again, this is what yarn has to say about it:
warning package-lock.json found. Your project contains lock files generated by tools
other than Yarn. It is advised not to mix package managers in order to avoid resolution
inconsistencies caused by unsynchronized lock files. To clear this warning, remove
package-lock.json.
Since to me it is not any harm to using both them into one project.
I use npm and yarn (50/50) in dev environment.
But on ci/di i use only yarn because it is faster, and i reduce build minutes thanks yarn.
Also they both create different .lock file names.
Nobody told about the lock files.
Imagine you use yarn on dev environment, and yarn on your build/production servers. When you install a package using yarn, and your project works on your computer, you probably would want to keep it working on a production environment (your server).
That being sad, you would commit you yarn.lock file, that "saves" the exact versions of each package you have, when the project ran on your computer.
On your buid/production server you should call yarn install, but asking to keep all the same versions with --frozen-lockfile parameter. Some even say "yarn install --frozen-lockfile should be the default behavior", and I agree.
Then... another dev jump in the project you are working and install a package using npm (other than yarn). That new package will not be included in your yarn.lock file, but, a new package-json.lock file would be created, telling the exact packages versions it is using.
When that commit arrives on your build/production server, it will crash, fail, because that new package doesn't exist on yarn.lock file. Someone would need to pull that changes, call a yarn to install the dependences and update the lock file with the new package dependences, and push it again to the repo.
A quick point about using the lock file or not. If you call a 'yarn install' on your build/production server some weeks after the last install on your machine, the server would have many other new versions than your last "stable" version. It already happened to me many times.
I published recently the package-locks-checks, which help ensure you have not just one lock file but also locked each package version on your project.
There will be a point that one or both will no longer work and your project will be stuck at only using the existing lock file. Meaning, the issue probably will involve installation fails if you opt to reinstall without a lock file. And that also means failure to create a new lock file, so you are stuck with the existing one that you are trying to get rid off in the first place. We are actually encountering this issue in one of our projects. Because it is so big, no one tries to fix the issue and just rely on the existing lock file.
So, even if we say it's a rare case that it won't cause harm. Mixing npm and yarn should be avoided.
Here https://classic.yarnpkg.com/en/docs/migrating-from-npm/ we may find a confirmation that Yarn's resolution algorithm is compatible with NPM resolution algorithm.
Inside a npm project (with package.json) if you run yarn it will read your node_modules folder (using the resolution algorithm) and create a yarn.lock file with your project's locked dependency tree.
Based on that I assume that they are compatible inside the same project.
Update 30/04/2021
My original reply refers to yarn 1 (classic), although I've just created a React app with create-react-app tool and it creates the project's repository with package.json + yarn.lock by default. Again, another demonstration that it's fine (even with the warning mentioned by Dave Pile).
At the end of the day this is a matter of putting both together to work and checking yourself...
Plus you get a warning from yarn as Dave Pile said because we have to push *-lock.json files changes you have to consider using npm version >= 7 to make sure whenever you install packages by npm it will update your yarn-lock.json file too.
Because whenever you install the packages either by npm or yarn depends on what you have chosen for updating a dependency in the package.json (Using tilde ( ~ ) which gives you bug fix releases and caret ( ^ ) gives you backward-compatible new functionality) it will update you.lock file and since you have to push it might happen that you have different version of lock files.
Yarn creates a yarn.lock file after you perform a yarn install.
Should this be committed to the repository or ignored? What is it for?
Yes, you should check it in, see Migrating from npm
What is it for?
The npm client installs dependencies into the node_modules directory non-deterministically. This means that based on the order dependencies are installed, the structure of a node_modules directory could be different from one person to another. These differences can cause works on my machine bugs that take a long time to hunt down.
Yarn resolves these issues around versioning and non-determinism by using lock files and an install algorithm that is deterministic and reliable. These lock files lock the installed dependencies to a specific version and ensure that every install results in the exact same file structure in node_modules across all machines.
Depends on what your project is:
Is your project an application? Then: Yes
Is your project a library? If so: No
A more elaborate description of this can be found in this GitHub issue where one of the creators of Yarn eg. says:
The package.json describes the intended versions desired by the original author, while yarn.lock describes the last-known-good configuration for a given application.
Only the yarn.lock-file of the top level project will be used. So unless ones project will be used standalone and not be installed into another project, then there's no use in committing any yarn.lock-file – instead it will always be up to the package.json-file to convey what versions of dependencies the project expects then.
I see these are two separate questions in one. Let me answer both.
Should you commit the file into repo?
Yes. As mentioned in ckuijjer's answer it is recommended in Migration Guide to include this file into repo. Read on to understand why you need to do it.
What is yarn.lock?
It is a file that stores the exact dependency versions for your project together with checksums for each package. This is yarn's way to provide consistency for your dependencies.
To understand why this file is needed you first need to understand what was the problem behind original NPM's package.json. When you install the package, NPM will store the range of allowed revisions of a dependency instead of a specific revision (semver). NPM will try to fetch update the dependency latest version of dependency within the specified range (i.e. non-breaking patch updates). There are two problems with this approach.
Dependency authors might release patch version updates while in fact introducing a breaking change that will affect your project.
Two developers running npm install at different times may get the different set of dependencies. Which may cause a bug to be not reproducible on two exactly same environments. This will might cause build stability issues for CI servers for example.
Yarn on the other hand takes the route of maximum predictability. It creates yarn.lock file to save the exact dependency versions. Having that file in place yarn will use versions stored in yarn.lock instead of resolving versions from package.json. This strategy guarantees that none of the issues described above happen.
yarn.lock is similar to npm-shrinkwrap.json that can be created by npm shrinkwrap command. Check this answer explaining the differences between these two files.
You should:
add it to the repository and commit it
use yarn install --frozen-lockfile and NOT yarn install as a default both locally and on CI build servers.
(I opened a ticket on yarn's issue tracker to make a case to make frozen-lockfile default behavior, see #4147).
Beware to NOT set the frozen-lockfile flag in the .yarnrc file as that would prevent you from being able to sync the package.json and yarn.lock file. See the related yarn issue on github
yarn install may mutate your yarn.lock unexpectedly, making yarn claims of repeatable builds null and void. You should only use yarn install to initialize a yarn.lock and to update it.
Also, esp. in larger teams, you may have a lot of noise around changes in the yarn lock only because a developer was setting up their local project.
For further information, read upon my answer about npm's package-lock.json as that applies here as well.
This was also recently made clear in the docs for yarn install:
yarn install
Install all the dependencies listed within package.json
in the local node_modules folder.
The yarn.lock file is utilized as follows:
If yarn.lock is present and is enough to satisfy all the dependencies
listed in package.json, the exact versions recorded in yarn.lock are
installed, and yarn.lock will be unchanged. Yarn will not check for
newer versions.
If yarn.lock is absent, or is not enough to satisfy
all the dependencies listed in package.json (for example, if you
manually add a dependency to package.json), Yarn looks for the newest
versions available that satisfy the constraints in package.json. The
results are written to yarn.lock.
If you want to ensure yarn.lock is not updated, use --frozen-lockfile.
From My experience I would say yes we should commit yarn.lock file. It will ensure that, when other people use your project they will get the same dependencies as your project expected.
From the Doc
When you run either yarn or yarn add , Yarn will generate a yarn.lock file within the root directory of your package. You don’t need to read or understand this file - just check it into source control. When other people start using Yarn instead of npm, the yarn.lock file will ensure that they get precisely the same dependencies as you have.
One argue could be, that we can achieve it by replacing ^ with --. Yes we can, but in general, we have seen that majority of npm packages comes with ^ notation, and we have to change notation manually for ensuring static dependency version.But if you use yarn.lock it will programatically ensure your correct version.
Also as Eric Elliott said here
Don’t .gitignore yarn.lock. It is there to ensure deterministic dependency resolution to avoid “works on my machine” bugs.
Not to play the devil's advocate, but I have slowly (over the years) come around to the idea that you should NOT commit the lock files.
I know every bit of documentation they have says that you should. But what good can it possibly do?! And the downsides far outweigh the benefits, in my opinion.
Basically, I have spent countless hours debugging issues that have eventually been solved by deleting lock files. For example, the lock files can contain information about which package registry to use, and in an enterprise environment where different users access different registries, it's a recipe for disaster.
Additionally, the lock files can really mess up your dependency tree. Because yarn and npm create a complex tree and keep external modules of different versions in different places (e.g. in the node_modules folder within a module in the top node_modules folder of your app), if you update dependencies frequently, it can create a real mess. Again, I have spent tons of time trying to figure out what an old version of a module was still being used in a dependency wherein the module version had been updated, only to find that deleting the lock file and the node_modules folder solved all the hard-to-diagnose problems.
I even have shell aliases now that delete the lock files (and sometimes node_modules folders as well!) before running yarn or npm.
Just the other side of the coin, I guess, but blindly following this dogma can cost you........
I'd guess yes, since Yarn versions its own yarn.lock file:
https://github.com/yarnpkg/yarn
It's used for deterministic package dependency resolution.
Yes! yarn.lock must be checked in so any developer who installs the dependencies get the exact same output! With npm [that was available in Oct 2016], for instance, you can have a patch version (say 1.2.0) installed locally while a new developer running a fresh install might get a different version (1.2.1).
Yes, You should commit it. For more about yarn.lock file, refer the official docs here
I would like to use a headless OpenOffice instance (to interface it with PHP) on my webhost's server. Of course I can't install stuff there with the usual package mechanism, since I am just an ordinary user. However, I would be allowed to run (more or less) whatever I want, and I have ssh access.
Is there a way to install LibreOffice/OpenOffice in some directory under my home directory, and also get all dependencies there? It would be advantageous if I didn't have to run the actual build on the webhost, since it would clog the server... But maybe I could manage to build it on some other FreeBSD host... I would want the build to be as static as possible, i.e. with as few outside dependencies as possible.
Can this be done?
For the record, the webhost is running FreeBSD 8.2-RELEASE-p3 on amd64.
Hmm, you may try to install ports-mgmt/pbi-manager port (it's basically a bunch of shellscripts) and use pbi_add command to install libreoffice PBI.
PBI packages contain all necessary dependencies and, AFAIK, can be installed without root permissions.
Goal
I am trying to create a port (Macports) for an open source tool based on Eclipse which doesn't need installation, in other words, it's just "extract and use" case. Users can download the tool from the official project site and use just like that. So there is no DESTROOT variable set.
Since many Mac users got used to the convenience of Macports, I would like to add the tool there, so that users could instantly install or uninstall the tool.
** Important notice: once users start the tool, it creates "/workspace" directory in the same place the tool was installed to keep users' preferences, settings, and other necessary files. So, when users starts the tool, the program should have access to write in the same directory it was installed. The current version of the tool doesn't provide a way to choose the workspace location.
Problem
How should I organize the Portfile?
I have set the following configurations where I tell Macports to not use configure, build, and destroot phases.
set cm_workspace /workspace
universal_variant no
use_configure no
supported_archs noarch
post-extract {
file mkdir ${worksrcpath}${cm_workspace}
destroot.keepdirs-append ${worksrcpath}${cm_workspace}
}
build {}
destroot {}
As I understand,
extract phase untars the file,
and install phases should archive those files,
and finally activate phase should move the files to the destroot.
But I keep getting errors.
---> Extracting cubridmanager
---> Configuring cubridmanager
---> Building cubridmanager
---> Staging cubridmanager into destroot
Error: No files have been installed in the destroot directory!
Error: Please make sure that this software supports 'make install DESTDIR=${destroot}' or implement an alternative destroot mechanism in the Portfile.
Error: Files might have been installed directly into your system, check before proceeding.
Error: Target org.macports.destroot returned: Staging cubridmanager into destroot failed
Log for cubridmanager is at: /opt/local/var/macports/logs/_Users_nbp_macports_databases_cubridmanager/cubridmanager/main.log
Error: Status 1 encountered during processing.
To report a bug, see <http://guide.macports.org/#project.tickets>
I want to contribute to that open source community, but I can't pass this step.
You misunderstood the phases, the usual workflow is as follows:
extract untars the downloaded file
patch applies any local patches
configure runs ./configure
build runs make
destroot runs make install DESTDIR=${destroot}
install packs the file in the destroot area into an archive
activate moves the files into ${prefix}
So, in your case, you don't need steps 2, 3 and 4. But you still need to copy the files to the destroot area in step 5, the destroot phase. Otherwise MacPorts does not know which files it is supposed to install.
supported_archs noarch
use_configure no
build {}
destroot {
copy ${worksrcpath} ${destroot}${prefix}/some/path
}
Note that MacPorts does discourage installing files outside the prefix directory, as the installation is meant to be self-contained. The path /workspace sounds like a pretty bad idea. Rather, you should use a path inside the users home directory to save any data as otherwise this cannot be used on a computer with multiple user accounts. Of course, the actual executable files can reside in the MacPorts prefix.
Normally, UNIX software separates binaries, libraries and shared data in /usr (or in the MacPorts case,/opt/local) from user-specific data in the home directory. If your tool does not follow this convention, this needs to be fixed by the developers first.
I don't think that tool fits with macports for related reasons
All files from macports should be in one of the supported directories i.e. destroot and ending up in /opt/local
The project tries to write to sub directories which is not good here
The directories written to bu macports can only be written to by the user macports so as to minimize the ability to affect the build and run environment.
In a multiuser system who owns the directory to write to? e.g. macports are installed as user macports and are run as someone else - Also if there are more than one normal user who writes to the directory?
I think you need to patch the tool so that it is passed a directory to create the workspace in when a normal user runs it but the tool is install as ownwd by macports in /opt/local/bin