For a while we deployed our local packages to a specific folder packages.stableand added that as a path repo. For testing purposes each developer has a copy of that machine and has an additional similar folder packages.mine were personal versions are stored (e.g. a new feature). Depending on the developer the folder packages.mine had newer or older versions compared to packages.stable, as well as some packages missing. As it was a local repo, we had the version attributein all composer.json files.
This allowed to composer req my/package1:*#dev and composer took the best alternative, either stable if my packages were old or missing, or mine if they were newer. This way no developer needed to care about all packages.
Now we wanted to move the stable folder packages to multiple VCS. Suddenly the overwrite is not working anymore, as composer up always ignores the local path repo packages.mine and takes "newest commit" from the VCS.
Is there a way to achieve the former scenario if stable is a VCS? Or is that some unpredicted behaviour bound to local path repos which only worked by chance?
This is indeed no problem at all if one uses a different prerelease suffix. -dev is a special reserved suffix and dev-masteralways points to some commit which is then set to -dev (dev-master might point to a branch, but still). That is actually the reason why -dev123 is no valid composer version, also it is a valid semver. So avoiding-dev means ignoring the latest commit dev-master.
If the "local" code use an reserved suffix which is not -devit works, e.g. -alpha. This allows the local code to have numbered suffixes like -alpha123as well. Adding a composer req my/package1:*#alpha creates the desired state where local path repo takes precedence over any VCS version.
See https://getcomposer.org/doc/04-schema.md#minimum-stability
Related
Our CI pipeline publishes wheels for branches that have a version of
<base version>-dev<timestamp>+<branch name>.p<pipeline id>
So if I am working on cool-stuff on in the xyzzy branch, it might upload a wheel for version 1.2.3.dev202211221111+xyzzy.p1234
Somebody else, working in the foobar branch, might cause 1.2.3.dev202211221115+foobar.p1235 to be created.
How can I get pip to install the latest version from the xyzzy branch? I tried pip install cool-stuff>1.2.3.dev*+xyzzy but it complained that it could not find a matching version (even though the available versions that it listed included a +xyzzy tag.)
pip install cool-stuff==1.2.3.dev202211221111+xyzzy.p1234 did work, but I would prefer not have have to update the time stamp and pipeline number each time. I am hoping to put cool-stuff >= <magic> in my config file and just run pip install -e . whenever I need new dependencies.
What format do I need to use here?
As far as I know, it is not possible. I can not think of a workable solution that would be based on the version string only. The "local" part of the version string (in other words the part after the plus sign +) can not be used to differentiate between two releases as you intend to.
If I were in your situation, I think I would investigate a solution where the CI/CD pipelines generate distributions with a name customized according to the git branch. For example in your case the pipelines should generate wheels for Library-foobar or Library-xyzzy, depending on what branch is currently being worked on (while still keeping the same top-level import names of course). This assumes that you can customize your pipelines and processes deeply enough to support such a workflow.
I've just come across the https://github.com/FriendsOfPHP/security-advisories tool which looks a great way to automatically scan for the vulnerabilities that are in that community-contributed database.
It scans a composer.lock file for packages with vulnerabilities. However, it's made me realise that my understanding of Composer is not what it should be!
I have a project that has a composer.json file that requires a single demo/package. That demo package also has requirements, like demo/dep.
The result of running composer install --no-dev is that I have a composer.lock file which includes:
demo/package version 1.0
demo/dep version 1.2
All good so far, and running symfony security:check /path/to/my/project/composer.lock gives me a green light, no vulnerabilities.
However on close inspection of the files now in my vendor dir, I can see there's a vendor/demo/package/composer.lock file, which contains references to demo/dep at version 1.1 - which has a security vulnerability against it.
As I understand, I have the safer 1.2 version installed - so says my project's composer.lock file, but why is a composer.lock file included with the vendor's package?
Does that mean that the dodgy code is installed somewhere, too? Or can I just simply ignore the composer.lock files if there's a composer.lock file in a dir above it or such? composer show does not list the versions in the nested lock file. Or maybe I should ignore composer.lock files if there's no sibling ./vendor/ dir?
Why not simply inspect your folders to find a vulnerable version? If there was any, you should find a vendor folder within that package, that's where that package could have installed stuff from it's own composer.lock
Usually, only the composer.json of a package is evaluated to install dependencies. If there is a lock file within one package's folder, you should ask the maintainer of that package why this is the case, but for installing dependencies on your system, this does not matter.
Side note: writing "usually" refers to the standard model of installations. I've seen some crude stuff where Composer plugins put other rules in place, but this cannot be said for your project without knowing more about the structure.
Yarn creates a yarn.lock file after you perform a yarn install.
Should this be committed to the repository or ignored? What is it for?
Yes, you should check it in, see Migrating from npm
What is it for?
The npm client installs dependencies into the node_modules directory non-deterministically. This means that based on the order dependencies are installed, the structure of a node_modules directory could be different from one person to another. These differences can cause works on my machine bugs that take a long time to hunt down.
Yarn resolves these issues around versioning and non-determinism by using lock files and an install algorithm that is deterministic and reliable. These lock files lock the installed dependencies to a specific version and ensure that every install results in the exact same file structure in node_modules across all machines.
Depends on what your project is:
Is your project an application? Then: Yes
Is your project a library? If so: No
A more elaborate description of this can be found in this GitHub issue where one of the creators of Yarn eg. says:
The package.json describes the intended versions desired by the original author, while yarn.lock describes the last-known-good configuration for a given application.
Only the yarn.lock-file of the top level project will be used. So unless ones project will be used standalone and not be installed into another project, then there's no use in committing any yarn.lock-file – instead it will always be up to the package.json-file to convey what versions of dependencies the project expects then.
I see these are two separate questions in one. Let me answer both.
Should you commit the file into repo?
Yes. As mentioned in ckuijjer's answer it is recommended in Migration Guide to include this file into repo. Read on to understand why you need to do it.
What is yarn.lock?
It is a file that stores the exact dependency versions for your project together with checksums for each package. This is yarn's way to provide consistency for your dependencies.
To understand why this file is needed you first need to understand what was the problem behind original NPM's package.json. When you install the package, NPM will store the range of allowed revisions of a dependency instead of a specific revision (semver). NPM will try to fetch update the dependency latest version of dependency within the specified range (i.e. non-breaking patch updates). There are two problems with this approach.
Dependency authors might release patch version updates while in fact introducing a breaking change that will affect your project.
Two developers running npm install at different times may get the different set of dependencies. Which may cause a bug to be not reproducible on two exactly same environments. This will might cause build stability issues for CI servers for example.
Yarn on the other hand takes the route of maximum predictability. It creates yarn.lock file to save the exact dependency versions. Having that file in place yarn will use versions stored in yarn.lock instead of resolving versions from package.json. This strategy guarantees that none of the issues described above happen.
yarn.lock is similar to npm-shrinkwrap.json that can be created by npm shrinkwrap command. Check this answer explaining the differences between these two files.
You should:
add it to the repository and commit it
use yarn install --frozen-lockfile and NOT yarn install as a default both locally and on CI build servers.
(I opened a ticket on yarn's issue tracker to make a case to make frozen-lockfile default behavior, see #4147).
Beware to NOT set the frozen-lockfile flag in the .yarnrc file as that would prevent you from being able to sync the package.json and yarn.lock file. See the related yarn issue on github
yarn install may mutate your yarn.lock unexpectedly, making yarn claims of repeatable builds null and void. You should only use yarn install to initialize a yarn.lock and to update it.
Also, esp. in larger teams, you may have a lot of noise around changes in the yarn lock only because a developer was setting up their local project.
For further information, read upon my answer about npm's package-lock.json as that applies here as well.
This was also recently made clear in the docs for yarn install:
yarn install
Install all the dependencies listed within package.json
in the local node_modules folder.
The yarn.lock file is utilized as follows:
If yarn.lock is present and is enough to satisfy all the dependencies
listed in package.json, the exact versions recorded in yarn.lock are
installed, and yarn.lock will be unchanged. Yarn will not check for
newer versions.
If yarn.lock is absent, or is not enough to satisfy
all the dependencies listed in package.json (for example, if you
manually add a dependency to package.json), Yarn looks for the newest
versions available that satisfy the constraints in package.json. The
results are written to yarn.lock.
If you want to ensure yarn.lock is not updated, use --frozen-lockfile.
From My experience I would say yes we should commit yarn.lock file. It will ensure that, when other people use your project they will get the same dependencies as your project expected.
From the Doc
When you run either yarn or yarn add , Yarn will generate a yarn.lock file within the root directory of your package. You don’t need to read or understand this file - just check it into source control. When other people start using Yarn instead of npm, the yarn.lock file will ensure that they get precisely the same dependencies as you have.
One argue could be, that we can achieve it by replacing ^ with --. Yes we can, but in general, we have seen that majority of npm packages comes with ^ notation, and we have to change notation manually for ensuring static dependency version.But if you use yarn.lock it will programatically ensure your correct version.
Also as Eric Elliott said here
Don’t .gitignore yarn.lock. It is there to ensure deterministic dependency resolution to avoid “works on my machine” bugs.
Not to play the devil's advocate, but I have slowly (over the years) come around to the idea that you should NOT commit the lock files.
I know every bit of documentation they have says that you should. But what good can it possibly do?! And the downsides far outweigh the benefits, in my opinion.
Basically, I have spent countless hours debugging issues that have eventually been solved by deleting lock files. For example, the lock files can contain information about which package registry to use, and in an enterprise environment where different users access different registries, it's a recipe for disaster.
Additionally, the lock files can really mess up your dependency tree. Because yarn and npm create a complex tree and keep external modules of different versions in different places (e.g. in the node_modules folder within a module in the top node_modules folder of your app), if you update dependencies frequently, it can create a real mess. Again, I have spent tons of time trying to figure out what an old version of a module was still being used in a dependency wherein the module version had been updated, only to find that deleting the lock file and the node_modules folder solved all the hard-to-diagnose problems.
I even have shell aliases now that delete the lock files (and sometimes node_modules folders as well!) before running yarn or npm.
Just the other side of the coin, I guess, but blindly following this dogma can cost you........
I'd guess yes, since Yarn versions its own yarn.lock file:
https://github.com/yarnpkg/yarn
It's used for deterministic package dependency resolution.
Yes! yarn.lock must be checked in so any developer who installs the dependencies get the exact same output! With npm [that was available in Oct 2016], for instance, you can have a patch version (say 1.2.0) installed locally while a new developer running a fresh install might get a different version (1.2.1).
Yes, You should commit it. For more about yarn.lock file, refer the official docs here
First I have read this answer: Vendoring in Go 1.6, then I use it as my example.
My gopath is GOPATH="/Users/thinkerou/xyz/", and the follow like:
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ pwd
/Users/baidu/xyz/src/ou
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ ls
main.go vendor
Now, I use go get, then becomes this:
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ ls
main.go vendor
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ cd vendor/
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou/vendor$ ls
vendor.json
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou/vendor$ cd ../..
thinkerou#MacBook-Pro-thinkerou:~/xyz/src$ ls
github.com ou
thinkerou#MacBook-Pro-thinkerou:~/xyz/src$ cd github.com/
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/github.com$ ls
zenazn
vendor.json is this:
{
"comment": "",
"package": [
{
"path": "github.com/zenazn/goji"
}
]
}
then, I should use what commands? why have no use vendor? My go version is 1.6.2.
With Go1.6, vendoring is built in as you read. What does this mean? Only one thing to keep in mind:
When using the go tools such as go build or go run, they first check to see if the dependencies are located in ./vendor/. If so, use it. If not, revert to the $GOPATH/src/ directory.
The actual "lookup paths" in Go 1.6 are, in order:
./vendor/github.com/zenazn/goji
$GOPATH/src/github.com/zenazn/goji
$GOROOT/src/github.com/zenazn/goji
With that said, go get will continue to install into you $GOPATH/src; and, go install will install into $GOPATH/bin for binaries or $GOPATH/pkg for package caching.
So, how do I use ./vendor?!?!
Hehe, armed with the knowledge above, it's pretty simple:
mkdir -p $GOPATH/src/ou/vendor/github.com/zenazn/goji
cp -r $GOPATH/src/github.com/zenazn/goji/ $GOPATH/src/ou/vendor/github.com/zenazn/goji
In short, to use vendoring, you copy the files using the same github.com/zenazn/goji full path, into your vendor director.
Now, the go build/install/run tooling will see and use your vendor folder.
An easier way instead of copying everything manually
Instead of finding and copying all 25+ vendor items, managing their versions, updating other projects etc... It would be better to use a dependency management tool. There are many out there and a little googling will point to you several.
Let me mention two that works with the vendor folder and doesn't fight you:
godep
govendor
In short, these tools will inspect your ou code, find the remote dependencies, and copy them from your $GOPATH/src to your $GOPATH/src/ou/vendor directory (actually, whatever current directory you are in when you run them).
For example, say you have all of your dependencies installed and working normally in your $GOPATH/src/ou/ project using the normal GOPATH/src/github installation of your dependencies. Your project runs and your tests validate everything is working with the exact version of the repos you have. With Godep as an example, you'd run this from your project root folder $GOPATH/src/ou/:
godep save ./...
This would copy all dependencies your project uses into your ./vendor folder.
Godep is by far and large the most popular. They have their own Slack channel on the Gopher Slack group. And, it's the one I use on my teams.
Govendor is another alternative I read has a nice sync feature. I haven't used it though.
Over Usage of Dependency Management Tool
This is purely opinion, and I'm sure haters will downvote... But as I need to finish my blog post on the subject, let me mention here that most people worry too much about depdency management in Go.
Yes, there is a need to lock in a repo to a version you depend on so you can ensure your system builds in production. Yes there is a need to ensure no breaking changes to a way a dependency is interrupting something.
Use dependency management for those, absolutely.
But, there is overuse of simple projects that lock in huge amounts of dependencies when in reality...
You may only need to lock in only 1 dependencies; otherwise, you want the latest version of MySQL drivers and test assertion frameworks for bug fixes.
This is where using the ./vendor/ folder apart from dependency managrment tools can really shine: you'd only need to copy that repo that need you lock in.
You selectively pick the one misbehaving repo and put it into your ./vendor/ folder. By doing this, you are telling your consumers:
Hey, this one repo needs to be held back at this revision. All others are fine and use the latest of those and update often with go get -u ./...; but, this one failed with newer versions so don't upgrade this one repo.
But if blanketly saving all your dependencies with a dependency management tool, you are basically telling your consumers:
There may or may not be a problem with one or more repos out of the 20 in the vendor folder. You may or may not be able to update them. You may or may not be able to get the latest MySQL driver. We simply don't know which may or may not be causing problems and just locked in something that worked at the time that I ran godep save. So yeah, upgrade at your own risk.
Personally, I have ran into this several times. A dependency was updated with a breaking change, and we have dozens of repos dependent on it. Vendoring just that one repo in /vendor allows us to use that one version of dependency, while go get ./... continues to run normally for all other repos to get the latest. We run with the latest bug fixes in PSQL and MySQL and others (there are constant fixes for these!) and so on.
On Git Bash's download site, it says you can clone it to update it. I downloaded it to install it, but I am starting to really get into Git and would like to clone it every time I want to update it from now on. I think I found the place where we need to clone it to, after much searching & less documentation than usual. If I am right, it is in the home directory, right under my nose the whole time!
I checked the version number with git --version beforehand, then cloned in that directory, then rechecked the version number Both times it said git version 1.8.3.msysgit.0, but on the download site it says the latest version is 1.8.4.3.
Have I not found the elusive git folder, or does the Git Bash team not let you directly clone the latest version until some time after it comes out? I know VirtualBox does that, for example.
I know this is sort of a duplicate of this question, but I tried its answer & the PATH variable didn't even have Git!
I would recommend to not try to update it, but simply download the portable version of msysgit and unzip it in a dedicated folder, that you then add to your path:
Current Portable msysgit download.
=> I unzip the latest one in (for instance) C:\prgs\git\PortableGit-1.8.4-preview20130916.
=> I add 'c:\prgs\git\PortableGit-1.8.4-preview20130916\bin' to my PATH.
Now if you want to automate that download/update process, I am building a Powershell script which does just that (for git and 30 other programs, including Mercurial, Subversion, but also Python, Ruby, Go, ...):
senv: download the zip archive of senv, unzip it in a path and call:
senv
The first time, you specify where you want all those programs to be downloaded: those are portable software only: no registry modification, no system variables modified.
If you don't want one of those program anymore, simply delete their installation folder.
That's it.
But each time you will call senv -u, it will check their respective web page and, if a new version is detected, it will download/upzip said new version in its dedicated folder, without removing the previous one.