We have a collection of related, and partly interdependent, packages. They live in separate Git repositories. Like:
git#bitbucket.company:common
git#bitbucket.company:libfoo
depends on common
git#bitbucket.company:libbar
depends on common
git#bitbucket.compnay:libbarex
depends on libbar
git#bitbucket.company:daemon
depends on common
final-image (just an image recipe)
depends on libfoo, libbar, libbarex, daemon
The packages are currently built by setting up the Yocto environment, checking out all of them and running bitbake with the image target that depends on all of them. The checkout is done with logic containing special logic. That is
git clone https://git.yoctoproject.org/git/poky
git clone -b $BRANCH git#bitbucket.company:common
git clone -b $BRANCH git#bitbucket.company:libfoo
git clone -b $BRANCH git#bitbucket.company:libbar
git clone -b $BRANCH git#bitbucket.company:libbarex
git clone -b $BRANCH git#bitbucket.company:daemon
. poky/oe-init-build-env
../common/add-layers.sh
bitbake final-image
The disadvantage is that the CI server does not understand the configuration management side of things, so it can't do things like building each pull-request, not to mention reporting it back to the Git repository manager to gate the pull-requests on it.
So I would like to start splitting the build to build each package separately, pulling the dependencies according to some specification that shall be updated as needed.
But how do I do that?
I can add
bitbake -c populate_sdk final-image
to the main build, and then use the SDK to build individual components. However that means that any changes in common have to be merged, and the SDK rebuilt, before I can build libfoo, libbar or daemon and additionally changes to libbar have to be merged before I can build libbarex.
Can I push the results of these incremental builds to some kind of repository and pull them back to build the dependent component without having to rebuild the SDK?
Related
I always use composer packages in Laravel but I never changed one. This is my first time and I don't want to do it incorrect.
I need to use and change a packages foo/bar. Everything that follows now is just guessed:
I forked the repo
I created a develop branch
I added a vcs to my composer.json
"require": {
//...
"foo/bar": "dev-develop",
},
"repositories": [
{
"type": "vcs",
"url": "https://github.com/thisisme/bar"
}
],
composer update
Now I have the thisisme/bar fork in my vendor folder in foo.
So far so good. Now I can use my own fork.
But currently, as I don't know what is good practice to modify the repo, I cloned the repo to a completely different location. Then I push my changes there and run composer update in my project to get the changes. But this is a pain.
Do I need to have a sub git in my project in vendor/foo with
git remote add origin https://github.com/thisisme/bar.git. Because "git in git" feels wrong and finally is not really working as git commands seem to interact with the "parent git".
While VonCs answer is correct regarding git, I'm not certainly sure that git submodule support is well aligned with composer(1) vendor dir for packages from a VCS repository. At least I have not experimented much with it and when I use a composer configuration with a VCS git repository, I normally don't need that1.
While composer(1) has support for git for vendor packages, it is on repository level, that is, you can have your own repository for your package (as you have configured it shown in your question) and then composer takes care of updating (or giving a warnings about local changes).
composer(1) supports this with its own remote for the packages (non-bare) clone (in the source install, read on).
So yes, what you describe ("But this is a pain."), is as long as you don't use it to your benefit. While you develop your (cloned) package, you don't need to run composer update all the time.
.git
composer.json
vendor/foo/bar/.git
A Composer project with two Git repositories
This is why IMHO "git in git" must not feel wrong. Similar to git sub-modules, git supports this very well. By default it even keeps track in the parent project of the current revision (changes) of the sub-project but without having the information of the remote - as it is local (gitlink).
You won't see this thought as within the tree, the gitlink would be at vendor/foo/bar and commonly (& given that) vendor is git ignored, no version tracking in the main project for vendor/foo/bar/.git - but there in the sub-project.
This is not a problem as Composer manages that git sub-project for you (the initial clone and further checkouts) in terms of your main project.
And git realizes it is a different project.
You should be able to cd into the package directory within the vendor folder (vendor/foo/bar) and configure your remote(s) there. You can then work within that project and git(1) will work there and not within the parent repository.
To have this work with composer(1) it is important that you configure composer to prefer the source install variant for that repository. This is the preferred-install option and you can configure it for your repository specifically.
{
"config": {
"preferred-install": {
"foo/bar": "source"
}
}
}
From the wording in your question, I assume that you have not yet configured it.
And this is somewhat important as only with the source install, there will be a (non-bare) git clone in vendor/foo/bar and therefore the git checkout with the overall git configuration within the packages folder in the vendor directory (as you have Github configured as the repository source and composer optimizes to take the dist version by default IIRC).
After you changed your configuration to the source install and updated it, cd into vendor/foo/bar and then run git remote -v. It now should show you the "composer" remote(s) for that package.
As you use the develop branch, you can add changes locally but mind the gap that you would also need to push them to the remote repository (Github) before you use composer again to update (at least) that foo/bar package - as while you use git for the development of the foo/bar package now, in your main project you use composer to manage the dependency.
This is the price you have on the payroll using Github instead of a configuration that is more near to the place of work, but at least locally, you can handle the package with "git in git".
This is normally straight forward. One overall price remains thought, due to managing two instead of one repository but that you can't prevent with this kind of composer project [composer only versioned vendor folder]).
Note: If development takes longer than a few hours, it may also make sense to include the new Git sub-project in the backup routine of your parenting project, so that when you remove the folder vendor/foo/bar you have a backup of the (local) Git repository and working tree in it. However, this depends on the project configuration and is your own responsibility.
A bit of a workflow with some hints is also outlined in the composer documentation in Loading a package from a VCS repository.
1 There is a type of setup for a composer project where vendor itself is under git version control, with that git sub-modules can work (very well), but this is most likely not the kind of setup you have for your project, so I skip it for this answer.
If you're working with sail or docker-compose and linking the foo/bar project in the vendor dir is only a temporary until 'it works' solution you could just add it as a volume link. This is what I usually do.
Eg: I'm working on my-project in ~/projects/my-project, I clone the foo/bar repo to ~/projects/bar
Then in the docker-compose.yml I can add the volume:
volumes:
- .:/var/www/html
- ../bar:/var/www/html/vendor/foo/bar
Again, this has a huge assumption on docker being used, but I like to think that everybody is using it these days.
Do I need to have a sub git in my project in vendor/foo with git remote add origin https://github.com/thisisme/bar.git.
That could be achieved with a submodule which allows for your parent Git repository to only store a reference to another repository.
You would use git submodule add for that.
A git clone --recurse-submodule would therefore clone your project with the submodule Git repository in it cloned as well, and checked out to the exact reference you previously committed.
I am trying to build my first image with Yocto for a Xilinx Eval Board ZCU104. As I am just a beginner I installed the required packages to work with Poky and stuck to the following simple steps:
I cloned Poky: git clone git://git.yoctoproject.org/poky
source oe-init-build-env: a build directory was created.
In the build directory I cloned the meta-xilinx layers: git clone git#github.com:Xilinx/meta-xilinx.git
I read the README.md and README.building.md files from meta-xilinx/meta-xilinx-bsp to find what to do next.
I used the command bitbake-layers add-layer to add the following layers: meta-xilinx-bsp, meta-xilinx-standalone and meta-xilinx-contrib. I can see those layers in bblayers.conf.
I edited MACHINE in local.conf: MACHINE ?= zcu104-zynqmp
I think I am good to go so I call bitbake -k core-image-minimal and expect a succesful build.
Bitbake started to parse until it throwed the following error:
ERROR: ParseError at
//poky/build/meta-xilinx/meta-xilinx-bsp/recipes-bsp/uboot-device-tree/uboot-device-tree.bb:11:
Could not inherit file classes/xsctdt.bbclass
I looked for that class under /meta-xilinx-bsp/classes and it is not there, but I found it in the meta-xilinx repository so I started wondering why I didn't get that class (among others).
Note that I did not switch to any branch yet (either yocto or meta-xilinx) and I stayed on the master branches. I gave it a second try after switching to the latest tags from both projects but I got the following error:
ERROR: Layer xilinx is not compatible with the core layer which only
supports these series: gatesgarth (layer is compatible with zeus)
What should I try next? Should I checkout other branches? If so, is there any index to link the right yocto and meta-xilinx branches?
Anyways, how come I am missing necessary classes from the repository when I clone it?
Thanks a lot!
Checkout the branches named the same way in each layer you're using (poky included). The branches are usually named after Yocto releases. It's usually wise to avoid the master branch.
If there is no branch of a given Yocto release for a layer, look into what seems to be a recent branch and check in conf/layer.conf for the LAYERSERIES_COMPAT variable. Find a branch which has a LAYERSERIES_COMPAT with the Yocto release you want to use. If there are none, you'll need to add the Yocto release to said variable and fix things along the way until it compiles and everything is working fine.
c.f.: https://docs.yoctoproject.org/ref-manual/variables.html#term-LAYERSERIES_COMPAT
I want to build a project in Azure Pipelines, but I want to know what the idiomatic way is to obtain the latest tag, latest tag distance, and repo remote path/URL in order to pass those values into the actual build script that is inside the repository.
Previously our build script would invoke hg log -r . --template with a clever template, but we found when moving to Continua CI build server that the build agent doesn't have access to the actual repository during a build, and had to find another way.
I'm assuming the same issue would crop up with Azure Pipelines and haven't quite found the relevant docs yet on artifact versioning.
Many thanks in advance.
For git at least, Azure Pipelines does a full clone of the repo by default, unless you explicitly denote that you're doing a shallow clone (source: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/pipeline-options-for-git?view=azure-devops).
Deriving the version/tag can be done via normal git commands (i.e. git describe --tags or whatever you prefer), which can then be saved as VSO variables to be accessed in later steps in the same job (see https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-variables-using-expressions for more info on how to do that).
Trying to move to the officially supported Golang vendoring solution from legacy Godeps workflow.
Scenario:
Repo A===
\
========> Repo C (shared library code)
/
Repo B===
What is the best workflow I can choose for a mid-size (roughly 5-10 member) team of engineers to vendor Repo C for both Repo A and Repo B? Engineers of varying abilities, most of which probably shouldn't need to know the details of this at all?
I'm currently using govendor for this. I'd prefer not to switch but would if there is a tool that provides a better workflow.
This needs to integrate with a CI server running the builds. I can think of 3 scenarios:
Vendor Repo C into A & B:
Pros:
Reproducible Builds
Easy integration with CI
Cons:
Manual and Error Prone - Can easily vendor incorrect code
Engineers need decent knowledge of vendor tool and methodology
Symlink Repo C trunk branch into vendor folders of A & B:
Pros:
Engineers need no knowledge of vendor tool
Low Developer Maintenance
Cons:
Builds not (easily) reproducible
Possibility of including code in build that shouldn't be released
Less Flexible (Repo A and Repo B can't have differing versions of C)
Include Repo C as a git submodule or subtree in Repo A and Repo B (either utilizing vendor or not):
Pros:
Engineers need no knowledge of vendor tool
Easy Setup
Less Maintenance
Reproducible Builds
Cons:
Having to use git submodule or subtree
Finding surprisingly little about this question on the internet. Is there some idiomatic way of doing this? I'm sure there are other ways of doing this; what am I missing?
i suggest to use a manifest based approach with version constraint.
Project A == Manifest
|- Repo A#~1.0.1
|- Repo B#~1.0.1
Repo A == Manifest
|- Repo C#~1.0.1
Repo B == Manifest
|- Repo C#~1.0.5
Repo C == Manifest empty
Which will resolve into
Project A == Resolved Manifest
|- Repo A#1.0.1
|- Repo B#1.0.1
|- Repo C#1.0.5
where ~1.0.1 means >=1.0.1 <1.1.0.
As you see B and A dependency to C are independent, yet within project they are resolved correctly.
In the event A and B would define incompatible dependency to C, an error should occur as the project should not be build-able.
You may prefer to use caret ^ rather than tilde ~, ^1.0.1 -> >=1.0.1 < 2.0.0.
Note that you are not forced to use those 'helpers' such tilde and caret, you may define explicit version range.
You shall decide which constraint to apply given the level of confidence you give to the remote author to correctly upgrade its version number.
Finally,you can use glide to solve that for you.
Starting with Repo C, assuming you already tagged the repos, run glide init, git commit -am 'glide init', git push
Repo A, glide init, glide get git#repo.com/repoc, git commit -am 'glide init', git push
Repo B, glide init, glide get git#repo.com/repoc, git commit -am 'glide init', git push
Finally, Project A, glide init, glide get git#repo.com/repoa, glide get git#repo.com/repob, git commit -am 'glide init', git push
To re install the project, glide install, go build.
Nothing prevent you to tarball ProjectA with its vendor folder, in order to skip the glide install command when you execute the remote installation.
But you normally don t want to commit the vendor folder for a development environment. You d usually add vendor/ to your .gitignore file and run glide install or glide update.
Expect some difficulties in the begin, passed that step, things will work.
Once you jumped to that workflow, note that you ll have to bump every changes on your repos.
That is bloatware when you work both project A and repo B to reach a viable change, so in that case, rather than vendoring repoB into project A (you can leave the manifest definition, but get ride of the repoB folder into vendor/), install repoB as a go module with the go get command.
Doing so the changes are taken in effect immediately on re-build. Once the change set is completed, browse into each repos and bump them appropriately.
Finally you may want to use a version bumper to help you to get it done quick and fast, it happens i did one for my personal usage.
hope this helps.
Since GitLab 7.6, or thereabouts, there is a new option to use TeamCity directly from GitLab projects. In the setup there is this message:
The build configuration in Teamcity must use the build format number
%build.vcs.number% you will also want to configure monitoring of all
branches so merge requests build, that setting is in the vsc root
advanced settings.
I'm not sure how this works. Lets say I have a repository Foo.
I have setup a build on TeamCity to listen to Foo with branch specification: +:refs/pull/*/merge
I then fork Foo in gitlab as FooFork, make a change, then request a merge FooFork -> Foo.
But nothing happens to test this merge, which is what I was expecting GitLab to do. If I accept the merge than the build server jumps into action (immediately) and builds twice (master and /ref/master).
I've also set the build configuration to use exactly: %build.vcs.number% as the build number as prescribed, but gitlab doesn't seem to give me any information about the build result.
So I'm a bit confused really as to what exactly this GitLab -> TeamCity integration is supposed to do and whether I'm doing wrong.
I'm currently running GitLab 7.9 and TeamCity 8.1.4
Update:
Seems this use case was not supported prior to version 8 - https://github.com/gitlabhq/gitlabhq/issues/7240
I'm running GitLab 8.0.2 and TeamCity 9.1.1 and am able to run CI builds on branches and merge requests.
I trigger CI builds for specific branches by setting a VCS trigger together with the branch specification +:refs/heads/(xyz*) where xyz is the string for our ticket system prefix since all active branches need to be named after an entry in our issue tracker.
I trigger builds for merge requests via the branch specification +:refs/(merge-requests/*)
Everything works as as expected and lets us know the status of all feature / bug branches and merge requests automatically.
Thanks to Rob's comment linking to the GitLab 8 release notes entry on the merge request spec.
Same problem here. There might be another way, I'm evaluating right now. Since there's no direct way of getting the merged state from the target MR, you have to build it on your own:
IMO there's the following todos
1.) init a bare repo $ git init
2.) add your target repo $ git remote add origin git#your-repo:<origin.group>/<origin.repo>.git
3.) add the remote/feature/to-merge's $ git remote add target git#your-repo:<feature.group>/<feature.repo>.git
4.) checkout your feature branch $ git checkout -b <feature.branch> feature/<feature.branch>
5.) checkout your original branch $ git checkout -b <origin.branch> origin/<origin.branch>
6.) Rebase feature into your original branch $ git rebase <feature.branch>
As stated here [1], GitLab-CE can fire an event on creation of a merge-request,
so all you have to do is building some meta, that can evaluate the WebHooks.
[1] http://doc.gitlab.com/ce/web_hooks/web_hooks.html#merge-request-events