Question: What is the best way to maintain a project based on another OSS project, through Xcode and version managed by SVN?
I'd like to start a fork (?) of a reasonably popular open source project (it's allowed). Mostly, I want to build my own user interface written in Cocoa/ObjC for it and throw in a few custom features of my own as well.
Now, this OSS project isn't exactly small. The project itself has over 3000 files, and the build process is pretty intense- consisting of multiple stages and steps, which need to compile build tools, run those, then compile the results.
All this is fine and dandy in Xcode, since it's easy enough to setup build phases and rules to handle everything.
What I'm not clear on, is how best to manage patches from upstream. They are constantly working on the project and I'd like to be able to keep up to date with those patches as easily as possible, as many of the diff files effect sometimes up to a hundred (!) files at once.
So maintaining a pristine unmodified copy of that source tree so I can apply patches to it seems like a smart thing to do, because I really don't want to be sorting through hundreds of files every few weeks merging patches by hand.
What I'm thinking of doing in this regard is:
1) Setup an "upstream" SVN repo to hold a copy of the upstream source, plus the bare minimum required to compile it in Xcode (so an xcproject, a few xcconfigs, some prefix header files and that's it)
2) Setup my own "downstream" SVN repo where I do all my work and apply my own modifications.
Whenever upstream releases a patch, I can apply it to #1 then synchronize across to #2, and deal with any issues created by my own modifications.
What I'm not clear about, is if this is a sane way of handling things- or if there's some better practice I should be following.
Is this the best way to handle things, or should I be looking at doing this some other way?
In SVN-world it was named "Vendor Branches" long time ago and intensively used by many teams (you can additionally google this phrase)
Technically it's
one SVN repo
at least one special branch (special in terms of usage, nothing more), which, with svn:externals, linked to 3-rd party repo of upstream code
your place for changes (trunk or any other place, I prefer trunk), initially created as copy of vanilla code and there you perform all code-hacks
If (or "when") vendor branch got updates from upstream, you have just merge branch to /your place/, integrate changes and continue to work
Related
My team is working on translating several legacy mobile applications to Xamarin Forms apps. Currently each application is in its own solution, which is not ideal when it comes to the fact that they all use a common set of backend software libraries. We were planning to consolidate all the smaller solutions into a single solution, containing the apps as well as the common libraries.
However, one of my teammates brought up a valid concern about how with a single Xamarin Forms app, several projects could get generated (core, Android, iOS, etc.), with the eventual result of a generally unwieldy solution. I agree with him that the current setup probably would not scale too well as we add more apps -- even if we group projects in solution folders, Visual Studio will eventually slow to a crawl after a certain amount of projects exist in the solution.
So we are considering just going back to having each app in its own solution, each solution containing the few Xamarin Forms projects for that app, as mentioned above. But this brings us back to the question of how to reasonably manage the shared library code. My current thought would be to just use shared project(s) for the libraries, or maybe assemble them into NuGet package(s) the app solutions would consume. Am I on the right track here, or does anyone know of a better way to do this?
There are several different ways to manage a shared code project using subtrees, submodules, NuGet packages, etc. There are pros and cons to each so it's best to decide based on the expected use case for that project.
Subtrees essentially take a copy of the remote repo and pull it into the parent repo. This makes it easy to pull in changes from the remote repo but if changes are expected to be pushed back it can be significantly more difficult since it has no knowledge of the remote repo. While it is possible to push changes back it can take a significant among of time to do depending on the amount of history of the repos.
Submodules are similar to subtrees except that instead of taking a copy it tracks the remote repo based on a specific commit it's pointed to. This essentially can be thought of as another repo inside of the parent that makes pushing changes back to the remote repo much easier but at the cost of making pulling/updating from it a little bit more difficult.
NuGet packages are extremely convenient to install, update, and release to others without having to make the source code public, but that comes with a bit more initial setup to generate each package version and comes at the cost of making it more difficult to debug than with the actual source code. This is particularly a great option if the shared code library will be distributed to others.
For most projects, if changes are expected to be potentially made to that shared project from a consuming one I'd recommend a repo for each project and set up the shared one as a submodule in each. It does take a bit of learning to get used to the different processes of checking out and updating a submodule but actually isn't all that difficult and worth learning the few git commands required. The docs provide a great example of how to get started using submodules.
In Countinous Delivery book, it's recommended to keep everything - including CI scripts - in the version control. Actually, current CI systems like gitlab CI already follow this rule of thumb and search for CI scripts in the same codebase.
On the other hand, we are versioning our codebase (and it's built artifacts) whenever it changes. And we follow semantic versioning for that; incrementing patch field for bugfixes, minor for non-breaking features, and so on...
And we make sure the version is incremented between commits by checking it in the CI.
But, there are commits that only change the CI scripts; i.e. adding an analysis job, optimizing another, etc.
My question, after this long boring preface, is that what is the best practice to versioning such changes to the CI? Since it possibly can affect the final built artifact (e.g. changing a build flag in the CI job for optimization or ...).
Is it ok to increment the version in this case?
Git is a revision control system. Every time you commit something to a git repo, it labels the content of the repo with a content hash value that represents that version of the repo. Semantic versioning of a git repo's content is redundant and pointless. The whole point of SemVer is to provide a means for producers to communicate risk to consumers. In other words, semantic versioning is intended for build product labeling, not the bits that go into producing the build.
If you attempt to apply SemVer semantics to the repo, you are labeling the product inputs, not the product itself. You should not apply a SemVer string until after all unit/regression/acceptance tests have been performed. How else can you have any certainty whether the code/build-script changes have broken anything?
Pre-build labeling cannot work. Build processes that are capable of reproducing the exact same output twice in a row, are extremely rare, if any exist at all. It is a violation of best practice to have multiple API's/packages in the world with the same SemVer string attached to them. If you label the repo content and then forward that label to the build output, every time you run the build, you produce a package with different content. There will always be some risk that more than one of those outputs will be released into the wild. Many security conscious consumers pay close attention to the content hash of packages they consume. Detecting that a particular producer has released multiple package hashes without bumping the version number, will raise red flags and lead to mistrust of that producers internal processes.
This is a very deep topic that can't be fully covered here. Other issues to consider are OS/Compiler/Tool chain updates. Will you also be committing the entire build tool chain to the same repo? This is an untenable approach, full of hazards I cannot fully enumerate, without taking a few months off work to document them.
Best practice:
Use semantic commit messages that clearly state the developer's intent.
Validate build outputs prior to packaging/labeling.
Always keep humans in the loop, for non-prerelease publications.
Just for clarity, let me add that maintaining build scripts and tool manifests in the repo is considered a best practice. It ties the versions of your scripts and tools, to the versions of the code you are building. Git does do this job quite well, by creating a commit hash that encompasses the state of the entire repo (minus the tags if I recall correctly). But there will be issues eventually, with older versions of tools, being withdrawn from file shares/feeds, particularly when they are found to create security vulnerabilities.
It will sometimes be the case, that older versions of your products, cannot be reproduced using the earlier build process. Checking in the binaries is often promoted as a fix for this issue, but I would argue that it's an anti-pattern. Binaries you are likely never going to want or need in the future, should not be stored in your repo. It just clogs everything up.
Consider using an alternate archival system. Maintaining a separate archive of older tools isn't a bad idea, but you will often find that you simply can't run them on current hardware and OS's, without significant reconfiguration of build machine(s) and re-introducing well known security risks. You should frequently prune such an archive, based on the latest known risks and weighing the cost of having to do some additional work, if/when the day ever comes, that you need to build from a really old commit hash.
It is better to maintain an up-to-date build system, that can build all of your code base, back to some reasonable point in its history. That point is usually the oldest bits that you are willing to actively support with bug fixes.
These days I'm using SemVer compatible Headver; https://github.com/line/HeadVer and feeling happy.
It is very CI friendly thanks to the automatic incremental versioning, but still be able to announce when breaking changes happens by allowing to define major version number manually.
I am new to git and while I can see the immense benefits of the branch/merge process on individual projects, it also seems to me that another useful application of Git is to use it for centralizing the core custom code and templates that you often use in all your projects, with each project starting off as its own branch.
For example I have a large set of custom code that is just a starting place that I use for all apps. If I tweak it here and there, or find bugs and fix them, I want that to replicate through all projects that use the code.
By working on tweaks in a separate branch off the master, then merging back to the master, and then merging the master with other branches that occupy separate projects, it would seem I can achieve this quite well.
Are there any major drawbacks to this approach to Git?
Also while I know that submodules exist, I prefer not to use them at this time since they are not natively supported in Xcode.
Several projects will tend to advance on their own terms. If you have all together on the same branch, you see intermingled commits, and there isn't any sense of history to be discerned. If you place each project on a separate branch, you'll have to jump from one to the other, and there won't be any advantage against separate git repositories (git is very lightweight!).
A DVCS does add overhead (if there are several repositories with the same contents, it certainly adds disk space; but that is dirt cheap nowadays...). It has the advantage that you can work anywhere, freely (no need to wait for a slow connection), and (at least with git) the operations are normally so fast that until you get used to it you have this lingering fear that you are being cheated and nothing has been done...
For development, the fact that branching and merging works and is cheap means experiments and exploration are encouraged. The fact that commits are instantaneous and use up very little space leads to microcommits (one logical change, even a minimal bugfix or typo, per commit), instead of "a morning worth of work" dumps. This makes history understandable, and tools like git bisect extremely useful.
I have a bunch of git repositories with stuff I want to preserve history on, but not necessarily ever share with anybody else. Some more important of that stuff I keep also in remote repositories (not much more than git and ssh at the other end, and a bit of discipline to push there now and then, are required) for backup. In that sense, I use git more as a local VCS, which also has the advantage that I can share with others ;-)
I've started working on a new Perl module and I've decided that I want to make a Ruby version of it as well (once I finish the Perl version). Do people tend to make separate repositories for each language? Or put them in the same repository?
I can easily see how the two sets of code are different enough to be treated as separate projects. But at the same time it's the same functionality written in two languages, so from that perspective it seems like a single project with two language ports.
What's considered best practice in this situation?
FWIW, I'm using git.
EDIT: I should be more clear here. These aren't modules in the sense of git submodules. They're modules that will be submitted to CPAN and RubyGems. Users of this project will likely be installing it via cpan or gem and then using/requiring it in the normal fashion.
In the course of my group's research, we have a couple repos, some with different technologies in each. We divide the repos by research question and checkout only the projects we are working on, with the repos having a uniform hierarchal directory structure that is the same for all projects. Since we already know the repo directory structural, running scripts and finding data becomes much easier.
I would recommend taking the same approach. The higher the division between the two technologies, the easier it will be to contribute to one of them without being confused by the presence of the other.
In the end ask yourself this: If I were to add another language, would I still keep it in one repo? If the answer is yes, keep doing what you're doing. If not, keep these libraries in two separate repos and manage the projects and contributers distinctly.
my experience in this kind of case is to have 2 smaller git repos for each of the modules and either cloning one branch into the consumer projects repo makes it quite simple. another way is to create a naked clone from the module's repo into the consumer projects repo, then just keep updating it as each module's development progresses. the consumer project should ignore the injected repos.
once other dev clones module A, and/or B, then he/she can just push to consumer project, as permissions allow. this is either a pro or a con depends.
Just moved to subversion...from visual studio. I love it already! Can someone briefly explain
Repository
Branches
Tags
Trunk
Do I need to create a new repository for every project? Or a new trunk?
Thanks
You don't need a separate repository, but you can if you want. I recommend reading the book at http://svnbook.red-bean.com/. Grab the pdf version or whatever. It doesn't take too long, and it explains some things pretty well. I read it, and found that I'm glad I did.
Remember that subversion is just a fancy filesystem that supports versioning. Think of a repository as a "drive root" like "C:/".
Each project gets a trunk, tags and branches directory. All of your day to day work happens in the trunk. Experimental code is done in a branch and then merged back into the trunk at a later date. Tags are for when you release the software. These are not to be edited. When you release the software, you create a tag with a unique name based on what is currently in the trunk.
I can't say whether or not you need a separate repository for each project, there are pros and cons. This blog posting details them:
Simplified administration. One set of hooks to deploy. One repository
to backup. etc.
Branch/tag flexibility. With the code all in one repository it makes it
easier to create a branch or tag
involving multiple projects.
Move code easily. Perhaps you want to take a section of code from
one project and use it in another, or
turn it into a library for several
projects. It is easy to move the code
within the same repository and retain
the history of the code in the
process.
Here are some of the drawbacks to the
single repository approach, advantages
to the multiple repository approach.
Size. It might be easier to deal with many smaller repositories than
one large one. For example, if you
retire a project you can just archive
the repository to media and remove it
from the disk and free up the storage.
Maybe you need to dump/load a
repository for some reason, such as to
take advantage of a new Subversion
feature. This is easier to do and with
less impact if it is a smaller
repository. Even if you eventually
want to do it to all of your
repositories, it will have less impact
to do them one at a time, assuming
there is not a pressing need to do
them all at once.
Global revision number. Even though this should not be an issue,
some people perceive it to be one and
do not like to see the revision number
advance on the repository and for
inactive projects to have large gaps
in their revision history.
Access control. While Subversion's authz mechanism allows
you to restrict access as needed to
parts of the repository, it is still
easier to do this at the repository
level. If you have a project that only
a select few individuals should
access, this is easier to do with a
single repository for that project.
Administrative flexibility. If you have multiple repositories, then
it is easier to implement different
hook scripts based on the needs of the
repository/projects. If you want
uniform hook scripts, then a single
repository might be better, but if
each project wants its own commit
email style then it is easier to have
those projects in separate
repositories
I agree, read the svnbook. It's a great resource.
Do I need to create a new repository for every project? Or a new trunk?
Kevin covered the single/multiple repository trade-offs pretty well. When we started with svn, we used one repository for all of our development projects. It worked well and had all the advantages mentioned. However, as the repository got bigger it got more difficult to administer because of the size of the dump file and resulting issues during backup. It also became an issue that projects couldn't easily be archived out of the repository - it's certainly possible but it requires dumping and pulling out projects from the repository. They aren't issues you can't get around but it's something to keep in mind.
Repository
Branches
Tags
Trunk
Branches, tags and the trunk are just copies of your files contained in the repository. It allows you to segregate and check-mark your files at whatever time you feel appropriate (usually at a release or a feature branch).
An important thing to keep in mind about branches, tags and trunk is that they just conventions in svn. There is no functional difference between the three locations, they are just an accepted usage model and they can be changed or organized differently if you have a good reason. I'm not recommending that you organize differently but you'll find that svn is very flexible because there isn't really a forced organizational structure other than convention.
Depending on how many projects you decide to have in your repository, you may organize differently.
You can have the subdirectories with projects under it:
\repo
\branches
\...
\tags
\...
\trunk
\..
or you can have projects contain the subdirectories:
\repo
\Project1
\branches
\tags
\trunk
\Project2
\branches
\tags
\trunk
There are trade-offs that are covered in the svnbook. The first method is usually used if you only have one project per repository and the second if there is more than one project in your repository.
The nice thing is that you can just start using svn and then figure out what you prefer. You should have some sort of organization but, with cheap copies, you can always re-arrange the folders as your situation or workflow changes.
An important thing to remember with SVN, compared to other version control systems like CVS or Git, is that SVN doesn't really have a concept or branching or tagging. As far as SVN is concerned it's all just a bunch of folders and files. So while you'll see a lot of people using the branches/tags/trunk setup, this is not required and you are able to deviate from this if you so choose.
Generally speaking 'trunk' is where you keep your active development going. So this is where you do all your commits. Whether or not you checkout trunk or use tags/branches instead is entirely up to you.
Branches, as I've used them, are usually for when you need to do large changes to your application but don't want them in trunk because you want to be able to continue developing against trunk without deploying your other changes. In this case you may have something like
\repo
\trunk
\branches
\version_two
In this case you can develop in both trunk and version_two separately and, assuming your live site is a checkout of trunk, you don't need to worry about 'accidentally' breaking your live site with your other changes. And when those changes are done and ready you just merge them back into trunk whenever you want.
Tags can be used similarly to branches, in that instead of checking out trunk and just using 'svn up' to update your repository you instead of several tags, each representing one release. So your repo may look something like
/repo
/trunk
/branch
/version_one
/version_two
/tags
/1.0.0
/1.0.1
/1.1.0
In this case the general idea is that when you're ready to do a deploy you do an
svn copy
To copy trunk over to a tag (in this case the next one could be 1.1.1, 1.2.0, 2.0.0, etc). How you name your tags it entirely up to you though and, again, depends on your project and requirements. With this route instead of doing a regular 'svn up' you would have to do an svn switch. So you have to deploy with
svn switch https://svn.yourrepo.com/repo/tags/1.1.0
The switch will automatically do updates, adds and deletes on the appropriate files.
When it comes to one repo for many projects or separate repos for each one I am an advocate of one repo per project. It provides the additional benefits of easily managing access to it. But most importantly it means that each project has a separate commit history and separate logs. This m
Reading your tags I see you started using VisualSVN instead of your old VSS system. (Your question says you stopped using Visual Studio.. which makes VisualSVN a strange choice).
One of the major differences between SourceSafe and VSS is that you can choose different tools to access the same repository (and you can switch every time you like as they all share the same workingcopy).
E.g.:
TortoiseSVN for Explorer integration.
The normal subversion client for scripts.
VisualSVN as Visual Studio frontend for TortoiseSVN
AnkhSVN as real SCC (VAPI) package in Visual Studio.