I face a problem, common I guess.
I have a project which is store in github.
I need it to run for: production, testing and different developers.
The project uses maven profiles to set some parameters.
The project has also a spring profile: currently only DEV which initialize a DB.
The project uses external software with specfic configuration files.
I need to have the project in production but being able to serve development versions.
The question: can I use git ignore locally within developers to distribute the code and still store the files in github?
Or would you have a different solution?
The actual question is:
Can I use git ignore locally for some configuration files and still have the files on the main repository? Updated by only the production users?
One way is to store:
one different configuration file per environment
one template file
one script able to detect the current environment the git repo is cloned in, and generate the actual configuration file (which isn't versioned) used for that local environment.
That generation can be automated on checkout, with a smudge script declared as a content filter driver.
The way we have solved this is with a configuration script that pulls the server-appropriate credential file for local, staging, or production from a separate repository.
After each server pulls the appropriate file, we rename it for consistency across the different environments, and then we have our settings file which is tracked by git include the credentials file.
Related
Due to the relative simplicity of our site and limited resources, we're using one machine as both our development and staging server using a self-hosted github runner. It connects via SSH and runs actions triggered by pushes to two different branches. What I'd like to do is have two github actions scripts execute for two different branches of the same repository, we'll call them dev and stable for repository projectA. dev would deploy to ~/actions_runner/_work/projectA/projectA-dev and stable to ~/actions_runner/_work/projectA/projectA-stable, then use individual ENV files to connect them to the appropriate versions of their support services. It's unclear how to configure each script to rename the directory for their branch. Currently, the development workflow (a single script) deploys to ~/actions_runner/_work/projectA/projectA automatically... it seems like it should be possible to change the second instance of the repository name in the path to something more customized. Is it?
I looked at the github actions documentation about deploying to specific locations and haven't seen anything. I also searched github, reddit and google. Nothing I've found has helped! Thanks in advance.
It's unclear how to configure each script to rename the directory for their branch
Ideally, you do not rename anything, but use each action to checkout in your runner workspace to two different paths.
That way, you can use projectA-dev/stable in each action.
- name: Checkout tools repo
uses: actions/checkout#v3
with:
repository: my-repo
path: projectA-dev
When a private agent build starts in VSTS, it gets assigned a directory, e.g. C:\vstsagent_work\1\s
Is there a way to set this to a different path? On other CI servers, like Jenkins, I can define a custom workspace for a job. I'm dealing with a huge monorepo and have dozens of build definitions around the same repository. It makes sense (to me anyway) to share a single directory on the build agent computer.
The benefit to me is that my builds can use pre-built components from upstream repositories, if they have already been built.
Thanks for any help
VSTS build always creates a working directory per build definition. This leaves you two options:
Create a single build definition and use conditionals on steps to skip certain steps in order to only run what is needed. This allows you to use the standard steps and may require a powershell script to figure out which steps to run and which ones to skip. Set variables from powershell using the special logging commands.
Disable the get sources step and add a step that manually fetches sources. You'll need to clean the working directory, checkout the right commit, basically replicating the actions in the get sources step manually. It may require some fidgeting to get all the behavior correctly for normal build, pull request builds etc. That way you can take full control over the location where sources are checked out.
I'd also recommend you investigate the 2017 project formats that use the new <packageReference> in the project files to fetch packages. The new system supports configuring a version range which can always fetch the latest available version of packages. It's a better long-term solution.
No, it isn’t available in VSTS build system.
You can change working directory of agent (C:\vstsagent_work) (Re-configure it and specify another working folder), but it won’t uses the same source folder for different build definitions, the folder would be 1, 2, 3 ….
In the infrastructure I've inherited, the guys use jenkins and maven to construct jars and config files.
So, jenkins checks out some code, then maven builds something. Jenkins post build task then copies files from mavens target directory and copies them to a "build share" (eeks), for example
Source
**/target/*.jar
Destination
releases/env_name/project/lib
Source
**/target/classes/some-service.xml
Destination
releases/env_name/project/conf
I dare say this is not ideal.
In the new provisioning project, the jars from the build, are deployed into Nexus. In turn, I then use the maven rpm plugin to build RPMs using GAVs to identify which jars are packaged. I haven't captured the config files.
Could I publish the config files to Nexus? Yes they're small, but still artifacts. They would also have a maven snapshot or release coordinate, which could fit in with my releases. I have not tried, but I imagine I can add a section to my existing RPM build pom so that the desired plain text files are uploaded to Nexus when I run mvn deploy.
Some of the files require modification post deployment to suit an environment - this adds further head ache to this.
The poms that define the RPMs will also grow significantly in length. Not a problem per se, if I can ascertain this is the right way.
The question: How have others managed jars and config files deployment in a multi-environment set up?
Since you are asking how others have done it, I will say how I do, which may or may not be appropriate in your situation.
The binaries (.jar or whatnot) are build by Jenkins from the source code in SVN.
The SVN source code has configuration files, but most of the time, those are configured for DEV or Local environments. I let them have it there so that developers' local builds can be simple, but really as far as I am concerned they don't exist.
The SVN also has a separate location (not accessible by DEV, only by Release Managers and Systems teams), and this location contains configuration files for all environments (including DEV environment, when deployed from the CI server)
Since we have a lot of environment specific settings, it makes sense to keep separate copies of each environment.
If a new configuration parameter is added, it is merged from DEV config files all the way to PROD, just like other source code changes are merged, with the exception that only RM or Systems team can do it.
The deployment process (in our case a shell script, but could be anything) takes care of pulling the correct configuration files from SVN and pushing them, along with binaries, to the remote server. Once server is deployed, the "in-jar" configuration is wholly replaced by the configuration files that came from SVN.
I organize the configuration file by ENV, then by TYPE, for example: QA/WEB, QA/API and then PROD/WEB, PROD/API. So I don't have to maintain configuration files for each remote machine.
However there are times when configuration is specific to the remote machine, such as an IP address of the machine. In these cases, the SVN configuration file contains a token, like [local_ip]. When the deployment script pushes this file to remote server, it then knows to replace the token with real IP of the remote machine
TeamCity appears to store the definitions for builds, projects, templates etc as XML internally.
This is exposed in the "Administration > Audit" view where you can see diffs that people made to individual configurations, at URLs like http://teamcityserver/admin/settingsDiffView.html?id=project:project10&versionBefore=8&versionAfter=9&actionId=3151
I'd like to manage a TeamCity setup partially from outside the web interface - e.g. for example keep the build definitions in version control and perhaps programmatically generate them.
Is there any way I can directly upload definitions in this format (or any similar alternative)? I'm aware that there are various APIs and extension points to TeamCity but haven't managed to find any that gives direct access to anything like this.
I can live with the format changing with TeamCity versions if necessary - it would be a reasonable price to pay for the other benefits.
For TeamCity 9.x and newer
As reported by Ganesh in the comments to this answer, an option was added in 9.x that supports changes and versioning through Source Code Management (SCM) tools. Please see his answer for 9.x and beyond.
For TeamCity 8.x and older
It might not be the "approved" way, but you can edit the project files on disk, and those changes will appear in your build configs. I have successfully edited them outside of the Web UI after they were created.
So, you could probably open that folder up as a restricted network share or set up ssh.
You'll find it at $TeamCityData/config/projects/ and then they are stored in subfolders such as $projectName/buildTypes/$buildFile.xml
An example is:
E:\TeamCityData\config\projects\CSandbox\buildTypes\CSandbox_Project1TrunkBuildUnitTest.xml
TeamCity 9 adds a new "Versioned Settings" feature which keeps these XML files under version control and allows changes to be made via the VCS.
In TeamCity 9.0 this can be git or mercurial, and the upcoming TeamCity 9.1 will add support for Perforce and Subversion.
I've been using it with git for a few months and it works quite nicely in practice.
I sometimes have trouble persuading TeamCity to notice changes coming in from the VCS - particularly when deleting projects - but otherwise it's been really useful for standardising configuration and spinning up new job chains quickly.
Another slight annoyance is that you can't configure the location within the repository that the settings come from - it's always .teamcity in the root - so I've had to use multiple branches or repositories to manage multiple TeamCity servers.
As a consultant, I have multiple clients that I'm doing work for. Each client utilizes their own internal Maven repository that is also set up to mirror Central and other external repositories. I need to configure my maven installation on my laptop so that when I'm doing work for one client, it utilizes their internal repository for everything.
I had thought I would be able to utilize profiles to handle this, but mirror settings cannot be changed per-profile.
Does anybody have suggestions on how to approach this maven configuration?
Note: A similar question is here: How do I configure maven to access maven central if nexus server is down?, but that question deals with switching between Central coming from a mirror or not. What I need is for Central (and others) to come from one mirror or a different one based on some property/setting/variable etc.
Create two shell aliases:
alias build_at_home="mvn -s $HOME/.m2/home_settings.xml"
alias build_at_work="mvn -s $HOME/.m2/work_settings.xml"
The "-s" option is handy for explicitly stating which environment settings file to use.
We use this mechanism on our shared build server to ensure each project build is isolated.
Obviously on windows you could create a set of batch files.
I have a very similar requirement in my project too. I created two separate settings.xml files, named them as settings_one.xml and settings_two.xml and saved them in the MAVEN_HOME. Depending on which file I need, I have a small script (a bat file on windows) which overwrites the existing settings.xml with one of the two settings files.
del C:\Users\<username>\.m2\settings.xml
copy C:\Users\<username>\.m2\settings_one.xml C:\Users\tadigotl\.m2\settings.xml
The simplest solution i can suggest is to install git and commit you .m2/settings.xml (of course ignore the repository itself via .gitignore) into git and make appropriate branches for the customers. Change the settings will be done by:
git checkout CUSTOMER_BRANCH
and furthermore any change is tracked by a SCM.