I'm using GitLab CI Auto DevOps to compile a project based on Maven based in a DinD (docker-in-docker) runner.
CI job start, and buildpack for Maven is correctly detected (based on herokuish).
How can I configure a custom settings file without switching to a custom .gitlab-ci.yml?
I would like to continue using Auto DevOps.
Because maven needs access to a private repository based on Nexus it is convenient to configure a custom settings.xml (and version it with your source code)
The easiest solution is to include a custom settings.xml. This feature was included in the corresponding buildpack used by Auto DevOps some time ago, as you can see in this article from Heroku buildpacks about "Using a Custom Maven Settings File".
So I defined MAVEN_SETTINGS_PATH variable in .gitlab-ci.yml config file:
variables:
- MAVEN_SETTINGS_PATH: ".m2/settings.xml"
Then, included the file settings.xml in the repository.
Avoid to include secrets or another sensible information
When using a private maven repository with credentials
Finally, you can define in Gitlab some variables to be used in settings.xml. Using Gitlab UI or API add variables for the user, password, and repository url, to be included as environment variables from Gitlab into the job. Then you can use it in settingx.xml like ${env.VARIABLE_NAME}
Example of Gitlab-CI configuration file:
include:
- template: Auto-DevOps.gitlab-ci.yml
variables:
MAVEN_SETTINGS_PATH: ".m2/settings.xml"
AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES: NEXUS_REPO_USER,NEXUS_REPO_PASSWORD,NEXUS_REPO_URL
As a final recommendation, you should avoid to use passwords in environment variables, use native methods from your environment for credentials storage is recommended.
Related
I am setting up 2 jenkins instances on same server. I would like to create 2 local maven repositories for both jenkins seprately. Jenkins 1 is already set up and operational and I woudnt like to touch it.
Can we have 2 local maven repositories for single user as both jenkins are running as the same user ?
Is there any way I can point maven from jenkins to the new repository ?
Thanks
AI
Thanks for your replies, I figured it out, every maven command I am running, I am adding extra parameter
-Dmaven.repo.local=/path/to/alternate/local/repository/
and it overrides default local maven repository.
Cheers,
AI
You can specify a local repository in the settings.xml under <settings><localRepository>.
Alternatively, you can also specify it on the command line as described here:
https://stackoverflow.com/a/7071791/927493
Maven configuration occurs at 3 levels:
◾ Project - most static configuration occurs in pom.xml
◾ Installation - this is configuration added once for a Maven installation
◾ User - this is configuration specific to a particular user
Maven reads the settings from the settings.xml file which can be located in ${M2_HOME}/conf/settings.xml, as well as ${user.home}/.m2/settings.xml
<settings>
...
<localRepository>/path/to/local/repo/</localRepository>
...
</settings>
The default value for the local maven repository is ${user.home}/.m2/repository/
Even though you are are constrained in that you are using the same use for both Jenkins, remember the jobs run on nodes and those can be launched by different users (if via SSH), but it's that node's user's ${user.home}, not the user running Jenkins itself which is read (See Node note below).
What's the simplest way to get two different <localRepository> for a given user?
Jenkins has a Global Tool Configuration (${JENKINS_URL}/configureTools) for Maven:
Further down the the same page you must configure the Maven installations
You could choose the Global Tool Configuration | Maven Configuration to NOT use the default maven settings (the two mentioned above) and specify one from the filesystem:
One one instance, choose ${user.home}/.m2/settings.J1.xml, and on the other, choose ${user.home}/.m2/settings.J2.xml
Alternatively, you could even choose two different "Maven installations", with a different MAVEN_HOME, then have a different ${M2_HOME}/conf/settings.xml in each (awkward, but sometimes useful).
JOB SPECIFIC REPO?
But, if disk space is not really an issue, you could go a step further and give every single job its own private local repository. This is especially handy when building shared libraries in parallel, parallel branches and other scenarios using -SNAPSHOT. Under the Advanced ... options in the maven step, select [ X ] use private Maven repository. That repository ends up residing in ${WORKSPACE}/.repository. Suggest adding the Workspace Cleanup plugin to help manage your space.
You can also, on a per-job basis, specify a specific file-system settings.xml.
NODE SPECIFIC REPO?
Also, each Node has its own Node Properties which again let you customize the "Tool Locations". You can override the "(Maven) Home" location here (but not settings location).
[
PIPELINE
All this is also supported and configurable if using a Pipeline, as described in the Pipeline Maven Integration Plugin.
Extra Plugins
Finally, if you don't like the idea of having all these settings.xml lying all over the filesystems, you can install the Config File Provider plugin which lets you store the custom settings.xml within Jenkins. After installation, the Global Tool Configuration (shown) and Job steps now has the added option to choose from the "provided settings.xml" options you create:
In our CI environment, we currently have one build server (based on Atlassian Bamboo) and two SonarQube instances (versions 6.0 and 6.5). Initially, our CI server was configured to communicate with the 6.0 SonarQube instance. This has been configured in the /home/bamboo/.gradle/gradle.properties file on our CI server like this:
systemProp.sonar.host.url=<http url of SonarQube 6.0 instance>
systemProp.sonar.login=<username here>
systemProp.sonar.password=<password here>
Now we have another Gradle-based project running on our CI server which shall talk to the new SonarQube 6.5 instance. I tried configuring this but failed all the time.
Things I have done so far:
Added commandline arguments to gradle wrapper command:
I have tried adding -Dsonar.host.url=, -Dsonar.login=, -Dsonar.password= to the Gradle command. As this didn't seem to work, I have also tried to set commandline arguments as SonarQube system properties using -DsystemProp.sonar.host.url=, -DsystemProp.sonar.login=, -DsystemProp.sonar.password=. This didn't work either.
Added properties to the build.gradle file
- Added properties to the build.gradle file like this:
sonarqube {
properties {
property "sonar.host.url", "<http url of SonarQube 6.0 instance>"
property "sonar.login", "<username here>"
property "sonar.password", "<password here>"
...<other SonarQube analysis settings here>...
}
}
In all cases, the CI server talked to the wrong SonarQube instance (6.0). My question is, whether it is possible to configure a single project to talk to another SonarQube instance. As you have seen, we use Gradle 3.2.1 as a build tool. And we are using the org.sonarqube Gradle plugin too.
Thank you for any help.
André
Your first try did not work, because you set the system properties from the commandline, but setting it from the project properties later on resets the system properties to the configured values.
Your second try did not work, because the systemProp.sonar.login syntax is only suppored in gradle.properties files, not via -P commandline project properties.
Your third try did not work because the SonarQube scanner prefers the system property values over the value configured via the DSL, so that one can change what is configured in the build script with the help of local configuration.
You need to set the system properties in your build script manually, this then overwrite what was automatically set from the project property. Using the project gradle.properties file does not work as the user file overwrite the project file. So you need something like System.properties.'sonar.login' = '...' in your build script. You can either hard-code it there, or then use project properties that you can set in your gradle.properties file or via -P parameters.
Besides that, I'd never depend on having any configuration in Gradle User dir on a build server. Most buildservers use build agents that might run on distributed machines, so you would always have to make sure that all build agents are configured the same and so on. I'd always configure in the build setup of the build server the according configuration, either by setting system properties, or environment properties or commandline arguments.
Just my 2ct.
In order to bootstrap Gradle-Wrapper, I need to pull the Gradle distribution from an Artifactory which requires HTTP Basic-Auth. There's no way for my build environment to access the outside world - this is blocked by the corporate proxy. My problem is how to provide the credentials so that Gradle can bootstrap.
The Gradle documentation suggests putting the username & password into gradle-werapper.properties.
If I put gradle-wrapper.properties into my project then anybody who has access to my source code would would have access to my credentials. Alternatively, if I put the gradle-wrapper.properties file into my build image then all of my builds will be tied to the same credentials. Neither of these are acceptable.
What I'd much rather do is have Gradle Wrapper pick up it's credentials from environment variables. My run-time environment makes it very easy to provide the credentials in the right way - but is there a way to make Gradle consume the credentials from an environment variable?
From the documents you gave.
In {user.home} directory create .gradle folder if it does not exist.
enter gradle.properties:
systemProp.gradle.wrapperUser=username
systemProp.gradle.wrapperPassword=password
now all you need is distributionUrl to point to your URL, and gradle will handle credentials.
There are three ways to provide credentials:
In folder {user.home} \ .gradle create file gradle.properties with
systemProp.gradle.wrapperUser=username
systemProp.gradle.wrapperPassword=password
pass throw system properties ( note: username, password can be environment variables)
./gradlew -Dgradle.wrapperUser=$username -Dgradle.wrapperPassword=$password
add system properties to GRADLE_OPTS
export GRADLE_OPTS=-Dgradle.wrapperUser=$username -Dgradle.wrapperPassword=$password
I'm using Gitlab CI to deploy TYPO3 projects onto a target server and I'm trying to remove the PackageStates.php from the git repository and generate it on the target server with EXT:typo3_console instead.
But I need to set the TYPO3_ACTIVE_FRAMEWORK_EXTENSIONS environment variable in order to have the necessary system extensions loaded.
How do I set this ENV variable?
What's a good way to store these information in my project to have it available in the deployment process? I could use Gitlab variables but I feel like this information should be included in the git repository.
You don't need to set and use the env variable if you don't want to. You can just require the TYPO3 core extensions you need in any package of your project and typo3_console will take care to populate the environment variable for you as needed during composer install time.
If you want to, you can however still override this env var from command line for individual calls.
EDIT: This feature has been integrated in typo3_console 3.3.0
I am in process of moving configuration parameters out of Java application. I discover that the best approach is to extend your classpath and use .properties files (leave ZooKeeper alone for another requirement).
So my WAR file no longer have any hosts/IPs/URLs, users/passwords.
DevOps distribute configs manually across test, stage, stable installations.
Now time for Jenkins to run tests. But they fail as there are no required .propeties files in classpath.
How can I load this config files to Jenkins and how to make in available in test classpath?
maven-surefire-plugin allow extending classpath and passing system-properties.
So only question how to get separate directory in Jenkins hosting server and load files to this directory and create alias/placeholder/envvar per build job to refer to this path in build config.
This job can be done with SSH access, but I think that this is "wrong way". I expect that this can be done via Jenkins UI (any manager can upload file in WEB browser).
UPDATE I have no requirements for distributed slave/master builds but it whould nice to have solution that migrate configuration files to slaves automatically...
In this way sshing to host or ftp/scp - bad thing.
I read most of Jenkins docs, ask at mail list and IRC. Yea - Jenkins community is silent. At docs I found link to Config File Provider Plugin, after that I visit http://builder.evil.com/jenkins/pluginManager/available page and look for config keyword.
There are a lot related plug-ins with various usefulness to my subject (most useless first):
https://wiki.jenkins-ci.org/display/JENKINS/Envfile+Plugin - This plugin enables you to set environment variables via a file.
https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin - Allows credentials to be bound to environment variables for use from miscellaneous build steps.
https://wiki.jenkins-ci.org/display/JENKINS/Environment+Script+Plugin - Allows you to run a script before each build that generates environment variables for it.
https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin - This plugin makes it possible to have an isolated environment for your jobs.
https://wiki.jenkins-ci.org/display/JENKINS/Copy+Data+To+Workspace+Plugin - Copies data to workspace directory for each project build.
https://wiki.jenkins-ci.org/display/JENKINS/Copy+To+Slave+Plugin - This plugin allows to copy a set of files, from a location somewhere on the master node, to jobs' workspaces. It also allows to copy files back from the workspaces of jobs located on a slave node to their workspaces on the master one.
https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin - Adds the ability to provide configuration files (i.e., settings.xml for maven, XML, groovy, custom files, etc.) loaded through the Jenkins UI which will be copied to the job's workspace.
Only last plug-in - Config File Provider Plugin allow editing configs via Jenkins WEB interface. And it have brother - Managed Script Plugin - for uploading/managing/editing custom scripts. No question now I use Config File Provider Plugin!
You should keep the configs required for the tests together with the rest of source code, so that after compilation, your unit tests can run.
After deploying the .war, the DevOps team should overwrite the in-war configs with whatever per-environment configs that they have.