I am trying to set up a GitLab CI configuration that sends an email after a pipeline's job completes with a link of the artifacts to the upload site. The pipeline builds based upon pom.xml, then tests with sonarqube and then uploads the artifacts using curl to a specific artifactory location. The folder structure and link of the artifact directory depends upon the CI_PIPELINE_ID. After all of these succeeds, I need to send this link for downloading the artifacts to a list of people via mail. My .gitlab-config.yml looks like the following:
image: maven:3.3.9-jdk-8
variables:
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
MAVEN_CLI_OPTS: "-U --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
REPO_NAME: "<artifactory url>"
cache:
paths:
- .m2/repository
- ./target/
stages:
- build
compile_commit:
stage: build
only:
- cr_integrate
before_script:
- git submodule sync --recursive
- git submodule update --init --recursive --remote
script:
- mvn -f pom.xml -s settings.xml $MAVEN_CLI_OPTS clean install $MAVEN_OPTS
- curl -i -u<username>:<token> -T "target/<artifact-1>.zip" "${REPO_NAME}/${CI_PIPELINE_ID}/<artifact-1>.zip"
- curl -i -u<username>:<token> -T "target/<artifact-1>.zip" "${REPO_NAME}/${CI_PIPELINE_ID}/<artifact-2>.zip"
- - curl -i -u<username>:<token> -T "target/<artifact-1>.zip" "${REPO_NAME}/${CI_PIPELINE_ID}/<artifact-3>.zip"
tags:
- <tagname>
How do I send a mail to some people after this with the link?
I built a solution for this, sharing it here.
The following tools were used for this:
GitLab release api
Python-GitLab api
Docker
Microsoft Teams
Sharepoint
The process flow can be outlined as follows:
A new pipeline is triggered
After successful build, codescan and publish, a release job is run
The release job uses a python script written with the help of
python-gitlab api to create a release using gitlab release api. It
inserts external artifactory links for downloading artifacts under
release assets and adds links to release note and other documents.
GitLab sends a release mail to the appropriate notification channel,
a group email id created by Microsoft Teams and Sharepoint, so that
the entire team receives the release mail.
The python script is given below:
import os
import gitlab
from datetime import datetime
if __name__ == '__main__':
access_token = os.environ['RELEASE_TOKEN']
gitlab_url = os.environ['GITLAB_URL']
project_id = int(os.environ['CI_PROJECT_ID'])
tag_name = os.environ['CI_PIPELINE_ID']
ref = os.environ['CI_COMMIT_REF_NAME']
# artifactory_links
artifactory_link = os.environ['ARTIFACTORY_PATH']
group_name = os.environ['GROUP_NAME']
project_name = os.environ['CI_PROJECT_NAME']
directory = f'{datetime.now():%Y%m%d}'
artifact_name = os.environ['ARTIFACT_NAME']
package_type = os.environ['PACKAGE_TYPE']
# artifacts_links
artifacts_links = f'{artifactory_link}/{group_name}/{project_name}/{directory}/{artifact_name}-{tag_name}.{package_type}'
# release note
release_note = os.environ['RELEASE_NOTE']
# authenticate with gitlab
gl = gitlab.Gitlab(gitlab_url, private_token=access_token)
gl.auth()
# obtain the project object by id
project = gl.projects.get(project_id)
# creating the project tags
project.tags.create({'tag_name': tag_name, 'ref': ref})
# creating the project releases
release = project.releases.create(
{
'name': f'Release for Pipeline ID {ref}',
'tag_name': tag_name,
'description': release_note,
'assets': {
'links': [{'name': artifact_name, 'url': artifacts_links}],
}
}
)
The script requires the following environment variables:
RELEASE_TOKEN – GitLab access token
GITLAB_URL – GitLab base URL.
ARTIFACTORY_PATH – Artifactory base URL.
GROUP_NAME – In case the project is under a group.
ARTIFACT_NAME – The artifact name
PACKAGE_TYPE – Artifact package type
RELEASE_NOTE – Link to release note and any other document.
These variables can be provided as GitLab CI variables. If there are more than one artifacts, the python script can be modified accordingly.
Since the python script needs to be called during the pipeline event and adding the script in the project would be modifying the project codebase, dockerizing the script is the best solution. That way, it can be pulled directly from docker hub. The dockerfile contents for this are as follows:
FROM python:3.7-alpine
COPY release_api.py /bin
RUN pip install python-gitlab
ENTRYPOINT ["/bin/release_api.py"]
CMD ["/bin/bash"]
In order to send a release mail to every member of the team, irrespective of their individual GitLab notification and subscription preferences, a team needs to be set up using Microsoft Teams. When a team is created in Teams application, a corresponding sharepoint site is created, along with a team email id. This set up takes some time.
Once a team is created, under Files section, there’s an option to open it in sharepoint (screenshot below).
The sharepoint site has a link in the left sidebar called Conversations. Once the sharepoint site is fully ready, clicking this link will open the inbox of the Teams email.
Under the settings for the group, the option Edit Group can be found and there the group email id can be found. This group email id will be used to send the release mail to everyone in the team.
Under user settings of GitLab, the group email needs to be added. Once the mail is added and verified, the notification channel can be set up under Notifications. Once this is done, all notifications for that group (or project) will go to the group mail, and everyone in the team will get them. The last activity left is to set up notification preference to send a notification when a new release is available.
The gitlab script line below can be used to send email. You will need to have the ssmtp linux program which I suggest building a docker container and installing the ssmtp dependency. Personally I went with an API. What's nice about this solution is you are using core protocols and can build the email message in plain-text and don't have to worry about character escaping an API request.
image: registry.gitlab.com/gitlab-group/dev-ops/REFERENCE-TO-IMAGE-OF-DOCKERFILE-BELOW
script:
- | # Build the core smtp configuration
echo "root=noreply#emailaddress.com" > /etc/ssmtp/ssmtp.conf
echo "mailhub=EMAIL_SERVER:EMAIL_SERVER_PORT" >> /etc/ssmtp/ssmtp.conf
echo "FromLineOverride=YES" >> /etc/ssmtp/ssmtp.conf
echo "AuthUser=EMAIL_SERVER_USERNAME" >> /etc/ssmtp/ssmtp.conf
echo "AuthPass=EMAIL_SERVER_PASSWORD" >> /etc/ssmtp/ssmtp.conf
echo "UseTLS=false" >> /etc/ssmtp/ssmtp.conf
echo "Debug=YES" >> /etc/ssmtp/ssmtp.conf
- "echo 'From: from#emailaddress.com' > msg.txt" # Build the message
- echo "EMAIL SUBJECT" >> msg.txt
- echo "" >> msg.txt
- echo "EMAIL BODY" >> msg.txt
- ssmtp recipient#emailaddress.com < msg.txt # Send the email
Here's a Dockerfile that you can use to get your ssmtp dependency
FROM alpine:3.9
RUN apk add --update --no-cache \
bash=4.4.19-r1 \
ssmtp \
Related
I'm working with a React Native project, setting up Xcode Cloud builds.
I keep getting this error:
unable to open configuration settings file
Pods-XXX.debug.xcconfig:1
The files in my workspace look like the following:
|-- XXX
|-- Pods
|. -- Podfile
|. -- Targets Support Files
|. -- Pods-XXX
|. -- Pods-XXX.debug
Guess what? XCode is garbage but doc can help us here.
This what I'm using to smoothly build my workflow. I'l share exact bash scripts here.
Why it's failing?
First open logs on your workflow window side bar.
Check if you're installing necessary dependencies before running archive process. You're using virtual machine so any dependencies like cocoapods or yarn aren't installed by default there.
If you haven't read and skipped to the SOLUTION:
Here is the steps:
Create ci_scripts folder inside ios folder.
Create 3 files inside ci_scripts folder:
ci_post_clone.sh
ci_post_xcodebuild.sh
ci_pre_xcodebuild.sh
Inside your ci_post_clone.sh file add this:
#!/bin/zsh
# fail if any command fails
echo "🧩 Stage: Post-clone is activated .... "
set -e
# debug log
set -x
# Install dependencies using Homebrew. This is MUST! Do not delete.
brew install node yarn cocoapods fastlane
# Install yarn and pods dependencies.
# If you're using Flutter or Swift
# just install pods by "pod install" command
ls && cd .. && yarn && pod install
echo "🎯 Stage: Post-clone is done .... "
exit 0
Inside your ci_pre_xcodebuild.sh file add this:
#!/bin/zsh
echo "🧩 Stage: PRE-Xcode Build is activated .... "
# You can add additional scripts here...
echo "🎯 Stage: PRE-Xcode Build is DONE .... "
exit 0
Inside your ci_post_xcodebuild.sh file add this:
#!/bin/zsh
echo "🧩 Stage: POST-Xcode Build is activated .... "
# You can add additional scripts here...
echo "🎯 Stage: POST-Xcode Build is DONE .... "
exit 0
I've had this same issue recently, there is now a default workflow for xcode cloud that performs the post clone step that was provided by #BEK ROZ
I also had the same issue locally when building a VueJS app with Ionic/Capacitor
Before now my knowledge of iOS builds is precisely zero, so excuse innocence, but I found the issues were relating to the *xcconfig files. Every capacitor module that you install with npm will have it's own xcconfig file for debug and release - Stored in:
ios/App/Pods/Target Support Files/{{ Module Name }}/{{ Module Name }}.debug.xcconfig
ios/App/Pods/Target Support Files/{{ Module Name }}/{{ Module Name }}.release.xcconfig
I've only installed two plugins, so I uninstalled one of them, which was cordova-plugin-screen-orientation, synced and then archived (steps below) and this time it passed.
# Remove a capacitor module with npm e.g. cordova-plugin-screen-orientation
npm uninstall cordova-plugin-screen-orientation
# Sync the iOS plugins to remove the plugin from your Podfile (ios/App/Podfile)
npx cap sync ios
So I'd recommend going through all of the plugins you've installed and see if any of them are playing up. Then go back to xcode, run "Product -> Archive" from the menus to start a new build or push your changes and let xcode cloud do the heavy lifting for you.
Good luck
I have a sonar instance that I run in my fastfile but now I wan to add an API call to SonarQube section in Fastlane to ensure the PR decoration happens correctly.
this is my sonar
sonar(
project_configuration_path: './fastlane/sonar-project.properties',
sources_path: File.expand_path('../'),
project_version: project_version,
branch_name: branch_name
)
my properties file is
sonar.projectKey=iOS
sonar.projectName=iOS
sonar.host.url=http://sonar.xxxxx.com
sonar.login=xxxxxxxx
sonar.language=swift
the url I am looking to add is curl -u "${token}:" -X POST 'https://sonar.....
Any help on how I can add this curl cmd in fastlane
I have the .env file saved with my firebase web API key saved in my local working directory.
I use it in my project as
const firebaseConfig = {
apiKey : process.env.REACT_APP_API_KEY,
}
Now I set up firebase hosting and use GitHub actions to automatically deploy when changes are pushed to GitHub.
However the deployed app by github gives me an error in my console saying Your API key is invalid, please check you have copied it correctly .And my app won't work as it is missing api key.
It seems like github actions is unable to access process.env.REACT_APP_API_KEY
I feel the problem is
As .env file is not pushed to github repo that's why my the build server is unable to access the command process.env.REACT_APP_API_KEY
How can I tackle this issue ?
Github workflow was automatically setup while setting up firebase hosting.
Is this the error or there is something else to take care ?
Below is my firebase-hosting-merge.yml
# This file was auto-generated by the Firebase CLI
# https://github.com/firebase/firebase-tools
name: Deploy to Firebase Hosting on merge
'on':
push:
branches:
- master
jobs:
build_and_deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- run: npm ci && npm run build --prod
- uses: FirebaseExtended/action-hosting-deploy#v0
with:
repoToken: '${{ secrets.GITHUB_TOKEN }}'
firebaseServiceAccount: '${{ secrets.FIREBASE_SERVICE_ACCOUNT_EVENTS_EASY }}'
channelId: live
projectId: myprojectname
env:
FIREBASE_CLI_PREVIEWS: hostingchannels
How can I make my .env file variables accessible to github build server ?
Do I need to change my firebaseConfig ? Or there is any way so that I can make my .env file available to build server and later delete it once the build finishes ?
const firebaseConfig = {
apiKey : process.env.REACT_APP_API_KEY,
}
a quick solution here could be having a step in github actions to manually create the .env file before you need it.
- name: Create env file
run: |
touch .env
echo API_ENDPOINT="https://xxx.execute-api.us-west-2.amazonaws.com" >> .env
echo API_KEY=${{ secrets.API_KEY }} >> .env
cat .env
I have setup Jenkins project piper (https://sap.github.io/jenkins-library/). I have then setup a basic SAP Cloud Application Programming model app with integration for the SAP Cloud SDK pipeline with default configuration and uncommented the 'productionDeployment' stage and completed cloud foundry endpoints/orgs/spaces etc. I have committed the applicatino to the master branch in the git repo.
The pipeline executes successfully but is skipping the production deployment step.
Pipeline execution results
When checking the logs I see:
[Pipeline] // stageenter code here
[Pipeline] stage
[Pipeline] { (Production Deployment)
Stage "Production Deployment" skipped due to when conditional
When I look at the script (https://github.com/SAP/cloud-s4-sdk-pipeline/blob/master/s4sdk-pipeline.groovy) I see:
stage('Production Deployment') {
*when { expression { commonPipelineEnvironment.configuration.runStage.PRODUCTION_DEPLOYMENT }* }
//milestone 80 is set in stageProductionDeployment
steps { stageProductionDeployment script: this }
}
Can anyone explain what is required to pass the commonPipelineEnvironment.configuration.runStage.PRODUCTION_DEPLOYMENT check in order to execute the stageProductionDeployment script?
My pipeline_config.yml file (anonymized) is:
###
# This file configures the SAP Cloud SDK Continuous Delivery pipeline of your project.
# For a reference of the configuration concept and available options, please have a look into its documentation.
#
# The documentation for the most recent pipeline version can always be found at:
# https://github.com/SAP/cloud-s4-sdk-pipeline/blob/master/configuration.md
# If you are using a fixed version of the pipeline, please make sure to view the corresponding version from the tag
# list of GitHub (e.g. "v15" when you configured pipelineVersion = "v15" in the Jenkinsfile).
#
# For general information on how to get started with Continuous Delivery, visit:
# https://blogs.sap.com/2017/09/20/continuous-integration-and-delivery
#
# We aim to keep the pipeline configuration as stable as possible. However, major changes might also imply breaking
# changes in the configuration. Before doing an update, please check the the release notes of all intermediate releases
# and adapt this file if necessary.
#
# This is a YAML-file. YAML is a indentation-sensitive file format. Please make sure to properly indent changes to it.
###
### General project setup
general:
productiveBranch: 'master'
### Step-specific configuration
steps:
setupCommonPipelineEnvironment:
collectTelemetryData: true
cloudFoundryDeploy:
dockerImage: 'ppiper/cf-cli'
smokeTestStatusCode: '200'
cloudFoundry:
org: 'XXXXXX'
space: 'XXXXXX'
appName: 'MTBookshopNode'
manifest: 'mta.yaml'
credentialsId: 'CF_CREDENTIALSID'
apiEndpoint: 'https://api.cf.XX10.hana.ondemand.com'
### Stage-specific configuration
stages:
# This exclude is required for the example project to be successful in the pipeline
# Remove it when you have added your first test
s4SdkQualityChecks:
jacocoExcludes:
- '**/OrdersService.class'
# integrationTests:
# credentials:
# - alias: 'mySystemAlias'
# credentialId: 'mySystemCredentialsId'
# s4SdkQualityChecks:
# nonErpDestinations:
# - 'myCustomDestination'
productionDeployment:
cfTargets:
- org: 'XXXXXX'
space: 'XXXXXX'
apiEndpoint: 'https://api.cf.XX10.hana.ondemand.com'
appName: 'myAppName'
manifest: 'mta.yaml'
credentialsId: 'CF_CREDENTIALSID'
My Jenkins file is unchanged:
#!/usr/bin/env groovy
/*
* This file bootstraps the codified Continuous Delivery pipeline for extensions of SAP solutions, such as SAP S/4HANA.
* The pipeline helps you to deliver software changes quickly and in a reliable manner.
* A suitable Jenkins instance is required to run the pipeline.
* The Jenkins can easily be bootstraped using the life-cycle script located inside the 'cx-server' directory.
*
* More information on getting started with Continuous Delivery can be found in the following places:
* - GitHub repository: https://github.com/SAP/cloud-s4-sdk-pipeline
* - Blog Post: https://blogs.sap.com/2017/09/20/continuous-integration-and-delivery
*/
/*
* Set pipelineVersion to a fixed released version (e.g. "v15") when running in a productive environment.
* To find out about available versions and release notes, visit: https://github.com/SAP/cloud-s4-sdk-pipeline/releases
*/
String pipelineVersion = "master"
node {
deleteDir()
sh "git clone --depth 1 https://github.com/SAP/cloud-s4-sdk-pipeline.git -b ${pipelineVersion} pipelines"
load './pipelines/s4sdk-pipeline.groovy'
}
Any ideas what I am missing for a production deployment and how I get through this check in the script for production deployment?
Regards
Neil
the pipeline was built for multi-branch pipelines and will not work correctly in a single-branch pipeline job. There is no problem with running a project that has a single branch in a multi-branch pipeline job. To avoid confusion, we added a check to the pipeline in a recent version as documented here https://blogs.sap.com/2019/11/21/new-versions-of-sap-cloud-sdk-3.8.0-for-java-1.13.1-for-javascript-and-v26-of-continuous-delivery-toolkit/#cd-toolkit
Kind regards
Florian
I'm configuring my Circleci config right now to try to deploy a staging build for every new branch and then sends a message to my slack team with the link to the staging demo.
I've done the sending message to slack part, but now CircleCI sends a message to slack every time I push. I would like to limit that to happen only once for a specific branch. I know there's CIRCLE_BRANCH env that I can use to identify the current branch, but how do I save that variable in some kind of cache so I can run a conditional check on that variable to avoid running the same command twice?
I've checked the CircleCI docs and they offered cache on files but didn't mention anything about saving a variable as cache.
My config.yml file for CircleCI looks like this:
slackMessage:
docker:
- image: circleci/node
working_directory: ~/client
steps:
- attach_workspace:
at: ~/client
# - run: echo "$CIRCLE_BRANCH" > _branch_check
# - restore_cache:
# keys:
# - pr-{{ checksum "_branch_check" }}
- run:
command: |
PR_NUMBER=${CIRCLE_PULL_REQUEST##*/}
# yolo=pr-`echo -n $CIRCLE_PULL_REQUEST | md5sum`
# if [ -f "$yolo" ]; then
# touch $yolo
curl -X POST <Slack API webhook curl url>
# fi
# - save_cache:
# key: pr-{{ checksum "_branch_check" }}
# paths:
# - pr-{{ checksum "_branch_check" }}
The lines that are commented are the saving to cache part. With these lines commented CircleCI would send a message to Slack on every push. Without the comment the expected behavior is for CircleCI to send the slack message only once for each branch name.
Very cool question. I'll share some ideas that might work.
So, is the staging site URL created based on the branch name? Would it always be generated the same way?
If so, on CircleCI I would check for the existence of the staging URL first. If it's there, it was already created, which means this branch has already posted to Slack, and the execution can end right there.
Another idea would be to touch .staging-created a file into the FS, and save it into CircleCI cache using the branch name ({{ .Branch }}) as part of the key. Then, before sending the message to Slack, check for that file after cache has been restored.