Gitlab CI/CD - sending comments/alerts to the gitlab UI? - bash

Currently I have this line in my .gitlab-ci.yml file:
if (( $coverage < $MIN_COVERAGE )) ; then echo "$coverage% of code coverage below threshold of $MIN_COVERAGE%" && exit 1 ; else exit 0 ; fi
$coverage is the test coverage of the code, determined with pytest-cov
$MIN_COVERAGE is a specified minimum level of test coverage which $coverage shouldn't drop below
Currently, this causes the pipeline to fail if, for instance, coverage is 70% and min_coverage is 80%. A message is also printed to the terminal: "$coverage% of code coverage below threshold of $MIN_COVERAGE%"
However, this message is only displayed in the terminal of the gitlab job, so if someone wanted to see why and by how much their pipeline failed they would need to go into the job terminal and look at the output.
Instead of having this echo to the job terminal, is there a way to get this message to output somewhere on the gitlab UI?

Here's how to create a new Merge Request Note/Comment using the GitLab API.
script:
# Project -> Settings -> Access Tokens, Create token with API scope.
# Project -> Settings -> CI/CD -> Variables, Store as CI_API_TOKEN
# GET /merge_requests?scope=all&state=opened&source_branch=:branch_name
- |
merge_request_iid=$( \
curl --request GET \
--header "PRIVATE-TOKEN: ${CI_API_TOKEN}" \
"${CI_API_V4_URL}/merge_requests?scope=all&state=opened&source_branch=${CI_COMMIT_REF_NAME}" | \
jq .[0].iid \
)
# POST /projects/:id/merge_requests/:iid/notes
- json_data='{"body":"Your message, here"}'
- |
echo $json_data |
curl --request POST \
--header "PRIVATE-TOKEN: ${CI_API_TOKEN}" \
--header "Content-Type: application/json" \
--data #- \
"${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${merge_request_iid}/notes"

If you have a GitLab Premium subscription or higher, you can use metrics reports to expose any metric, including coverage percentage, in the MR UI.
In all tiers of GitLab, coverage visualization is also available, but it's unclear to me if this displays the overall coverage percentage.
Alternatively, you can use the API to add comments to the merge request (you can get the MR ID from predefined variables in the job). However, you will need to supply an API token to the CI job -- you cannot use the builtin job token to add comments.

In addition, you can use the GitLab CLI tool (glab) in your CI pipeline:
comment-mr:
image: registry.gitlab.com/gitlab-org/cli:latest
variables:
GIT_STRATEGY: none
GITLAB_TOKEN: $MR_AUTOMATION_TOKEN # Project -> Settings -> Access Tokens (api, api_read scopes)
script:
- if [ -z "$CI_OPEN_MERGE_REQUESTS" ]; then echo "No opened MR"; exit 0; fi
- glab mr --repo "$CI_PROJECT_PATH" comment $(echo "$CI_OPEN_MERGE_REQUESTS" | cut -d '!' -f2) --unique=true
--message "šŸ‹ Branch-based docker image - \`$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-branch\`"

Related

Getting the last 100 commits in repositories of a GitHub user/organisation in Bash?

Context
I wrote the following code to get the last n commits of a repository of a GitHub user/organisation:
# Get commits
commits_json=$(curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/$github_username/$github_repo_name/commits?per_page=1&page=1)
echo "commits_json=$commits_json"
echo ""
# Get the first commit.
readarray -t branch_commits_arr < <(echo "$commits_json" | jq ".[].sha")
echo "branch_commits_arr=$branch_commits_arr"
Issue
I noticed I get into the reported rate limits of 60 API calls per hour when I try to do this for all repositories in a GitHub user/organisation.
Attempt I
I tried the more general format to get the commit lists in a single API call:
curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/$some_user/commits?per_page=10&page=1
Which returned:
{ "message": "Not Found",
"documentation_url": "https://docs.github.com/rest/reference/repos#get-a-repository"
}
Attempt II
Another approach to get the data without triggering the API rate limit would be to parse the atom format of each repository, however, it seems like this is an undesirable hack/more boilerplate code than needed.
Question
Hence, I was wondering, how can one get a list/json of/containing all/the most recent 100/n commits across all the repositories of a GitHub user/organisation, using the GitHub API in Bash?

How to send a message to Slack using curl from gitlab-ci.yml?

I am using GitLab for a CI/CD process. I want to send messages to my channel in Slack. Following the API works from the terminal:
curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/xxx/yyyy/zzzz
However, when I put this line into my .yml file, it gives me a "yaml invalid error". Complete block is here:
slack_jar:
stage: slack
before_script:
- echo "hi there"
script:
- curl -F file=#target/springApp-0.0.1.jar -F channels=#application_dev_backend -F token='xoxb-1111-2222-yyyyyy' https://slack.com/api/files.upload
only:
- dev
slack_message:
stage: slack
script:
- echo "Send Slack Messages"
- curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/xxxx/yyyy/zzzz
only:
- dev
The first stage (sending file) is correct, but the second one is not working. This is the error message I get:
Status: syntax is incorrect Error: jobs:slack_message:script config should be a string or an array of strings
Based on your error message, the curl command in slack_message is incorrect. Try wrapping the entire command in quotes and escaping the internal quotes. The way you have it, the YAML parser thinks the Content-type: application/json is a key:value pair of a dictionary.
Try this instead:
slack_message:
stage: slack
script:
- echo "Send Slack Messages"
- "curl -X POST -H 'Content-type: application/json' --data '{\"text\":\"Hello, World!\"}' https://hooks.slack.com/services/xxxx/yyyy/zzzz"
only:
- dev
Pro Tip
You can use the CI Lint tool to validate the contents of gitlab-ci.yaml. You can access this in the CI/CD > Pipelines screen. See CI Lint.
There is also a useful website http://www.yamllint.com/ where you can input YAML, and it will (a) validate it, and (b) return a UTF-8 version. If you have string problems, the UTF-8 version will look mangled (which is what happens with your YAML).
Please give required privilege to bot User to post in channel.
follow the below ci.yaml
#please select the image which has curl command
slack_notification:
image: ubuntu:latest
script:
- echo "Get user id with curl from Slack"
- curl -X GET -H 'Authorization:Bearer <bot token>' https://slack.com/api/users.lookupByEmail?email=$GITLAB_USER_EMAIL | jq -r '.user.id'
# From the above if you change the | jq -r '.user.name' then you will get the name of the user from slack.
- echo "Slack post request"
- >
curl -X POST -H 'Authorization:Bearer <bot token>' -H 'Content-type: application/json' --data '{"channel":"<channel id which start CXXXX>","text":"Your job has been finished please validate Job Url '"$CI_PIPELINE_URL"'"}' https://slack.com/api/chat.postMessage
CI_PIPELINE_URL: This will give the job url.
Slack reference: https://api.slack.com/web

Variable expansion of trigger branch property prevents downstream pipeline from being created

A branch job in which the branch property of the trigger property is using a variable will always fail with reason: downstream pipeline can not be created.
Steps to reproduce
Set up a downstream pipeline with a trigger property as you would normally.
Add a branch property to the trigger property. Write the name of an existing branch on the downstream repository, like master/main or the name of a feature branch.
Run the pipeline and observe that the downstream pipeline is successfully created.
Now change the branch property to use a variable instead, like branch: $CI_TARGET_BRANCH.
Manually run the CI pipeline with that, setting variable through the GitLab GUI.
The job will instantly fail with reason: downstream pipeline can not be created.
Code example
The goal is to create a GitLab CI config that runs the pipeline of a specified downstream branch. The bug occurs when attempting to do it with a variable.
This works, creating a downstream pipeline like normal. But the branch name is hardcoded:
stages:
- deploy
deploy:
variables:
environment: dev
stage: deploy
trigger:
project: group/project
branch: foo
strategy: depend
This does not work; although TARGET_BRANCH is set successfully, the job fails because the downstream pipeline can not be created:
stages:
- removeme
- deploy
before_script:
- if [ -z "$TARGET_BRANCH" ]; then TARGET_BRANCH="main"; fi
- echo $TARGET_BRANCH
test_variable:
stage: removeme
script:
- echo $TARGET_BRANCH
deploy:
variables:
environment: dev
stage: deploy
trigger:
project: group/project
branch: $TARGET_BRANCH
strategy: depend
If you know what I'm doing wrong, or you have something that does work with variable expansion of the branch property, please share it (along with your GitLab version). Alternate solutions are also welcome, but this one seems like it should work.
GitLab Version on which bug occurs
Self-hosted GitLab Community Edition 12.10.7
What is the current bug behavior?
The job always fails for reason: downstream pipeline can not be created.
What is the expected correct behavior?
The branch property should be set to the value of the variable and the downstream pipeline should be created as normal, just as if you simply hardcoded/typed the name of the branch.
More details
The ability to use variable expansion in the trigger branch property was added in v12.4, and it's explicitly mentioned in the docs.
I searched for other .gitlab-ci.yml / GitLab config files. Every single one that attempted to use variable expansion in the branch property had it commented out, saying it was bugged for an unknown reason (example.
I haven't been able to find a repository in which someone claimed to have a working variable expansion for the branch property of the trigger property.
Unfortunately, the alternate solutions are either (a) hardcoding every downstream branch name into the GitLab CI config of the upstream project, or (b) not being able to test changes to the downstream GitLab CI config without first committing them to master/main, or having to use only/except.
TL;DR: How to use the value of a variable for the branch property of a bridge job? My current solution makes it so the job fails and the downstream pipeline isn't created.
this is a 'works as designed', and gitlab will improve in upcoming releases.
trigger job will pretty weak b/c it is not a full job that runs on a runner. Therefore most of the trigger configuration needs to be hardcoded.
I use direct API calls to trigger downstream jobs passing the CI_JOB_TOKEN which links the upstream job to downstream as the trigger does
API calls give you full control
curl -X POST \
-s \
-F token=${CI_JOB_TOKEN} \
-F "ref=${REF_NAME}" \
-F "variables[STAGE]=${STAGE}" \
"${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/trigger/pipeline"
now this will not wait and monitor for when the job is done so you will need to code for that if you need to wait for the downstream job to finish,
Moreover, CI_JOB_TOKEN cannot be used to get the status of the downstream job, so you will another token for that.
- |
DOWNSTREAM_RESULTS=$( curl --silent -X POST \
-F token=${CI_JOB_TOKEN} \
-F "ref=${DOWNSTREAM_PROJECT_REF}" \
-F "variables[STAGE]=${STAGE}" \
-F "variables[SLS_PACKAGE_PATH]=.serverless-${STAGE}" \
-F "variables[INVOKE_SLS_TESTS]=false" \
-F "variables[UPSTREAM_PROJECT_REF]=${CI_COMMIT_REF_NAME}" \
-F "variables[INSTALL_SLS_PLUGINS]=${INSTALL_SLS_PLUGINS}" \
-F "variables[PROJECT_ID]=${CI_PROJECT_ID}" \
-F "variables[PROJECT_JOB_NAME]=${PROJECT_JOB_NAME}" \
-F "variables[PROJECT_JOB_ID]=${PROJECT_JOB_ID}" \
"${CI_SERVER_URL}/api/v4/projects/${DOWNSTREAM_PROJECT_ID}/trigger/pipeline" )
echo ${DOWNSTREAM_RESULTS} | jq .
DOWNSTREAM_PIPELINE_ID=$( echo ${DOWNSTREAM_RESULTS} | jq -r .id )
echo "Monitoring Downstream pipeline ${DOWNSTREAM_PIPELINE_ID} status..."
DOWNSTREAM_STATUS='running'
COUNT=0
PIPELINE_API_URL="${CI_SERVER_URL}/api/v4/projects/${DOWNSTREAM_PROJECT_ID}/pipelines/${DOWNSTREAM_PIPELINE_ID}"
echo "Pipeline api endpoint => ${PIPELINE_API_URL}"
while [ ${DOWNSTREAM_STATUS} == "running" ]
do
if [ $COUNT -eq 0 ]
then
echo "Starting loop"
fi
if [ ${COUNT} -ge 350 ]
then
echo 'TIMEOUT!'
DOWNSTREAM_STATUS="TIMEOUT"
break
elif [ $(( ${COUNT} % 60 )) -eq 0 ]
then
echo "Downstream pipeline status => ${DOWNSTREAM_STATUS}"
echo "Count => ${COUNT}"
sleep 10
else
sleep 10
fi
DOWNSTREAM_CALL=$( curl --silent --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" ${PIPELINE_API_URL} )
if [ $COUNT -eq 0 ]
then
echo ${DOWNSTREAM_CALL} | jq .
fi
DOWNSTREAM_STATUS=$( echo ${DOWNSTREAM_CALL} | jq -r .status )
COUNT=$(( ${COUNT} + 1 ))
done
#pipeline status is running, failed, success, manual
echo "PIPELINE STATUS => ${DOWNSTREAM_STATUS}"
if [ ${DOWNSTREAM_STATUS} != "success" ]
then
exit 2
fi

How to use WebAPI in bash for sonarqube?

I want to write a shell script to login and get bugs for a project. I want the dashboard values like bugs, Vulnerabilities, code smells and coverage.
The url of dashboard is: http://www.example.com/dashboard?id=example_project_name.
Here is what I tried:
curl GET -u username:password http://www.example.com/api/issues/search?project=example_project_name&types=BUG.
So, this prints all the data. I just need the value show in the below image:
Basically What I want to achieve is that Iā€™m using a Sonarqube plugin in Jenkins, so I use extended email plugin to send email for job execution and in that email I want to give details like number of bugs in the repository after the build.
Is there any other way?
Finally after reading the documentation carefully, I got the values. Here is the script that I created.
#!/bin/bash
vul=$(curl -sX GET -u username:password 'http://www.example.com/api/issues/search?projectKeys=example_project_name&types=VULNERABILITY');
bug=$(curl -sX GET -u username:password 'http://www.example.com/api/issues/search?projectKeys=example_project_name&types=BUG');
no_vul=$(echo $vul | jq -r .total);
no_bug=$(echo $bug | jq -r .total);
echo "Total number of VULNERABILITIES are $no_vul"
echo "Total number of BUGS are $no_bug"
Here is the API documentation URL.

Is it possible to permanently update the value of a TeamCity build parameter as a result of a custom run?

Is it possible to permanently update the value of a build parameter as a result of a custom run?
For example, consider a build which is configured to have the build number format:
%Major%.%Minor%.%Patch%.%build.counter%
Major, Minor and Patch and defined in the build configuration to have certain values. For the sake of an example, lets say this gives a build number of 3.1.2.36.
It is possible to change the build number by clicking '...' next to run and then changing the value of one of the params. Changing Minor from 1->2 and patch from 2->0 would give the next build the number 3.2.0.37.
I'm not overly concerned that 37 hasn't be reset to 0, but the problem is that the next build which is triggered (not as a result of a custom run) will have the build number 3.1.2.38 which is a lower number. Is it possible that when you run a custom build and change the numbers that the new values are persisted?
I am looking for a way that users with no TeamCity admin rights can cause the version number to be incremented according to the changes they have made.
We are running v8.1.2 (build 29993).
To fix the issue I used the TeamCity REST API. I created a new build param of type prompt called 'ReleaseType' which can be either Patch, Minor or Major. This is then used in this command line script which is set up as a TeamCity build step:
IF "%ReleaseType%"=="Major" (
set /a newVersion=%VersionMajor%+1
curl -v --request PUT -d 0 --Header "Content-Type: text/plain" http://username:password#servername:8080/httpAuth/app/rest/projects/%system.teamcity.projectName%/parameters/VersionMinor
curl -v --request PUT -d 0 --Header "Content-Type: text/plain" http://username:password#servername:8080/httpAuth/app/rest/projects/%system.teamcity.projectName%/parameters/VersionPatch
)
IF "%ReleaseType%"=="Minor" (
set /a newVersion=%VersionMinor%+1
curl -v --request PUT -d 0 --Header "Content-Type: text/plain" http://username:password#servername:8080/httpAuth/app/rest/projects/%system.teamcity.projectName%/parameters/VersionPatch
)
IF "%ReleaseType%"=="Patch" (
set /a newVersion=%VersionPatch%+1
)
curl -v --request PUT -d %%newVersion%% --Header "Content-Type: text/plain" http://username:password#servername:8080/httpAuth/app/rest/projects/%system.teamcity.projectName%/parameters/Version%ReleaseType%
curl -v --request PUT -d 0 --Header "Content-Type: text/plain" http://username:password#servername:8080/httpAuth/app/rest/buildTypes/id:%dep.Dependant_BuildName.system.teamcity.buildType.id%/settings/buildNumberCounter
This increments the specified build number and resets downstream version parts to 0.
For example, a minor version increase on 3.2.12.122 goes to 3.3.0.0.
Note - in my particular example above the build counter is reset on a dependant build and not the configuration which is running. This may or may not be what you are after. Replace
%dep.Dependant_BuildName.system.teamcity.buildType.id%
with
%system.teamcity.buildType.id%
if you want to reset the current running build configuration.

Resources