I'm configuring my Circleci config right now to try to deploy a staging build for every new branch and then sends a message to my slack team with the link to the staging demo.
I've done the sending message to slack part, but now CircleCI sends a message to slack every time I push. I would like to limit that to happen only once for a specific branch. I know there's CIRCLE_BRANCH env that I can use to identify the current branch, but how do I save that variable in some kind of cache so I can run a conditional check on that variable to avoid running the same command twice?
I've checked the CircleCI docs and they offered cache on files but didn't mention anything about saving a variable as cache.
My config.yml file for CircleCI looks like this:
slackMessage:
docker:
- image: circleci/node
working_directory: ~/client
steps:
- attach_workspace:
at: ~/client
# - run: echo "$CIRCLE_BRANCH" > _branch_check
# - restore_cache:
# keys:
# - pr-{{ checksum "_branch_check" }}
- run:
command: |
PR_NUMBER=${CIRCLE_PULL_REQUEST##*/}
# yolo=pr-`echo -n $CIRCLE_PULL_REQUEST | md5sum`
# if [ -f "$yolo" ]; then
# touch $yolo
curl -X POST <Slack API webhook curl url>
# fi
# - save_cache:
# key: pr-{{ checksum "_branch_check" }}
# paths:
# - pr-{{ checksum "_branch_check" }}
The lines that are commented are the saving to cache part. With these lines commented CircleCI would send a message to Slack on every push. Without the comment the expected behavior is for CircleCI to send the slack message only once for each branch name.
Very cool question. I'll share some ideas that might work.
So, is the staging site URL created based on the branch name? Would it always be generated the same way?
If so, on CircleCI I would check for the existence of the staging URL first. If it's there, it was already created, which means this branch has already posted to Slack, and the execution can end right there.
Another idea would be to touch .staging-created a file into the FS, and save it into CircleCI cache using the branch name ({{ .Branch }}) as part of the key. Then, before sending the message to Slack, check for that file after cache has been restored.
Related
This is a question for the github workflow action dawidd6/action-download-artifact.
There is no discussion board in https://github.com/dawidd6/action-download-artifact, so asking this question in this forum.
This is how I wish to use this workflow in my GitHub repo:
A pull request is created.
This triggers an workflow – lets call it the “build workflow” - to build the entire repo and uploads the build artifacts.
Then another workflow – lets call it the “test workflow” - should start, that should download the build artifact using action-download-artifact and run some other actions.
Now if I put the trigger for the “test workflow” as pull_request, then how can I make it wait for the corresponding “build workflow” to complete? Do I specify the run_id ?
For now I am using “workflow_run” as the trigger for the run WF. But then when a PR is created, it does not show the “test workflow” as one of the checks for the PR. Can you help me figure out the correct way of using the download-artifact action that would help for my purpose?
You could write two workflows where the first builds when the pull request is opened or edited, and the second executes the test when the pull request is closed and merged. The HEAD commit SHA could be used to identify the artifact name between the two workflows.
I'm going to reword your requirements slightly.
Build everything and upload the artifacts when a pull request is opened or edited (e.g. new commits added).
Download the artifact and test it when a pull request is closed and merged.
Here are two sample workflows that would accomplish that. You will need to create a token to share the artifacts between workflows (see secrets.GITHUB_TOKEN below).
Build.yml
name: Build
on:
pull_request:
jobs:
Build:
steps:
- name: Environment Variables
shell: bash
run: |
ARTIFACTS_SHA=$(git rev-parse HEAD)
BUILD_ARTIFACTS=BuildArtifacts_${ARTIFACTS_SHA}
echo "ARTIFACTS_DIR=$ARTIFACTS_DIR" >> $GITHUB_ENV
- name: Build
run: make
- name: Capture Artifacts
uses: actions/upload-artifact#2
with:
name: Artifacts_${{ env.ARTIFACTS_SHA }}
path: path/to/artifact/
Test.yml
name: Test
on:
pull_request:
types: [closed]
jobs:
Test:
steps:
- name: Environment Variables
shell: bash
run: |
ARTIFACTS_SHA=$(git rev-parse HEAD)
BUILD_ARTIFACTS=BuildArtifacts_${ARTIFACTS_SHA}
echo "ARTIFACTS_DIR=$ARTIFACTS_DIR" >> $GITHUB_ENV
- name: Download Artifacts
uses: dawidd6/action-download-artifact#v2
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
workflow: Build.yml
name: ${{ env.BUILD_ARTIFACTS }}
I have the following Github Action workflow that is intended to read our lines of code coverage from a coverage.txt file and print the coverage as a comment.
name: Report Code Coverage
on:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Use Node.js 12.x
uses: actions/setup-node#v1
with:
node-version: 12.x
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm run test:coverage
## I need help around here, as test:coverage generates a file and I need to get a value from a file
- uses: mshick/add-pr-comment#v1
with:
message: |
**{{Where I want the code coverage value to go}}**
🌏
!
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]' # The user.login for temporary GitHub tokens
allow-repeats: false # This is the default
Where I am struggling is on taking the output of the file, which I can obtain with bash using awk '/^Lines/ { print $2 }' < coverage.txt (not sure if there is a better alternative way) and inserting it into the message, which currently just says hello.
I found this article on yaml variables, but when I added some of that code to my own file, it just was not recognized any longer by GitHub actions (which is the only way I know to test yaml). Normally it would fail somewhere and give me an error, but after multiple attempts, the only thing that worked was to remove it.
It is quite possible that I am missing something obvious as I do not know yaml very well nor even what certain key words might be.
Alternatively, is it easier to just dump the contents of the file into the message? That could be acceptable as well.
You can create a step that gets the coverage to an output variable that then can be accessed by the next step.
Notice that for utilizing this method you will need to give the step and id and the set output variable a variable name so that you can access it in follow up steps within the same job.
Sample with your workflow below, but if you want to see a running demo here is the repo https://github.com/meroware/demo-coverage-set-output
- name: Run tests
run: npm run test:coverage
- name: Get coverage output
id: get_coverage
run: echo "::set-output name=coverage::$(awk '/^Lines/ { print $2 }' < test.txt)"
- uses: mshick/add-pr-comment#v1
with:
message: |
Coverage found ${{steps.get_coverage.outputs.coverage}}
🌏
!
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]' # The user.login for temporary GitHub tokens
allow-repeats: false # This is the default
I make a lot of github actions tutorials here if you're looking to grasp more on this topic. Cheers!!
I have the .env file saved with my firebase web API key saved in my local working directory.
I use it in my project as
const firebaseConfig = {
apiKey : process.env.REACT_APP_API_KEY,
}
Now I set up firebase hosting and use GitHub actions to automatically deploy when changes are pushed to GitHub.
However the deployed app by github gives me an error in my console saying Your API key is invalid, please check you have copied it correctly .And my app won't work as it is missing api key.
It seems like github actions is unable to access process.env.REACT_APP_API_KEY
I feel the problem is
As .env file is not pushed to github repo that's why my the build server is unable to access the command process.env.REACT_APP_API_KEY
How can I tackle this issue ?
Github workflow was automatically setup while setting up firebase hosting.
Is this the error or there is something else to take care ?
Below is my firebase-hosting-merge.yml
# This file was auto-generated by the Firebase CLI
# https://github.com/firebase/firebase-tools
name: Deploy to Firebase Hosting on merge
'on':
push:
branches:
- master
jobs:
build_and_deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- run: npm ci && npm run build --prod
- uses: FirebaseExtended/action-hosting-deploy#v0
with:
repoToken: '${{ secrets.GITHUB_TOKEN }}'
firebaseServiceAccount: '${{ secrets.FIREBASE_SERVICE_ACCOUNT_EVENTS_EASY }}'
channelId: live
projectId: myprojectname
env:
FIREBASE_CLI_PREVIEWS: hostingchannels
How can I make my .env file variables accessible to github build server ?
Do I need to change my firebaseConfig ? Or there is any way so that I can make my .env file available to build server and later delete it once the build finishes ?
const firebaseConfig = {
apiKey : process.env.REACT_APP_API_KEY,
}
a quick solution here could be having a step in github actions to manually create the .env file before you need it.
- name: Create env file
run: |
touch .env
echo API_ENDPOINT="https://xxx.execute-api.us-west-2.amazonaws.com" >> .env
echo API_KEY=${{ secrets.API_KEY }} >> .env
cat .env
I am trying to set up a GitLab CI configuration that sends an email after a pipeline's job completes with a link of the artifacts to the upload site. The pipeline builds based upon pom.xml, then tests with sonarqube and then uploads the artifacts using curl to a specific artifactory location. The folder structure and link of the artifact directory depends upon the CI_PIPELINE_ID. After all of these succeeds, I need to send this link for downloading the artifacts to a list of people via mail. My .gitlab-config.yml looks like the following:
image: maven:3.3.9-jdk-8
variables:
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
MAVEN_CLI_OPTS: "-U --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
REPO_NAME: "<artifactory url>"
cache:
paths:
- .m2/repository
- ./target/
stages:
- build
compile_commit:
stage: build
only:
- cr_integrate
before_script:
- git submodule sync --recursive
- git submodule update --init --recursive --remote
script:
- mvn -f pom.xml -s settings.xml $MAVEN_CLI_OPTS clean install $MAVEN_OPTS
- curl -i -u<username>:<token> -T "target/<artifact-1>.zip" "${REPO_NAME}/${CI_PIPELINE_ID}/<artifact-1>.zip"
- curl -i -u<username>:<token> -T "target/<artifact-1>.zip" "${REPO_NAME}/${CI_PIPELINE_ID}/<artifact-2>.zip"
- - curl -i -u<username>:<token> -T "target/<artifact-1>.zip" "${REPO_NAME}/${CI_PIPELINE_ID}/<artifact-3>.zip"
tags:
- <tagname>
How do I send a mail to some people after this with the link?
I built a solution for this, sharing it here.
The following tools were used for this:
GitLab release api
Python-GitLab api
Docker
Microsoft Teams
Sharepoint
The process flow can be outlined as follows:
A new pipeline is triggered
After successful build, codescan and publish, a release job is run
The release job uses a python script written with the help of
python-gitlab api to create a release using gitlab release api. It
inserts external artifactory links for downloading artifacts under
release assets and adds links to release note and other documents.
GitLab sends a release mail to the appropriate notification channel,
a group email id created by Microsoft Teams and Sharepoint, so that
the entire team receives the release mail.
The python script is given below:
import os
import gitlab
from datetime import datetime
if __name__ == '__main__':
access_token = os.environ['RELEASE_TOKEN']
gitlab_url = os.environ['GITLAB_URL']
project_id = int(os.environ['CI_PROJECT_ID'])
tag_name = os.environ['CI_PIPELINE_ID']
ref = os.environ['CI_COMMIT_REF_NAME']
# artifactory_links
artifactory_link = os.environ['ARTIFACTORY_PATH']
group_name = os.environ['GROUP_NAME']
project_name = os.environ['CI_PROJECT_NAME']
directory = f'{datetime.now():%Y%m%d}'
artifact_name = os.environ['ARTIFACT_NAME']
package_type = os.environ['PACKAGE_TYPE']
# artifacts_links
artifacts_links = f'{artifactory_link}/{group_name}/{project_name}/{directory}/{artifact_name}-{tag_name}.{package_type}'
# release note
release_note = os.environ['RELEASE_NOTE']
# authenticate with gitlab
gl = gitlab.Gitlab(gitlab_url, private_token=access_token)
gl.auth()
# obtain the project object by id
project = gl.projects.get(project_id)
# creating the project tags
project.tags.create({'tag_name': tag_name, 'ref': ref})
# creating the project releases
release = project.releases.create(
{
'name': f'Release for Pipeline ID {ref}',
'tag_name': tag_name,
'description': release_note,
'assets': {
'links': [{'name': artifact_name, 'url': artifacts_links}],
}
}
)
The script requires the following environment variables:
RELEASE_TOKEN – GitLab access token
GITLAB_URL – GitLab base URL.
ARTIFACTORY_PATH – Artifactory base URL.
GROUP_NAME – In case the project is under a group.
ARTIFACT_NAME – The artifact name
PACKAGE_TYPE – Artifact package type
RELEASE_NOTE – Link to release note and any other document.
These variables can be provided as GitLab CI variables. If there are more than one artifacts, the python script can be modified accordingly.
Since the python script needs to be called during the pipeline event and adding the script in the project would be modifying the project codebase, dockerizing the script is the best solution. That way, it can be pulled directly from docker hub. The dockerfile contents for this are as follows:
FROM python:3.7-alpine
COPY release_api.py /bin
RUN pip install python-gitlab
ENTRYPOINT ["/bin/release_api.py"]
CMD ["/bin/bash"]
In order to send a release mail to every member of the team, irrespective of their individual GitLab notification and subscription preferences, a team needs to be set up using Microsoft Teams. When a team is created in Teams application, a corresponding sharepoint site is created, along with a team email id. This set up takes some time.
Once a team is created, under Files section, there’s an option to open it in sharepoint (screenshot below).
The sharepoint site has a link in the left sidebar called Conversations. Once the sharepoint site is fully ready, clicking this link will open the inbox of the Teams email.
Under the settings for the group, the option Edit Group can be found and there the group email id can be found. This group email id will be used to send the release mail to everyone in the team.
Under user settings of GitLab, the group email needs to be added. Once the mail is added and verified, the notification channel can be set up under Notifications. Once this is done, all notifications for that group (or project) will go to the group mail, and everyone in the team will get them. The last activity left is to set up notification preference to send a notification when a new release is available.
The gitlab script line below can be used to send email. You will need to have the ssmtp linux program which I suggest building a docker container and installing the ssmtp dependency. Personally I went with an API. What's nice about this solution is you are using core protocols and can build the email message in plain-text and don't have to worry about character escaping an API request.
image: registry.gitlab.com/gitlab-group/dev-ops/REFERENCE-TO-IMAGE-OF-DOCKERFILE-BELOW
script:
- | # Build the core smtp configuration
echo "root=noreply#emailaddress.com" > /etc/ssmtp/ssmtp.conf
echo "mailhub=EMAIL_SERVER:EMAIL_SERVER_PORT" >> /etc/ssmtp/ssmtp.conf
echo "FromLineOverride=YES" >> /etc/ssmtp/ssmtp.conf
echo "AuthUser=EMAIL_SERVER_USERNAME" >> /etc/ssmtp/ssmtp.conf
echo "AuthPass=EMAIL_SERVER_PASSWORD" >> /etc/ssmtp/ssmtp.conf
echo "UseTLS=false" >> /etc/ssmtp/ssmtp.conf
echo "Debug=YES" >> /etc/ssmtp/ssmtp.conf
- "echo 'From: from#emailaddress.com' > msg.txt" # Build the message
- echo "EMAIL SUBJECT" >> msg.txt
- echo "" >> msg.txt
- echo "EMAIL BODY" >> msg.txt
- ssmtp recipient#emailaddress.com < msg.txt # Send the email
Here's a Dockerfile that you can use to get your ssmtp dependency
FROM alpine:3.9
RUN apk add --update --no-cache \
bash=4.4.19-r1 \
ssmtp \
I have a job defined in a yaml file for gitlab that runs when a tag is made
dev_tests:
stage: tagging
tags:
- pro1
- shared
only:
- tags
except:
- branches
script:
- echo running dev tests
- echo $CI_COMMIT_REF_NAME
- /usr/local/bin/phpunit -c phpunit_config.xml
The variable $CI_COMMIT_REF_NAME gives me the tag name but I need the branch name as well.
Is there a way to get this ? None of the other variables seem to do the job
Thanks
I suppose your question is: "Can I get the branch the tag originates from?"
In that case this SO post could be the answer (abstract):
git branch --contains tags/$CI_COMMIT_REF_NAME