My problem is basically this:
I have a build job and a deploy job in my gitlab-ci.yml.
build:
extends: .node_base
artifacts:
paths:
- artifact_folder
stage: deploy
script:
- npm start
deploy:
tags:
- linux-docker
stage: deploy
when: manual
image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
script:
- aws --endpoint-url $AWS_HOST s3 sync artifact_folder/ s3://$AWS_S3_BUCKET --delete --acl public-read
dependencies:
- build
The build job downloads files from an external location and saves them in an artifact for my deploy job to use.
The deploy takes the files from the build job artifact and uploads them to an s3 bucket.
So far so good. The problem is that everytime I want to deploy new changes I will have to first re-run the build job to get the updated files from the external location, before I re-run the deploy job.
Its not a big issue but I would like to, if possible, only have one job that does both the build step and the deploy step.
My first idea was to simply run the - npm start in the build job as a before_script in the deploy job. However, I am limited by the infrastructure setup by devops atm, which means the build job runs on an environment where npm is installed, and the deploy job runs on an environment where npm is not installed.
Is there anyway I can run these two jobs separately, but somehow also only need one button in gitlab to start both of these scripts.
Or perhaps force the build job to always re-run before the deploy job runs, or vice versa. And disable the deploy job from being able to run independently of the build job?
Related
Hello there i have a particular question regarding using different images under a single gitlab-ci pipeline job. Initially i am using a maven 3.6.1-jdk-8 because the project runs per those requirements. What i want to do is to use another image maven 3.6.3-jdk-11 before running the script so the sonarqube scanner be able to run properly and does not fails. Both images are required for the job.
sonarqube:
image: maven:3.6.1-jdk-8
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: "0" # Tells git to fetch all the branches of the project, required by the analysis task
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- mvn verify sonar:sonar -Dsonar.qualitygate.wait=true
allow_failure: true
only:
- master
I'm new with Gitlab CI. Every time Gitlab CI run, it replace old folder on server. I have small problem when I want to reduce time Gradle build for project which include DL4J (very big size and take time to build). So I want it keep build folder from last version. I follow this to reduce time build by gradle.
Question: Is that possible to skip some folder by config of gitlab ci to keep it exist. This is my gitlab ci
stages:
- build
something_run:
stage: build
script:
- gradle build
- systemctl restart myproject
tags:
- ml
only:
- master
When it run, gradle will build project and time to build quite long. So I want next time CI run it will not delete last build version.
Take a look at cache (https://docs.gitlab.com/ee/ci/yaml/#cache)
cache is used to specify a list of files and directories which should be cached between jobs.
GitLab CI/CD provides a caching mechanism that can be used to save time when your jobs are running.
See also https://docs.gitlab.com/ee/ci/caching/index.html
I've got a java project I want to implement Gitlab CI into. Everytime I make a commit, I want it to compile the jar (via maven) and then start it on my VPS, and keep it running until the next commit where I update it, ect.
I've tried to do this, but I can't get the deploy stage right. It runs the Jar file, but it runs it as part of the pipeline, instead of backgrounding it, and running it on the VPS.
My .gitlab-ci.yml:
image: maven:3-jdk-8
compile:
stage: build
script:
- "pwd"
- "mvn compile assembly:single"
deploy:
stage: deploy
script:
- "nohup java -jar target/botaco_rewrite-1.0.0-jar-with-dependencies.jar dev.livaco.botaco_rewrite.Botaco &"
It's looks from this line image: maven:3-jdk-8
like you running the job inside a docker container so when the job is done the container is destroyed and all the running process are killed with it.
You should either run the job on the host directly (assuming you running your own gitlab runner) or better to my opinion copy the jar file to another machine and run it there.
P.S.
(you probably want to upload the jar to some artifact repository.)
Fixed this by simply moving to CentOS.
I have a gitlab pipeline running on a windows machine with Windows 7 and powershell 4.0.
The .yaml has the typical 3 stages: build, test and deploy.
For the second stage I want to perform some simple tests that generate a log file which should be available after the test stage finishes.
Here the script section from the test:
script:
- '$exitCode = (start-process C:\app_versions\app_20181211\bin\app.exe -PassThru -Wait).ExitCode'
- 'cat .\TestLogs\BasicFunctionsTestPlan.log'
- 'exit $exitCode'
artifacts:
paths:
- .\TestLogs
expire_in: 1 year
Here I had one problem, after the test run has finished the stage finishes always successfully even if the test themselves failed. Then I had to force the script exit with an error code in case the application tells me that the tests failed.
This caused the second problem: the artifacts link do not get created even they are available (my test produce it anyway).
Probably if I knew how to tell gitlab that the test failed in a more clean way, the artifacts would be available anyway.
I agree that the log file is not an artifact but I would like to keep that file in order to check how the tests have performed, maybe there is a better way to save this file.
Thanks in advance for your help!
EDIT:
Looks like there were more people having the same issue here, maybe it helps understanding better the problem.
I had the same question, but it's easily solved:
You can use artifacts:when to upload artifacts on job failure or despite the
failure.
artifacts:when
source: Gitlab CI yaml reference: artifacts:when
Introduced in GitLab 8.9 and GitLab Runner v1.3.0.
artifacts:when is used to upload artifacts on job failure or despite the
failure.
artifacts:when can be set to one of the following values:
on_success - upload artifacts only when the job succeeds. This is
the default.
on_failure - upload artifacts only when the job
fails.
always - upload artifacts regardless of the job status.
Example:
To upload artifacts only when job fails:
job:
artifacts:
when: on_failure
allow_failure
BTW: you can tell Gitlab CI to continue to the next job after a job failure with allow_failure: true
source: Gitlab CI yaml Reference: allow_failure
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
So combined it could look something like:
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
artifacts:
when: always # or 'on_failure'
paths:
- resulting_artifacts
Currently, I have two main project.
1-) Vue Project which contains (webviews for IOS and Android, websites, and renderer for our Electron ) they are sharing components & API's.
2-) Electron Project which builds desktop app for (windows, darwin, linux)
i would like to automate our building, releasing process. my current setup..
before_script:
- apt-get update
- apt-get install zip unzip
- rm -rf vue-project
- git clone vue-project
- cd vue-project
- git checkout dev
- git pull
- sed -i "/\b\(areaCode\|inline-svg-loader\)\b/d" ./packages/devtool/package.json
- yarn install
- ln -s vue-project/packages/desktop/ web
- npm install
build_darwin:
stage: build
script:
- npm run package -- darwin --deploy
cache:
paths:
- vue-project/node_modules
- node_modules
which basically before bundling electron project it's cloning vue-project install dependencies and bundling electron-renderer then when it's finish. i'm running package.
I would like to separate this two different job from each other. is there anyway i could use artifacts from different project gitlab-CI pipelines ?
any help would be an appreciated.
Gitlab has a API for do a lot of tricks.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test"
for download it as a file.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" -o artifacts.zip "http://gitlab.example.net/api/v4/projects/<projectnumber>/jobs/artifacts/master/download?job=build_desktop
Gitlab can certainly support this. To accomplish this, follow these steps:
ARTIFACT GENERATION
In your Vue Project, modify your job(s) of interest to store artifacts relevant to the Electron project. Each job's artifacts are defined using Gitlab Job Artifacts notation, and are uploaded at job completion to Gitlab, and stored associated to your Project, Branch, and Job.
Note: Branch is often overlooked, and matters when you want to retrieve your artifacts, more on this later.
Illustrating:
Vue Project .gitlab_ci.yml
stages:
- stage1
- ...
vue-job1:
stage: stage1
script:
- echo "vue-job1 artifact1" > ./artifact1
- echo "vue-job1 artifact2" > ./artifact2
artifacts:
when: always
paths:
- ./artifact1
- ./artifact2
expire_in: 90 days
vue-job2:
stage: stage1
script:
# error, would overwrite job1's artifacts since working
# directory is a global space shared by all pipeline jobs
# - echo "vue-job2 artifact1" > ./artifact1
- echo "vue-job2 artifact1" > ./artifact3
artifacts:
when: always
paths:
- ./artifact3
expire_in: 90 days
The artifacts generated above are written to the working directory, which is a clone of your project's repo. So be careful with filename conflicts. To be safe, put your artifacts in a subdirectory (eg: cat "foo" > ./subdir/artifact) and reference them in paths the same way (paths: - ./subdir/artifact). You can use 'ls' in your script to view the working directory.
When your job completes, you can confirm the artifacts stored in Gitlb by using the Gitlab UI. View the job output, and use the Browse button under Job Artifacts on the right panel.
ARTIFACT RETRIEVAL
In your Electron Project, modify your job(s) of interest to retrieve artifacts stored in the Vue Project using the Gitlab Job Artifacts API and curl. In order to access the Vue artifacts, you will need the Vue Project, Branch, and Job that the artifacts were created under.
Project: For Project, use the Project ID displayed in the Gitlab UI Project Details screen.
Branch: Usually master, but depends on the branch your pipeline executes against. Although this is not relevant for your problem, if you are generating and consuming artifacts across executions of the same pipeline, use the Gitlab variable $CI_COMMIT_BRANCH for the branch.
Job: Generally the Job Name that generated the artifacts for your Project. But if you need artifacts produced by a specific Job, then use the Job Number and the corresponding retrieval API.
Illustrating:
Electron Project .gitlab_ci.yml
stages:
- stage1
- ...
electron-job1:
stage: stage1
script:
- curl -o ./artifact1 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact1?job=vue-job1
- curl -o ./artifact2 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact2?job=vue-job1
- curl -o ./artifact3 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact3?job=vue-job2
This script retrieves artifacts individually to the working directory of your Electron Project. There are also options for retrieving all artifacts for your job at once as a zip archive.
MISCELLANEOUS
Although this is not in the problem posed, it is worth noting that you can use artifacts both within the lifespan of a single pipeline execution to pass information between jobs. You can also use this to pass information across pipeline executions within the same project.
With recent versions of gitlab, this can be achieved simply by using either the multi-project pipeline feature (then starting a pipeline in one project triggers a build from the other project): see the documentation
Or you can also use the "needs:project" mechanism which allows one job to download artifacts from other pipelines (see the documentation Use needs:project to download artifacts from up to five jobs in other pipelines.)