This question is very specific to Bluemix DevOps.
I have a Java backend application that has a sizeable JavaScript front-end. So I created a GRUNT task to do the needed: uglify, minify, CDNify etc. My current setup is to have the Bluemix build just running mvn -B package and the Grunt task beforehand as a script on my local machine:
#!/bin/bash
grunt build
git add --all
git commit
git push origin master
But that precludes any edit using the online editor. So I'd like to have both task to run by the pipeline. I see 3 options:
Run both tasks in one build block triggered by git push as separate tasks
Run them in one build script triggered by git push
Run 2 pipeline steps, the first triggered by git push, the second by the completion of the first
something else
I haven't tried it yet (shame on me), just wanted to ask if someone did that before (If yes - cool, if no I will post my findings later on)
Solved it. This is what I tried:
modify the script in build and prefix with npm install npm or mvn (depends on what I selected) wasn't found)
add 2 jobs to one build stage, one grunt one maven (the deploy task would not find the war file)
Use 2 pipeline stages (see picture below) : Horray --- that worked.
None of the build steps required setting a directory, which is a little trap, since mvn sets target as default directory, so remove this. The script for Bower/Grunt is this:
#!/bin/bash
npm install
grunt build
the script for the maven task:
#!/bin/bash
mvn -B package
works like a charm (just be careful not to add npm modules you don't actually need, it slows the build quite a bit)
Related
I am running an azure pipeline for a maven project which has windows commands for maven installation, by calling multiple methods i.e.,
call mvn ... clean install
call mvn ... clean package (authentication)
call mvn ... clean package (restapi)
The build environment is linux Hence I am converting all the commands in the batch file to sh commands. For maven installation I initially added a maven installation task by mentioning the path and commands in the task. This failed.
So I am currently changing the windows commands to sh commands. The other commands except for maven installation have been converted using a batch to sh commands reference article.
Could some one guide me to convert the above mentioned installation commands to sh commands?
If the Apache Maven has been installed and added to the system path on the agent machine where your pipeline runs, you can just try directly calling the mvn command in the bash script, like you call it on Windows.
For example:
mvn ... clean install
mvn ... clean package (authentication)
mvn ... clean package (restapi)
Before you execute the bash script in the pipeline, you firstly should try and debug it on your local machine to make sure it can work as expected on the local machine. Then move the bash script to the pipeline on Azure DevOps.
If the bash script runs failed in the pipeline, for us to investigate this issue further, please share us with the complete debug logs of the failed pipeline run. To get the debug logs, you need to set the pipeline variable System.Debug to true, then trigger the pipeline.
From the official document from GitLab, it seems that a CI/CD stage can only be skipped based on if a file is changed. Is it possible to skip a step based on if a file / folder exists on the machine to be deployed?
The case is that it is common to use a package management tool. (e.g. composer in PHP or NPM in Node.js) Currently the rule is to check if the respective configuration file is changed (e.g. composer.json for composer or package.json for NPM) to see if it is necessary to run the install step (i.e. composer install or npm install) However, although it seldom happens, when a new machine is used for deployment, the CI/CD would crash because the install step is skipped.
Currently, the problem is solved by manually trigger the install step, but is it possible to auto detect if the file exists at the hosting machine to determine if the install step should be run?
I don't think that there is a way to do what you describe. The only/except feature in GitLab runs on the GitLab server, before any jobs are created. To be able to know if a file already exists on a server, there needs to be a job created and it needs to be assigned to a gitlab runner.
They have this issue, where it looks like it will be possible to start a job and then determine that it should be skipped by using exit codes:
https://gitlab.com/gitlab-org/gitlab/issues/16733
In the mean time, you can make it so that your install job always runs, and then just use your build scripts to check for this file. From the pipeline view in GitLab, you will always see your install step. The actual job can just skip to do anything if the file already exists, and then you don't need to remember to run the job manually.
I am trying to wrap my head around Travis-ci scripts but I am trying to figure out if what I want is even possible:
Repo #1:
- my app to be built via travis-ci
Repo #2:
- contains selenium/nightwatch tests that should run once repo #1 is finished building
So this is the workflow I am aiming for:
a PR in Repo #1 gets merged
Travis-ci builds Repo #1 and completes successfully & deploys
Travis-ci tells Repo #2 to start building
Repo #2 installs the repo which triggers browserstack to begin running the selenium tests
Is this at all possible? I've been researching this for a few days and couldn't find a way to trigger a separate repo to build.
Any help is appreciated.
Thanks!
Possible Duplicate of: Triggering builds of dependent projects in Travis CI
Nevertheless, pasting the answer here: Yes, it is possible to trigger another Travis job after a first one succeeds. You can use the trigger-travis.sh script that is part of the plume-lib library.
The script's documentation tells how to use it -- set an environment variable and add three lines to your .travis.yml file.
You can setup repo#1 and repo#2 using different jobs and use the above approach to trigger downstream job after successfully deployment has completed using job 1
Currently, I have two main project.
1-) Vue Project which contains (webviews for IOS and Android, websites, and renderer for our Electron ) they are sharing components & API's.
2-) Electron Project which builds desktop app for (windows, darwin, linux)
i would like to automate our building, releasing process. my current setup..
before_script:
- apt-get update
- apt-get install zip unzip
- rm -rf vue-project
- git clone vue-project
- cd vue-project
- git checkout dev
- git pull
- sed -i "/\b\(areaCode\|inline-svg-loader\)\b/d" ./packages/devtool/package.json
- yarn install
- ln -s vue-project/packages/desktop/ web
- npm install
build_darwin:
stage: build
script:
- npm run package -- darwin --deploy
cache:
paths:
- vue-project/node_modules
- node_modules
which basically before bundling electron project it's cloning vue-project install dependencies and bundling electron-renderer then when it's finish. i'm running package.
I would like to separate this two different job from each other. is there anyway i could use artifacts from different project gitlab-CI pipelines ?
any help would be an appreciated.
Gitlab has a API for do a lot of tricks.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test"
for download it as a file.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" -o artifacts.zip "http://gitlab.example.net/api/v4/projects/<projectnumber>/jobs/artifacts/master/download?job=build_desktop
Gitlab can certainly support this. To accomplish this, follow these steps:
ARTIFACT GENERATION
In your Vue Project, modify your job(s) of interest to store artifacts relevant to the Electron project. Each job's artifacts are defined using Gitlab Job Artifacts notation, and are uploaded at job completion to Gitlab, and stored associated to your Project, Branch, and Job.
Note: Branch is often overlooked, and matters when you want to retrieve your artifacts, more on this later.
Illustrating:
Vue Project .gitlab_ci.yml
stages:
- stage1
- ...
vue-job1:
stage: stage1
script:
- echo "vue-job1 artifact1" > ./artifact1
- echo "vue-job1 artifact2" > ./artifact2
artifacts:
when: always
paths:
- ./artifact1
- ./artifact2
expire_in: 90 days
vue-job2:
stage: stage1
script:
# error, would overwrite job1's artifacts since working
# directory is a global space shared by all pipeline jobs
# - echo "vue-job2 artifact1" > ./artifact1
- echo "vue-job2 artifact1" > ./artifact3
artifacts:
when: always
paths:
- ./artifact3
expire_in: 90 days
The artifacts generated above are written to the working directory, which is a clone of your project's repo. So be careful with filename conflicts. To be safe, put your artifacts in a subdirectory (eg: cat "foo" > ./subdir/artifact) and reference them in paths the same way (paths: - ./subdir/artifact). You can use 'ls' in your script to view the working directory.
When your job completes, you can confirm the artifacts stored in Gitlb by using the Gitlab UI. View the job output, and use the Browse button under Job Artifacts on the right panel.
ARTIFACT RETRIEVAL
In your Electron Project, modify your job(s) of interest to retrieve artifacts stored in the Vue Project using the Gitlab Job Artifacts API and curl. In order to access the Vue artifacts, you will need the Vue Project, Branch, and Job that the artifacts were created under.
Project: For Project, use the Project ID displayed in the Gitlab UI Project Details screen.
Branch: Usually master, but depends on the branch your pipeline executes against. Although this is not relevant for your problem, if you are generating and consuming artifacts across executions of the same pipeline, use the Gitlab variable $CI_COMMIT_BRANCH for the branch.
Job: Generally the Job Name that generated the artifacts for your Project. But if you need artifacts produced by a specific Job, then use the Job Number and the corresponding retrieval API.
Illustrating:
Electron Project .gitlab_ci.yml
stages:
- stage1
- ...
electron-job1:
stage: stage1
script:
- curl -o ./artifact1 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact1?job=vue-job1
- curl -o ./artifact2 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact2?job=vue-job1
- curl -o ./artifact3 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact3?job=vue-job2
This script retrieves artifacts individually to the working directory of your Electron Project. There are also options for retrieving all artifacts for your job at once as a zip archive.
MISCELLANEOUS
Although this is not in the problem posed, it is worth noting that you can use artifacts both within the lifespan of a single pipeline execution to pass information between jobs. You can also use this to pass information across pipeline executions within the same project.
With recent versions of gitlab, this can be achieved simply by using either the multi-project pipeline feature (then starting a pipeline in one project triggers a build from the other project): see the documentation
Or you can also use the "needs:project" mechanism which allows one job to download artifacts from other pipelines (see the documentation Use needs:project to download artifacts from up to five jobs in other pipelines.)
I have a Jenkins job that uses a script to build my project. On the following line, the script fails mvn -e -X -Dgit='$git' release:prepare.
Because I want to search for the cause of this, I want to go to the Jenkins server and run mvn -e -X -Dgit='$git' release:prepare from the command line, to see if it works.
Does Jenkins store the projects' source code somewhere, such that I can go to that folder and call Maven?
If yes, then where?
Yes, It Stores the project files for the job by default at
/var/lib/jenkins/workspace/{your-job-name}
This is where jenkins suppose the project files to be present or it pulls it from a source before start working/building from it.
Quote from Andrew M.:
"Hudson/Jenkins doesn't quite work that way. It stores configurations and job information in /var/lib/jenkins by default (if you're using the .deb package). If you want to setup persistence for a specific application, that's something you'll want to handle yourself - Hudson is a continuous integration server, not a test framework.
Check out the Wiki article on Continuous Integration for an overview of what to expect."
From this Question on serverfault.
This worked for me:
/var/jenkins/workspace/JobNameExample
but, if your build machine (node) is a different than the one where Jenkins is running (manager), You need specify it:
/var/jenkins/workspace/JobNameExample/label/NodeName
Where you can define label too:
jenkins stores its workspace files currently in /var/jenkins_home/workspace/project_name
I am running from docker though!