I am stuck trying to let GitLab CI run a shell file that builds Unity Player. It works when I do it but not when GitLab Tries it - shell

The Problem
I used a tutorial to create a shell file that would build an Unity player for me. Tutorial I used. But when I try to let GitLab's CI call this shell script it seems only clean-up the directory and just ignore the Unity build command. Console output
My .yml file looks like this: (Important is the build-job: section all the other stuff works)
stages: # List of stages for jobs, and their order of execution
- build
- test
build-job: # This job runs in the build stage, which runs first.
stage: build
script:
- echo "Compiling the code..."
- ./Build/build.sh
- echo "Compile complete."
unit-test-job: # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- echo "Running unit tests... This will take about 60 seconds."
- C:/"Program Files"/Unity/Hub/Editor/2020.3.20f1/Editor/Unity.exe \
-batchmode \
-projectPath C:/Users/joshu/Desktop/Gamedev/Testing/testing-asignment-2/"Testing_Asignment 2" \
-runTests -testPlatform editmode \
-logFile . \
-testResults ./unit-tests.xml \
-quit
- echo "Code is tested"
lint-test-job: # This job also runs in the test stage.
stage: test # It can run at the same time as unit-test-job (in parallel).
script:
- echo "Linting code... This will take about 10 seconds."
- echo "No lint issues found."
My shell script contains this:
echo Cleaning Up Build Folder
rm -rf ./Build/Builds
echo Starting Build Process
C:/"Program Files"/Unity/Hub/Editor/2020.3.20f1/Editor/Unity.exe -quit -projectPath ../"Testing Asignment 2" -executeMethod Building.MyBuildScript.PreformBuild
echo Ended Build Process
My building script contains this:
using System;
using System.IO;
using UnityEditor;
using UnityEditor.Build.Reporting;
namespace Building
{
public class MyBuildScript
{
public static void PreformBuild()
{
BuildPlayerOptions buildPlayerOptions = new BuildPlayerOptions();
buildPlayerOptions.scenes = new[] { "Assets/Scenes/SampleScene.unity" };
buildPlayerOptions.locationPathName = "../Build/Builds/WindowsBuild/Windows64Build.x86_64";
buildPlayerOptions.target = BuildTarget.StandaloneWindows64;
buildPlayerOptions.options = BuildOptions.None;
BuildReport report = BuildPipeline.BuildPlayer(buildPlayerOptions);
using StreamWriter writer = File.CreateText("../Build/Builds/WindowsBuild/results.txt");
writer.Write(
$"Build result: {report.summary.result} \n" +
$"Process time: \n" +
$" start: {report.summary.buildStartedAt} \n" +
$" end: {report.summary.buildEndedAt} \n" +
$" total: {report.summary.totalTime} \n" +
$"{report.summary.totalErrors} Errors found{(report.summary.totalErrors > 0 ? "!" : "")}");
}
}
}
I think that there is some thing wrong with the access of my Unity.exe but when I open security settings on windows the excess to write/execute is enabled.
Things I tried
I also tried just not using a shell script and calling the MyBuildScript from the .yml directly.
build-job: # This job runs in the build stage, which runs first.
stage: build
script:
- echo "Compiling the code..."
- C:/"Program Files"/Unity/Hub/Editor/2020.3.20f1/Editor/Unity.exe -quit -projectPath ./"Testing Asignment 2" -executeMethod Building.MyBuildScript.PreformBuild
- echo "Compile complete."
But this also seemed to not execute Unity and just skip the command.
Furthermore I tried using the -buildWindows64Player instead of the -executeMethod.
build-job: # This job runs in the build stage, which runs first.
stage: build
script:
- echo "Compiling the code..."
- C:/"Program Files"/Unity/Hub/Editor/2020.3.20f1/Editor/Unity.exe -quit -projectPath ./"Testing Asignment 2" -buildWindows64Player ./Build/Builds/WindowsBuild
- echo "Compile complete."
But this also seemed have skipped Unity. However this way of trying did take the longest to finish. All the other options were done in a few seconds but doing it like this it took about a minute or two. I don't know why but if I had to guess it would be because it actually did start-up Unity and failed somewhere on the way.
Note
I know that I should add -batchmode to the shell commands but first I want to see Unity open so that I know it is doing something.
Little update as of 12-29-2021 23:48. I noticed that the unit-test also wasn't realy working but I fixed this by changing my Unity version to 2021.2.7f1, changing the Unity project folder structure and updating the unit-test section to this:
unit-test-job: # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- echo "Running unit tests... This will take about 60 seconds."
- C:/"Program Files"/Unity/Hub/Editor/2021.2.7f1/Editor/Unity.exe -batchmode -projectPath C:/Users/joshu/Desktop/Gamedev/Testing/testing-asignment-2/"Testing Asignment 2" -runTests -testPlatfrom editmode -logFile -testResults ./unit-tests.xml | Out-Default

Well turns out I did something stupid. I had my runner installed on my laptop and kept trying to push new changes from my pc. However I used the projectPath that pointed to my local project on the laptop and not the project that the runner was cloning from GitLab. So the -projectPath now points to ./"Testing Asignment 2". This also was the reason I thought changing my Unity version helped because I pulled the project to my laptop while installing a new version.
I also discoverd that my test needed to have "| Out-Default" behind it to actually sort of read the test data. StackOverflow page that helped with this
I don't realy use the shell script anymore now but I just wanted it building Unity, and that works!
The .yml file I am using right now looks like this:
stages: # List of stages for jobs, and their order of execution
- build
- test
build-job: # This job runs in the build stage, which runs first.
stage: build
script:
- echo "Compiling the code..."
- C:/"Program Files"/Unity/Hub/Editor/2021.2.7f1/Editor/Unity.exe -batchmode -quit -projectPath ./"Testing Asignment 2" -executeMethod Building.MyBuildScript.PreformBuild | Out-Default
- echo "Compile complete."
unit-test-job: # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- echo "Running unit tests... This will take about 60 seconds."
- C:/"Program Files"/Unity/Hub/Editor/2021.2.7f1/Editor/Unity.exe -batchmode -quit -projectPath ./"Testing Asignment 2" -runTests -testPlatfrom editmode -logFile -testResults ./unit-tests.xml | Out-Default
(I also changed my gitignore to not ignore Library\ScriptAssemblies from Unity. But I don't know if this is needed.)

Related

How to use an anchor to prevent repetition of code sections?

Say I have a number of jobs that all do similar series of scripts, but need a few variables that change between them:
test a:
stage: test
tags:
- a
interruptible: true
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
script:
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t a -f Dockerfile.a .
test b:
stage: test
tags:
- b
interruptible: true
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
script:
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t b -f Dockerfile.b .
All I need is to be able to define e.g.
- docker build -t ${WHICH} -f Dockerfile.${which} .
If only I could make an anchor like:
.x: &which_ref
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t $WHICH -f Dockerfile.$WHICH .
And include it there:
test a:
script:
- export WHICH=a
<<: *which_ref
This doesn't work and in a yaml validator I get errors like
Error: YAMLException: cannot merge mappings; the provided source object is unacceptable
I also tried making an anchor that contains some entries under script inside of it:
.x: &which_ref
script:
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t $WHICH -f Dockerfile.$WHICH .
This means I have to include it from one step higher up. So this does not error, but all this accomplishes is cause the later declared script section to override the first one.
So I'm losing hope. It seems like I will just need to abstract the sections away into their own shell scripts and call them with arguments or whatever.
The YAML merge key << is a non-standard extension for YAML 1.1, which has been superseded by YAML 1.2 about 14 years ago. Usage is discouraged.
The merge key works on mappings, not on sequences. It cannot deep-merge. Thus what you want to do is not possible to implement with it.
Generally, YAML isn't designed to process data, it just loads it. The merge key is an outlier and didn't find its way into the standard for good reasons. You need a pre- or postprocessor to do complex processing, and Gitlab CI doesn't offer anything besides simple variable expension, so you're out of luck.

How do I assign exe output to a variable in gitlab ci scripts?

When running my gitlab ci I need to check whether a specified svn directory exists.
I was using the script:
variables:
DIR_CHECK: "default"
stages:
- setup
- test
- otherDebugJob
.csharp:
only:
changes:
- "**/*.cs"
- "**/*.js"
setup:
script:
- $DIR_CHECK = $(svn ls https://server.fsl.local:port/svn/myco/personal/TestNotReal --depth empty)
- echo $DIR_CHECK
test:
script:
- echo "DIR_CHECK is blank"
- echo $DIR_CHECK
rules:
- if: $DIR_CHECK == ''
otherDebugJob:
script:
- echo "DIR_CHECK is not blank"
- echo $DIR_CHECK
rules:
- if: $DIR_CHECK != ''
the svn command works and echos back the correct reply but $DIR_CHECK does not get set to anything but the original default. It does not store the returned string from the svn command.
How do I store the returned string from an exe in a variable in gitlab ci?
Test run:
Executing "step_script" stage of the job script 00:00 $ $DIR_CHECK =
$(svn ls https://server.fsl.local:port/svn/myco/personal/TestNotReal
--depth empty) svn: E170000: Illegal repository URL https://server.fsl.local:port/svn/myco/personal/TestNotReal' $ echo
$DIR_CHECK Cleaning up file based variables 00:01 Job succeeded
Passing variables between jobs
Unfortunately, you cannot use DIR_CHECK variable the way you described. List of steps to be executed generates before steps actually runs, that means for all of the steps DIR_CHECK will be equal to default. First of all there are few tips how you can pass variables between jobs:
First way
You can add desired command to the before_script section in your .csharp template:
.csharp:
before_script:
- export DIR_CHECK=$(svn ls https://server.fsl.local:port/svn/myco/personal/TestNotReal --depth empty)
and extend other steps with this .csharp.
Second way
You can pass variables between jobs with job artifacts:
setup:
stage: setup
script:
- DIR_CHECK=$(svn ls https://server.fsl.local:port/svn/myco/personal/TestNotReal --depth empty)
- echo "DIR_CHECK=$DIR_CHECK" > dotenv_file
artifacts:
reports:
dotenv:
- dotenv_file
Thirds way
You can trigger or use parent/child pipelines to pass variables into pipelines.
staging:
variables:
DIR_CHECK: "you are awesome, guys!"
stage: deploy
trigger: my/deployment
In the triggered pipeline your variable will exists at the very start moment, and all the rules will be applied correctly.
Solution
In your case, if you really don't want to include otherDebugJob step in your pipeline you can do the following:
First approach
This is quite easy way and this will work, but looks like not a best practice. So, we are already know how to pass our DIR_CHECK variable from setup step , just add some check in the test step script block:
script:
- |
if [ -z "$DIR_CHECK" ]; then
exit 0
fi
- echo "DIR_CHECK is blank"
- echo $DIR_CHECK
Do the almost same thing for the otherDebugJob but check if DIR_CHECK is not empty with if [ -n "$DIR_CHECK" ].
This approach is helpful when your pipeline not contains a lot of steps, but after the test and otherDebugJob follows another few steps.
Second approach
You can fail your setup step and then handle this fail in otherDebugJob step:
setup:
script:
- DIR_CHECK=$(svn ls https://server.fsl.local:port/svn/myco/personal/TestNotReal --depth empty)
- |
if [ -z "$DIR_CHECK" ]; then
exit 1
fi
otherDebugJob:
script:
- echo "DIR_CHECK is not blank"
when: on_failure
This approach is useful if you only want to make some debug stuff after this setup step. After all on_failure jobs, pipeline will be marked as Failed and stopped.

Variable expansion of trigger branch property prevents downstream pipeline from being created

A branch job in which the branch property of the trigger property is using a variable will always fail with reason: downstream pipeline can not be created.
Steps to reproduce
Set up a downstream pipeline with a trigger property as you would normally.
Add a branch property to the trigger property. Write the name of an existing branch on the downstream repository, like master/main or the name of a feature branch.
Run the pipeline and observe that the downstream pipeline is successfully created.
Now change the branch property to use a variable instead, like branch: $CI_TARGET_BRANCH.
Manually run the CI pipeline with that, setting variable through the GitLab GUI.
The job will instantly fail with reason: downstream pipeline can not be created.
Code example
The goal is to create a GitLab CI config that runs the pipeline of a specified downstream branch. The bug occurs when attempting to do it with a variable.
This works, creating a downstream pipeline like normal. But the branch name is hardcoded:
stages:
- deploy
deploy:
variables:
environment: dev
stage: deploy
trigger:
project: group/project
branch: foo
strategy: depend
This does not work; although TARGET_BRANCH is set successfully, the job fails because the downstream pipeline can not be created:
stages:
- removeme
- deploy
before_script:
- if [ -z "$TARGET_BRANCH" ]; then TARGET_BRANCH="main"; fi
- echo $TARGET_BRANCH
test_variable:
stage: removeme
script:
- echo $TARGET_BRANCH
deploy:
variables:
environment: dev
stage: deploy
trigger:
project: group/project
branch: $TARGET_BRANCH
strategy: depend
If you know what I'm doing wrong, or you have something that does work with variable expansion of the branch property, please share it (along with your GitLab version). Alternate solutions are also welcome, but this one seems like it should work.
GitLab Version on which bug occurs
Self-hosted GitLab Community Edition 12.10.7
What is the current bug behavior?
The job always fails for reason: downstream pipeline can not be created.
What is the expected correct behavior?
The branch property should be set to the value of the variable and the downstream pipeline should be created as normal, just as if you simply hardcoded/typed the name of the branch.
More details
The ability to use variable expansion in the trigger branch property was added in v12.4, and it's explicitly mentioned in the docs.
I searched for other .gitlab-ci.yml / GitLab config files. Every single one that attempted to use variable expansion in the branch property had it commented out, saying it was bugged for an unknown reason (example.
I haven't been able to find a repository in which someone claimed to have a working variable expansion for the branch property of the trigger property.
Unfortunately, the alternate solutions are either (a) hardcoding every downstream branch name into the GitLab CI config of the upstream project, or (b) not being able to test changes to the downstream GitLab CI config without first committing them to master/main, or having to use only/except.
TL;DR: How to use the value of a variable for the branch property of a bridge job? My current solution makes it so the job fails and the downstream pipeline isn't created.
this is a 'works as designed', and gitlab will improve in upcoming releases.
trigger job will pretty weak b/c it is not a full job that runs on a runner. Therefore most of the trigger configuration needs to be hardcoded.
I use direct API calls to trigger downstream jobs passing the CI_JOB_TOKEN which links the upstream job to downstream as the trigger does
API calls give you full control
curl -X POST \
-s \
-F token=${CI_JOB_TOKEN} \
-F "ref=${REF_NAME}" \
-F "variables[STAGE]=${STAGE}" \
"${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/trigger/pipeline"
now this will not wait and monitor for when the job is done so you will need to code for that if you need to wait for the downstream job to finish,
Moreover, CI_JOB_TOKEN cannot be used to get the status of the downstream job, so you will another token for that.
- |
DOWNSTREAM_RESULTS=$( curl --silent -X POST \
-F token=${CI_JOB_TOKEN} \
-F "ref=${DOWNSTREAM_PROJECT_REF}" \
-F "variables[STAGE]=${STAGE}" \
-F "variables[SLS_PACKAGE_PATH]=.serverless-${STAGE}" \
-F "variables[INVOKE_SLS_TESTS]=false" \
-F "variables[UPSTREAM_PROJECT_REF]=${CI_COMMIT_REF_NAME}" \
-F "variables[INSTALL_SLS_PLUGINS]=${INSTALL_SLS_PLUGINS}" \
-F "variables[PROJECT_ID]=${CI_PROJECT_ID}" \
-F "variables[PROJECT_JOB_NAME]=${PROJECT_JOB_NAME}" \
-F "variables[PROJECT_JOB_ID]=${PROJECT_JOB_ID}" \
"${CI_SERVER_URL}/api/v4/projects/${DOWNSTREAM_PROJECT_ID}/trigger/pipeline" )
echo ${DOWNSTREAM_RESULTS} | jq .
DOWNSTREAM_PIPELINE_ID=$( echo ${DOWNSTREAM_RESULTS} | jq -r .id )
echo "Monitoring Downstream pipeline ${DOWNSTREAM_PIPELINE_ID} status..."
DOWNSTREAM_STATUS='running'
COUNT=0
PIPELINE_API_URL="${CI_SERVER_URL}/api/v4/projects/${DOWNSTREAM_PROJECT_ID}/pipelines/${DOWNSTREAM_PIPELINE_ID}"
echo "Pipeline api endpoint => ${PIPELINE_API_URL}"
while [ ${DOWNSTREAM_STATUS} == "running" ]
do
if [ $COUNT -eq 0 ]
then
echo "Starting loop"
fi
if [ ${COUNT} -ge 350 ]
then
echo 'TIMEOUT!'
DOWNSTREAM_STATUS="TIMEOUT"
break
elif [ $(( ${COUNT} % 60 )) -eq 0 ]
then
echo "Downstream pipeline status => ${DOWNSTREAM_STATUS}"
echo "Count => ${COUNT}"
sleep 10
else
sleep 10
fi
DOWNSTREAM_CALL=$( curl --silent --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" ${PIPELINE_API_URL} )
if [ $COUNT -eq 0 ]
then
echo ${DOWNSTREAM_CALL} | jq .
fi
DOWNSTREAM_STATUS=$( echo ${DOWNSTREAM_CALL} | jq -r .status )
COUNT=$(( ${COUNT} + 1 ))
done
#pipeline status is running, failed, success, manual
echo "PIPELINE STATUS => ${DOWNSTREAM_STATUS}"
if [ ${DOWNSTREAM_STATUS} != "success" ]
then
exit 2
fi

How to trigger gitlab ci when a comment like "test please" is written in merge request discussion?

I saw there are several ways to trigger the CI.
Even for merge requests
https://docs.gitlab.com/ee/ci/merge_request_pipelines/
What I want to do is to trigger a gitlab CI pipeline not for all merge request and not for all commits.
Only when someone comments:
'test please' or 'test gitlab' or some special keyword maybe defined by regex?
Is this possible?
That was requested in gitlab-org/gitlab-foss issue 39215, and shipped with 11.0
rspec:
script: ...
only:
variables:
- $CI_COMMIT_MESSAGE =~ /some-regexp/
You also have the following workaround for pipelines (windows cmd shell):
Process the job only if commit message doesn't contain [CI Release]
script:
- git show -s --format=%%B | findstr /C:"[CI Release]" >nul 2>&1 && (exit 0) || (set errorlevel=0)
- cd beUcb
- call mvn -N -Pver resources:resources
- REM ... rest of script ...

Gitlab-ci: extend script section

I have an unity ci-project.
.gitlab-ci.yml contains base .build job with one script command. Also I have multiple specified jobs for build each platform which extended base .build. I want to execute some platform-specific commands for android, so I have created separated job generate-android-apk. But if it's failing the pipeline will be failed too.(I know about allow_failure). Is it possible to extend script section between jobs without copy-pasting?
UPDATE:
since gitlab 13.9 it is possible to use !reference tags from other jobs or "templates" (which are commented jobs - using dot as prefix)
actual_job:
script:
- echo doing something
.template_job:
after_script:
- echo done with something
job_using_references_from_other_jobs:
script:
- !reference [actual_job, script]
after_script:
- !reference [.template_job, after_script]
Thanks to #amine-zaine for the update
FIRST APPROACH:
You can achieve modular script sections by utilizing 'literal blocks' (using |) like so:
.template1: &template1 |
echo install
.template2: &template2 |
echo bundle
testJob:
script:
- *template1
- *template2
See Source
ANOTHER SOLUTION:
Since GitLab 11.3 it is possible to use extend which could also work for you.
.template:
script: echo test template
stage: testStage
only:
refs:
- branches
rspec:
extends: .template1
after_script:
- echo test job
only:
variables:
- $TestVar
See Docs
More Examples

Resources