GITLAB CI/CD Improving Pipeline Speed with cache or Artifacts? - caching

Question

I really don't understand how cache works and how i can really speed up my CI/CD. I would like to just make it run in less than 2 minutes but all i got is a 30m/40m/1h pipeline.
I have already tried a lot of caching techniques (Global cache, per job cache)
I am not going to write all of them here (i am working on this 3 days by now so i tried a lot of stuff and read a lot of stuff too..).
The project is a .NET MVC Project with some static bundling through NPM.
The CI/CD is running on a Windows Server 2008 R2.
No docker images. Just gitlab-runner inside the C:\gitlab_runner folder and all the tools are natively installed on the machine tagged as deploy-vmdev
(The project is called Iom.Site and it is saved inside the Iom.Site folder)
The best i was able to do was this (but it runs on average of 30m and sometimes it breaks up because depedencies caches, node_modules and packages, aren't present.):
stages:
- install_dependencies
- build_app
- deploy
#Execute dependencies only on package-lock.json e packages.config changes
install_dependencies:
stage: install_dependencies
only:
changes:
- Iom.Site/package-lock.json
- Iom.Site/packages.config
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- Iom.Site/node_modules
- packages/
tags:
- deploy-vmdev
script:
- if(!((nuget sources) -like "*Promoscience*").Length -gt 0) { nuget sources add -Name Promoscience -Source http://nupromo.promoscience.com/nuget };
- nuget restore
- cd Iom.Site
- npm install
#Bundling through gulp npm packages static files (SCSS, JS,..)
#Copying .\site\content inside .\dist folder at root for packaging
build_static_files:
stage: build_app
only:
changes:
- Iom.Site\scripts\bundle\**\*
- Iom.Site\scripts\react\**\*
- Iom.Site\scss\**\*
- ".gitlab-ci.yml"
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- Iom.Site/node_modules
policy: pull
tags:
- deploy-vmdev
script:
- gulp build
- cd ..
- Copy-Item Iom.Site\content .\dist\
artifacts:
paths:
- .\dist
when: on_success
expire_in: 30 days
#Building .NET App through MSBUILD. Msbuild profile saves the output in .\dist folder
#for packaging the whole app
build_app:
stage: build_app
only:
changes:
- "**/*.cs"
- "**/*.cshtml"
- "**/*.config"
- ".gitlab-ci.yml"
tags:
- deploy-vmdev
script:
- msbuild Iom.Site\Iom.Site.csproj /p:Configuration=test /p:PublishProfile=FolderProfile /p:transformConfigFiles=true /p:deployOnBuild=true
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- packages/
policy: pull
artifacts:
paths:
- .\dist
when: on_success
expire_in: 30 days
#Deploying: Downloading artifacts and merging them.
#Copying the content of dist folder inside the IIS Site
deploy_test:
stage: deploy
dependencies:
- build_static_files
- build_app
tags:
- deploy-vmdev
only:
- master
- dev
cache: {}
variables:
SiteAppPath: "E:\\websites\\CnrIom2020"
script:
- echo $env:SiteAppPath
- dir .\dist
- Copy-Item .\dist\* $env:SiteAppPath -Recurse -Force
UPDATE 1:
In the end i found this on gitlab ci wiki:
Caching is an optimization, but it isn’t guaranteed to always work. You need to be prepared to regenerate any cached files in each job that needs them.
And changed my pipeline in this way:
stages:
- build_app
- package
- deploy
build_static_files:
stage: build_app
before_script:
- cd Iom.Site
- if(-Not(Test-Path node_modules)){npm install} #Installiamo le dipendenze di npm solo se non presenti
only:
changes:
- Iom.Site\scripts\bundle\**\*
- Iom.Site\scripts\react\**\*
- Iom.Site\scss\**\*
- ".gitlab-ci.yml"
tags:
- deploy-vmdev
cache:
key: static_${CI_COMMIT_REF_SLUG}
paths:
- Iom.Site/node_modules
policy: pull-push
script:
- gulp build
- cd ..
- Copy-Item Iom.Site\content .\dist\content -Recurse -Force #Copiamo i file statici in dist
artifacts:
paths:
- .\dist
when: on_success
expire_in: 2 hours
build_app:
stage: build_app
before_script: #Installiamo le dipendenze di Nuget solo se già non presenti
- if(-Not(Test-Path packages)){ nuget restore }
only:
changes:
- "**/*.cs"
- "**/*.cshtml"
- "**/*.config"
- ".gitlab-ci.yml"
tags:
- deploy-vmdev
cache:
key: app_${CI_COMMIT_REF_SLUG}
paths:
- packages
policy: pull-push
script:
- msbuild Iom.Site\Iom.Site.csproj /p:Configuration=test /p:PublishProfile=FolderProfile /p:transformConfigFiles=true /p:deployOnBuild=true
artifacts:
paths:
- .\dist
when: on_success
expire_in: 2 hours
package_app:
stage: package
dependencies:
- build_static_files
- build_app
tags:
- deploy-vmdev
only:
- master
- dev
- release
cache: { }
script:
- dir .\dist
artifacts:
paths:
- .\dist
expire_in: never
deploy_test:
stage: deploy
dependencies:
- package_app
tags:
- deploy-vmdev
only: #TODO: When: manual
- master
- dev
- release
cache: {}
variables:
SiteAppPath: "E:\\websites\\CnrIom2020"
script:
- echo $env:SiteAppPath
- dir .\dist
- Copy-Item .\dist\* $env:SiteAppPath -Recurse -Force
# deploy_release: To define