Github Actions: No such file or Directory error - windows

Background
I am trying to get a Github Action working with Windows and Bakeware because I am trying to create a release using it.
However, I am having an unexpected issue with creating folders.
Code
Following is a github action that is a Minimal Working Example of the issue.
Basically, all steps work, except for the creation of a single folder:
name: build
env:
MIX_ENV: test
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
name: Build on Windows
runs-on: windows-2019
env:
CC: gcc
MAKE: make
steps:
- uses: actions/checkout#v2
- uses: erlef/setup-beam#v1
with:
elixir-version: '1.13.x' # Define the elixir version [required]
otp-version: '24.2.x' # Define the OTP version [required]
- name: Install choco
shell: powershell
run: |
Set-ExecutionPolicy -ExecutionPolicy Bypass
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
- name: Install bakeware dependencies
shell: powershell
run: choco install -y zstandard make
- name: Install Dependencies
shell: powershell
run: mix deps.get
- name: Run credo code analyser
shell: powershell
run: mix credo --strict
I am even using powershell to do it (even though I am not really sure if this is needed).
Problem
However my GitHub Actions code comes back with this error:
==> bakeware
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/launcher"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"
mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress] Error 1
could not compile dependency :bakeware, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile bakeware", update it with "mix deps.update bakeware" or clean it with "mix deps.clean bakeware"
** (Mix) Could not compile with "make" (exit status: 2).
For the careful reader, you will notice the part saying:
mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
Research
By changing the GitHub Action a little and adding an additional step, I was able to inspect some behaviour:
- name: Compile everything
shell: powershell
run: |
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/launcher"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"
ls .\_build\test\lib\bakeware\obj\zstd\lib\decompress
Test-Path -Path "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"
mix deps.compile --all
mix compile
env:
CC: C:\ProgramData\Chocolatey\bin\gcc.exe
MAKE: C:\ProgramData\Chocolatey\bin\make.exe
In this snippet I manually create the needed folders and then I assert on them. The output is as follows:
Directory: D:\a\market_manager\market_manager\_build\test\lib\bakeware
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2/2/2022 11:06 AM obj
d----- 2/2/2022 11:06 AM launcher
Directory: D:\a\market_manager\market_manager\_build\test\lib\bakeware\obj\zstd\lib
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2/2/2022 11:06 AM decompress
True
We can see that:
All directories are created
Test-Path returns True, which means that 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress' does exist.
This only makes things more confusing to me as the error has still not changed:
mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress] Error 1
could not compile dependency :bakeware, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile bakeware", update it with "mix deps.update bakeware" or clean it with "mix deps.clean bakeware"
** (Mix) Could not compile with "C:\ProgramData\Chocolatey\bin\make.exe" (exit status: 2).
I really can't understand the error No such file or directory when every command I used shows beyond reasonable doubt that the folder does exist.
Question
Why do I keep getting this error?

Answer
After smashing my head against a wall for several days, I think I finally figured it out.
The command was failing because of how cmd mkdir behaves, in that, it behaves different from the one in poewrshell.
By changing the Makefile to only create folders if they don't exist already, I was able to fix the issue while still mantaining backwards compatibility, which was one of my requirements.
I have submitted a PR with this fix and at the time of this writing, I am happy to announce it was already merged into master:
https://github.com/bake-bake-bake/bakeware/pull/128

Related

golangci-lint: exec: "git": executable file not found in $PATH

I'm trying to integrate golangci-lint on my project. I use golangci-lint v1.49.0 that I install from snap. I've added ci config on my project but when I try to run it, it throws a very bizarre error.
$ golangci-lint run -v
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 7 linters: [errcheck gosimple govet ineffassign staticcheck typecheck unused]
INFO [loader] Go packages loading at mode 575 (files|name|types_sizes|deps|exports_file|compiled_files|imports) took 805.161254ms
ERRO Running error: context loading failed: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: github.com/user/repo#v1.3.4: git init --bare in /home/user/snap/golangci-lint/94/go/pkg/mod/cache/vcs/88980456e822acee6ee17242ca10c17af0281944db0d695e6f13a17951f304ee: exec: "git": executable file not found in $PATH
go: github.com/user/repo#v1.3.4: git init --bare in /home/user/snap/golangci-lint/94/go/pkg/mod/cache/vcs/88980456e822acee6ee17242ca10c17af0281944db0d695e6f13a17951f304ee: exec: "git": executable file not found in $PATH
INFO Memory: 10 samples, avg is 27.8MB, max is 27.8MB
INFO Execution took 809.752753ms
github.com/user/repo isn't a real url, the one I use is private.
These are my config files
# .gitlab-ci.yml
stages:
- test
- build
- sast
- deploy
- cleanup
golangci-lint:
stage: test
image: registry.gitlab.com/gitlab-org/gitlab-build-images:golangci-lint-alpine
script:
- golangci-lint run -v
# .golangci.yml
run:
tests: false
timeout: 10m
skip-dirs:
- "service/mocks"
output:
format: colored-line-number
Why is it unable to read git in $PATH? I've been using git for years using this machine, this is my first time I got an error like this. I also just reinstalled using sudo apt-get install git just to make sure, but it still throws the same error. It looks like Git has already been installed, but it keeps throwing that error.
My OS is Ubuntu v20.04
I've also added the git path by running export PATH=$PATH:/usr/bin/git, but it still keeps throwing the same error.

How to set Bakeware env vars on GitHub Actions for a Windows release?

Background
I am trying to get a Github Action working with Windows and Bakeware because I am trying to create a release using it.
However, I am having issues with the environment variables.
Code
In Bakeware's setup page it is mentioned that we have to set the MAKE and CC environment variables:
https://github.com/bake-bake-bake/bakeware#powershell
In my Github Action, that is exactly what I do (I think):
name: build
env:
MIX_ENV: test
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
name: Build on Windows
runs-on: windows-2019
env:
CC: gcc
MAKE: make
steps:
- uses: actions/checkout#v2
- uses: erlef/setup-beam#v1
with:
elixir-version: '1.13.x' # Define the elixir version [required]
otp-version: '24.2.x' # Define the OTP version [required]
- name: Install choco
shell: powershell
run: |
Set-ExecutionPolicy -ExecutionPolicy Bypass
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
- name: Install bakeware dependencies
shell: powershell
run: choco install -y zstandard make mingw
- name: Install Dependencies
shell: powershell
run: mix deps.get
- name: Run credo code analyser
shell: powershell
run: mix credo --strict
I am even using powershell to do it (even though I am not really sure if this is needed).
Problem
However my GitHub Actions code comes back with this error:
==> bakeware
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/launcher"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"
mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress] Error 1
could not compile dependency :bakeware, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile bakeware", update it with "mix deps.update bakeware" or clean it with "mix deps.clean bakeware"
** (Mix) Could not compile with "make" (exit status: 2).
It says it cannot compile with make.
Question
I have tried copy/pasting the section:
env:
CC: gcc
MAKE: make
To every section in that file I could think of, but I always end up with the same issue.
What am I doing wrong?
Answer
The environment vars, in this case, were being correctly created and set. The problem was more deep, it was related to the library itself not being able to create needed folders.
Because the make tool used by the library needed these folders, the tool crashed itself and reported as such, thus making me believe there was an issue with my environmental setup since I was getting the error:
(Mix) Could not compile with "make" (exit status: 2).
In reality however, this issue was a bug in the library, an issue that is already fixed in master:
https://github.com/bake-bake-bake/bakeware/issues/127

Gitlab runner stop after npm install

For using the pipeline in Gitlab, i've created the following .gitlab-ci.yml file:
image: node:8.2.1
cache:
paths:
- node_modules/
TestIt:
script:
- npm install
- '/node_modules/#angular/cli/bin/ng test --single-run=true --browsers PhantomJS --watch=false'
When the runner start the job it is doing the npm install successfully but by that it ends. It doesn't continue to the second script (like it ignores it from some reason).
This is the output:
What can be the cause for that?
If you are on windows, you probably ran into this problem (nothing else executing after a "npm" command):
https://gitlab.com/gitlab-org/gitlab-runner/issues/2730
TL;DR: Use call npm install instead of npm install, then the second command will execute too. Downside: Then your CI config is not platform-independent anymore.
I still didn't found the reason why this happens but as A workaround After a long search instead of using ng test, i'm using npm test, like that:
TestIt:
script:
- npm test
in
Karma.config.js
I changed from autoWatch: true to false and from singleRun: false to true to prevent continuous testing.
I took out the - npm install

GitLab CI - Cache not working

I'm currently using GitLab in combination with CI runners to run unit tests of my project, to speed up the process of bootstrapping the tests I'm using the built-in cache functionality, however this doesn't seem to work.
Each time someone commits to master, my runner does a git fetch and proceeds to remove all cached files, which means I have to stare at my screen for around 10 minutes to wait for a test to complete while the runner re-downloads all dependencies (NPM and PIP being the biggest time killers).
Output of the CI runner:
Fetching changes...
Removing bower_modules/jquery/ --+-- Shouldn't happen!
Removing bower_modules/tether/ |
Removing node_modules/ |
Removing vendor/ --'
HEAD is now at 7c513dd Update .gitlab-ci.yml
Currently my .gitlab-ci.yml
image: python:latest
services:
- redis:latest
- node:latest
cache:
key: "$CI_BUILD_REF_NAME"
untracked: true
paths:
- ~/.cache/pip/
- vendor/
- node_modules/
- bower_components/
before_script:
- python -V
# Still gets executed even though node is listed as a service??
- '(which nodejs && which npm) || (apt-get update -q && apt-get -o dir::cache::archives="vendor/apt/" install nodejs npm -yqq)'
- npm install -g bower gulp
# Following statements ignore cache!
- pip install -r requirements.txt
- npm install --only=dev
- bower install --allow-root
- gulp build
test:
variables:
DEBUG: "1"
script:
- python -m unittest myproject
I've tried reading the following articles for help however none of them seem to fix my problem:
http://docs.gitlab.com/ce/ci/yaml/README.html#cache
https://fleschenberg.net/gitlab-pip-cache/
https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/336
Turns out that I was doing some things wrong:
Your script can't cache files outside of your project scope, creating a virtual environment instead and caching that allows you to cache your pip modules.
Most important of all: Your test must succeed in order for it to cache the files.
After using the following config I got a -3 minute time difference:
Currently my configuration looks like follows and works for me.
# Official framework image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python
image: python:latest
# Pick zero or more services to be used on all builds.
# Only needed when using a docker container to run your tests in.
# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-service
services:
- mysql:latest
- redis:latest
cache:
untracked: true
key: "$CI_BUILD_REF_NAME"
paths:
- venv/
- node_modules/
- bower_components/
# This is a basic example for a gem or script which doesn't use
# services such as redis or postgres
before_script:
# Check python installation
- python -V
# Install NodeJS (Gulp & Bower)
# Default repository is outdated, this is the latest version
- 'curl -sL https://deb.nodesource.com/setup_8.x | bash -'
- apt-get install -y nodejs
- npm install -g bower gulp
# Install dependencie
- pip install -U pip setuptools
- pip install virtualenv
test:
# Indicate to the framework that it's being unit tested
variables:
DEBUG: "1"
# Test script
script:
# Set up virtual environment
- virtualenv venv -ppython3
- source venv/bin/activate
- pip install coverage
- pip install -r requirements.txt
# Install NodeJS & Bower + Compile JS
- npm install --only=dev
- bower install --allow-root
- gulp build
# Run all unit tests
- coverage run -m unittest project.tests
- coverage report -m project/**/*.py
Which resulted in the following output:
Fetching changes...
Removing .coverage --+-- Don't worry about this
Removing bower_components/ |
Removing node_modules/ |
Removing venv/ --`
HEAD is now at 24e7618 Fix for issue #16
From https://git.example.com/repo
85f2f9b..42ba753 master -> origin/master
Checking out 42ba7537 as master...
Skipping Git submodules setup
Checking cache for master... --+-- The files are back now :)
Successfully extracted cache --`
...
project/module/script.py 157 9 94% 182, 231-244
---------------------------------------------------------------------------
TOTAL 1084 328 70%
Creating cache master...
Created cache
Uploading artifacts...
venv/: found 9859 matching files
node_modules/: found 7070 matching files
bower_components/: found 982 matching files
Trying to load /builds/repo.tmp/CI_SERVER_TLS_CA_FILE ...
Dialing: tcp git.example.com:443 ...
Uploading artifacts to coordinator... ok id=127 responseStatus=201 Created token=XXXXXX
Job succeeded
For the coverage report, I used the following regular expression:
^TOTAL\s+(?:\d+\s+){2}(\d{1,3}%)$

cache installation of deps and afterwards build in travis

Is it possible to separate the install deps and caching from the build of the source code?
I have:
sudo: required
language: cpp
matrix:
include:
- env: GCC_VERSION="4.9"
os: linux
dist: trusty
compiler: gcc
cache:
directories:
- /usr/local/include
- /usr/local/lib
- /usr/local/share
addons:
apt:
packages:
- gcc-4.9
- g++-4.9
sources:
- ubuntu-toolchain-r-test
# Install dependencies
install:
- export BUILD_DEPS="OFF"
- export BUILD_GRSF="ON"
- export CHECKOUT_PATH=`pwd`;
- chmod +x $CHECKOUT_PATH/travis/install_${TRAVIS_OS_NAME}.sh
- . $CHECKOUT_PATH/travis/install_${TRAVIS_OS_NAME}.sh
script:
- chmod +x $CHECKOUT_PATH/travis/build.sh
- . $CHECKOUT_PATH/travis/build.sh
notifications:
email: false
Because my build takes too long (more than 50 minutes with building dependencies and the source code) I proceed in the following way:
I set
BUILD_DEPS="ON" # build only deps
BUILD_GRSF="OFF"
which only builds the dependencies and caches them, afterwards I set
BUILD_DEPS="OFF"
BUILD_GRSF="ON" # build only source
in the .travis.yaml file which then builds only the source code.
This seems to work but is cumbersome? Is there a better solution to this? Maybe directly on travis modifying the .travis.yaml and make a new commit "travis cached, build source now". which will then trigger another travis build (which now builds the source)
Your dependency install script could look for a marker file your script leaves after successful installation in a cached dir and only if that's not found you would re-run the build.
That way you don't need any modifications to the travis spec at least.
It seems travis can only cache in $HOME:
https://github.com/travis-ci/travis-ci/issues/6115#issuecomment-222817367

Resources