CTest.exe on Windows splits PATH environment property on semicolons - windows

I have a Jenkins pipeline which runs tests on a Windows VM using ctest.exe in a bat script, like this:
// Run all tests in parallel, timeout after 1 hour, exclude any tests needing opengl
bat ''' "C:/Program Files/CMake/bin/ctest.exe" --timeout 3600 --verbose -C Release -j4 -E "(_gl)|(GL)" '''
In my CTest setup, I'm setting the environment using the code below. I have already been looking around and found that I need to escape the backslashes in the PATH to avoid CMake interpreting it as a list, but that doesn't seem to make it all the way through to the end:
set(test_dll_path)
set(PATH_STRING "")
# Add the paths I want to a list
list(APPEND test_dll_path "${PROJECT_SOURCE_DIR}/../bin64")
list(APPEND test_dll_path "${PROJECT_SOURCE_DIR}/../bin64/plugins")
# Concatenate that list and convert it to a string
cmake_path(APPEND_STRING test_dll_path ";$ENV{PATH}")
cmake_path(CONVERT "${test_dll_path}" TO_NATIVE_PATH_LIST test_dll_path NORMALIZE)
# Escape the backslashes to avoid CTest interpreting it as a list
string(REPLACE ";" "\\;" test_dll_path "${test_dll_path}")
# Join it to the existing path using more escaped backslashes
list(JOIN test_dll_path "\\;" PATH_STRING)
# Assign the environment to blank first ("append_string" only works if the property existed already)
set_property( TEST ${TEST_TARGET} PROPERTY ENVIRONMENT "" )
set_property( TEST ${TEST_TARGET} APPEND_STRING PROPERTY ENVIRONMENT "PATH=${PATH_STRING}" )
# This comes through as one full string when printed as a message, but still interpreted as a list when CTest uses it later :(
message(DEBUG "Test ${TEST_TARGET} PATH = \"${PATH_STRING}\"")
Then, when I run the jenkins pipeline, the --verbose setting prints the environment for the test.
When this happens, the PATH is split on each semicolon and the tests act as though only the first item is in the PATH:
1: Test command: C:\jenkins\workspace\ows-IncrementalBuild-Test_PR-856\source\Tests\tut.exe "-cppunitreport"
1: Environment variables:
1: PATH=C:\jenkins\workspace\ows-IncrementalBuild-Test_PR-856\source\bin64
1: C:\jenkins\workspace\ows-IncrementalBuild-Test_PR-856\source\bin64\plugins
1: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow
1: C:\Program Files (x86)\MSBuild\14.0\bin\amd64
1: C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64
1: C:\Windows\Microsoft.NET\Framework64\v4.0.30319
...etc

From a week of searching I hadn't seen this, but a colleague just sent me this answer:
How do I make a list in CMake with the semicolon value?
So the answer is, if in doubt, keep adding more backslashes. I needed three when joining the path up, like so:
# Escape the backslashes to avoid CTest interpreting it as a list
string(REPLACE ";" "\\\;" test_dll_path "${test_dll_path}")
# Join it to the existing path using more escaped backslashes
list(JOIN test_dll_path "\\\;" PATH_STRING)

Related

Need to convert back slashes to forward slashes in github action work flow

In my actions yml file, I set an environment variable pointing to the root directory for my ctest input image files (I run ctest to test an image codec decompressor, and these are the input images).
env:
DATA_ROOT: ${{github.workspace}}/data
On windows, this is gives me something like c:\Foo\Bar/data, and I would like to convert it to
c:/Foo/Bar/data
I can do the conversion in PowerShell:
$temp = ${DATA_ROOT}
$pattern = '[\\]'
$temp = $temp -replace $pattern, '/'
but how do I then reset ${DATA_ROOT} to equal $temp ?
I want subsequent steps to use the new ${DATA_ROOT} .
Note this is hurting me right now with ${{github.workspace}} using \ path separators on windows, breaking any "run" actions using the bash shell. So this doesn't work;
defaults:
run:
shell: bash
jobs:
build:
steps:
- name: Configure CMake
run: cmake -B ${{github.workspace}}/build
A workaround is to put single quotes around '${{github.workspace}}/build' which prevents bash from treating the \ path separators as escapes.
I'm not sure why this is tagged CMake, but you can set environment variables in GitHub Actions by writing a line containing NAME=VALUE to $GITHUB_ENV.
From the GitHub Actions documentation:
echo "{name}={value}" >> $GITHUB_ENV
Creates or updates an environment variable for any actions running next in a job. The action that creates or updates the environment variable does not have access to the new value, but all subsequent actions in a job will have access. Environment variables are case-sensitive and you can include punctuation.
Example
steps:
- name: Set the value
id: step_one
run: |
echo "action_state=yellow" >> $GITHUB_ENV
- name: Use the value
id: step_two
run: |
echo "${{ env.action_state }}" # This will output 'yellow'
Multiline strings
For multiline strings, you may use a delimiter with the following syntax.
{name}<<{delimiter}
{value}
{delimiter}
For a CMake solution, if you have a path with potentially mixed slashes, you can use file(TO_CMAKE_PATH)
file(TO_CMAKE_PATH "c:\\Foo\\Bar/data" path)
message(STATUS "path = ${path}") # prints: path = c:/Foo/Bar/data
Note that this function assumes the incoming path is a native path for the platform on which CMake is running. So on Linux, an input with a drive letter would be invalid (the : would be interpreted as a PATH separator and converted to a ;)

Dot symbol in docker compose environment variables doesn't work

I have a docker-compose file with environment variables set like:
identity-api:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=http://0.0.0.0:80
- SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104
It's just an example from https://github.com/dotnet-architecture/eShopOnContainers/blob/dev/src/docker-compose.override.yml.
The issue: I replaced underscore symbol with dot symbol here: ESHOP_EXTERNAL_DNS_NAME_OR_IP -> ESHOP_EXTERNAL_DNS.NAME_OR_IP.
After that, If I try to build docker-compose project in VS, it will not work. The error looks like:
Error DT1001 Invalid interpolation format for "environment" option in service "identity-api":"SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104"
docker-compose C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets 202
So the question: what is wrong with the variable name here? Is dot symbol supported or not? Or maybe it can be escaped somehow?
In general, environment variable names with dots in them are allowed at a technical level but aren't well supported in any context, and I'd choose a different name.
In Docker Compose, it looks like the regexps used to identify variable expansions only support ASCII letters, digits, and underscores. This suggests dots in environment variable names aren't supported, and there's no way to escape them.
I think variables with dots in them also aren't supported in shell scripts, but I'm having trouble proving that to myself reading the POSIX spec. A Name also only consists of ASCII letters, digits, and underscores; parameters either have a name, a number, or a single symbol; and variables are parameters with names; but none of the formal description of this says a "name" is a Name.
One easy thing to demonstrate is running the bash shell in the ubuntu Docker image. We can use the docker run -e option to set arbitrary environment variables (that don't contain =) but we can't really use any shell features to expand variables with dots:
host$ docker run --rm -it -e foo=A -e foo.bar=B ubuntu
# Demonstrate the environment variable is set:
root#227bec28c674:/# env | grep foo
foo.bar=B
foo=A
# It won't get expanded without braces:
root#227bec28c674:/# echo $foo.bar
A.bar
# With braces, it's considered illegal syntax:
root#227bec28c674:/# echo ${foo.bar}
bash: ${foo.bar}: bad substitution

bash: Unable to run build commands via bash scripts

I am trying to run 'make' command from my bash script to build the code.
I can see that all parameters got assigned and able to display the command that i am trying to run. I could not see any issue with the command. But the issue is when it tries to run the command via bash script it fails.
My command is :- ./build_script LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm"
Script Snippet of parsing all the arguments and constructing make command:-
makeargs=""
for build_flag do
if [[ "$build_flag " = "PROJECTS="* ]]; then
apps =`echo $build_flag |sed "s/PROJECTS=//" `
makeargs="$makeargs PROJECTS=\"$apps \""
else
makeargs="$makeargs $build_flag"
fi
done
echo "make DCOV=1 $makeargs $maketest"
make DCOV=1 $makeargs $maketest
When i run the script, I can see the build command has constructed properly.
Output :-
make DCOV=1 LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm" run_all
GNUmakefile:16: warning: overriding commands for target `/'
GNUmakefile:19: warning: ignoring old commands for target `/'
make: *** No rule to make target `bfm"'. Stop.
I try to print PROJECTS variable in my 'GNUmakefile' present in build_main folder. I can see the output : PROJECTS is "bfm . It is not taking whole "cuda bfm" as a whole string.
Note:- When i try to run the same build command :- make DCOV=1 LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm" run_all explicitly it works fine.
Seems like issue with Interpreting variables with makefile.
Any solution for this ? Please help.
Thanks!
Change makeargs string to array before passing it as an arguments group.
eval makeargs_array=( $makeargs )
make UVC=1 "${makeargs_array[#]}" $maketest
Without converting to array, if you enable debug, it shows last line interpretation as
make DCOV=1 LIC=1 DOCUMNETS=1 'PROJECTS="cuda' bfm '"'
Which is clearly ignoring double-quote and considering space as separator.
Even double-quote is getting passed as a separate argument in this case.
Explanation:
Word-splitting
It says,
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
If we use "$makeargs" i.e. surrounded by double-quote, it is not considered by word-splitting and results in LIC=1 DOCUMNETS=1 "PROJECTS=cuda bfm"
But again its a complete string, while requirement is to split the string to use as arguments.
So now using $makeargs.
Word-splitting gets in action as per the default IFS (space, tab, newline), we get result as LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm "
Double-quoted part of string didn't affect the word-splitting since, subject to splitting is complete string here.
Why array worked here?
Array itself expands each element as separate word when using # and here no further word-splitting requires after expansion.
Arrays

Bash imported variables break configure script when double-quoted

I have a bash script which imports same variables from another static file, which itself uses environment variables set by another file when the script is invoked.
This is the file that gets imported and sets some variables.
# package.mk
PKG_NAME="binutils"
PKG_VERSION="2.24"
PKG_URL_TYPE="http"
PKG_URL="http://ftp.gnu.org/gnu/binutils/${PKG_NAME}-${PKG_VERSION}.tar.bz2"
PKG_DEPENDS=""
PKG_SECTION="devel"
PKG_CONFIGURE_OPTS="--prefix=${TOOLS} \
--target=${TARGET} \
--with-sysroot=${TOOLS}/${TARGET} \
--disable-nls \
--disable-multilib"
It is used by the builds script as so:
#!/bin/bash
# Binutils
. settings/config
pkg_dir="$(locate_package 'binutils')"
. "${pkg_dir}/package.mk"
# etc...
"${CLFS_SOURCES}/${PKG_NAME}-${PKG_VERSION}/configure" "${PKG_CONFIGURE_OPTS}"
# etc...
This script first imports the settings/config file which has a bunch of global variables used by this script and others, and exports them so they are available as environment variables. It then locates the correct package.mk file for the specific component we are building, and imports it as well. So far, so good.
However when I double-quote the options (PKG_CONFIGURE_OPTS) for the configure script:
"${CLFS_SOURCES}/${PKG_NAME}-${PKG_VERSION}/configure" "${PKG_CONFIGURE_OPTS}"`
I get the following error:
gcc: error: unrecognized command line option ‘--with-sysroot=/root/LiLi/target/cross-tools/arm-linux-musleabihf’
If I leave it not quoted like:
"${CLFS_SOURCES}/${PKG_NAME}-${PKG_VERSION}/configure" ${PKG_CONFIGURE_OPTS}`
it works fine (--with-sysroot= is indeed a valid configure flag for binutils).
Why is this? What can I change so that I can double-quote that portion (going by the bash wisdom that one should double-quote just about everything).
Quoting the variable means the entire thing is passed as a single argument, spaces and newlines included. You want word splitting to be performed so that the string is treated as multiple arguments. That's why leaving it unquoted works.
If you're looking for the "right" way to handle this, I recommend using an array. An array can hold multiple values while also preserving whitespace properly.
PKG_CONFIGURE_OPTS=(--prefix="$TOOLS"
--target="$TARGET"
--with-sysroot="$TOOLS/$TARGET"
--disable-nls
--disable-multilib)
...
"$CLFS_SOURCES/$PKG_NAME-$PKG_VERSION/configure" "${PKG_CONFIGURE_OPTS[#]}"

Shell string with double quotes - RTC command

I am trying to run a RTC 4.x command to add components to a workspace. The list of components have spaces in the names so they need to be surrounded by quotes. I am storing this list in a simple string variable:
COMPONENTS="\"TestComp\" \"Common Component\""
When I just echo out COMPONENTS it displays correctly, but when I use it in a scm command odd things happen to the quotes. I am running this in Jenkins so I can get some additional output, but the same thing happens when I run it on the command line so this is not a Jenkins issue.
From the console log:
+ COMPONENTS='"TestComp" "Common Component"'
+ echo '"TestComp"' '"Common' 'Component"'
"TestComp" "Common Component"
The command is trying to run the following:
+ scm workspace add-components TEST_Workspace -s Test_Stream '"TestComp"' '"Common' 'Component"'
Which produces:
Problem running 'workspace add-components':
Unmatched component ""Common".
Typically, you need to use an array to store items that may themselves contain whitespace:
components=("TestComp" "Common Component")
scm workspace add-components TEST_Workspace -s Test_Stream "${components[#]}"
Quoting an array expansion indexed with # produces a sequence of words, one per element of the array, rather than a single word.

Resources