Variable mysteriously disappears? AWS CodeBuild - bash

What follows is my buildspec.yml
build:
commands:
- 'IMAGE_TAG=$(cat package.json | grep version | head -1 | awk -F: ''{ print $2 }'' | sed ''s/[",]//g'')'
- echo $IMAGE_TAG
- docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
- docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
Here's the output from the relevant build:
[Container] 2018/12/12 22:06:42 Running command IMAGE_TAG=$(cat package.json | grep version | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g')
[Container] 2018/12/12 22:06:42 Running command echo $IMAGE_TAG <<< GOOD
1.0.0 <<<< PERFECT
[Container] 2018/12/12 22:06:42 Running command docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
invalid argument "gotbot-air:" for t: invalid reference format
See 'docker build --help'. <<<<<< OH NO
[Container] 2018/12/12 22:06:42 Command did not exit successfully docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG . exit status 125
As you can clearly see on the marked lines, somehow the variable $IMAGE_TAG is set correctly to 1.0.0 when echo'd, yet on the very next line of execution in my build script, it seems to have disappeared.
Please note I am using version 0.2 of the specification.
EDIT: It may be important that my other environment variables are either declared at the top in env or native code build variables, it might be that im getting some different execution environment when running the docker command?

Check the version of your BuildSpec file. Change it to version: 0.2.
In version 0.1, AWS CodeBuild runs each build command in a separate instance of the default shell in the build environment. In version 0.2, AWS CodeBuild runs all build commands in the same instance of the default shell in the build environment.

Related

sed fails with error "\1 not defined in the RE" when running in Gitlab CI

I am trying to update contents of a file from a variable with sed during Gitlab CI job. The variable comes from artifacts of the previous stage version. If simplified, my job looks something like this:
build-android-dev:
stage: build
dependencies:
- version
only:
- mybranch
before_script:
- PUBSPEC_VERSION="`cat app-pubspec-version`"
- echo "Pubspec version - $PUBSPEC_VERSION"
script:
- >
sed -i -E "s/^(version: )(.+)$/\1${PUBSPEC_VERSION}/g" pubspec.yaml
- cat pubspec.yaml | grep version
interruptible: true
tags:
- macmini-flutter
Unfortunatelly, the job fails with the following error message:
$ sed -i -E "s/^(version: )(.+)$/\1${PUBSPEC_VERSION}/g" pubspec.yaml
sed: 1: "s/^(version: )(.+)$/\14 ...": \1 not defined in the RE
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit status 1
PUBSPEC_VERSION coming from artifacts is the following:
$ echo "Pubspec version - $PUBSPEC_VERSION"
Pubspec version - 4.0.0+2
I am able to execute the command on my local Ubuntu (Linux) machine without any issues:
$ export PUBSPEC_VERSION=4.0.0+2
$ sed -i -E "s/^(version: )(.+)$/\1${PUBSPEC_VERSION}/g" pubspec.yaml
$ cat pubspec.yaml | grep version
version: 4.0.0+2
The remote machine where Gitlab Runner is started is MacOS. Not sure whether it matters.
As you can see, I also use folding style in my CI configuration like proposed here in order to avoid inproper colon interpretation.
I googled for solutions to solve the issue but it seems that I don't need to escape (though I also tried) group parentheses in my regular expression because I use extended regular expression.
So I'm stuck on it...
P.S. I don't have access to the shell of remote MacOS.
is MacOS.
-i takes a suffix argument, so -E is the backup suffix to create. Yuo would want:
- sed -i '' -E 's/...'

Sending list of filenames in current directory as a single string to a Docker container's STDIN

I have a Docker container and need to pass a single string (an array of strings would work too, but as far as I know this is not possible) containing the names of the file present in the current directory.
My first strategy to get the filenames inside the container was running the command docker build -t doc-validate . && docker run doc-validate (printf "%s " $(ls -p | grep -v /)) , to send the output of (printf "%s " $(ls -p | grep -v /)) directly into the container's STDIN, however, I get zsh: bad pattern: (printf %s Dockerfile returned as an error; seems to me that the container or the shell is trying to get 'Dockerfile' executed somewhere and I don't know why this happens, as running this same printf command directly in the terminal works as expected (printing only file names in the current directory).
My second approach was trying to send these filenames as an environment variable to the container. Running PROJECT_FILES=(printf "%s " $(ls -p | grep -v /)) in the terminal works as expected, $PROJECT_FILES outputs these filenames. However, if I try to pass it directly into the container like this: docker build -t doc-validate . && docker run --env PROJECT_FILES=(printf "%s " $(ls -p | grep -v /)) doc-validate I get the same zsh: bad pattern: PROJECT_FILES=(printf %s Dockerfile as an error.
Seems to me that it the commands are failing before even entering the container due to 'zsh:' being in the error output, however, I don't see why running them standalone in the terminal works and why passing them as parameters to the container is being such a headache.
How can I get the output of running (printf "%s " $(ls -p | grep -v /)) in my local machine as a value accessible inside the container?
Dockerfile:
FROM node:16
WORKDIR /moduleRoot
COPY package.json /moduleRoot
RUN npm install
COPY . /moduleRoot/
# Authorize execution of entrypoint bash script
RUN chmod +x /moduleRoot/docker-container-entrypoint.sh
# these values below will be set later, just register them explicitly here
ENV PROJECT_FILES=""
CMD [ "/moduleRoot/docker-container-entrypoint.sh" ]
Bash entrypoint (docker-container-entrypoint.sh file in CMD):
#!/bin/bash
echo "Args are: $#"
echo "PROJECT_FILES=$PROJECT_FILES"
First thing is that you are using the node:16 and according to the entrypoint of the project it will call node to execute anything you pass as an argument.
On your Dockerfile you are calling a CMD and not changing the ENTRYPOINT, so when you execute the container it will try to execute the bash script using node and you must be getting an error like this:
/moduleRoot/docker-container-entrypoint.sh:2
echo "Args are: $#"
^^^^^^^^^^^^^^
SyntaxError: Unexpected string
at Object.compileFunction (node:vm:353:18)
at wrapSafe (node:internal/modules/cjs/loader:1039:15)
at Module._compile (node:internal/modules/cjs/loader:1073:27)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1138:10)
at Module.load (node:internal/modules/cjs/loader:989:32)
at Function.Module._load (node:internal/modules/cjs/loader:829:14)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
at node:internal/main/run_main_module:17:47
If your bash script is there only to help you troubleshoot, then you can use the online heredoc to pass the arguments you need:
» docker run -t --rm doc-validate <<< echo $(ls -a1)
Args are: docker-container-entrypoint.sh Dockerfile package.json
PROJECT_FILES=
On the other hand, if you need the bash script then you'll have to change the CMD with ENTRYPOINT. Another thing worth noting is that Linux's ls doesn't have the -p flag.
Maybe this question/answers should work for you if you are using a CI pipeline with Jenkins: How to mount docker volume with jenkins docker container?

Saving bash output in Dockerfile as ENV var

I am setting up a random password to be used on the initial login within my Dockerfile. I have tried
ENV PASSWORD=RUN date +%s | sha256sum | base64 | head -c 32
Which does not work because Docker commands are in separate containers. I also tried
RUN export PASSWORD= date +%s | sha256sum | base64 | head -c 32
Which has also failed.
I have also tried to redirect my output to a file, which also fails, the file stays empty and no error is given.
RUN date +%s | sha256sum | base64 | head -c 32 > test.txt
How can I save the output from this command as an ENV var to be used in my Docker container?
The following Dockerfile:
FROM alpine:latest
RUN echo "bd" > /test.txt
when run has the test.txt file present:
docker build . -t test:latest
docker run -it --rm test:latest
> cat /test.txt
> 'bd'
You could change your command to write a file in a sourceable manner (RUN echo "my_env_var=$(echo 'stuff')" > /test.txt) and then have your entrypoint include . /test.txt

Issue in executing docker command in bash script

I have following bash script code try to run a docker container through bash script but I am retreiving error.
#!/bin/bash
name=sudo docker ps | grep 'test' | awk '{print
$1}'
sudo docker exec -it $name bash
error:
docker exec requires at least two arguments
Assuming that you have a docker container actually called test running, the way the id is attained is incorrect. In order to expand a command into a variable, it needs to be contained within $() and so:
#!/bin/bash
name=$(docker ps | grep 'test' | awk '{print $1}')
sudo docker exec -it $name bash
A further note is the fact that you don't need sudo permissions to run docker ps ...
Additionally, you never need to pipe grep into awk as awk can do this for you:
name=$(docker ps | awk '/test/ {print $1}')
Further more, there is no need to pipe at all given the native capabilities built into docker-ps and so:
name=$(docker ps -q -f name=test)
This will print only the container id of a container named test. I'm assuming that the name is test here but it could be something else i.e. the label that is named test, in which case the filter would need to change:
-f, --filter=[]
Filter output based on these conditions:
- exited=<int> an exit code of <int>
- label=<key> or label=<key>=<value>
- status=(created|restarting|running|paused|exited|dead)
- name=<string> a container's name
- id=<ID> a container's ID
- before=(<container-name>|<container-id>)
- since=(<container-name>|<container-id>)
- ancestor=(<image-name>[:tag]|<image-id>| â¨image#digestâ©) - conâ
tainers created from an image or a descendant.
- volume=(<volume-name>|<mount-point-destination>)
- network=(<network-name>|<network-id>) - containers connected to
the provided network

set inside .gitlab-ci.yml a variable reading the name of artifactId from pom

Inside .gitlab-ci.yml we define a variable (which is just the artifactId name for the project) ARTIFACT_ID: myMicroservice-1
This variable ARTIFACT_ID is sent to a general microservice which has all the scripts to publish/deploy docker, etc.
How can I read this variable direct from POM file?
pom:
<artifactId>myMicroservice-1</artifactId>
.gitlab-ci.yml:
variables:
SKIP_UNIT_TESTS_FLAG: "true"
ARTIFACT_ID: myMicroserverName
IS_OSL: "true"
KUBERNETES_NAMESPACE: test
Here is how we do it.
Value is extracted from the pom.xml based on its XPath.
We use xmllint tool from libxml2-utils, but there are various other tools for that.
Then value is saved as an environment variable in a file, which is passed to further GitLab jobs as artifact.
stages:
- prepare
- build
variables:
VARIABLES_FILE: ./variables.txt # "." is required for sh based images
POM_FILE: pom.xml
get-version:
stage: prepare
image: ubuntu
script:
- apt-get update
- apt-get install -y libxml2-utils
- APP_VERSION=`xmllint --xpath '/*[local-name()="project"]/*[local-name()="version"]/text()' $POM_FILE`
- echo "export APP_VERSION=$APP_VERSION" > $VARIABLES_FILE
artifacts:
paths:
- $VARIABLES_FILE
build:
stage: build
image: docker:latest
script:
- source $VARIABLES_FILE
- echo "Here use $APP_VERSION as you like"
This is the script to grab information from pom, in this case, artifactId:
- export myARTIFACT_ID=$(mvn exec:exec -q -Dexec.executable=echo -Dexec.args='${project.artifactId}')
- if [[ "$myARTIFACT_ID" == *finishWithWhatEverName]]; then export myVariable="false"; else export myVariable="true";
Then you can use myVariable to whatever you want.
I tried Angel's solution, but got an error:
/bin/bash: line 87: mvn: command not found
Finally I succeeded when I used the following to extract the tag value:
- export ARTIFACT_ID=$(cat pom.xml | grep "<artifactId>" | head -1 | cut -d">" -f2 | cut -d"<" -f1 | awk '{$1=$1;print}')
jobname:
stage: stage
before_script:
- export "MAVEN_ID=$(mvn help:evaluate -Dexpression=project.id -q -DforceStdout)"
- >
IFS=: read -r MAVEN_GROUPID MAVEN_ARTIFACTID MAVEN_PACKAGING MAVEN_VERSION <<< ${MAVEN_ID}
script:
- >
echo -e "groupId: ${MAVEN_GROUPID}\nartifactId: ${MAVEN_ARTIFACTID}\nversion: ${MAVEN_VERSION}\npackaging: ${MAVEN_PACKAGING}"
mvn help:evaluate -Dexpression=project.id -q -DforceStdout prints artifact identification information in the following format: com.group.id:artifactid:packaging:version
MAVEN_ID variable is parsed using IFS based on the colon (:) as a separator to get common maven variables like artifactId, groupId, version and packaging (explanation)
later these variables can be used in the code, for example for echoing values
IFS is a bash feature, thus corresponding GitLab runner should have bash installed

Resources