I am trying to run maven through a docker image using a shell script
When running docker in the shell, I use sed to remove single quotes:
bash script:
docker run $(echo "-e CUCUMBER_FILTER_TAGS=$CUCUMBER_FILTER_TAGS $RUN_ARGUMENT $INPUT_MAVEN_COMMAND $MAVEN_ARGUMENTS $AUTHENTICATION" | sed "s/'//g")
is translated into
docker run -e 'CUCUMBER_FILTER_TAGS="#bidrag-person' and not '#ignored"' --rm -v /home/deployer/actions-bidrag-cucumber-backend/ws/bidrag-cucumber-backend/bidrag-cucumber-backend:/usr/src/mymaven -v /home/deployer/.m2:/root/.m2 -w /usr/src/mymaven maven:3.6.3-openjdk-15 mvn test -e -DUSERNAME=j104364 -DINTEGRATION_INPUT=json/integrationInput.json -DUSER_AUTH=*** -DTEST_AUTH=*** -DPIP_AUTH=***
how can I remove those extra single quotes around and within CUCUMBER_FILTER_TAGS that seems to pop up from nowhere?
I cannot solve this and are seeking a solution. This script (https://github.com/navikt/bidrag-maven/blob/feature/filter.tags/cucumber-backend/cucumber.sh) is being run from a cron job on GitHub (GitHub Actions, part of a GitHub workflow)
The other variables (which are not inputs to this script) are set as environment variables from GitHub secrets in a GitHub workflow
AUTHENTICATION="-DUSER_AUTH=$USER_AUTHENTICATION -DTEST_AUTH=$TEST_USER_AUTHENTICATION -DPIP_AUTH=$PIP_USER_AUTHENTICATION"
are set in in a GitHub workflow yaml file like this:
- uses: navikt/bidrag-maven/cucumber-backend#v6
with:
maven_image: maven:3.6.3-openjdk-15
username: j104364
env:
USER_AUTHENTICATION: ${{ secrets.USER_AUTHENTICATION }}
TEST_USER_AUTHENTICATION: ${{ secrets.TEST_USER_AUTHENTICATION }}
PIP_USER_AUTHENTICATION: ${{ secrets.PIP_USER_AUTHENTICATION }}
You should be using an array.
docker_options=(
-e "CUCUMBER_FILTER_TAGS=$CUCUMBER_FILTER_TAGS"
"$RUN_ARGUMENT"
"$INPUT_MAVEN_COMMAND"
"$MAVEN_ARGUMENTS"
"$AUTHENTICATION"
)
docker run "${docker_options[#]}"
while these answers work to some degree, they will not function in my use case... I have recently upgraded the server where these scripts are used, and I have moved on to other scripting languages...
Conclusion:
Bash scripting is hard and painstaking... Both of these suggestions are functioning (sort of), but not as intended...
Related
I wrote following pipeline:
image: maven:3-openjdk-11
variables:
TARGET_LOCATION: "/tmp/uploads/"
stages:
- deploy
deploy-job:
stage: deploy
before_script:
- export MAVEN_ARTIFACT_VERSION=$(mvn --non-recursive help:evaluate -Dexpression=project.version | grep -v '\[.*'| tail -1)
- export MAVEN_ARTIFACT=app-${MAVEN_ARTIFACT_VERSION:+$MAVEN_ARTIFACT_VERSION.jar}
script:
- eval $(ssh-agent -s)
(SSH STUFF HERE...)
- scp -o HostKeyAlgorithms=ssh-rsa -p /builds/xxxxx/app/target/$MAVEN_ARTIFACT user#host:${TARGET_LOCATION}
I expected the $MAVEN_ARTIFACT in scp command change to something like app-BETA-0.1.jar and TARGET_NAME change it's value but it's not parsing and I got variable name in both places. I tried with brackets as well but I can't achieve what I want.
Is there any way to pass variables generated during script execution as arguments to other programs executed in the same script section?
Below is a piece of logs from pipeline execution:
$ scp -o HostKeyAlgorithms=ssh-rsa -p /builds/xxxxx/app/target/$MAVEN_ARTIFACT user#host:${TARGET_LOCATION}
You're using these correctly, and it is working.
The GitLab pipeline logs show the command exactly as you write it in your script. It will not replace variables with their values. If you need to verify the value or confirm it's set, use standard debugging techniques like printing the value with something like echo $MAVIN_ARTIFACT.
From my gitlab-ci I would need to pass an environment variable with the spring profiles to docker compose. Such variable is defined for each server environment where we deploy.
So, in my gitlab-ci I have this:
.deploy_template: &deploy_template
script:
- echo $ENV_SPRING_PROFILES
# start containers
- $SSH_COMMAND user#$CI_ENVIRONMENT_URL "cd $REMOTE_DEPLOY_DIR/docker && SPRING_ACTIVE_PROFILES=$ENV_SPRING_PROFILES && DOCKER_HOST=tcp://localhost:2375 && docker-compose up -d"
deploy_811AC:
<<: *deploy_template
stage: deploy
when: manual
only:
- /^feature.*$/
- /^fix.*$/
environment:
name: ccvli-ecp626
url: 10.135.XXX.XXX
variables:
ENV_SPRING_PROFILES: "mock"
When I run the runner, I can see with this - echo $ENV_SPRING_PROFILES the value of the variable. However, it seems not be replaced in the SSH command as docker-compose say the variable SPRING_ACTIVE_PROFILES is empty.
It is becoming a kind of nightmare so any clue is welcome.
Thanks in advance
I have not to much experience with gitlab ci, but what I think is that the way variables are "declared" is not with "=" but like this:
variables:
SPRING_ACTIVE_PROFILES: $SPRING_ACTIVE_PROFILES
script:
...
- $SSH_COMMAND user#$CI_ENVIRONMENT_URL "cd $REMOTE_DEPLOY_DIR/docker && $ENV_SPRING_PROFILES && DOCKER_HOST=tcp://localhost:2375 && docker-compose up -d"
Once it is declare on the first block, you can start using it on the code :)
I am trying to dynamically set the image name and tag for AWS Elastic Beanstalk in my Dockerrun.aws.json file:
"Image": {
"Name": "IMAGETAG",
"Update": "true"
}
with the following sed command as a script in my GitLab CI file:
sed -i.bak "s|IMAGETAG|$CONTAINER_TEST_IMAGE|" Dockerrun.aws.json && rm Dockerrun.aws.json.bak; eb deploy Production
Where $CONTAINER_TEST_IMAGE is a verified good environment variable (tested by doing echo $CONTAINER_TEST_IMAGE as a script). $CONTAINER_TEST_IMAGE contains the structure of the following content (where ... is the full id):
gitlab.company.com:123/my-group/my-project:core_7de09851...8f_testing
The problem I am facing is that sed does not work during the CI pipeline. I am failing to understand why considering if I set the environment variable locally and run the same command, it will successfully replace the value of Name to the same structure URL. This testing was done on a Macbook.
I know that it is not updating the file because the Gitlab CI log reports
WARN: Failed to pull Docker image IMAGETAG:latest, retrying...
I've tried a few things that did not work:
Running the sed and eb deploy commands as separate scripts (two different lines in the CI file)
Switch the variable that I am seeking to replace in Dockerrun.aws.json to <IMAGE>
While it was at <IMAGE>, running sed -i='' "s|<IMAGE>|$CONTAINER_RELEASE_IMAGE|" Dockerrun.aws.json instead of doing the .bak and then rm'ing it (I read somewhere that sed has inconsistencies on OSX with the -i='' version)
Does anyone have any thoughts on what the issue might be and how it can be resolved?
There were two aspects of this that were going wrong:
The sed command was not executing correctly on the runner, but was working locally
eb deploy was ignoring the updated file
For part 1, he working sed command is:
sed -ri "s|\"IMAGETAG\"|\"$1\"|" Dockerrun.aws.json
where the line in Dockerrun.aws.json is "Name": "IMAGETAG",. sed still confuses me here so I can't explain why this one works vs the original command.
For part 2, apparently eb deploy will always look at the latest commit if it can, rather than the current working directory. Makes sense, I guess. To get around this, run the command as eb deploy --staged. You can read more about this flag on AWS's site.
Also, note that my .gitlab-ci.yml simply calls a script to run all of this rather than doing it there.
- chmod +x ./scripts/ebdeploy.sh
- ./scripts/ebdeploy.sh $CONTAINER_TEST_IMAGE
From the GitLab CI documentation the bash shell is supported on Windows.
Supported systems by different shells:
Shells Bash Windows Batch PowerShell
Windows ✓ ✓ (default) ✓
In my config.toml, I have tried:
[[runners]]
name = "myTestRunner"
url = xxxxxxxxxxxxxxxxxxx
token = xxxxxxxxxxxxxxxxxx
executor = "shell"
shell = "bash"
But if my .gitlab-ci.yml attempts to execute bash script, for example
stages:
- Stage1
testJob:
stage: Stage1
when: always
script:
- echo $PWD
tags:
- myTestRunner
And then from the folder containing the GitLab multi runner I right-click and select 'git bash here' and then type:
gitlab-runner.exe exec shell testJob
It cannot resolve $PWD, proving it is not actually using a bash executor. (Git bash can usually correctly print out $PWD on Windows.)
Running with gitlab-runner 10.6.0 (a3543a27)
Using Shell executor...
Running on G0329...
Cloning repository...
Cloning into 'C:/GIT/CI_dev_project/builds/0/project-0'...
done.
Checking out 8cc3343d as bashFromBat...
Skipping Git submodules setup
$ echo $PWD
$PWD
Job succeeded
The same thing happens if I push a commit, and the web based GitLab CI terminal automatically runs the .gitlab-ci script.
How do I correctly use the Bash terminal in GitLab CI on Windows?
Firstly my guess is that it is not working as it should (see the comment below your question). I found a workaround, maybe it is not what you need but it works. For some reason the command "echo $PWD" is concatenated after bash command and then it is executed in a Windows cmd. That is why the result is "$PWD". To replicate it execute the following in a CMD console (only bash is open):
bash && echo $PWD
The solution is to execute the command inside bash with option -c (it is not the ideal solution but it works). The .gitlab-ci.yml should be:
stages:
- Stage1
testJob:
stage: Stage1
when: always
script:
- bash -c "echo $PWD"
tags:
- myTestRunner
I have gitlab deployment activem and I want to get the deploy script to have some custom information about the deployment process (like $CI_PIPELINE_ID).
However, the script doesn't get the variables, instead it gets the "raw text".
the call performed by the script is: $ python deploy/deploy.py $CI_COMMIT_TAG $CI_ENVIRONMENT_URL $CI_PIPELINE_ID
How can i get it to use the variables?
My .gitlab-ci.yml:
image: python:2.7
before_script:
- whoami
- sudo apt-get --quiet update --yes
- sudo chmod +x deploy/deploy.py
deploy_production:
stage: deploy
environment: Production
only:
- tags
- trigger
except:
# - develop
- /^feature\/.*$/
- /^hotfix\/.*$/
- /^release\/.*$/
script:
- python deploy/deploy.py $CI_COMMIT_TAG $CI_ENVIRONMENT_URL $CI_PIPELINE_ID
It looks like potentially that you could be using a different environmental variable that you should be using.
bash/sh $variable
windows batch %variable%
PowerShell $env:variable
See using CI variables in your job script.
I don't get what you mean with "raw text", but you can declare the variables in your project settings. Also, have you configured you're runner?
Go to Settings->CI/CD->Secret Variables and just put them right there.
You can also find valuable information in the documentation.