I'm unable to execute AWS CLI via Bash shell script called from a Jenkinsfile.
If the shell script is executed directly inside the Jenkins Docker, the scripts executes without issue, which suggests the assumed IAM role for the Jenkins Docker has the correct permissions.
Any ideas?
I found the issue - in the Jenkinsfile it authorises the script to run via binding authorisation credentials to a certain IAM role, the IAM user assigned to the shell script was was not for the environment in which it was running!
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
credentialsId: 'jenkins-ci-aws-iam-user']]) {
Related
Problem
I am trying to use a Windows Docker container to run GitHub Actions.
I want to run scripts before and after the job (e.g. to clean the directory).
I have successfully done this before on a computer not running docker, so I figured the same should work in docker.
What I have tried
I found here that you can do that using Environment Variables.
I used the following two commands in command prompt to set the environment variables.
Pre-Job Script:
setx ACTIONS_RUNNER_HOOK_JOB_STARTED C:\actions-runner-resources\scripts\pre-post-build\pre-run-script.ps1
Post-Job Script:
setx ACTIONS_RUNNER_HOOK_JOB_COMPLETED C:\actions-runner-resources\scripts\pre-post-build\post-run-script.ps1
The scripts do not run.
I have tried restarting the docker container.
I have tried restarting the actions runner service.
I am new to docker, so I am wondering if I am doing something wrong with the environment variables that does not work with docker.
How do I get the actions runner to run pre/post job scripts in docker?
You can safely add them to your environment variable by doing this recommended method;
Inside the actions-runner directory, locate the .env file, edit it by adding your environment variable. Save and restart the runner service.
I'm running CI jobs on a self-hosted GitLab instance plus 10 GitLab Runners.
For this minimal example, two Runners are needed:
Admin-01
A shell runner with Docker installed.
It can execute e.g. docker build ... to create new images, which are then pushed to the private Docker registry (also self-hosted / part of the GitLab installation)
Docker-01
A docker runner, which executes the previously build image.
On a normal bare-metal, virtual machine or shell runner, I would modify e.g. ~/.profile to execute commands before before_script or script sections are executed. In my use case I need to set new environment variables and source some configuration files provided by the tools I want to run in an image. Yes, environment variables could be set differently, but there seams to be no way to source Bash scripts automatically before before_script or script sections are executed.
When sourcing the Bash source file manually, it works. I also notice, that I have to source it again in script block. So I assume the Bash session is ended between before_script block to script block. Of cause, it's no nice solution to manually source the tools Bash configuration script in every .gitlab-ci.yml file manually by the image users.
myjobn:
# ...
before_script:
- source /root/profile.additions
- echo "PATH=${PATH}"
# ...
script:
- source /root/profile.additions
- echo "PATH=${PATH}"
# ...
The mentioned modifications for e.g. shell runners does not work in images executed by GitLab Runner. It feels like the Bash in the container is not started as login shell.
The minimal example image is build as follows:
fetch debian:bullseye-slim from Docker Hub
use RUN commands in Dockerfile to modify with some echo outputs
/etc/profile
/root/.bashrc
/root/.profile
# ...
RUN echo "echo GREETINGS FROM /ROOT/PROFILE" >> /root/.profile \
&& echo "echo GREETINGS FROM /ETC/PROFILE" >> /etc/profile \
&& echo "echo GREETINGS FROM /ROOT/BASH_RC" >> /root/.bashrc
When the job starts, non of the echos is printing messages, while a cat shows, the echo commands have been put at the right places while building the image.
At next I tried to modify
SHELL ["/bin/bash", "-l", "-c"]
But I assume, this has only effects in RUN commands in the Dockerfile, but not on an executed container.
CMD ["/bin/bash", "-l"]
I see no behavior changes.
Question:
How to start the Bash in the Docker image managed by GitLab Runner as login shell so it ready configuration scripts?
How to modify the environment in a container before before_script or script runs. Modifying means environment variables and execution / sourcing a configuration script or patched default script like ~/.profile.
How does GitLab Runner execute a job with Docker?
This is not documented by GitLab in the official documentation ...
What I know so far, it jumps between Docker images specified by GitLab and user defined images and shares some directories/volumes or so.
Note:
Yes, the behavior can be achieved with some Docker arguments in docker run, but as I wrote GitLab Runner is managing the container. Alternatively, how to configure, how GitLab Runner launches the images? To my knowledge, there is no configuration option available / documented for this situation.
A shell runner with Docker installed. It can execute e.g. docker build ...
Use docker-in-docker or use kaniko. https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
Shell executor is like "the last resort", where you want specifically to make changes to the server, or you are deploying your application "into" this server.
How to start the Bash in the Docker image managed by GitLab Runner as login shell so it ready configuration scripts?
Add ENTRYPOING bash -l to your image. Or set the entrypoint from gitlab-ci.yml. See docker documentation on ENTRYPOINT and gitlab-ci.yml documentation on image: entrypoint: .
How to modify the environment in a container before before_script or script runs.
Build the image with modified environment. Consult Dockerfile documentation on ENV statements.
Or set the environment from gitlab-ci.yml file. Read documentation on variables: in gitlab-ci.
How to prepare the shell environment in an image executed by GitLab Runner?
Don't. The idea is that the environment is reproducible, ergo, there should be no changes beforehand. Add variables: in gitlab-ci file and use base images if possible.
How does GitLab Runner execute a job with Docker?
This is not documented by GitLab in the official documentation ...
Gitlab is open-source.
What I know so far, it jumps between Docker images specified by GitLab and user defined images and shares some directories/volumes or so.
Yes, first a gitlab-runner-helper is executed - it has git and git-lfs and basically clones the repository and downloads and uploads the artifacts. Then the container specified with image: is run, cloned repository is copied into it and a specially prepared shell script is executed in it.
A Bash script is CRON-ed to run on server A every day and connects to a remote MySQL database to run a query, dump the results to a file, and send an email.
In my freestyle project, I have set up my build to pull that Bash script from my Github repo and execute it. Jenkins starts it but the script fails. Based on my discovery,
Jenkins runs my script in a clone of my repo on its own server, not on server A where it is CRON-ed
The above explains why my shell script fails to find the mysql client executable
Based on my understanding, this means that my freestyle project needs to connect to server A where my script is CRON-ed to run like so
ssh serverA_username#serverA #using ServerA user key
SCRIPT="$WORKSPACE/shell/bash_script_name.sh"
sh -x $SCRIPT
Am I correct? Any pointers welcome. Thank you!
I am trying to create a "aws_configure.bat" file which will run aws commands. I need to configure "aws_configure.bat" file as windows task. I created my script with below content.
aws configure set AWS_ACCESS_KEY_ID <mykey>
aws configure set aws_secret_access_key <myskey>
aws configure set region us-west-2
aws dynamodb list-tables
When I am trying to run this script then its printing the first line in cmd window. Can someone please suggest what is the problem here. Why my script is not able to run the aws cli commands. (I have installed aws cli in my system and when i am running these commands directly in cmd window, everything is working fine).
You should consider creating and configuring your AWS credentials outside of your batch file, then referencing the named profile from the batch file.
Run aws configure --profile myprofile, and provide the information required.
Then from your batch file, call aws dynamodb list-tables --profile myprofile.
To setup the prefered/default profile, set AWS_PROFILE=myprofile in system environment. With this method, you will not need to reference the profile in the batch file.
I have a deployment script checked in the Github which I checkout with the source code in the pipeline. I need to execute this shell script to the remote server to perform the tasks (create backup, start / stop Jboss). I am using ssh agent to ssh into the remote server. How can I execute this within the Jenkins pipeline, and get the output from the commands back to the pipeline?
Thank you