How to pull image from Docker hub to EC2 using bitbucket pipeline - amazon-ec2

I am trying to implement CICD using bitbucket pipelines.
So far I was able to create the image and push it to docker hub. Seems straightforward and the internet is full of tutorials.
But, to pull the image from an EC2 instance and run the image I didnt find anything.
I have this bitbucket-pipeline.yml file:
image: atlassian/default-image:latest
pipelines:
default:
- step:
services:
- docker
script:
- export IMAGE_NAME=juanibe/vinimayapi:$BITBUCKET_COMMIT
- docker build -t $IMAGE_NAME .
- docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- docker push $IMAGE_NAME
And I have this script, but I dont know where tu put it:
#!bin/bash
sudo docker ps
echo 'Login in to docker'
docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD // How can I set env variable here?
echo 'Fetching latest image'
sudo docker pull user/vinimayapi:latest
echo 'Stoping current container'
sudo docker stop cont_docker_app_test
echo 'Removing old container'
sudo docker rm cont_docker_app_test-old
echo 'Rename stoped container'
sudo docker rename user/cont_docker_app_test user/cont_docker_app_test_old
echo 'Starting new container'
sudo docker run -d --name cont_docker_app_test -p 443:3333 -p 8001:8001 --link my-mongo-testing:my-mongo-testing user/vinimayapi:latest
Any help will be really appreciated, I've been trying to create a pipeline for days without success.

Add to your pipeline additional step:
- pipe: "atlassian/ssh-run:0.2.4"
variables:
SSH_USER: user
SERVER: ip_server
SSH_KEY: sshkey
MODE: script
COMMAND: script.sh
It should look like below:
image: atlassian/default-image:latest
pipelines:
default:
- step:
services:
- docker
script:
- export IMAGE_NAME=juanibe/vinimayapi:$BITBUCKET_COMMIT
- docker build -t $IMAGE_NAME .
- docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- docker push $IMAGE_NAME
- pipe: "atlassian/ssh-run:0.2.4"
variables:
SSH_USER: user
SERVER: ip_server
SSH_KEY: sshkey
MODE: script
COMMAND: script.sh
script.sh file located in that case is located in the same directory as bitbucket_pipelines.yml

Related

gitlab-ci predefined is defined in script step of deploy stage but undefined inside bash script run via 'bash -s'

I am trying to deploy a branch other than the default (master) branch. For some reason the predefined variables are not visible inside the roll-out.sh script. But the echo statements before calling the script do print the variables correctly.
I have another script that rolls out the master branch. In this script it is able to run docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY with no problems.
I have tried with the branch both protected and not protected. The variables are still undefined for some reason.
What I am doing wrong here?
build:
stage: build
image: docker:20.10.12
services:
- docker:20.10-dind
rules:
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH # not master branch
allow_failure: true
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker pull $CI_REGISTRY_IMAGE/client/base:latest || true
- docker build --cache-from $CI_REGISTRY_IMAGE/client/base:latest --cache-from $CI_REGISTRY_IMAGE/client:latest -t $CI_REGISTRY_IMAGE/client:$CI_COMMIT_BRANCH .
- docker push $CI_REGISTRY_IMAGE/client:$CI_COMMIT_BRANCH
deploy:
variables:
branch: $CI_COMMIT_BRANCH
stage: deploy
image: alpine
rules:
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH # not master branch
before_script:
- apk add openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
script:
- echo $CI_COMMIT_REF_NAME
- echo $CI_COMMIT_BRANCH
- echo $branch
- ssh -o StrictHostKeyChecking=no $SERVER_USER#$DOMAIN 'bash -s' < ./script/roll_out.sh
If you are sshing to another machine within your ci script I don’t think it would have accsess to the variables you are echoing out in the lines before because it’s a new session on a different machine.
You could try a few different things though to achieve what you want;
Try to send the variables as arguments (not great sending information to a machine through ssh).
Install a gitlab runner on the host you are trying to ssh to, tag this runner so it only runs the specific deployment job and then you’ll have the variables available on the host.
The problem was solved by explicitly passing the environment variables to the script
- ssh -o StrictHostKeyChecking=no $SERVER_USER#$DOMAIN "CI_REGISTRY_IMAGE=$CI_REGISTRY_IMAGE CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH bash -s" < ./script/roll_out_tagged.sh
I had another developer with lots of experience take a look at it.
He had no idea why docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY worked. That is what lead me to believe that all of the environment variables would be available on the remote. But seems like those three variables are some sort of exception.

GItLab is not deploying laravel app to AWS ec2

I am trying to deploy my Laravel app to AWS ec2 instance. I m using GitLab for code management and pipeline process.
Here is my .gitlab-ci.yml file.
# Node docker image on which this would be run
image: node:8.9.0
#This command is run before actual stages start running
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- npm install
stages:
- test
- deploy
- production
#Production stage
production:
stage: production
before_script:
#generate ssh key
- mkdir -p ~/.ssh
- echo -e "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy.sh
environment:
name: production
url: MY_HOST_IP_ADDRESS
when: manual
# lint and test are two different jobs in the same stage.
# This allows us to run these two in parallel and making build faste
And here is my .gitlab-deploy.sh file
#!/bin/bash
#Get servers list
set -f
string=$DEPLOY_SERVER
array=(${string//,/ })
#Iterate servers for deploy and pull last commit
for i in "${!array[#]}"; do
echo "Deploying information to EC2 and Gitlab"
echo "Deploy project on server ${array[i]}"
ssh ubuntu#${array[i]} "cd /var/www/html && git pull origin"
done
When I push my code it's processing fine. As you can see below image.
After a successful process when I list the directory of /var/www/html it's still empty. I am using Apache2 on Ubuntu. I want to deploy my code directly AWS EC2 instance.
Thanks in advance!

./deploy.sh not working on gitlab ci

My problem is the bash script I created got this error "/bin/sh: eval: line 88: ./deploy.sh: not found" on gitlab. Below is my sample script .gitlab-ci.yml.
I suspect that gitlab ci is not supporting bash script.
image: docker:latest
variables:
IMAGE_NAME: registry.gitlab.com/$PROJECT_OWNER/$PROJECT_NAME
DOCKER_DRIVER: overlay
services:
- docker:dind
stages:
- deploy
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
- docker pull $IMAGE_NAME:$CI_BUILD_REF_NAME || true
production-deploy:
stage: deploy
only:
- master#$PROJECT_OWNER/$PROJECT_NAME
script:
- echo "$PRODUCTION_DOCKER_FILE" > Dockerfile
- docker build --cache-from $IMAGE_NAME:$CI_BUILD_REF_NAME -t $IMAGE_NAME:$CI_BUILD_REF_NAME .
- docker push $IMAGE_NAME:$CI_BUILD_REF_NAME
- echo "$PEM_FILE" > deploy.pem
- echo "$PRODUCTION_DEPLOY" > deploy.sh
- chmod 600 deploy.pem
- chmod 700 deploy.sh
- ./deploy.sh
environment:
name: production
url: https://www.example.com
And this also my deploy.sh.
#!/bin/bash
ssh -o StrictHostKeyChecking=no -i deploy.pem ec2-user#targetIPAddress << 'ENDSSH'
// command goes here
ENDSSH
All I want is to execute deploy.sh after docker push but unfortunately got this error about /bin/bash thingy.
I really need your help guys. I will be thankful if you can solve my problem about gitlab ci bash script got error "/bin/sh: eval: line 88: ./deploy.sh: not found".
This is probably related to the fact you are using Docker-in-Docker (docker:dind). Your deploy.sh is requesting /bin/bash as the script executor which is NOT present in that image.
You can test this locally on your computer with Docker:
docker run --rm -it docker:dind bash
It will report an error. So rewrite the first line of deploy.sh to
#!/bin/sh
After fixing that you will run into the problem that the previous answer is addressing: ssh is not installed either. You will need to fix that too!
docker:latest is based on alpine linux which is very minimalistic and does not have a lot installed by default. For example, ssh is not available out of the box, so if you want to use ssh commands you need to install it first. In your before_script, add:
- apk update && apk add openssh
Thanks. This worked for me by adding bash
before_script:
- apk update && apk add bash
Let me know if that still doesn't work for you.

How can I solve "crontab: your UID isn't in the passwd file. bailing out."?

Hi I'm using Docker and whenever to write cron schedule rules, but when I run whenever --update-crontab in my docker container this errors is showing to me.
crontab: your UID isn't in the passwd file.
bailing out.
[fail] Couldn't write crontab; try running `whenever' with no options to ensure your schedule file is valid.
Dockerfile
FROM ruby:2.4.1-slim
RUN apt-get update && apt-get -y install cron
ENV RAILS_ENV production
ENV INSTALL_PATH /app
RUN mkdir -p $INSTALL_PATH
WORKDIR $INSTALL_PATH
COPY Gemfile Gemfile.lock ./
RUN bundle install --binstubs --jobs 20 --retry 5
COPY . .
RUN chown -R nobody:nogroup /app
USER nobody
# use docker run -it --entrypoint="" demo "ls -la" to skip
EXPOSE 3000
CMD puma -C config/puma.rb
Docker Version: Docker version 17.05.0-ce, build 89658be
My Docker compose file
chatbot_web:
container_name: chatbot_web
depends_on:
- postgres
- chatbot_redis
- chatbot_lita
user: "1000:1000"
build: .
image: dpe/chatbot
ports:
- '3000:3000'
volumes:
- '.:/app'
restart: always
How can I solve this?
EDIT:
When I use:
host$ docker run -it dpe/chatbot bash
container $ whenever --update-cron
[write] crontab file updated
Works, but when I use:
host$ docker exec -it chatbot_web bash
I have no name!#352c6a7500d2:/app$ whenever --update-cron
crontab: your UID isn't in the passwd file.
bailing out.
[fail] Couldn't write crontab; try running `whenever' with no options to ensure your schedule file is valid.
Don't Work =(
To fix I use same user in Dockerfile and docker-compose
Dockerfile
RUN chown -R nobody:nogroup /app
USER nobody
Docker Compose
chatbot_web:
user: "nobody:nogroup"

How to set hosts in docker for mac

When I use docker before, I can use docker-machine ssh default to set hosts in docker's machine /etc/hosts, but in docker for mac I can't access it's VM because of it don't have it.
So, the problem is how to set hosts in docker for mac ?
My secondary domain wants to point the other ip.
I found a solution, use this command
screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Now, edit the /etc/hosts in the Docker VM.
To exit screen, use Ctrl + a + d.
Here's how I do it with a bash script so the changes persist between Docker for Mac restarts.
cd ~/Library/Containers/com.docker.docker/Data/database
git reset --hard
DFM_HOSTS_FILE="com.docker.driver.amd64-linux/etc/hosts"
if [ ! -f ${DFM_HOSTS_FILE} ]; then
echo "appending host to DFM /etc/hosts"
echo -e "xxx.xxx.xxx.xxx\tmy.special.host" > ${DFM_HOSTS_FILE}
git add ${DFM_HOSTS_FILE}
git commit -m "add host to /etc/hosts for dns lookup"
fi
You can automate it via this script, run this scrip on start up time or login time will save you..
#!/bin/sh
# host entry -> '10.4.1.4 dockerreigstry.senz.local'
# 1. run debian image
# 2. check host entry exists in /etc/hosts file
# 3. if not exists add it to /etc/hosts file
docker run --name debian -it --privileged --pid=host debian nsenter \
-t 1 -m -u -n -i sh \
-c "if ! grep -q dockerregistry.senz.local /etc/hosts; then echo -e '10.4.1.4\tdockerregistry.pagero.local' >> /etc/hosts; fi"
# sleep 2 seconds
# remove stopped debian container
sleep 2
docker rm -f debian
I have created a blog post with more information about this topic.
https://medium.com/#itseranga/set-hosts-in-docker-for-mac-2029276fd448
You must have to create an docker-compose.yml file. This file will be on the same route of your Dockerfile
For example, I use this docker-compose.yml file:
version: '2'
services:
app:
hostname: app
build: .
volumes:
- ./:/var/www/html
working_dir: /var/www/html
depends_on:
- db
- cache
ports:
- 80:80
cache:
image: memcached:1.4.27
ports:
- 11211:11211
rabbitmq:
image: rabbitmq:latest
ports:
- 5672:5672
db:
image: postgres:9.5.3
ports:
- 5432:5432
environment:
- TZ=America/Mazatlan
- POSTGRES_PASSWORD=root
- POSTGRES_DB=restaurantcore
- POSTGRES_USER=rooms
- POSTGRES_PASSWORD=rooms
The ports are binding with the ports of your host docker machine.

Resources