is there any way to enable extglob from ansible?
- name: copy files
sudo: yes
shell: shopt -s extglob
but I got error:
failed: [host] => {"changed": true, "cmd": "shopt -s extglob", "delta": "0:00:00.001410", "end": "2015-10-20 09:10:36.438309", "rc": 127, "start": "2015-10-20 09:10:36.436899", "warnings": []}
stderr: /bin/sh: 1: shopt: not found
FATAL: all hosts have already failed -- aborting
I need extglob to be on to run this command. this command excluding directory vendor from being copied.
cp -pav --parents `git diff --name-only master feature/deploy_2186 !(vendor)` /tmp/current
The command is running properly from terminal, but not from ansible task. and after reading some article, it needs to enable extglob, so I can use !(vendor) pattern to exlcude vendor directory.
error when run copy from ansible task
failed: [host] => {"changed": true, "cmd": "cp -pav --parents `git diff --name-only master new_release !(vendor)` /tmp/current", "delta": "0:00:00.003255", "end": "2015-10-20 09:22:16.387262", "rc": 1, "start": "2015-10-20 09:22:16.384007", "warnings": []}
stderr: fatal: ambiguous argument '!': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
cp: missing destination file operand after '/tmp/current'
Try 'cp --help' for more information.
my ansible task to do copy, which if I remove !(vendor) it works perfectly but it has vendor inside:
- name: copy files
shell: cp -pav --parents `git diff --name-only master feature/deploy_2186 !(vendor)` /tmp/current
args:
chdir: /var/www
tags: release
There are 3 ways you can address this.
1) Put the command into a shell script copy.sh and set shell: copy.sh.
#!/bin/bash
shopt -s extglob
cp -pav --parents `git diff --name-only master feature/deploy_2186 !(vendor)` /tmp/current
2) Use grep -v instead of extglob:
shell: cp -pav --parents `git diff --name-only master feature/deploy_2186 * | grep -v /vendor/` /tmp/current
3) Use bash to set extglob and run the cp command. You need to pass two separate lines to the ansible task variable. Since the syntax is Yaml it boils down to embed a newline in the shell string. Test it yourself.
shell: |
bash -c 'shopt -s extglob
cp -pav --parents `git diff --name-only master feature/deploy_2186 !(vendor)` /tmp/current'
or
shell: "bash -c 'shopt -s extglob \n cp -pav --parents `git diff --name-only master feature/deploy_2186 !(vendor)` /tmp/current'"
You may need to do something like this so that the shopt and cp commands are actually run within the same shell instance:
- name: copy files
sudo: yes
shell: shopt -s extglob && cp -pav --parents `git diff --name-only master feature/deploy_2186 !(vendor)` /tmp/current
Related
Bash script:
clonePath=/data/config/
git branch -r | fgrep -v 'origin/HEAD' | sed 's| origin/|git checkout |' > checkoutAllBranches.sh
chmod +x checkoutAllBranches.sh
echo "Fetch branch: `cat checkoutAllBranches.sh`"
./checkoutAllBranches.sh
git checkout master
git remote rm origin
rm checkoutAllBranches.sh
for config_dir in `ls -a`; do
cp -r $config_dir $clonePath/;
done
echo "API Config update complete..."
Dockerfile which issues this script execution
ENTRYPOINT ["sh","config-update-force.sh","|| true"]
The error below causes the container startup failure despite setting the command status to 0 manually using || true
ERROR:
Error:
cp: cannot create regular file '/data/./.git/objects/pack/pack-27a9d...fb5e368e4cf.pack': Permission denied
cp: cannot create regular file '/data/./.git/objects/pack/pack-27a9d...fbae25e368e4cf.idx': Permission denied
I am looking for 2 options here:
Change these file permissions and then store them in the remote with rwx permissions
Do something to the docker file to ignore this script failure error and start the container.
DOCKERFILE:
FROM docker.hub.com/java11-temurin:latest
USER root
RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get install -y rsync telnet vim wget git
RUN mkdir -p /opt/config/clone/data
RUN chown -R 1001:1001 /opt/config
USER 1001
ADD build/libs/my-api-config-server.jar .
ADD config-update-force.sh .
USER root
RUN chmod +x config-update-force.sh
USER 1001
EXPOSE 8080
CMD java $BASE_JAVA_OPTS $JAVA_OPTS -jar my-api-config-server.jar
ENTRYPOINT ["sh","config-update-force.sh","|| true"]
BASH SCRIPT:
#!/bin/bash
set +e
set +x
clonePath=/opt/clone/data/data
#source Optumfile.properties
echo "properties loaded: example ${git_host}"
if [ -d my-api-config ]; then
rm -rf my-api-config;
echo "existing my-api-config dir deleted..."
fi
git_url=https://github.com/my-api-config-server
git clone https://github.com/my-api-config-server
cd my-api-config-server
git branch -r | fgrep -v 'origin/HEAD' | sed 's| origin/|git checkout |' > checkoutAllBranches.sh
chmod +x checkoutAllBranches.sh
echo "Fetch branch: `cat checkoutAllBranches.sh`"
./checkoutAllBranches.sh
git checkout master
git remote rm origin
rm checkoutAllBranches.sh
for config_dir in `ls -a`; do
cp -r $config_dir $clonePath/;
done
echo "My API Config update complete..."
When you do in the script...
chmod +x checkoutAllBranches.sh
...than why not before cp
chmod -R +rwx ${clonePath}
...or if the stderr message 'wont impact anything'...
cp -r $config_dir $clonePath/ 2>/dev/null;
...even cp dont copy -verbosly.
?
When your Dockerfile declares an ENTRYPOINT, that command is the only thing the container does. If it also declares a CMD, the CMD is passed as additional arguments to the ENTRYPOINT; it is not run on its own unless the ENTRYPOINT makes sure to execute it.
Shell errors are not normally fatal, and especially if you explicitly set +e, even if a shell command fails the shell script will keep running. You see this in your output where you get multiple cp errors; the first error does not terminate the script.
You need to do two things here. The first is to set the ENTRYPOINT to actually run the CMD; the simplest and most common way to do this is to end the script with
exec "$#"
The second is to remove the || true from the Dockerfile. As you have it written out currently, this is passed as the first argument to the entrypoint wrapper – it is not run through a shell and it is not interpreted as a "or" operator. If your script begins with a "shebang" line and is marked executable (both of these are correct in the question) the you do not explicitly need the sh interpreter.
# must be a JSON array; no additional "|| true" argument; no sh -c wrapper
ENTRYPOINT ["./config-update-force.sh"]
# any valid CMD will work with `exec "$#"
CMD java $BASE_JAVA_OPTS $JAVA_OPTS -jar my-api-config-server.jar
I'm trying to deploy my flask application to AWS EC2 instance using gitlab ci runner.
.gitlab.ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master # Run this job only on changes for stage branch
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
if [ ${shell} -d "/flask-ci-cd" ] # check if directory exists
then
eval "${shell} cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "${shell} git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
Error: .gitlab-deploy-prod.sh: line 7: -o: command not found
How can i check if directory existing??
What i've tried.
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
eval "${shell}" # i thought gitlab would provide me with shell access
echo "Deploy project on server ${SERVER_URL}"
if [-d "/flask-ci-cd" ] # check if directory exists
then
eval "cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
I've tried to log into the ssh shell before executing the scripts inside if else. But it doesn't works the way intended.
Your script has some errors.
Do not use eval. No, eval does not work that way. eval is evil
When storing a command to a variable, do not use normal variables. Use bash arrays instead to preserve "words".
Commands passed via ssh are double escaped. I would advise to prefer to use here documents, they're simpler to get the quoting right. Note the difference in expansion when the here document delimiter is quoted or not.
i thought gitlab would provide me with shell access No, without open standard input the remote shell will just terminate, as it will read EOF from input. No, it doesn't work that way.
Instead of doing many remote connection, just transfer the execution to remote side once and do all the work there.
Take your time and research how quoting and word splitting works in shell.
git_token=$DEPLOY_TOKEN No, set variables are not exported to remote shell. Either pass them manually or expand them before calling the remote side. (Or you could also use ssh -o SendEnv=git_token and configure remote ssh with AcceptEnv=git_token I think, never tried it).
Read documentation for the utilities you use.
No, git clone doesn't take branch name after url. You can specify branch with --branch or -b option. After url it takes directory name. See git clone --help. Same for git pull.
How can i check if directory existing??
Use bash arrays to store the command. Check if the directory exists just by executing the test command on the remote side.
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
if "${shell[#]}" [ -d "/flask-ci-cd" ]; then
...
In case of directory name with spaces I would go with:
if "${shell[#]}" sh <<'EOF'
[ -d "/directory with spaces" ]
EOF
then
Pass set -x to sh to see what's happening also on the remote side.
For your script, try rather to move the execution to remote side - there is little logic in making 3 separate connections. I say just
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
EOF
But instead of getting the quoting in some cases right, use declare -p and declare -f to transfer properly quoted stuff to remote side. That way you do not need case about proper quoting - it will work naturally:
echo "Deploy project on server ${SERVER_URL}"
work() {
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:"${git_token}"#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
{
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
$(declare -p git_token) # transfer variables you need
$(declare -f work) # transfer function you need
work # call the function.
EOF
Updated answer for future reads.
.gitlab-ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d flask-ci-cd ]; then
echo "\n Cloning into remote repo..."
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git
# Create and activate virtualenv
echo "\n Creating virtual env"
python3 -m venv env
else
echo "Pulling remote repo origin..."
cd flask-ci-cd
git pull
cd ..
fi
# Activate virtual env
echo "\n Activating virtual env..."
source env/bin/activate
# Install packages
cd flask-ci-cd/
echo "\n Installing dependencies..."
pip install -r requirements.txt
EOF
There is a test command which is explicit about checking files and directories:
test -d "/flask-ci-cd" && eval $then_commands || eval $else_commands
Depending on the AWS instance I'd expect "test" to be available. I'd recommend putting the commands in variables. (e.g. eval $then_commands)
I'm on ubuntu 15.04, my version of ssh client is
OpenSSH_6.9p1 Ubuntu-2ubuntu0.2, OpenSSL 1.0.2d 9 Jul 2015
When I try to run the following command ssh admin#server bash -c 'cd /path/to/repo && git pull' the cd is not effective and i got
fatal: Not a git repository (or any of the parent directories): .git
However If I do
ssh admin#server bash -c 'echo test && cd /path/to/repo && git pull'
then it works
Already up-to-date.
Of course I'm well aware echo is not supposed to change anything but after trying several time, several days on several different servers (though all on debian) I'm now sure to have this error.
On other servers I tried the command cd /tmp && pwd , and I got my home directory, and if i do echo toto && /tmp && pwd I go /tmp printed...
Unfortunately, ssh passes through a single command line string to $SHELL -c on the remote. Your quotes aren't being effective.
When you run
ssh admin#server bash -c 'cd /path/to/repo && git pull'
this is being run on the remote server (with $SHELL -c):
bash -c cd /path/to/repo && git pull
So Bash is given single command (cd) and an unused argument, and then separately, you're also running git pull in the home directory.
On the other hand, when you run
ssh admin#server bash -c 'echo test && cd /path/to/repo && git pull'
this is being run on the remote server:
bash -c echo test && cd /path/to/repo && git pull
The first part is again useless, but the shell running the whole command then does cd /path/to/repo and git pull. Which works.
What you probably want to do is
ssh admin#server 'cd /path/to/repo && git pull'
The existing answer by ephemient is entirely correct in terms of cause.
To add an alternate solution -- one which works when your remote code contains constructs which sh -c will misinterpret -- consider:
repo=/path/to/repo ## here, this works even when your path contains
## nonprintable or otherwise surprising characters
printf -v repo_q '%q' "$repo" ## ...because we're asking your local copy of bash
## to generate a quoted/escaped copy of the value
## that will 'eval' back to its original meaning
## when interpreted by bash
## to ensure that it's interpreted by bash, we pass 'bash -s' as the command to ssh
## with an *unquoted* heredoc (<<EOF, vs <<'EOF'), with the escaped value expanded
ssh admin#server 'bash -s' <<EOF
cd $repo_q && git pull
EOF
I am using the following to export all the changed files from my master branch:
git archive --output=test_v2.zip HEAD $(git diff --name-only master)
however it gives me the following error:
error: unknown option `name-only'
usage: git archive [<options>] <tree-ish> [<path>...]
or: git archive --list
or: git archive --remote <repo> [--exec <cmd>] [<options>] <tree-ish> [<path>...]
or: git archive --remote <repo> [--exec <cmd>] --list
--format <fmt> archive format
--prefix <prefix> prepend prefix to each pathname in the archive
-o, --output <file> write the archive to this file
--worktree-attributes
read .gitattributes in working directory
-v, --verbose report archived files on stderr
-0 store only
-1 compress faster
-9 compress better
-l, --list list supported archive formats
--remote <repo> retrieve the archive from remote repository <repo>
--exec <command> path to the remote git-upload-archive command
what am I doing wrong?
it creates test_v2.zip but it is empty.
based on my research I think I should not use $ sign but I don't know what to use then.
Your example works fine in bash. But you try to run it in cmd.exe. The $() syntax is Bash syntax to start a subshell, execute the enclosed command and replace the construct with the output.
Either do the same in Git Bash (anyway the better choice) or use a cmd.exe specific construct instead if one exists.
I'm trying to get this task to run locally (on the machine that is running the playbook) :
- name: get the local repo's branch name
local_action: git branch | awk '/^\*/{print $2}'
register: branchName
I tried plenty of variations with no success
all other tasks are meant to run on the target host, which is why running the whole playbook local is not an option
TASK: [get the local repo's branch name] **************************************
<127.0.0.1> REMOTE_MODULE git branch | awk '/^\*/{print $2}'
<127.0.0.1> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1407258765.57-75899426008172 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1407258765.57-75899426008172 && echo $HOME/.ansible/tmp/ansible-tmp-1407258765.57-75899426008172']
<127.0.0.1> PUT /tmp/tmpQVocvw TO /home/max/.ansible/tmp/ansible-tmp-1407258765.57-75899426008172/git
<127.0.0.1> EXEC ['/bin/sh', '-c', '/usr/bin/python /home/max/.ansible/tmp/ansible-tmp-1407258765.57-75899426008172/git; rm -rf /home/max/.ansible/tmp/ansible-tmp-1407258765.57-75899426008172/ >/dev/null 2>&1']
failed: [portal-dev] => {"failed": true}
msg: this module requires key=value arguments (['branch', '|', 'awk', '/^\\*/{print $2}'])
FATAL: all hosts have already failed -- aborting
update:
I have followed bkan's suggestion (bellow), and got a bit further, but
- name: get the local repo's branch name
local_action: command git branch | (awk '/^\*/{print $2}')
sudo: no
register: branchName
now the git command gets launched but not correctly (see error below).
note that this command runs perfectly as a "shell" but unfortunately there is no local_shell equivalent of local_action ...
failed: [portal-dev] => {"changed": true, "cmd": ["git", "branch", "|", "(awk", "/^\\*/{print $2})"], "delta": "0:00:00.002980", "end": "2014-08-05 18:00:01.293632", "rc": 129, "start": "2014-08-05 18:00:01.290652"}
stderr: usage: git branch [options] [-r | -a] [--merged | --no-merged]
or: git branch [options] [-l] [-f] <branchname> [<start-point>]
or: git branch [options] [-r] (-d | -D) <branchname>...
or: git branch [options] (-m | -M) [<oldbranch>] <newbranch>
...
The format for local_action is:
local_action: <module_name> <arguments>
In your example, Ansible thinks you are trying to use the git module and throws an error because you don't have the correct arguments for the git module. Here is how it should look:
local_action: shell git branch | awk '/^\*/{print $2}'
Source: http://docs.ansible.com/playbooks_delegation.html#delegation