Multi-line bash script in yaml for EC2 Image Builder - amazon-ec2

I'm trying to create a custom component document. While I've tested the yaml file using various yaml linters, EC2 Image builder is complaining with the below error
Failed to create component. Fix the error(s) and try again:
The value supplied for parameter 'data' is not valid. Parsing step 'ConfigureMySQL' in phase 'build' failed. Error: line 4: cannot unmarshal map into string.
And I'm unable to figure out what is wrong with my yaml file
name: MyJavaAppTestDocument
description: This is JavaApp Document
schemaVersion: 1.0
phases:
- name: build
steps:
- name: InstallSoftware
action: ExecuteBash
inputs:
commands:
- sudo yum update -y
- sudo yum install -y java-1.8.0
- sudo amazon-linux-extras install -y tomcat8.5
- sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
- sudo yum install -y mysql-community-server
- name: ConfigureTomcat
action: ExecuteBash
inputs:
commands:
- sudo sed -i 's/<\/tomcat-users>/\n<role rolename="manager-gui"\/>\n <role
rolename="manager-script"\/>\n <role rolename="admin-gui"\/>\n <user username="admin"
password="admin" roles="manager-gui,manager-script,admin-gui"\/>\n<\/tomcat-users>/'
/etc/tomcat/tomcat-users.xml
- sudo systemctl start tomcat
- sudo systemctl enable tomcat
- name: ConfigureMySQL
action: ExecuteBash
inputs:
commands:
- sudo systemctl start mysqld
- sudo systemctl enable mysqld
- mysqlpass=$(sudo grep 'temporary password' /var/log/mysqld.log | sed 's/.*root#localhost: //')
- mysql -u root -p$mysqlpass --connect-expired-password -h localhost -e "ALTER USER 'root'#'localhost' IDENTIFIED BY 'whyDoTh1s#2020'"
- |
sudo cat <<EoF > /tmp/mysql-create-user.sql
CREATE USER 'admin'#'%' IDENTIFIED BY 'whyDoTh1s#2020';
GRANT ALL PRIVILEGES ON *.* TO 'admin'#'%' WITH GRANT OPTION;
EoF
- sudo mysql -u root -pwhyDoTh1s#2020 -h localhost < /tmp/mysql-create-user.sql
Appreciate if someone could help me find the error. The objective is to build an AMI with pre-configured software and settings.

You're getting that error because of the : in this line:
- mysqlpass=$(sudo grep 'temporary password' /var/log/mysqld.log | sed 's/.*root#localhost: //')
The YAML parser is interpreting that line as creating a map instead of creating a string entry in an existing map. A workaround I was able to use in my own YAML was to surround the : with single quotes, so the problem line would become
- mysqlpass=$(sudo grep 'temporary password' /var/log/mysqld.log | sed 's/.*root#localhost':' //')
According to my own use and this online YAML parser I tested with that solution applied, that should do the trick.

Although #micah-l-c answer can help in OP's case.
I would suggest an alternate approach which can also help and can be more correct solution towards this problem.
Erroneous line here is
- mysqlpass=$(sudo grep 'temporary password' /var/log/mysqld.log | sed 's/.*root#localhost: //')
This can be rewritten as
- |-
mysqlpass=$(sudo grep 'temporary password' /var/log/mysqld.log | sed 's/.*root#localhost: //')
OR
- >-
mysqlpass=$(sudo grep 'temporary password' /var/log/mysqld.log | sed 's/.*root#localhost: //')
Now : will be escaped fine with above approach.
I faced a similar issue and I will give an example where #micah-l-c solution didn't work and above approach worked fine.
Below was line i was adding in the ec2 image builder component
- echo "user1 ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
I put single quotes around : and it made /etc/sudoers malformed and thus it didn't work.
I replace above line like this
- |-
echo "user1 ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
References :
How to escape indicator characters (i.e. : or - ) in YAML

Related

Getting error when try to list all users on the hosts

I am trying to get all the users I created on the hosts machine. When I run the following command on terminal, I get all the users on the machine.
sudo getent passwd {1000..6000} | cut -d":" -f1
However when I try to run it using ansible, I get an error. I tried all like enquoting in double quotes, escaping the brackets, piping the output to cat etc but nothing is working.
---
- name: "run commands"
become: true
gather_facts: no
hosts: all
tasks:
- name: list all users
shell: getent passwd {1000..6000} | cut -d":" -f1
register: getent
- debug: var=getent.stdout_lines
Note that, per default, Ansible is using /bin/sh, as pointed in the synopsis of the command.
It is almost exactly like the ansible.builtin.command module but runs the command through a shell (/bin/sh) on the remote node.
Source: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/shell_module.html#synopsis
But sh won't interpret a sequence construct like {0..10}.
There are two ways you can overcome this:
Using seq rather:
- shell: getent passwd $(seq 1000 6000) | cut -d":" -f1
register: getent
Specifying to the shell task that you want it executed via bash:
- shell: getent passwd {1000..6000} | cut -d":" -f1
register: getent
args:
executable: /bin/bash

Execute sudo command non-interactively

I would like to execute the following command without interaction:
sudo grep -e "test" /etc/sudoers
I have tried the following method:
tester#compute:~$ echo 'clouduser' | sudo -S grep -e "test" /etc/sudoers
[sudo] password for tester: test ALL=(ALL) NOPASSWD: ALL
The problem is that I am getting the [sudo] password for tester: in front of the response.
How I can cut that part from the front of the answer?
Thanks!
I will answer to my question - maybe someone else will need it:
(echo 'clouduser' | sudo -Si >/dev/null 2>&1); sudo grep -e test /etc/sudoers
Add the following line to your /etc/sudoers file in order to turn on password-less sudo. In this case, I use john as the login account. Change to your own account id.
john ALL=(ALL:ALL) ALL
Alternatively, and perhaps better is to put that line into a file called /etc/sudoers.d/john.

Getting Lego error passing email address using --email

To renew a LetsEncrypt certificate I was following this Bitnami doc and:
sudo /opt/bitnami/letsencrypt/lego --tls --email="somebody#example.com" --domains="example.com" --domains="www.example.com" --path="/opt/bitnami/letsencrypt" renew --days 90
kept returning the error:
You have to pass an account (email address) to the program using --email or -m
I tried retyping the hyphens and quotations in case they were being changed to en dashes and inverted commas in copy/paste, but no luck. Appreciate pointers to what I may be doing wrong or could do differently.
Bitnami Engineer here. It seems you have an old version of the lego binary in your instance, you have 2 options:
Use that previous version: In this case you will need to run this command instead
sudo /opt/bitnami/letsencrypt/lego --email="somebody#example.com" --domains="example.com" --domains="www.example.com" --path="/opt/bitnami/letsencrypt" renew --days 90
Note: I removed the --tls parameter
Update the lego binary
cd /tmp
curl -Ls https://api.github.com/repos/xenolf/lego/releases/latest | grep browser_download_url | grep linux_amd64 | cut -d '"' -f 4 | wget -i -
tar xf lego_v2.6.0_linux_amd64.tar.gz
sudo mkdir -p /opt/bitnami/letsencrypt
sudo mv lego /opt/bitnami/letsencrypt/lego
Note: 2.6.0 was the latest version of the lego binary when writing this message
and then run your command again
sudo /opt/bitnami/letsencrypt/lego --tls --email="somebody#example.com" --domains="example.com" --domains="www.example.com" --path="/opt/bitnami/letsencrypt" renew --days 90
I hope this helps

Ansible playbook put find exhaust to text file with sed

How to start a playbook when pulling a playbook and insert it into the config? for example in /etc/nginx/sites-available/default
need to insert this
fastcgi_pass unix: /var/run/php/php7.0-fpm.sock;
How to insert for example 7.1 or 7.2 instead of 7.0 if the PHP versions will differ?
Partially found the solution using find, but how now to insert the result of find into the file?
I use the command: find /var/run/php/ -name *.sock
exhaust such: /var/run/php/php7.2-fpm.sock
Now this case should be inserted into the file by sed
An option would be to use lineinfile
- shell: "php -n -v | head -n 1 | cut -d ' ' -f 2 | cut -d '.' -f 1,2"
register: php_version
- lineinfile:
path: /etc/nginx/sites-available/default
regexp: "^fastcgi_pass unix:"
line: "fastcgi_pass unix: /var/run/php/php{{ php_version.stdout }}-fpm.sock;"

Passing a private key through Fly Cli

The company I work for has recently started to use Concourse CI to do all our CI needs. At the current moment one of my jobs consists of a task with a script that scp's and ssh's into our aws ec2 instances and configures those servers. The issue I am having, however, is getting the private key to ssh into those instances. One way discussed here (https://concourse-ci.org/fly-set-pipeline.html) is to pass the key in through a variable. In my script, I take that variable passed in and echo it to a new .pem file and I set the permissions to 600. When I echo just the variable and later cat the new .pem file, they look exactly the same as the original .pem file. The container I am trying to ssh from is the standard ubuntu docker image.
When I try to use this file to scp and ssh I am confronted with the prompt about entering the passphrase. If I try to ssh with the original file I don't get this prompt at all. Is there something I am missing? I would greatly appreciate some insight into this issue.
pipeline.yml
jobs:
- name: edge-priceconfig-deploy
plan:
- aggregate:
- get: ci-git
- get: pricing-config
trigger: true
- task: full-price-deploy
file: ci-git/ci/edge/edge-price-config-task.yml
params:
USER_AND_SERVER: {{edge_user_and_server}}
DEPLOY_KEY_PAIR: {{deploy_key_pair}}
task.yml
---
platform: linux
image_resource:
type: docker-image
source: {repository: ubuntu}
inputs:
- name: ci-git
- name: pricing-config
run:
path: ./ci-git/ci/edge/edge-priceconfig-deploy.sh
task.sh
#!/bin/bash
touch DeployKeyPair.pem
echo $DEPLOY_KEY_PAIR
echo $DEPLOY_KEY_PAIR > DeployKeyPair.pem
cat DeployKeyPair.pem
apt-get update && apt-get -y install sudo
sudo apt-get -y install openssh-client
sudo chmod 400 ci-git/key/DeployKeyPair.pem
sudo chmod 600 DeployKeyPair.pem
mkdir company-price-config-edge
mv pricing-config/fsconfig/conf/com.company.api.v1.pricing/*.xlsx company-price-config-edge/
commandstr="sudo rm -f /etc/company/edge/fsconfig/*xlsx; \
ls -l /etc/company/edge/fsconfig; \
sudo mv /home/ec2-user/company-price-config-edge/*xlsx /etc/company/edge/fsconfig/; \
sudo rm -rf /home/ec2-user/company-price-config-edge;"
scp_link="$USER_AND_SERVER:/home/ec2-user/"
scp_link="$(echo $scp_link | tr -d ' ')"
echo $scp_link
sudo echo -ne '\n' | scp -r -oStrictHostKeyChecking=no -i DeployKeyPair.pem company-price-config-edge $scp_link
sudo ssh -oStrictHostKeyChecking=no -i DeployKeyPair.pem $USER_AND_SERVER $commandstr
credentials.yml
username: |
username
password: |
password
access_token: |
token
ci_scripts_github: |
ci-script-link
edge_user_and_server: |
server.com
staging_user_and_server: |
staging
training_user_and_server: |
training
production_user_and_server: |
production
deploy_key_pair:
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
Had this same problem today and took me a bit to figure out what was going on.
You're specifying the private key as a multiline yaml property, but when it's being echoed out to a file it's stripping out the linebreaks and replacing them with spaces, so your key is a large single line file.
I had to use sed to replace the space with newlines to get it to work
echo $DEPLOY_KEY_PAIR | sed -e 's/\(KEY-----\)\s/\1\n/g; s/\s\(-----END\)/\n\1/g' | sed -e '2s/\s\+/\n/g' > DeployKeyPair.pem
First sed command uses two sets of replacement groups, both with the same purpose
First group - Grab header text and break it out to a new line
s/\(KEY-----\)\s/\1\n/g
\(KEY-----\) puts the last part of the header block in a capture group
\s matches whitespace
/1\n/g Puts capture group (#1) back in, then adds a newline
/g Unnecessary global capture that I forgot to remove
Second sed group does the same but grabs the space and first part of the footer text to put it on a new line
Third group does a global replacement on whitespace and replaces it with newline. This had to be a separate command so processing was finished for first command and it was then a 3-line file/stream
'2s/\s\+/\n/g'
2s/ Perform replacement only on line 2
\s\+ matches any whitespace
\n replace with newline
/g global match to grab every instance
Two things.
One: You need to declare that these particular environment variables are being used by your task
task.yml:
---
platform: linux
image_resource:
type: docker-image
source: {repository: ubuntu}
inputs:
- name: ci-git
- name: pricing-config
params:
USER_AND_SERVER:
DEPLOY_KEY_PAIR:
run:
path: ./ci-git/ci/edge/edge-priceconfig-deploy.sh
Two: I think you need to add a | to your credentials.yml file.
credentials.yml:
deploy_key_pair: |
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
you can also use a keystore integration, e.g. credhub, with concourse.
see https://docs.pivotal.io/p-concourse/credential-management.html

Resources