spring boot with ElasticSearch in docker - spring-boot

I am trying to connect my Spring boot micro services to an ElasticSearch service running in docker. due to the massive change in ElasticSearch in 8.x, I am not finding any relevant, but I did find a good docker-compose.xml example that does NOT contain Logstash, but does have multinode Elasticsearch and Kibana. I am able to docker compose up and log in with the password stored in the .env file. So far so good....but now I don't know how to get my Spring Boot 2.7 services to push logging statements to ElasticSearch so I can view them in Kibana.
Here is my docker-compose:
version: "2.2"
services:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
user: "0"
command: >
bash -c '
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable in the .env file";
exit 1;
fi;
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: es01\n"\
" dns:\n"\
" - es01\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: es02\n"\
" dns:\n"\
" - es02\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: es03\n"\
" dns:\n"\
" - es03\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
interval: 1s
timeout: 5s
retries: 120
es01:
depends_on:
setup:
condition: service_healthy
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
- esdata01:/usr/share/elasticsearch/data
ports:
- ${ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es02,es03
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
es02:
depends_on:
- es01
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
- esdata02:/usr/share/elasticsearch/data
environment:
- node.name=es02
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es01,es03
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es02/es02.key
- xpack.security.http.ssl.certificate=certs/es02/es02.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es02/es02.key
- xpack.security.transport.ssl.certificate=certs/es02/es02.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
I tried adding Logstash to this, but it fails because port 5000 seems to be allocated. And, from what I can tell its not needed explictly anymore?
My question is, I have this available on http://localhost:5601 but I could really use some help on how to get my logging output to ElasticSearch. I am using pretty standard Spring Boot configuration. Naturally this will eventually be cloud deployed, so ideally a solution for both local and AWS can be found.

Related

unable to execute a bash script in k8s cronjob pod's container

Team,
/bin/bash: line 5: ./repo/clone.sh: No such file or directory
cannot run above file but I can cat it well. I tried my best and still trying to find but no luck so far..
my requirement is to mount bash script from config map to a directory inside container and run it to clone a repo but am getting below message.
cron job
spec:
concurrencyPolicy: Allow
jobTemplate:
metadata:
spec:
template:
metadata:
spec:
containers:
- args:
- -c
- |
set -x
pwd && ls
ls -ltr /
cat /repo/clone.sh
./repo/clone.sh
pwd
command:
- /bin/bash
envFrom:
- configMapRef:
name: sonarscanner-configmap
image: artifactory.build.team.com/product-containers/user/sonarqube-scanner:4.7.0.2747
imagePullPolicy: IfNotPresent
name: sonarqube-sonarscanner
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /repo
name: repo-checkout
dnsPolicy: ClusterFirst
initContainers:
- args:
- -c
- cd /
command:
- /bin/sh
image: busybox
imagePullPolicy: IfNotPresent
name: clone-repo
securityContext:
privileged: true
volumeMounts:
- mountPath: /repo
name: repo-checkout
readOnly: true
restartPolicy: OnFailure
securityContext:
fsGroup: 0
volumes:
- configMap:
defaultMode: 420
name: product-configmap
name: repo-checkout
schedule: '*/1 * * * *'
ConfigMap
kind: ConfigMap
metadata:
apiVersion: v1
data:
clone.sh: |-
#!bin/bash
set -xe
apk add git curl
#Containers that fail to resolve repo url can use below step.
repo_url=$(nslookup ${CODE_REPO_URL} | grep Non -A 2 | grep Name | cut -d: -f2)
repo_ip=$(nslookup ${CODE_REPO_URL} | grep Non -A 2 | grep Address | cut -d: -f2)
if grep ${repo_url} /etc/hosts; then
echo "git dns entry exists locally"
else
echo "Adding dns entry for git inside container"
echo ${repo_ip} ${repo_url} >> /etc/hosts
fi
cd / && cat /etc/hosts && pwd
git clone "https://$RU:$RT#${CODE_REPO_URL}/r/a/${CODE_REPO_NAME}" && \
(cd "${CODE_REPO_NAME}" && mkdir -p .git/hooks && \
curl -Lo `git rev-parse --git-dir`/hooks/commit-msg \
https://$RU:$RT#${CODE_REPO_URL}/r/tools/hooks/commit-msg; \
chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
cd ${CODE_REPO_NAME}
pwd
output pod describe
Warning FailedCreatePodSandBox 1s kubelet, node1 Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "sonarqube-cronjob-1670256720-fwv27": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:303: getting the final child's pid from pipe caused \"EOF\"": unknown
pod logs
+ pwd
+ ls
/usr/src
+ ls -ltr /repo/clone.sh
lrwxrwxrwx 1 root root 15 Dec 5 16:26 /repo/clone.sh -> ..data/clone.sh
+ ls -ltr
total 60
.
drwxr-xr-x 2 root root 4096 Aug 9 08:58 sbin
drwx------ 2 root root 4096 Aug 9 08:58 root
drwxr-xr-x 2 root root 4096 Aug 9 08:58 mnt
drwxr-xr-x 5 root root 4096 Aug 9 08:58 media
drwxrwsrwx 3 root root 4096 Dec 5 16:12 repo <<<<< MY MOUNTED DIR
.
+ cat /repo/clone.sh
#!bin/bash
set -xe
apk add git curl
#Containers that fail to resolve repo url can use below step.
repo_url=$(nslookup ${CODE_REPO_URL} | grep Non -A 2 | grep Name | cut -d: -f2)
repo_ip=$(nslookup ${CODE_REPO_URL} | grep Non -A 2 | grep Address | cut -d: -f2)
if grep ${repo_url} /etc/hosts; then
echo "git dns entry exists locally"
else
echo "Adding dns entry for git inside container"
echo ${repo_ip} ${repo_url} >> /etc/hosts
fi
cd / && cat /etc/hosts && pwd
git clone "https://$RU:$RT#${CODE_REPO_URL}/r/a/${CODE_REPO_NAME}" && \
(cd "${CODE_REPO_NAME}" && mkdir -p .git/hooks && \
curl -Lo `git rev-parse --git-dir`/hooks/commit-msg \
https://$RU:$RT#${CODE_REPO_URL}/r/tools/hooks/commit-msg; \
chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
cd code_dir
+ ./repo/clone.sh
/bin/bash: line 5: ./repo/clone.sh: No such file or directory
+ pwd
pwd/usr/src
Assuming the working directory is different thant /:
If you want to source your script in the current process of bash (shorthand .) you have to add a space between the dot and the path:
. /repo/clone.sh
If you want to execute it in a child process, remove the dot:
/repo/clone.sh

GitLab CI rules not working with extends and individual rules

Below are two jobs in the build stage.
Default, there is set some common condition, and using extends keyword for that, ifawsdeploy.
As only one of them should run, if variable $ADMIN_SERVER_IP provided then connect_admin_server should run, working that way.
If no value provided to $ADMIN_SERVER_IP then create_admin_server should run, but it is not running.
.ifawsdeploy:
rules:
- if: '$TEST_CREATE_ADMIN && $REGION && $ROLE_ARN && $PACKAGEURL && $TEST_CREATE_ADMIN == "aws" && $SUB_PLATFORM == "aws" && $ROLE_ARN != "" && $PACKAGEURL != "" && $REGION != ""'
variables:
TEST_CREATE_ADMIN:
#value: aws
description: "Platform, currently aws only"
SUB_PLATFORM:
value: aws
description: "Platform, currently aws only"
REGION:
value: "us-west-2"
description: "region where to deploy company"
PACKAGEURL:
value: "http://somerpmurl.x86_64.rpm"
description: "company rpm file url"
ACCOUNT_NAME:
value: "testsubaccount"
description: "Account name of sub account to refer in the deployment, no need to match in AWS"
ROLE_ARN:
value: "arn:aws:iam::491483064167:role/uat"
description: "ROLE ARN of the user account assuming: aws sts get-caller-identity"
tfenv_version: "1.1.9"
DEV_PUB_KEY:
description: "Optional public key file to add access to admin server"
ADMIN_SERVER_IP:
description: "Existing Admin Server IP Address"
ADMIN_SERVER_SSH_KEY:
description: "Existing Admin Server SSH_KEY PEM content"
#export variables below will cause the terraform to use the root account instead of the one specified in tfvars file
.configure_aws_cli: &configure_aws_cli
- aws configure set region $REGION
- aws configure set aws_access_key_id $AWS_FULL_STS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_FULL_STS_ACCESS_KEY_SECRET
- aws sts get-caller-identity
- aws configure set source_profile default --profile $ACCOUNT_NAME
- aws configure set role_arn $ROLE_ARN --profile $ACCOUNT_NAME
- aws sts get-caller-identity --profile $ACCOUNT_NAME
- aws configure set region $REGION --profile $ACCOUNT_NAME
.copy_remote_log: &copy_remote_log
- if [ -e outfile ]; then rm outfile; fi
- copy_command="$(cat $CI_PROJECT_DIR/scp_command.txt)"
- new_copy_command=${copy_command/"%s"/"outfile"}
- new_copy_command=${new_copy_command/"~"/"/home/ec2-user/outfile"}
- echo $new_copy_command
- new_copy_command=$(echo "$new_copy_command" | sed s'/\([^.]*\.[^ ]*\) \([^ ]*\) \(.*\)/\1 \3 \2/')
- echo $new_copy_command
- sleep 10
- eval $new_copy_command
.check_remote_log: &check_remote_log
- sleep 10
- grep Error outfile || true
- sleep 10
- returnCode=$(grep -c Error outfile) || true
- echo "Return code received $returnCode"
- if [ $returnCode -ge 1 ]; then exit 1; fi
- echo "No errors"
.prepare_ssh_key: &prepare_ssh_key
- echo $ADMIN_SERVER_SSH_KEY > $CI_PROJECT_DIR/ssh_key.pem
- cat ssh_key.pem
- sed -i -e 's/-----BEGIN RSA PRIVATE KEY-----/-bk-/g' ssh_key.pem
- sed -i -e 's/-----END RSA PRIVATE KEY-----/-ek-/g' ssh_key.pem
- perl -p -i -e 's/\s/\n/g' ssh_key.pem
- sed -i -e 's/-bk-/-----BEGIN RSA PRIVATE KEY-----/g' ssh_key.pem
- sed -i -e 's/-ek-/-----END RSA PRIVATE KEY-----/g' ssh_key.pem
- cat ssh_key.pem
- chmod 400 ssh_key.pem
connect-admin-server:
stage: build
allow_failure: true
image:
name: amazon/aws-cli:latest
entrypoint: [ "" ]
rules:
- if: '$ADMIN_SERVER_IP && $ADMIN_SERVER_IP != "" && $ADMIN_SERVER_SSH_KEY && $ADMIN_SERVER_SSH_KEY != ""'
extends:
- .ifawsdeploy
script:
- TF_IN_AUTOMATION=true
- yum update -y
- yum install git unzip gettext jq -y
- echo "Your admin server key and info are added as artifacts"
# Copy the important terraform outputs to files for artifacts to pass into other jobs
- *prepare_ssh_key
- echo "ssh -i ssh_key.pem ec2-user#${ADMIN_SERVER_IP}" > $CI_PROJECT_DIR/ssh_command.txt
- echo "scp -q -i ssh_key.pem %s ec2-user#${ADMIN_SERVER_IP}:~" > $CI_PROJECT_DIR/scp_command.txt
- test_pre_command="$(cat "$CI_PROJECT_DIR/ssh_command.txt") -o StrictHostKeyChecking=no"
- echo $test_pre_command
- test_command="$(echo $test_pre_command | sed -r 's/(ssh )(.*)/\1-tt \2/')"
- echo $test_command
- echo "sudo yum install -yq $PACKAGEURL 2>&1 | tee outfile ; exit 0" | $test_command
- *copy_remote_log
- echo "Now checking log file for returnCode"
- *check_remote_log
artifacts:
untracked: true
when: always
paths:
- "$CI_PROJECT_DIR/ssh_key.pem"
- "$CI_PROJECT_DIR/ssh_command.txt"
- "$CI_PROJECT_DIR/scp_command.txt"
after_script:
- cat $CI_PROJECT_DIR/ssh_key.pem
- cat $CI_PROJECT_DIR/ssh_command.txt
- cat $CI_PROJECT_DIR/scp_command.txt
create-admin-server:
stage: build
allow_failure: false
image:
name: amazon/aws-cli:latest
entrypoint: [ "" ]
rules:
- if: '$ADMIN_SERVER_IP != ""'
when: never
extends:
- .ifawsdeploy
script:
- echo "admin server $ADMIN_SERVER_IP"
- TF_IN_AUTOMATION=true
- yum update -y
- yum install git unzip gettext jq -y
- *configure_aws_cli
- aws sts get-caller-identity --profile $ACCOUNT_NAME #to check whether updated correctly or not
- git clone "https://project-n-setup:$(echo $PERSONAL_GITLAB_TOKEN)#gitlab.com/company-oss/project-n-setup.git"
# Install tfenv
- git clone https://github.com/tfutils/tfenv.git ~/.tfenv
- ln -s ~/.tfenv /root/.tfenv
- ln -s ~/.tfenv/bin/* /usr/local/bin
# Install terraform 1.1.9 through tfenv
- tfenv install $tfenv_version
- tfenv use $tfenv_version
# Copy the tfvars temp file to the terraform setup directory
- cp .gitlab/admin_server.temp_tfvars project-n-setup/$SUB_PLATFORM/
- cd project-n-setup/$SUB_PLATFORM/
- envsubst < admin_server.temp_tfvars > admin_server.tfvars
- rm -rf .terraform || exit 0
- cat ~/.aws/config
- terraform init -input=false
- terraform apply -var-file=admin_server.tfvars -input=false -auto-approve
- echo "Your admin server key and info are added as artifacts"
# Copy the important terraform outputs to files for artifacts to pass into other jobs
- terraform output -raw ssh_key > $CI_PROJECT_DIR/ssh_key.pem
- terraform output -raw ssh_command > $CI_PROJECT_DIR/ssh_command.txt
- terraform output -raw scp_command > $CI_PROJECT_DIR/scp_command.txt
- cp $CI_PROJECT_DIR/project-n-setup/$SUB_PLATFORM/terraform.tfstate $CI_PROJECT_DIR
- cp $CI_PROJECT_DIR/project-n-setup/$SUB_PLATFORM/admin_server.tfvars $CI_PROJECT_DIR
artifacts:
untracked: true
paths:
- "$CI_PROJECT_DIR/ssh_key.pem"
- "$CI_PROJECT_DIR/ssh_command.txt"
- "$CI_PROJECT_DIR/scp_command.txt"
- "$CI_PROJECT_DIR/terraform.tfstate"
- "$CI_PROJECT_DIR/admin_server.tfvars"
How to fix that?
I tried the below step from suggestions on comments section.
.generalgrabclustertrigger:
rules:
- if: '$TEST_CREATE_ADMIN && $REGION && $ROLE_ARN && $PACKAGEURL && $TEST_CREATE_ADMIN == "aws" && $SUB_PLATFORM == "aws" && $ROLE_ARN != "" && $PACKAGEURL != "" && $REGION != ""'
.ifteardownordestroy: # Automatic if triggered from gitlab api AND destroy variable is set
rules:
- !reference [.generalgrabclustertrigger, rules]
- if: 'CI_PIPELINE_SOURCE == "triggered"'
when: never
And included the above in extends of a job.
destroy-admin-server:
stage: cleanup
extends:
- .ifteardownordestroy
allow_failure: true
interruptible: false
But I am getting syntax error in the .ifteardownordestroy part.
jobs:destroy-admin-server:rules:rule if invalid expression syntax
You are overriding rules: in your job that extends .ifawsdeploy. rules: are not combined in this case -- the definition of rules: in the job takes complete precedence.
Take for example the following configuration:
.template:
rules:
- one
- two
myjob:
extends: .template
rules:
- a
- b
In the above example, the myjob job only has rules a and b in effect. Rules one and two are completely ignored because they are overridden in the job configuration.
Instead of uinsg extends:, you can use !reference to preserve and combine rules. You can also use YAML anchors if you want.
create-admin-server:
rules:
- !reference [.ifawsdeploy, rules]
- ... # your additional rules
If no value provided to $ADMIN_SERVER_IP then create_admin_server should run
Lastly, pay special attention to your rules:
rules:
- if: '$ADMIN_SERVER_IP != ""'
when: never
In this case, there are no rules that allow the job to run ever. You either need a case that will evaluate true for the job to run, or to have a default case (an item with no if: condition) in order for the job to run.
To get the behavior you expect, you probably want your default case to be on_success:
rules:
- if: '$ADMIN_SERVER_IP != ""'
when: never
- when: on_success
you can change your rules to :
rules:
- if: '$ADMIN_SERVER_IP != ""'
when: never
- when: always
or
rules:
- if: '$ADMIN_SERVER_IP == ""'
when: always
I have a sample in here: try-rules-stackoverflow-72545625 - GitLab and the pipeline record Pipeline no value - GitLab, Pipeline has value - GitLab

yq create complex object based on existing file

Given a docker compose file:
version: '3'
services:
service1:
image: image:name
environment:
- TEMP_ID=1928
volumes:
- type: bind
source: local/path/to
target: container/path/to
ports:
- 8900:8580
service2:
image: image:name
environment:
- TEMP_ID=1451
volumes:
- type: bind
source: local/path/to/1451
target: container/path/to
ports:
- 8901:8580
I am limited to writing a bash script to add a service to the above template based on its content. some of the values are being directly copied from the last service written in the services array, some fields needs modification.
I managed to extract and prepare all the values I need to add and I am stuck with creating the service object and add it to the existing file.
The script I have so far is:
#!/bin/bash
SERVICE_NAME=$1
TEMP_ID_ARG=$2
#extract data to copy from last configured client.
SERVICES=($(yq eval '.services | keys' docker-compose.yml))
LAST_SERVICE=${SERVICES[${#SERVICES[#]}-1]}
echo "adding user based on last service: $LAST_SERVICE"
IMAGE=$(yq -P -C eval '.services["'$LAST_SERVICE'"].image' docker-compose.yml)
ENVIRONEMNT_ENTRY="TEMP_ID=${TEMP_ID_ARG}"
TARGET_PATH_IN_CONTAINER=$(yq -P -C eval '.services["'$LAST_SERVICE'"].volumes[0].target' docker-compose.yml)
VOLUME_TYPE=$(yq -P -C eval '.services["'$LAST_SERVICE'"].volumes[0].type' docker-compose.yml)
LOCAL_PATH_TO_SOURCES=$(yq -P -C eval '.services["'$LAST_SERVICE'"].volumes[0].source' docker-compose.yml)
PATH_AS_ARRAY=($(echo $LOCAL_PATH_TO_SOURCES | tr "\/" " "))
PATH_AS_ARRAY[${#PATH_AS_ARRAY[#]}-1]=$TEMP_ID_ARG
NEW_PATH_TO_RESOURCE=$(printf "/%s" "${PATH_AS_ARRAY[#]}")
# extract port mapping, take first argument (exposed port) increment its value by 1 (no upper limitation)
# join back together with : delimiter.
PORT_MAPING_AS_ARRAY=($(yq -P -C eval '.services["'$LAST_SERVICE'"].ports[0]' docker-compose.yml | tr ":" " "))
# NO UPPER LIMITATION FOR PORT!!!
PORT_MAPING_AS_ARRAY[0]=$(expr $PORT_MAPING_AS_ARRAY + 1)
NEW_PORT_MAPPING=$(printf ":%s" "${PORT_MAPING_AS_ARRAY[#]}")
NEW_PORT_MAPPING=${NEW_PORT_MAPPING:1}
VOLUME_ENTRY=$(yq -P -C eval --null-input '.type = "'$VOLUME_TYPE'" | .source = "'$NEW_PATH_TO_RESOURCE'" | .target = "'$TARGET_PATH_IN_CONTAINER'"')
test=$(yq -P -C -v eval --null-input ' .'$SERVICE_NAME'.image = "'$IMAGE'" | .'$SERVICE_NAME'.environement = ["'$ENVIRONEMNT_ENTRY'"] | (.'$SERVICE_NAME'.volumes = ["'$VOLUME_ENTRY'"] | .'$SERVICE_NAME'.ports = ["'NEW_PORT_MAPPING'"])')
echo $test
when running, what I thought to be a working assembly command of all parts, returns the following error:
Error: cannot pass files in when using null-input flag
the expected output when calling add_service.sh service3 1234
given the above input file:
version: '3'
services:
service1:
image: image:name
environment:
- TEMP_ID=1928
volumes:
- type: bind
source: local/path/to
target: container/path/to
ports:
- 8900:8580
service2:
image: image:name
environment:
- TEMP_ID=1451
volumes:
- type: bind
source: local/path/to/1451
target: container/path/to
ports:
- 8901:8580
service3:
image: image:name
environment:
- TEMP_ID=1234
volumes:
- type: bind
source: local/path/to/1234
target: container/path/to
ports:
- 8902:8580
As my bash skills are not so strong I welcome any advice or better solution to my problem.
You should do it all with one yq call, passing the external values as variables; that will make the code safer and faster:
service_name="service3" \
temp_id="1234" \
environment="TEMP_ID=1234" \
yq eval '
.services[ .services | keys | .[-1] ] as $last |
.services[ strenv(service_name) ] = {
"image": $last.image,
"environment": [ strenv(environment) ],
"volumes": [ {
"type": $last.volumes[0].type,
"source": (
$last.volumes[0].source |
split("/") |
.[-1] = strenv(temp_id) |
join("/")
),
"target": $last.volumes[0].target
} ],
"ports": [ (
"" + $last.ports[0] |
split(":") |
.[0] tag = "!!int" |
.[0] += 1 |
join(":") |
. tag = ""
) ]
}
' docker-compose.yml
output:
version: '3'
services:
service1:
image: image:name
environment:
- TEMP_ID=1928
volumes:
- type: bind
source: local/path/to
target: container/path/to
ports:
- 8900:8580
service2:
image: image:name
environment:
- TEMP_ID=1451
volumes:
- type: bind
source: local/path/to/1451
target: container/path/to
ports:
- 8901:8580
service3:
image: image:name
environment:
- TEMP_ID=1234
volumes:
- type: bind
source: local/path/to/1234
target: container/path/to
ports:
- 8902:8580

Split string and extract variables with shell script

Question
Given this single-line string:
PG_USER=postgres PG_PORT=1234 PG_PASS=icontain=and*symbols
What would be the right way to assign each value to its designated variable so that I can use it afterward?
Context
I'm parsing the context of a k8s secret within a CronJob so that I can periodically call a Stored Procedure in our Postgres database.
To do so, I plan on using:
PG_OUTPUT_VALUE=$(PGPASSWORD=$PG_PASSWD psql -qtAX -h $PG_HOST -p $PG_PORT -U $PG_USER -d $PG_DATABASE -c $PG_TR_CLEANUP_QUERY)
echo $PG_OUTPUT_VALUE
The actual entire helm chart I'm currently trying to fix looks like this:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: {{ template "fullname" $ }}-tr-cleanup-cronjob
spec:
concurrencyPolicy: Forbid
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
volumes:
- name: postgres
secret:
secretName: {{ template "fullname" $ }}-postgres
containers:
- name: {{ template "fullname" $ }}-tr-cleanup-pod
image: postgres:12-alpine
imagePullPolicy: Always
env:
- name: PG_PROPS
valueFrom:
secretKeyRef:
name: {{ template "fullname" $ }}-postgres
key: postgres.properties
command:
- /bin/sh
- -c
- echo "props:" && echo $PG_PROPS && PG_USER=$(grep "^PG_USER=" | cut -d"=" -f2-) && echo $PG_USER && PG_TR_CLEANUP_QUERY="SELECT something FROM public.somewhere;" && echo $PG_TR_CLEANUP_QUERY && PG_OUTPUT_VALUE=$(PGPASSWORD=$PG_PASSWD psql -qtAX -h $PG_HOST -p $PG_PORT -U $PG_USER -d $PG_DATABASE -c $PG_TR_CLEANUP_QUERY) && echo PG_OUTPUT_VALUE
volumeMounts:
- name: postgres
mountPath: /etc/secrets/postgres
Current approach
As you can see, I'm currently using:
PG_USER=$(grep "^PG_USER=" | cut -d"=" -f2-)
That is because I initially thought the secret would be output on multiple lines, but it turns out that I was wrong. The echo $PG_USER displays an empty string.
The bash declare command is appropriate here, and is safer than eval.
Suppose the input contains something potentially malicious
line='PG_USER=postgres PG_PORT=1234 PG_PASS=icontain=and*symbols`ls`'
I'm assuming none of the values contain whitespace. Let's split that string
read -ra assignments <<< "$line"
Now, declare each one
for assignment in "${assignments[#]}"; do declare "$assignment"; done
Everywhere we examine the input, we maintain double quotes.
Let's see what we ended up with:
$ declare -p PG_USER PG_PORT PG_PASS
declare -- PG_USER="postgres"
declare -- PG_PORT="1234"
declare -- PG_PASS="icontain=and*symbols\`ls\`"
Option 1
This function can be reused to assign each variable individually:
extract() {
echo "$INPUT" | grep -o "$1=.*" | cut -d" " -f1 | cut -d"=" -f2- ;
}
And to use it:
PG_USER=$(extract PG_USER)
PG_PORT=$(extract PG_PORT)
PG_PASS=$(extract PG_PASS)
Option 2
Another potential solution, with a security concern, is to simply use:
eval "$INPUT"
It should only be used if you have validated the input.
Contextual complete answer
And because I've presented the k8s context in the question, here is the answer as plugged into that solution.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: {{ template "fullname" $ }}-cronjob
spec:
concurrencyPolicy: Forbid
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
volumes:
- name: postgres
secret:
secretName: {{ template "fullname" $ }}-postgres
containers:
- name: {{ template "fullname" $ }}-cronjob-pod
image: postgres:12-alpine
imagePullPolicy: Always
env:
- name: PG_PROPS
valueFrom:
secretKeyRef:
name: {{ template "fullname" $ }}-postgres
key: postgres.properties
command:
- /bin/sh
- -c
- >-
extract() { echo "$PG_PROPS" | grep -o "$1=.*" | cut -d" " -f1 | cut -d"=" -f2- ; } &&
export PGHOST=$(extract PG_HOST) &&
export PGPORT=$(extract PG_PORT) &&
export PGDATABASE=$(extract PG_DATABASE) &&
export PGUSER=$(extract PG_USER) &&
PG_SCHEMA=$(extract PG_SCHEMA) &&
PG_QUERY="SELECT tenant_schema FROM $PG_SCHEMA.tenant_schema_mappings;" &&
PGPASSWORD=$(extract PG_PASSWD) psql --echo-all -c "$PG_QUERY"
volumeMounts:
- name: postgres
mountPath: /etc/secrets/postgres

Auditbeat not picking up authentication events in CentOs 7

I am trying to ship the authentication related of my CentOS 7 to Elasticsearch. Strangely I am not getting any authentication events.
When I ran the debug command auditbeat -c auditbeat.conf -d -e "*" , I found something like below:
{
"#timestamp": "2019-01-15T11:54:37.246Z",
"#metadata": {
"beat": "auditbeat",
"type": "doc",
"version": "6.4.0"
},
"error": {
"message": "failed to set audit PID. An audit process is already running (PID 68504)"
},
"beat": {
"name": "env-cs-westus-devtest-66-csos-logs-es-master-0",
"hostname": "env-cs-westus-devtest-66-csos-logs-es-master-0",
"version": "6.4.0"
},
"host": {
"name": "env-cs-westus-devtest-66-csos-logs-es-master-0"
},
"event": {
"module": "auditd"
}
}
Also there was an error line like below:
Failure receiving audit events {"error": "failed to set audit PID. An audit process is already running (PID 68504)"}
Machine details
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Audibeat Configuration File
#================================ General ======================================
fields_under_root: False
queue:
mem:
events: 4096
flush:
min_events: 2048
timeout: 1s
max_procs: 1
max_start_delay: 10s
#================================= Paths ======================================
path:
home: "/usr/share/auditbeat"
config: "/etc/auditbeat"
data: "/var/lib/auditbeat"
logs: "/var/log/auditbeat/auditbeat.log"
#============================ Config Reloading ================================
config:
modules:
path: ${path.config}/conf.d/*.yml
reload:
period: 10s
enabled: False
#========================== Modules configuration =============================
auditbeat.modules:
#----------------------------- Auditd module -----------------------------------
- module: auditd
resolve_ids: True
failure_mode: silent
backlog_limit: 8196
rate_limit: 0
include_raw_message: True
include_warnings: True
audit_rules: |
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
-a always,exit -F dir=/home -F uid=0 -F auid>=1000 -F auid!=4294967295 -C auid!=obj_uid -F key=power-abuse
-a always,exit -F arch=b64 -S setuid -F a0=0 -F exe=/usr/bin/su -F key=elevated-privs
-a always,exit -F arch=b32 -S setuid -F a0=0 -F exe=/usr/bin/su -F key=elevated-privs
-a always,exit -F arch=b64 -S setresuid -F a0=0 -F exe=/usr/bin/sudo -F key=elevated-privs
-a always,exit -F arch=b32 -S setresuid -F a0=0 -F exe=/usr/bin/sudo -F key=elevated-privs
-a always,exit -F arch=b64 -S execve -C uid!=euid -F euid=0 -F key=elevated-privs
-a always,exit -F arch=b32 -S execve -C uid!=euid -F euid=0 -F key=elevated-privs
#----------------------------- File Integrity module -----------------------------------
- module: file_integrity
paths:
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etc
- /home/jenkins
exclude_files:
- (?i)\.sw[nop]$
- ~$
- /\.git($|/)
scan_at_start: True
scan_rate_per_sec: 50 MiB
max_file_size: 100 MiB
hash_types: [sha1]
recursive: False
#================================ Outputs ======================================
#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
enabled: True
hosts:
- x.x.x:9200
compression_level: 0
protocol: "http"
worker: 1
bulk_max_size: 50
timeout: 90
#================================ Logging ======================================
logging:
level: "info"
selectors: ["*"]
to_syslog: False
to_eventlog: False
metrics:
enabled: True
period: 30s
to_files: True
files:
path: /var/log/auditbeat
name: "auditbeat"
rotateeverybytes: 10485760
keepfiles: 7
permissions: 0600
json: False
Version of Auditbeat
auditbeat version 6.4.0 (amd64), libbeat 6.4.0
Have anyone faced a similar issue and got a resolution?.
Note: this configuration for auditbeat successfully captures authentication events for Ubuntu
So I posted the same in the Elastic beats forum and got a solution. You can find the same here
As per their suggestion, turning off the auditd service would allow Audit events to be captured by Audibeat. I tried the same and it worked for me. But I am not sure of the implications of turning the auditd off. So I might switch to a Filebeat based solution.

Resources