I am assembling a YAML build pipeline using bash as follows.
cat <<EOT >> ${BUILD_SOURCESDIRECTORY}/azure-pipelines.yml
- template: templates/deploy-to-env-ssh.yml#infrastructure
parameters:
dockerHostEndpoint: ${DOCKER_HOST}
jobName: ${BASENAME}
stackName: ${STACK_NAME}
composeFile: ${STACK_NAME}.yml
schedule: ???
$(cat schedule.yml)
tz: ${TZ}
EOT
What I want is to store the following YAML into schedule as a string which I can reuse in a another part of the pipeline.
version: 1.4
jobs:
DockerJob:
cmd: docker ps
time: "*"
notifyOnSuccess:
- type: stdout
data:
- stdout
But it seems it needs to be indented.
You can use the pr utility:
cat <<EOF >> ${BUILD_SOURCESDIRECTORY}/azure-pipelines.yml
- template: templates/deploy-to-env-ssh.yml#infrastructure
parameters:
dockerHostEndpoint: ${DOCKER_HOST}
jobName: ${BASENAME}
stackName: ${STACK_NAME}
composeFile: ${STACK_NAME}.yml
schedule: $(printf "\n" && pr -to 8 schedule.yml)
tz: ${TZ}
EOT
I use printf "\n" because you'd need to place the $(…) at the first column if you want to write $(…) on a new line, since every line including the first one will be offset by the given number of spaces.
Related
I've got a bash script in a GitHub action that greps for image tags, then replaces them in a docker-compose.yml.
# docker-compose.yml from a JSON payload
docker_compose='"services:\n api:\n proxy: ghcr.io/org/api:main-4095094301\n ports:\n - 0:80\n restart: always\n\n proxy:\n image: ghcr.io/org/proxy:main-4095124301\n depends_on:\n - api\n environment:\n - \"ENVIRONMENT=dev\"\n - \"PORT=8000\"\n ports:\n - 0:3000\n restart: always\n\n ..."'
# Extract the tag of each image, send to GitHub actions output
echo "proxy_tag=$(echo -n $docker_compose | grep -oP 'proxy:\K[-\w]+')" >> $GITHUB_OUTPUT
If I remove the >> operator to echo the output, everything looks OK:
proxy_tag=main-4095094301
But when I feed it into sed later in a different step of the pipeline, an extra newline character seems to come from nowhere:
echo "running s/PROXY_TAG/$proxy_tag/"
sed -i "s/PROXY_TAG/$proxy_tag/" docker-compose.yml
# running s/PROXY_TAG/main-4095094301
# /
# sed: -e expression #1, char 18: unterminated `s' command
# Error: Process completed with exit code 1.
I've tried some common suggestions eg. piping output through tr -d '\n':
echo "proxy_tag=$(echo -n $docker_compose | grep -oP 'proxy:\K[-\w]+' | tr -d '\n')" >> $GITHUB_OUTPUT
Is there something missing I don't understand about bash vars or Github actions?
See below a more complete context of how where these commands are being used.
name: Update Docker Compose
on:
workflow_dispatch:
inputs:
proxy_tag:
description: proxy_tag
type: string
required: false
# api_tag:
# site_tag:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Fetch docker-compose
id: get-tags
run: |
docker_compose=$(curl ...)
echo "proxy_tag=$(echo -n $docker_compose | grep -oP 'proxy:\K[-\w]+')" >> $GITHUB_OUTPUT
echo "api_tag=$(echo $docker_compose | grep -oP 'api:\K[-\w]+')" >> $GITHUB_OUTPUT
echo "site_tag=$(echo $docker_compose | grep -oP 'site:\K[-\w]+')" >> $GITHUB_OUTPUT
- name: Replace tags docker-compose
env:
proxy_tag: >
${{ github.event_name == 'workflow_dispatch'
&& github.event.inputs.proxy_tag
|| steps.get-tags.outputs.proxy_tag
|| 'latest'
}}
# api_tag: >
# site_tag: >
run: |
echo "Setting [$proxy_tag]"
sed -i "s/PROXY_TAG/$proxy_tag/" docker-compose.yml
sed -i "s/API_TAG/$api_tag/" docker-compose.yml
sed -i "s/SITE_TAG/$site_tag/" docker-compose.yml
# Outputs:
# Setting [main-4095094301 # <--- notice the newline
# ]
#
# sed: -e expression #1, char 18: unterminated `s' command
# Error: Process completed with exit code 1.
I suspect that the newline character being added is from the YAML's folding using > while setting your environment variables under env. Try >- to not add (strip) the last newline character.
This (https://yaml-multiline.info/) might be helpful for live experimentation with YAML.
Worst case, if you're using bash,
$: proxy_tag="main-4095094301
" # embedded newline for example
$: echo "[$proxy_tag]" # show with embedded newline
[main-4095094301
]
$: echo "[${proxy_tag%$'\n'}]" # show with embedded newline removed
[main-4095094301]
$: echo PROXY_TAG | sed "s/PROXY_TAG/$proxy_tag/" # syntax error with embedded newline
sed: -e expression #1, char 27: unterminated `s' command
$: echo PROXY_TAG | sed "s/PROXY_TAG/${proxy_tag%$'\n'}/" # removed, syntax ok
main-4095094301
That seems error-prone to me, though, and I prefer to do most of my basic string processing right in the interpreter unless it's huge and I need it faster, so I'd probably do something based on this -
$: docker_compose=$'services:\n api:\n proxy: ghcr.io/org/api:main-4095094301\n ports:\n - 0:80\n restart: always\n\n proxy:\n image: ghcr.io/org/proxy:main-4095124301\n depends_on:\n - api\n environment:\n - "ENVIRONMENT=dev"\n - "PORT=8000"\n ports:\n - 0:3000\n restart: always\n\n ...' # note $'...' for clean quotes and newlines
$: echo "[$docker_compose]"
[services:
api:
proxy: ghcr.io/org/api:main-4095094301
ports:
- 0:80
restart: always
proxy:
image: ghcr.io/org/proxy:main-4095124301
depends_on:
- api
environment:
- "ENVIRONMENT=dev"
- "PORT=8000"
ports:
- 0:3000
restart: always
...]
$: shopt -s extglob # enable extended globbing
$: tmp="${docker_compose##* proxy: *([^:]):}" # trim front - *(...) is "0 or more"
$: proxy_tag="${tmp%%$'\n'*}" # trim after, from 1st newline on
$: echo "[$proxy_tag]" # show that it's clean
[main-4095094301]
I can see a way to create kubectl command like this where I can pass some parameterize values ad well.
My question is, with what type of file we will save this, Is this a bash script? and how to run and supply the parameter?
export SECRET_NAME="my-app.secret"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: $SECRET_NAME
type: Opaque
data:
password: $(echo -n "s33msi4" | base64 -w0)
username: $(echo -n "jane" | base64 -w0)
EOF
Yes, it can be treated as a bash script. As Jetchisel already mentioned in his comment, it contains a structure called Here Document used with cat command but it also contains export command which sets and exports a variable. So as a whole it can be treated as a simple bash script with 2 instructions.
In order to run it and create a new Secret object (which is your ultimate goal), follow these steps:
Fix the indentation which is crucial in yaml files:
export SECRET_NAME="my-app.secret"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: $SECRET_NAME
type: Opaque
data:
password: $(echo -n "s33msi4" | base64 -w0)
username: $(echo -n "jane" | base64 -w0)
EOF
Save the above content as a file. You can call it secret.sh.
Source it (source and . are the same command):
. secret.sh
You should see the following message:
secret/my-app.secret created
Alternatvely you can paste it directly into the console. As you can see, it also works:
### don't copy it, this is the example console output
### you will see once you paste the above script in your bash shell
$ export SECRET_NAME="my-app.secret"
$
$ cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Secret
> metadata:
> name: $SECRET_NAME
> type: Opaque
> data:
> password: $(echo -n "s33msi4" | base64 -w0)
> username: $(echo -n "jane" | base64 -w0)
> EOF
secret/my-app.secret created
When I retrieve all records for a hosted zone in Cloudflare, e.g. of response, I need to create from it the following yaml structure:
name_zones: # this line we create
.zone_name: # the value is taken from the response
auth_key: XXX # this line we create
records: # this line we create
# iterate over all records
- name: .name
type: .type
priority: .priority # create line if value set|exist
content: .content
ttl: .ttl # create line if value set|exist
e.g. jq code which almost done this:
jq '.result[] | {name: .name, type: .type, content: .content, ttl: .ttl} + if has("priority") then {priority} else null end' | jq -n '.name_zone.zone_name.auth_key.records |= [inputs]' | yq r -P -
How to pass or create the value of zone_name and auth_key: XXX?
First, your two invocations of jq can be replaced by just one, as shown in the answer below.
Second, there are currently (at least) two yqs in the wild:
python-based yq (https://kislyuk.github.io/yq) - hereafter python-yq
go-based yq (https://github.com/mikefarah/yq)
For better and/or worse, version 4 of the go-based yq is significantly different from earlier versions, so if you want to use the current go-based version you may have to make adjustments accordingly. To simplify things (at least from my point of view), I will replace your yq r -P - by:
python-jq -y .
The following produces the output shown below.
< cf_response.json \
jq '{name_zones:
{zone_name: .result[0].zone_name,
auth_key: "XXX",
records:
[.result[]
| {name, type, content, ttl}
+ if has("priority")
then {priority}
else null end] }} '|
python-yq -y .
Output
name_zones:
zone_name: test.com
auth_key: XXX
records:
- name: test.com
type: A
content: 111.111.111.111
ttl: 1
- name: test.com
type: TXT
content: google-site-verification=content
ttl: 1
- name: test.com
type: MX
content: smtp.test.com
ttl: 1
priority: 0
auth_key
If you want to pass in the value of auth_key as a parameter to jq, you could use the command-line sequence --arg auth_key XXX, and then use $auth_key in the jq program.
I am trying to add a couple of lines in a text file with sed.
I think I have special characters that are giving me the issue.
I want to insert lines between
username: system:node:{{EC2PrivateDNSName}}
and
kind: ConfigMap
This is what I want to insert -
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
I have also tried using forward slashes around the special characters to no avail.
Here is the sed command I have now that does not work - it seems not to insert anything. I assume it can't find the line "username: system:node:{{EC2PrivateDNSName}}".
sed '/^username\: system:node\:{{EC2PrivateDNSName}}$/r'<(
echo " - groups:"
echo " - eks-role"
echo " - system:master"
echo " rolearn: arn:aws:iam::xxxxx:role/eks"
echo " username: eks"
echo " mapUsers: |"
echo " - userarn: arn:aws:iam::xxxxx:user/test-ecr"
echo " username: ecr"
echo " groups:"
echo " - eks-role"
) -i -- temp-aws-auth.yaml
Here is the contents of the file that I want to insert into -
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::xxxxx:role/eksctl-ops-nodegroup-linux-ng-sys-NodeInstanceRole-763ALQD2ZGXK
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
creationTimestamp: "2020-12-09T15:54:56Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1298"
UPDATE: Taking into consideration OPs answer/comment re: missing spaces, and a bit more fiddling, I was able to get the following sed command to work, too:
sed '/^.*username.*EC2PrivateDNSName.*$/r'<(cat replace.txt) temp-aws-auth.yaml
Assumptions:
OP is unable to use a yaml-aware tool to perform the edit
username ... EC2PrivateDNSName only shows up in one place in the file (or, alternatively, it shows up in multiple places and OP wishes to add a new line after each occurrence)
Replacement data:
$ cat replace.txt
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
NOTE: If the replacement data is in a variable it can fed into awk as a herestring.
One awk idea:
awk '
FNR==NR { a[FNR]=$0 # store first file (replacement data) into an array
next } # skip to next line in first file
{ print } # print current line of second file
/username.*EC2PrivateDNSName/ { for (i in a) # if we found our match then dump the contents of array a[] to stdout
print a[i]
next
}
' replace.txt temp-aws-auth.yaml
Or as a single-line:
awk 'FNR==NR {a[FNR]=$0; next} {print} /username.*EC2PrivateDNSName/ { for (i in a) print a[i]; next}' replace.txt temp-aws-auth.yaml
This generates:
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::xxxxx:role/eksctl-ops-nodegroup-linux-ng-sys-NodeInstanceRole-763ALQD2ZGXK
username: system:node:{{EC2PrivateDNSName}}
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
kind: ConfigMap
metadata:
creationTimestamp: "2020-12-09T15:54:56Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1298"
I found out the issue with my original command - Sed needs the spaces included in the line it is looking for!
Since the line I was looking for has spaces in it :
' username: system:node:{{EC2PrivateDNSName}}'
I had to add the spaces to my sed statement :
sed '/^ username\: system:node\:{{EC2PrivateDNSName}}$/r'<(
Thanks for the feedback!
Happy holidays!!
This might work for you (GNU sed & cat):
cat <<\! |sed ':a;/username: system:node:{{EC2PrivateDNSName}}/{n;/kind: ConfigMap/!ba;h;s/.*/cat -/ep;g}' file
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
!
Make a here-document with the lines to be inserted.
Pipe the here-document through to a sed command.
If a line contains username: system:node:{{EC2PrivateDNSName}}, print it and fetch the next line.
If the following line does not contain kind: ConfigMap go to the start of the sed cycle and start again.
Otherwise, copy the current line, replace/print the current line by the lines to inserted from the here-document and then over write the replacement by the copy in the hold space.
N.B. The replacement lines are inserted into the document by way of the substitute command and the e flag, that evaluates what is substituted into the pattern space i.e. the cat - that is the here-document that is passed through via the pipe.
I have file like test.yaml file, the text content in the file like below.
servers:
- uri: "http://demo.nginx1.com/status"
name: "NGinX Monitor1"
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I want to remove - uri line and immediate next line (start with name:) where host name = demo.nginx1.com.
I want out put like below.
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I tied like below..
cat test.yaml | grep -v demo.nginx1.com | grep -v Monitor1 >> test_back.yaml
mv test_back.yaml test.yaml
I am getting expected out put. But it's re creating the file and I don't want to re create the file
Please help me with suitable command that i can use..
Just a simple logic using GNU sed
sed '/demo.nginx1.com/,+1 d' test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
For in-place replacement, add a -i flag as -i.bak
sed -i.bak '/demo.nginx1.com/,+1 d' test.yaml
To see the in-place replacement:-
cat test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
As I dislike using regular expressions to hack something you can parse - here's how I'd tackle it, using perl and the YAML module:
#!/usr/bin/env perl
use strict;
use warnings;
use YAML;
use Data::Dumper;
#load yaml by reading files specified as args to stdin,
#or piped in. (Just like how you'd use 'sed')
my $config = Load ( do { local $/ ; <>} );
#output data structure for debug
print Dumper $config;
#use grep to filter the uri you don't want.
#{$config -> {servers}} = grep { not $_ -> {uri} =~ m/demo.nginx2/ } #{$config -> {servers}};
#resultant data structure
print Dumper $config;
#output YAML to STDOUT
print Dump $config;