I have a gitlab deployment CD yaml file which takes in inputs and sets up an environment.. but I need to sometimes cleanup the inputs in bash before I use certain variables (For instance I want to use the branch name in the environment being created in juju.. but juju models cannot contain / characters.
What is the proper way to templatize routines so I don't have to line-for-line repeat them? seems there are directives in gitlab CICD but those are static routines.. and I was hoping to have something dynamic
example .gitlab-ci.yaml
deploy-staging-lxd-upstream:
tags:
- Juju, staging
stage: deploy
environment:
name: staging-lxd
script:
- echo $UPSTREAM_PROJECT - $UPSTREAM_BRANCH - $UPSTREAM_PROJECT_ID - $UPSTREAM_PACKAGE_VERSION - $UPSTREAM_OP_FILENAME
- ls -la
- SAFE_BRANCH_NAME=$(echo "$UPSTREAM_BRANCH"|tr '/' '-')
- SAFE_BRANCH_NAME=$(echo "$SAFE_BRANCH_NAME"|tr '_' '-')
- juju switch "mahrio-upstream-$UPSTREAM_PROJECT_ID-$SAFE_BRANCH_NAME" || true
how can i extract this
- SAFE_BRANCH_NAME=$(echo "$UPSTREAM_BRANCH"|tr '/' '-')
- SAFE_BRANCH_NAME=$(echo "$SAFE_BRANCH_NAME"|tr '_' '-')
and just .. create a bash routine that I put in any branch name.. and a "safe" branch name comes back out for use in subsequent -script calls? or is it better to just do this in python or some other language than bash.. and put that logic in a pypi module across my CICD ?
Just use tr 'abc' '-'?
- SAFE_BRANCH_NAME=$(echo "$UPSTREAM_BRANCH"|tr '/_' '-')
See man tr for details on how it treats more characters in string1 than string2:
$ man tr
SYNOPSIS
tr [-Ccsu] string1 string2
tr [-Ccu] -d string1
tr [-Ccu] -s string1
tr [-Ccu] -ds string1 string2
[...]
In the first synopsis form, the characters in string1 are translated into
the characters in string2 where the first character in string1 is
translated into the first character in string2 and so on. If string1 is
longer than string2, the last character found in string2 is duplicated
until string1 is exhausted.
Related
I want to replace the value between http:// and :80 dynamically everytime with a IP address that I am fetching. I have written below script to get the value and replace the placeholder 'localhost' with the IP address. But I want it to replace value betwen http:// and :80 and not by matching with localhost as it may change.
Line to substitute value in:
"UIApiPath": "http://localhost:80/uisvc/v1/api",
"serviceApiPath": "http://localhost:80/servicesvc/v1/api",
"DBApiPath": "http://locahost:80/DBsvc/v1/api"
Existing script for fetching and replacing value by matching 'localhost':
#!/bin/bash
DEPLOYMENT_NAME=$1
# Getting the pod name from the deployment object
GETTING_POD_NAME=$(kubectl describe deploy $1 | grep 'NewReplicaSet:' | awk '{print $2}')
# Getting the ip-address from pod description
echo ''
GETTING_IP_ADDRESS=$(kubectl describe pod $GETTING_POD_NAME | sed -n '4p' | awk '{print $2}')
# Getting the correct ip-address
SERVER_IP=${GETTING_IP_ADDRESS#*/}
echo $SERVER_IP
# Path of book-store-app configmap file:
APP_CONFIGMAP_FILE_PATH=deployment/java-apps/app-configmap.yaml
# Configure the ip-address in the configmap
sed -i "s/localhost/$SERVER_IP/g" $APP_CONFIGMAP_FILE_PATH
Simple sed command based on // string:
sed "s#//[^:/]\+:80/#//$SERVER_IP:80/#" -i $APP_CONFIGMAP_FILE_PATH
Could do the job! (While $SERVER_IP variable don't hold any # character.)
Explanation
Note: Cited paragraph are copied from info sed pages.
s is normal replacment command under sed
The syntax of the 's' command is 's/REGEXP/REPLACEMENT/FLAGS'.
...
# is used as command separator;
The '/' characters may be uniformly replaced by any other single
character within any given 's' command...
[^:/] mean any character, but no slashes / and no colons :.
A "bracket expression" is a list of characters enclosed by '[' and ']'.
It matches any single character in that list; if the first character of
the list is the caret '^', then it matches any character *not* in the
list.
* mean any number (even zero) of previous character (no slashes nor colons)
'*'
Matches a sequence of zero or more instances of matches for the
preceding regular expression, which must be an ordinary character,
a special character preceded by '\', a '.', a grouped regexp (see
below), or a bracket expression.
...
'\+'
As '*', but matches one or more.
I insist: Please read carefully info sed!
I have a small script that downloads a value from a web page.
Before anyone looses their mind because I am not using an HTML parser, besides the headers, the whole web page only has 3 lines of text between one pair of pre tags. I am just after the number values - that is it.
</head><body><pre>
sym
---
12300
</pre></body></html>
This is the script :
#!/bin/bash
wget -O foocounthtml.txt "http://foopage"
tr -d "\n" foocounthtml.txt > foocountnonewlines.txt
Anyhow the tr command is throwing an error.
tr: extra operand ‘foocounthtml.txt’
Only one string may be given when deleting without squeezing repeats.
Try 'tr --help' for more information.
Yes, I could use sed for inplace modification with the -i tag. However I am perplexed by this tr error. Redirecting tr output works fine from command line, but not in a script.
The 'tr' command operates on SETs of text rather than files. From the man page:
NAME
tr - translate or delete characters
SYNOPSIS
tr [OPTION]... SET1 [SET2]
DESCRIPTION
Translate, squeeze, and/or delete characters from standard input, writing to standard output.
...
SETs are specified as strings of characters. Most represent themselves. Interpreted sequences are:
So tr is expecting the actual content you want to operate on rather than the target file. You can simply pipe the files contents to tr for the resuts you want
cat foocounthtml.txt | tr -d "\n" > foocountnonewlines.txt
or as #CHarlesDUffy points out, it would be faster to read directly from the file:
tr -d "\n" < foocounthtml.txt > foocountnonewlines.txt
In Google cloudbuild tutorial the example cloudbuild.yaml uses - | as one of the arguments.
args:
- '-c'
- |
if [ -d "environments/$BRANCH_NAME/" ]; then
...
What is the purpose of '- |'
That character is called "Literal Block Scalar" and it is used to span values across multiple lines. Spanning with | will include the newlines and any trailing spaces. You can also span with > but this will fold new lines to spaces.
Example:
include_newlines: |
exactly as you see
will appear these three
lines of poetry
fold_newlines: >
this is really a
single line of text
despite appearances
If you want to know more about the yaml syntax, you can visit this for information that may not be included in the cloud build documentation.
Background...
Trying to find which commit(s) last touched a specific file.
I can do this on the CLI piping from git-log to grep but I'm trying to wrap this in a zsh function, more for ease of memory.
Here's my function, and then here is the output I'd like to generate with it.
# match lines from git log that start with commit or include the
# filename I'm interested in and then pipe back through grep to color the output
glpg() {
\git log --name-only | \grep -E ‘“$"|^commit\s\S' | \grep -B1 --color -E ‘$'
}
Desired usage and output
dwight:assets (add-analytics*) $ glpg clickouts
commit 6662418b8e68e478b95e7254faa6406abdada30f
web/assets/app/viewmodels/clickouts.js
web/assets/app/views/clickouts.html
web/client/app/viewmodels/clickouts.js
web/client/app/views/clickouts.html
--
commit cee37549f613985210c9caf90a48e2cca28d4412
web/client/app/viewmodels/clickouts.js
web/client/app/views/clickouts.html
--
commit df9ea8cd90ff80b89a0c7e2b0657141b105d5e7e
web/client/app/viewmodels/clickouts.js
web/client/app/views/clickouts.html
Three problems.
You use Unicode apostrophes and quotes, ‘ and “. Replace them with ASCII quotes and doublequotes.
You can't use \s and \S to mean space or non-space with a standard (POSIX) grep. Use ' ' and [^ ] instead to be portable.
The list of all args is referenced with "$#" including the double quotes.
Can someone walk through what this code is doing?
if [ -f "saved.txt" ]; then // What does -f do?
rm saved.txt
fi
in=$(echo "{query}" | tr -d "\\") // How does this work?
// What does | tr -d "\\" mean?
echo "$in" > saved.txt // Is this simply putting the
// value of $in into saved.txt?
The initial if statement will test if the file is a regular file. More on file test operators here.
This script will echo the characters {query} and pipe it to the command tr, which with the -d will delete characters that are specified. tr stands for translate. In this case it takes a SET and per the man page, it will delete backslashes if you use \\.
The result is stored in $in.
Finally, the result stored in in will be outputted to saved.text.
NAME
tr - translate or delete characters
SYNOPSIS
tr [OPTION]... SET1 [SET2]
DESCRIPTION
Translate, squeeze, and/or delete characters from standard input, writing to standard output.
-c, -C, --complement
first complement SET1
-d, --delete
delete characters in SET1, do not translate
-s, --squeeze-repeats
replace each input sequence of a repeated character that is listed in SET1 with a single occurrence of that
character
-t, --truncate-set1
first truncate SET1 to length of SET2
--help display this help and exit
--version
output version information and exit
SETs are specified as strings of characters. Most represent themselves. Interpreted sequences are:
\NNN character with octal value NNN (1 to 3 octal digits)
\\ backslash
The first part tests if saved.txt exists before trying to remove it.
The second part copies the contents of query (I'm assuming a typo and that should be ${query}, not {query}) into in, minus any backslashes.
The third part, you are correct; it writes the value of in to the file saved.txt.