ansible playbook shell module escaping special characters - shell

Using in an ansible playbook for ansible 2.4.2.0
- name: Check the Tomcat version
shell: "unzip -c {{ mount_path_instance }}/tomcat9/lib/catalina.jar META-INF/MANIFEST.MF |awk '/Implementation-Version: / {sub(/[^ ]+ /, \"\"); print \$0}'"
register: tomcat_version
when: instance_check_v9.stat.exists
I get the error
TASK [redhat-tomcat-update : include_tasks] ************************************
fatal: [localhost]: FAILED! => {"reason": "Syntax Error while loading YAML.\n\n\nThe error appears to have been in '/home/ansible/work/play_redhat_tomcat_update/roles/redhat-tomcat-update/tasks/variables.yml': line 39, column 153, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n- name: Check the Tomcat version\n shell: \"unzip -c {{ mount_path_instance }}/tomcat9/lib/catalina.jar META-INF/MANIFEST.MF |awk '/Implementation-Version: / {sub(/[^ ]+ /, \\\"\\\"); print \\$0}'\"\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n\nexception type: <class 'yaml.scanner.ScannerError'>\nexception: while parsing a quoted scalar\n in \"<unicode string>\", line 39, column 10\nfound unknown escape character\n in \"<unicode string>\", line 39, column 153"}
running the next command works fine
ansible localhost -m shell -a "unzip -c {{ mount_path_instance }}/tomcat9/lib/catalina.jar META-INF/MANIFEST.MF |awk '/Implementation-Version: / {sub(/[^ ]+ /, \"\"); print \$0}'" --extra-vars='{"mount_path_instance": "/appl/tomcat/paul_1_uat", "instance_name": "paul_1_uat" }'
[WARNING]: Consider using unarchive module rather than running unzip
localhost | SUCCESS | rc=0 >>
9.0.7.redhat-12
does some one see what is wrong in the yml definition?

YAML expands backslashes in double-quoted strings, and \$ is not a legal escape in YAML (the \" is fine)
What you'll want is to use the alternate syntax that side-steps the double-quoting issue:
shell: |
unzip -c {{ mount_path_instance }}/tomcat9/lib/catalina.jar META-INF/MANIFEST.MF | awk '/Implementation-Version: / {sub(/[^ ]+ /, ""); print $0}'
Ironically, you don't actually have to escape that dollarsign because the awk command is already in single-quotes, so you could also just drop the \$ from your existing command, too, but I am pretty sure you'll get less backslash-itis using the shell: | syntax
With regard to:
ansible localhost -m shell -a "unzip -c {{ mount_path_instance }}/tomcat9/lib/catalina.jar META-INF/MANIFEST.MF |awk '/Implementation-Version: / {sub(/[^ ]+ /, \"\"); print \$0}'"
running fine, that's because your shell actually collapsed the \$ for you, which one can see with a simple echo:
echo "thingy \"\" \$0"

Related

Trailing newline included in variable assignment after grep, and sent to GitHub actions output

I've got a bash script in a GitHub action that greps for image tags, then replaces them in a docker-compose.yml.
# docker-compose.yml from a JSON payload
docker_compose='"services:\n api:\n proxy: ghcr.io/org/api:main-4095094301\n ports:\n - 0:80\n restart: always\n\n proxy:\n image: ghcr.io/org/proxy:main-4095124301\n depends_on:\n - api\n environment:\n - \"ENVIRONMENT=dev\"\n - \"PORT=8000\"\n ports:\n - 0:3000\n restart: always\n\n ..."'
# Extract the tag of each image, send to GitHub actions output
echo "proxy_tag=$(echo -n $docker_compose | grep -oP 'proxy:\K[-\w]+')" >> $GITHUB_OUTPUT
If I remove the >> operator to echo the output, everything looks OK:
proxy_tag=main-4095094301
But when I feed it into sed later in a different step of the pipeline, an extra newline character seems to come from nowhere:
echo "running s/PROXY_TAG/$proxy_tag/"
sed -i "s/PROXY_TAG/$proxy_tag/" docker-compose.yml
# running s/PROXY_TAG/main-4095094301
# /
# sed: -e expression #1, char 18: unterminated `s' command
# Error: Process completed with exit code 1.
I've tried some common suggestions eg. piping output through tr -d '\n':
echo "proxy_tag=$(echo -n $docker_compose | grep -oP 'proxy:\K[-\w]+' | tr -d '\n')" >> $GITHUB_OUTPUT
Is there something missing I don't understand about bash vars or Github actions?
See below a more complete context of how where these commands are being used.
name: Update Docker Compose
on:
workflow_dispatch:
inputs:
proxy_tag:
description: proxy_tag
type: string
required: false
# api_tag:
# site_tag:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Fetch docker-compose
id: get-tags
run: |
docker_compose=$(curl ...)
echo "proxy_tag=$(echo -n $docker_compose | grep -oP 'proxy:\K[-\w]+')" >> $GITHUB_OUTPUT
echo "api_tag=$(echo $docker_compose | grep -oP 'api:\K[-\w]+')" >> $GITHUB_OUTPUT
echo "site_tag=$(echo $docker_compose | grep -oP 'site:\K[-\w]+')" >> $GITHUB_OUTPUT
- name: Replace tags docker-compose
env:
proxy_tag: >
${{ github.event_name == 'workflow_dispatch'
&& github.event.inputs.proxy_tag
|| steps.get-tags.outputs.proxy_tag
|| 'latest'
}}
# api_tag: >
# site_tag: >
run: |
echo "Setting [$proxy_tag]"
sed -i "s/PROXY_TAG/$proxy_tag/" docker-compose.yml
sed -i "s/API_TAG/$api_tag/" docker-compose.yml
sed -i "s/SITE_TAG/$site_tag/" docker-compose.yml
# Outputs:
# Setting [main-4095094301 # <--- notice the newline
# ]
#
# sed: -e expression #1, char 18: unterminated `s' command
# Error: Process completed with exit code 1.
I suspect that the newline character being added is from the YAML's folding using > while setting your environment variables under env. Try >- to not add (strip) the last newline character.
This (https://yaml-multiline.info/) might be helpful for live experimentation with YAML.
Worst case, if you're using bash,
$: proxy_tag="main-4095094301
" # embedded newline for example
$: echo "[$proxy_tag]" # show with embedded newline
[main-4095094301
]
$: echo "[${proxy_tag%$'\n'}]" # show with embedded newline removed
[main-4095094301]
$: echo PROXY_TAG | sed "s/PROXY_TAG/$proxy_tag/" # syntax error with embedded newline
sed: -e expression #1, char 27: unterminated `s' command
$: echo PROXY_TAG | sed "s/PROXY_TAG/${proxy_tag%$'\n'}/" # removed, syntax ok
main-4095094301
That seems error-prone to me, though, and I prefer to do most of my basic string processing right in the interpreter unless it's huge and I need it faster, so I'd probably do something based on this -
$: docker_compose=$'services:\n api:\n proxy: ghcr.io/org/api:main-4095094301\n ports:\n - 0:80\n restart: always\n\n proxy:\n image: ghcr.io/org/proxy:main-4095124301\n depends_on:\n - api\n environment:\n - "ENVIRONMENT=dev"\n - "PORT=8000"\n ports:\n - 0:3000\n restart: always\n\n ...' # note $'...' for clean quotes and newlines
$: echo "[$docker_compose]"
[services:
api:
proxy: ghcr.io/org/api:main-4095094301
ports:
- 0:80
restart: always
proxy:
image: ghcr.io/org/proxy:main-4095124301
depends_on:
- api
environment:
- "ENVIRONMENT=dev"
- "PORT=8000"
ports:
- 0:3000
restart: always
...]
$: shopt -s extglob # enable extended globbing
$: tmp="${docker_compose##* proxy: *([^:]):}" # trim front - *(...) is "0 or more"
$: proxy_tag="${tmp%%$'\n'*}" # trim after, from 1st newline on
$: echo "[$proxy_tag]" # show that it's clean
[main-4095094301]

Ansible : how to run multiple steps shell command with special chars

Under my Playbook , i want to run a shell command which is the following :
for STACK in stackone stacktwo;do docker stack ps --format "table {{.ID}}\t{{.Name}}\t{{.CurrentState}}\t{{.Error}}\t{{.Node}}" $STACK | (read -r; printf "%s\n" "$REPLY"; sort -k 2|grep srcd |grep -v Shutdown ); done;
As you can see my shell command is quite complex , so when i am running it like this , it fails throwing always syntax errors
- name : Check running services
shell: for STACK in stackone srcd-pilote;do docker stack ps --format "table {{.ID}}\t{{.Name}}\t{{.CurrentState}}\t{{.Error}}\t{{.Node}}" $STACK | (read -r; printf "%s\n" "$REPLY"; sort -k 2|grep srcd |grep -v Shutdown ); done;
register: result
I see there is many characters which need to be passed as strings such as {{.Name}} and |grep ...
So i ve tried this :
- name : Check running services
shell: "for STACK in srcd-current stacktwo;do docker stack ps --format 'table {{'"{{.ID}}"'}}\t{{'"{{.Name}}"'}}\t{{'"{{.CurrentState}}"'}}\t{{'"{{.Error}}"'}}\t{{'"{{.Node}}"'}}' $STACK | (read -r; printf '%s\n' '$REPLY'; sort -k 2'"{{|}}"'grep srcd '"{{|}}"'grep -v Shutdown ); done; "
register: result
But it stills failing .
Suggesstions ??
I had to use a shell for loop in one of my ansible tasks. I use a Literal Block Scalar '|' to do so. Try something like this:
- name : Check running services
shell: |
for STACK in stackone srcd-pilote;do
docker stack ps --format "table {{ '{{' }}.ID{{ '}}' }}\t{{ '{{' }}.Name{{ '}}' }}\t{{ '{{' }}.CurrentState{{ '}}' }}\t{{ '{{' }}.Error{{ '}}' }}\t{{ '{{' }}.Node{{ '}}' }}" $STACK | (read -r; printf "%s\n" "$REPLY"; sort -k 2|grep srcd |grep -v Shutdown );
done;
More doc on the scalar can be found on the Ansible doc about YAML syntax.
You should also escape the {{ and }} from the jinja2, by using {{ '{{' }} and {{ '}}' }}.
Note 1: Here is a nice sed command to do so:
s/{{\([^}]*\)}}/{{ '{{' }}\1{{ '}}' }}/g
Note 2:
You should review the command to split it in several lines to improve the readability (with \ at the end of line or using the > scalar).
Ansible-Lint rules recommend line shorter than 160 characters (rule E204).

How to replace a line which has multiple special characters with ansible playbook

Trying to replace a line which has multiple special characters, which gets interpreted as escape characters by ansible.
Tried using "\" and "." for every character, and tried using !unsafe.
- name: Update file
replace:
path: /some/file
regexp: "*[0-9a-zA-Z._-]* )" #<<=== This is line to be replaced
replace: "*[0-9a-z._-]* )" #<<== With this
backup: yes
Getting errors like:
raise error, v # invalid expression\r\nsre_constants.error: nothing to repeat\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
You should escape the special characters with one backslash each:
regexp: '\*\[0\-9a\-zA\-Z\._\-\]\* \)'
And you should use single quotes.
Tried this and it worked:
regexp: '\*\[0-9a-zA-Z\._-\]\* \)'
replace: '*[0-9a-z._-]* )'

Probable quoting issue using the stdin argument of ansible's shell module

I've got a playbook with the following tasks:
- set_fact:
asg_filter: >
.AutoScalingGroups[] |
select(.Tags[] | select(.Key == "Role").Value == "myrole")
- shell: aws autoscaling --region us-west-2 describe-auto-scaling-groups | jq --compact-output "{{ asg_filter }}"
register: asgs_result
- set_fact:
stale_instance_filter: >
.LaunchConfigurationName as $lc |
.Instances[] |
select(.LaunchConfigurationName != $lc) |
.InstanceId
Now I want to use stale_instance_filter on asgs_result.stdout. The following works:
- shell: echo '{{ asgs_result.stdout }}' | jq -r '{{ stale_instance_filter }}'
But this doesn't:
- shell: jq -r '{{ stale_instance_filter }}'
args:
stdin: "{{ asgs_result.stdout }}"
I get the following error message: parse error: Invalid numeric literal at line 1, column 23 (which I believe is from the account number in the ARN for the ASG.) I think it's a quoting issue (maybe something about the double quotes in the JSON), but I've also tried asgs_result.stdout | quote to no avail. I also tried the command module; it didn't help either. Of course this all works if I do it directly on the CLI.
I realize I could combine the two jq filters but I want to reuse asgs_result for other things and don't want to have to make the query multiple times. How can I fix this so I can use the stdin argument?
Edit: I was asked to provide an example of the value of asgs_result, well here you go, here's the stdout attribute in it (since I don't use anything else):
"stdout": "{\"AutoScalingGroupARN\":\"arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev\",\"TargetGroupARNs\":[],\"SuspendedProcesses\":[],\"DesiredCapacity\":4,\"Tags\":[{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"dev\",\"Key\":\"Dimension\"},{\"ResourceType\":\"auto-scaling-group\",\"ResouJceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"true\",\"Key\":\"Monitored\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"myrole\",\"Key\":\"Name\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"myrole\",\"Key\":\"Role\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"2035-09-30 18:55:31 +0000\",\"Key\":\"cleaner-destroy-after\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"vpce-2c23ca45\",\"Key\":\"force_s3_endpoint_dependency\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"owned\",\"Key\":\"kubernetes.io/cluster/dev\"}],\"EnabledMetrics\":[],\"LoadBalancerNames\":[],\"AutoScalingGroupName\":\"myrole-dev\",\"DefaultCooldown\":300,\"MinSize\":4,\"Instances\":[{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2b\",\"InstanceId\":\"i-0141fd35e3cf3ad0a\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2c\",\"InstanceId\":\"i-01aec2b3546d75190\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2a\",\"InstanceId\":\"i-0830b227f034d2859\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2b\",\"InstanceId\":\"i-0f7d847e8c168040b\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"}],\"MaxSize\":4,\"VPCZoneIdentifier\":\"subnet-c348988e,subnet-79743210,subnet-156ee36e\",\"HealthCheckGracePeriod\":300,\"TerminationPolicies\":[\"Default\"],\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\",\"CreatedTime\":\"2018-02-20T22:35:32.183Z\",\"AvailabilityZones\":[\"us-east-2a\",\"us-east-2b\",\"us-east-2c\"],\"HealthCheckType\":\"EC2\",\"NewInstancesProtectedFromScaleIn\":false}"
Sorry that it is all on one line but I don't want to make anyone think there is a newline in there, because there isn't.
The JSON content seems to be interpreted before sent to the stdin, so looks like simple quotes are sent (seen in verbose mode with -vvv):
"stdin": "{'AutoScalingGroupARN': 'arn:aws:autoscaling:us-east-2:123456:autoScalin
gGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev', ...,
'AvailabilityZones': ['us-east-2a', 'us-east-2b', 'us-east-2c']}"
Which is not JSON valid:
$ echo "{'AutoScalingGroupARN': 'arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev', 'HealthCheckGracePeriod': 300}" | jq
parse error: Invalid numeric literal at line 1, column 23
$ echo '{"AutoScalingGroupARN": "arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev", "HealthCheckGracePeriod": 300}' | jq
{
"AutoScalingGroupARN": "arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev",
"HealthCheckGracePeriod": 300
}
So, you need to "escape" it.
Unfortunately, the to_json filter, escape to much:
"stdin": "\"{\\\"AutoScalingGroupARN\\\":\\\"arn:aws:autosca...
But the string filter fits perfectly:
"stdin": "{\"AutoScalingGroupARN\":\"arn:aws:autosca...
So, the correct way with stdin is this:
- shell: jq -r '{{ stale_instance_filter }}'
args:
stdin: "{{ asgs_result.stdout | string }}"

How to replace ${} placeholders in a text file?

I want to pipe the output of a "template" file into MySQL, the file having variables like ${dbName} interspersed. What is the command line utility to replace these instances and dump the output to standard output?
The input file is considered to be safe, but faulty substitution definitions could exist. Performing the replacement should avoid performing unintended code execution.
Update
Here is a solution from yottatsa on a similar question that only does replacement for variables like $VAR or ${VAR}, and is a brief one-liner
i=32 word=foo envsubst < template.txt
Of course if i and word are in your environment, then it is just
envsubst < template.txt
On my Mac it looks like it was installed as part of gettext and from MacGPG2
Old Answer
Here is an improvement to the solution from mogsie on a similar question, my solution does not require you to escale double quotes, mogsie's does, but his is a one liner!
eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
The power on these two solutions is that you only get a few types of shell expansions that don't occur normally $((...)), `...`, and $(...), though backslash is an escape character here, but you don't have to worry that the parsing has a bug, and it does multiple lines just fine.
Sed!
Given template.txt:
The number is ${i}
The word is ${word}
we just have to say:
sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt
Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation.
Use /bin/sh. Create a small shell script that sets the variables, and then parse the template using the shell itself. Like so (edit to handle newlines correctly):
File template.txt:
the number is ${i}
the word is ${word}
File script.sh:
#!/bin/sh
#Set variables
i=1
word="dog"
#Read in template one line at the time, and replace variables (more
#natural (and efficient) way, thanks to Jonathan Leffler).
while read line
do
eval echo "$line"
done < "./template.txt"
Output:
#sh script.sh
the number is 1
the word is dog
I was thinking about this again, given the recent interest, and I think that the tool that I was originally thinking of was m4, the macro processor for autotools. So instead of the variable I originally specified, you'd use:
$echo 'I am a DBNAME' | m4 -DDBNAME="database name"
Create rendertemplate.sh:
#!/usr/bin/env bash
eval "echo \"$(cat $1)\""
And template.tmpl:
Hello, ${WORLD}
Goodbye, ${CHEESE}
Render the template:
$ export WORLD=Foo
$ CHEESE=Bar ./rendertemplate.sh template.tmpl
Hello, Foo
Goodbye, Bar
template.txt
Variable 1 value: ${var1}
Variable 2 value: ${var2}
data.sh
#!/usr/bin/env bash
declare var1="value 1"
declare var2="value 2"
parser.sh
#!/usr/bin/env bash
# args
declare file_data=$1
declare file_input=$2
declare file_output=$3
source $file_data
eval "echo \"$(< $file_input)\"" > $file_output
./parser.sh data.sh template.txt parsed_file.txt
parsed_file.txt
Variable 1 value: value 1
Variable 2 value: value 2
Here's a robust Bash function that - despite using eval - should be safe to use.
All ${varName} variable references in the input text are expanded based on the calling shell's variables.
Nothing else is expanded: neither variable references whose names are not enclosed in {...} (such as $varName), nor command substitutions ($(...) and legacy syntax `...`), nor arithmetic substitutions ($((...)) and legacy syntax $[...]).
To treat a $ as a literal, \-escape it; e.g.:\${HOME}
Note that input is only accepted via stdin.
Example:
$ expandVarsStrict <<<'$HOME is "${HOME}"; `date` and \$(ls)' # only ${HOME} is expanded
$HOME is "/Users/jdoe"; `date` and $(ls)
Function source code:
expandVarsStrict(){
local line lineEscaped
while IFS= read -r line || [[ -n $line ]]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
# Escape ALL chars. that could trigger an expansion..
IFS= read -r -d '' lineEscaped < <(printf %s "$line" | tr '`([$' '\1\2\3\4')
# ... then selectively reenable ${ references
lineEscaped=${lineEscaped//$'\4'{/\${}
# Finally, escape embedded double quotes to preserve them.
lineEscaped=${lineEscaped//\"/\\\"}
eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
done
}
The function assumes that no 0x1, 0x2, 0x3, and 0x4 control characters are present in the input, because those chars. are used internally - since the function processes text, that should be a safe assumption.
here's my solution with perl based on former answer, replaces environment variables:
perl -p -e 's/\$\{(\w+)\}/(exists $ENV{$1}?$ENV{$1}:"missing variable $1")/eg' < infile > outfile
I would suggest using something like Sigil:
https://github.com/gliderlabs/sigil
It is compiled to a single binary, so it's extremely easy to install on systems.
Then you can do a simple one-liner like the following:
cat my-file.conf.template | sigil -p $(env) > my-file.conf
This is much safer than eval and easier then using regex or sed
Here is a way to get the shell to do the substitution for you, as if the contents of the file were instead typed between double quotes.
Using the example of template.txt with contents:
The number is ${i}
The word is ${word}
The following line will cause the shell to interpolate the contents of template.txt and write the result to standard out.
i='1' word='dog' sh -c 'echo "'"$(cat template.txt)"'"'
Explanation:
i and word are passed as environment variables scopped to the execution of sh.
sh executes the contents of the string it is passed.
Strings written next to one another become one string, that string is:
'echo "' + "$(cat template.txt)" + '"'
Since the substitution is between ", "$(cat template.txt)" becomes the output of cat template.txt.
So the command executed by sh -c becomes:
echo "The number is ${i}\nThe word is ${word}",
where i and word are the specified environment variables.
If you are open to using Perl, that would be my suggestion. Although there are probably some sed and/or AWK experts that probably know how to do this much easier. If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily, but you might just as well put it into a standard Perl script at that point.
perl -p -e 's/\$\{dbName\}/testdb/s' yourfile | mysql
A short Perl script to do something slightly more complicated (handle multiple keys):
#!/usr/bin/env perl
my %replace = ( 'dbName' => 'testdb', 'somethingElse' => 'fooBar' );
undef $/;
my $buf = <STDIN>;
$buf =~ s/\$\{$_\}/$replace{$_}/g for keys %replace;
print $buf;
If you name the above script as replace-script, it could then be used as follows:
replace-script < yourfile | mysql
file.tpl:
The following bash function should only replace ${var1} syntax and ignore
other shell special chars such as `backticks` or $var2 or "double quotes".
If I have missed anything - let me know.
script.sh:
template(){
# usage: template file.tpl
while read -r line ; do
line=${line//\"/\\\"}
line=${line//\`/\\\`}
line=${line//\$/\\\$}
line=${line//\\\${/\${}
eval "echo \"$line\"";
done < ${1}
}
var1="*replaced*"
var2="*not replaced*"
template file.tpl > result.txt
I found this thread while wondering the same thing. It inspired me to this (careful with the backticks)
$ echo $MYTEST
pass!
$ cat FILE
hello $MYTEST world
$ eval echo `cat FILE`
hello pass! world
Lots of choices here, but figured I'd toss mine on the heap. It is perl based, only targets variables of the form ${...}, takes the file to process as an argument and outputs the converted file on stdout:
use Env;
Env::import();
while(<>) { $_ =~ s/(\${\w+})/$1/eeg; $text .= $_; }
print "$text";
Of course I'm not really a perl person, so there could easily be a fatal flaw (works for me though).
It can be done in bash itself if you have control of the configuration file format. You just need to source (".") the configuration file rather than subshell it. That ensures the variables are created in the context of the current shell (and continue to exist) rather than the subshell (where the variable disappear when the subshell exits).
$ cat config.data
export parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA
export parm_user=pax
export parm_pwd=never_you_mind
$ cat go.bash
. config.data
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
If your config file cannot be a shell script, you can just 'compile' it before executing thus (the compilation depends on your input format).
$ cat config.data
parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA # JDBC URL
parm_user=pax # user name
parm_pwd=never_you_mind # password
$ cat go.bash
cat config.data
| sed 's/#.*$//'
| sed 's/[ \t]*$//'
| sed 's/^[ \t]*//'
| grep -v '^$'
| sed 's/^/export '
>config.data-compiled
. config.data-compiled
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
In your specific case, you could use something like:
$ cat config.data
export p_p1=val1
export p_p2=val2
$ cat go.bash
. ./config.data
echo "select * from dbtable where p1 = '$p_p1' and p2 like '$p_p2%' order by p1"
$ bash go.bash
select * from dbtable where p1 = 'val1' and p2 like 'val2%' order by p1
Then pipe the output of go.bash into MySQL and voila, hopefully you won't destroy your database :-).
In place perl editing of potentially multiple files, with backups.
perl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : ""/eg' \
-i.orig \
-p config/test/*
I created a shell templating script named shtpl. My shtpl uses a jinja-like syntax which, now that I use ansible a lot, I'm pretty familiar with:
$ cat /tmp/test
{{ aux=4 }}
{{ myarray=( a b c d ) }}
{{ A_RANDOM=$RANDOM }}
$A_RANDOM
{% if $(( $A_RANDOM%2 )) == 0 %}
$A_RANDOM is even
{% else %}
$A_RANDOM is odd
{% endif %}
{% if $(( $A_RANDOM%2 )) == 0 %}
{% for n in 1 2 3 $aux %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/passwd field #$n: $(grep $USER /etc/passwd | cut -d: -f$n)
{% endfor %}
{% else %}
{% for n in {1..4} %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/group field #$n: $(grep ^$USER /etc/group | cut -d: -f$n)
{% endfor %}
{% endif %}
$ ./shtpl < /tmp/test
6535
6535 is odd
$myarray[0]: a
/etc/group field #1: myusername
$myarray[1]: b
/etc/group field #2: x
$myarray[2]: c
/etc/group field #3: 1001
$myarray[3]: d
/etc/group field #4:
More info on my github
To me this is the easiest and most powerful solution, you can even include other templates using the same command eval echo "$(<template.txt):
Example with nested template
create the template files, the variables are in regular bash syntax ${VARIABLE_NAME} or $VARIABLE_NAME
you have to escape special characters with \ in your templates otherwhise they will be interpreted by eval.
template.txt
Hello ${name}!
eval echo $(<nested-template.txt)
nested-template.txt
Nice to have you here ${name} :\)
create source file
template.source
declare name=royman
parse the template
source template.source && eval echo "$(<template.txt)"
the output
Hello royman!
Nice to have you here royman :)
envsubst
please don't use anything else (ie. don't eval)

Resources