I've got a playbook with the following tasks:
- set_fact:
asg_filter: >
.AutoScalingGroups[] |
select(.Tags[] | select(.Key == "Role").Value == "myrole")
- shell: aws autoscaling --region us-west-2 describe-auto-scaling-groups | jq --compact-output "{{ asg_filter }}"
register: asgs_result
- set_fact:
stale_instance_filter: >
.LaunchConfigurationName as $lc |
.Instances[] |
select(.LaunchConfigurationName != $lc) |
.InstanceId
Now I want to use stale_instance_filter on asgs_result.stdout. The following works:
- shell: echo '{{ asgs_result.stdout }}' | jq -r '{{ stale_instance_filter }}'
But this doesn't:
- shell: jq -r '{{ stale_instance_filter }}'
args:
stdin: "{{ asgs_result.stdout }}"
I get the following error message: parse error: Invalid numeric literal at line 1, column 23 (which I believe is from the account number in the ARN for the ASG.) I think it's a quoting issue (maybe something about the double quotes in the JSON), but I've also tried asgs_result.stdout | quote to no avail. I also tried the command module; it didn't help either. Of course this all works if I do it directly on the CLI.
I realize I could combine the two jq filters but I want to reuse asgs_result for other things and don't want to have to make the query multiple times. How can I fix this so I can use the stdin argument?
Edit: I was asked to provide an example of the value of asgs_result, well here you go, here's the stdout attribute in it (since I don't use anything else):
"stdout": "{\"AutoScalingGroupARN\":\"arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev\",\"TargetGroupARNs\":[],\"SuspendedProcesses\":[],\"DesiredCapacity\":4,\"Tags\":[{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"dev\",\"Key\":\"Dimension\"},{\"ResourceType\":\"auto-scaling-group\",\"ResouJceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"true\",\"Key\":\"Monitored\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"myrole\",\"Key\":\"Name\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"myrole\",\"Key\":\"Role\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"2035-09-30 18:55:31 +0000\",\"Key\":\"cleaner-destroy-after\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"vpce-2c23ca45\",\"Key\":\"force_s3_endpoint_dependency\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"owned\",\"Key\":\"kubernetes.io/cluster/dev\"}],\"EnabledMetrics\":[],\"LoadBalancerNames\":[],\"AutoScalingGroupName\":\"myrole-dev\",\"DefaultCooldown\":300,\"MinSize\":4,\"Instances\":[{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2b\",\"InstanceId\":\"i-0141fd35e3cf3ad0a\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2c\",\"InstanceId\":\"i-01aec2b3546d75190\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2a\",\"InstanceId\":\"i-0830b227f034d2859\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2b\",\"InstanceId\":\"i-0f7d847e8c168040b\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"}],\"MaxSize\":4,\"VPCZoneIdentifier\":\"subnet-c348988e,subnet-79743210,subnet-156ee36e\",\"HealthCheckGracePeriod\":300,\"TerminationPolicies\":[\"Default\"],\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\",\"CreatedTime\":\"2018-02-20T22:35:32.183Z\",\"AvailabilityZones\":[\"us-east-2a\",\"us-east-2b\",\"us-east-2c\"],\"HealthCheckType\":\"EC2\",\"NewInstancesProtectedFromScaleIn\":false}"
Sorry that it is all on one line but I don't want to make anyone think there is a newline in there, because there isn't.
The JSON content seems to be interpreted before sent to the stdin, so looks like simple quotes are sent (seen in verbose mode with -vvv):
"stdin": "{'AutoScalingGroupARN': 'arn:aws:autoscaling:us-east-2:123456:autoScalin
gGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev', ...,
'AvailabilityZones': ['us-east-2a', 'us-east-2b', 'us-east-2c']}"
Which is not JSON valid:
$ echo "{'AutoScalingGroupARN': 'arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev', 'HealthCheckGracePeriod': 300}" | jq
parse error: Invalid numeric literal at line 1, column 23
$ echo '{"AutoScalingGroupARN": "arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev", "HealthCheckGracePeriod": 300}' | jq
{
"AutoScalingGroupARN": "arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev",
"HealthCheckGracePeriod": 300
}
So, you need to "escape" it.
Unfortunately, the to_json filter, escape to much:
"stdin": "\"{\\\"AutoScalingGroupARN\\\":\\\"arn:aws:autosca...
But the string filter fits perfectly:
"stdin": "{\"AutoScalingGroupARN\":\"arn:aws:autosca...
So, the correct way with stdin is this:
- shell: jq -r '{{ stale_instance_filter }}'
args:
stdin: "{{ asgs_result.stdout | string }}"
Related
I'm trying to take a YAML style array I get from an AWS command, and assign it to a key while updating my own YAML.
This represents what I have right now:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= "'"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')"'"' -i route53.yml
This is how route53.yml looks like before I run the command:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
and this is how route53.yml looks like after:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
ResourceRecordSets: |-
- Name: a.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: b.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: c.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: d.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: e.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
As you can see there's a |- right after the key, and it seems like it is treated as multiline string instead of an array of maps. How can I avoid it and assign the array as a YAML array? When I tried manually assigning an array in the style of ['a', 'b', 'c'] and the update works as it should, adding a YAML array under the key, how can I achieve it with the output of the aws command?
The reason why yq is interpreting the output of the $(...) expression as a multiline strine is because you have quoted it; your yq expression, simplified, looks like:
yq 'ResourceRecordSets |= "some string here"'
The quotes mean "this is a string, not a structure", so that's what you get. You could try dropping the quotes, like this:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= '"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')" route53.yml
That might work, but it is fragile. A better solution is to have yq parse the output of the subexpression as a separate document, and then merge it as a structured document rather than a big string. Like this:
yq \
'(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= input.ResourceRecordSets' \
route53.yml \
<(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml)
This takes advantage of the fact that you can provide jq (and hence yq) multiple files on the command line, and then refer to the input variable (described in the IO section of the jq manual). The above command line is structured like this:
yq <expression> <file1> <file2>
Where <file1> is route53.yml, and <file2> is a bash process substitution.
This solution simplifies issues around quoting and formatting.
(You'll note I dropped your use of -i here; that seems to throw yq for a loop, and it's easy to output to a temporary file and then rename it.)
Eventually I solved it using a combination of larsks answer about the unnecessary quotes, and using jq instead of yq (no matter what I did, using solely yq wouldn't work) to assign the array to a separate variable before editing the YAML document:
record_sets=$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" | jq '.[]')
yq '(.HostedZones[] | select(.Id=="ABC123")).ResourceRecordSets |= '"$record_sets"'' -i route53.yml
I am using [file lookup] which reads the whole file and stores the content in a variable. My play looks something like this:
- name: Store foo.xml contents in a variable
set_fact:
foo_content: "{{ lookup('file', 'foo.xml' ) | replace('\n', '')}}"
So the above code reads the foo.xml file and stores it in the variable, but the problem is when the foo.xml has line breaks in it, it also includes the line break in the variable.
My foo.xml is this file:
<?xml version="1.0" encoding="utf-8"?>
<initialize_param>
<secrets>
<my_secret id="99">3VMjII6Hw+pd1zHV5THSI712y421USUS8124487128745812sajfhsakjfasbfvcasvnjasjkvbhasdfasgfsfaj5G8A9+n8CkLxk7Dqu0G8Jclg0eb1A5xeFzR3rrJHrb2GBBa7PJNVx8tFJP3AtF6ek/F/WvlBIs2leX2fq+/bGryKlySuFmbcwBsThmPJC5Z5AwPJgGZx</my_secret>
</secrets>
</initialize_param>
The output removes line break \n but also incudes the tabs \r & \t
I need to got rid of the \n , need to get rid of extra formatting too (\r & \t), Moreover after the replace filter I get the error while firing a DB Update query as
stderr: /bin/sh: 1: cannot open ?xml: No such file
Use the Jinja trim filter:
"{{ lookup('file', 'foo.xml' ) | trim }}"
You can do that with the replace filter?
contents: "{{ lookup('file', '/etc/foo.txt') | replace('\n', '')}}"
You may use the regex_replace filter since the trim doesn't clear other line break characters as you mentioned in the question.
"{{ some_stdout_to_clear | regex_replace('[\\r\\n\\t]+','') }}"
I have a Ansible line that fails linting:
tags: "{{ deployment_id | resource_tags('asg', base_resource_tags, deployment=deployment_id, deployment_env=deployment_env, deployment_name=deployment_name, purpose=deployment_purpose_tag, cpu_utilization=deployment_cpu_utilization_tag, disk_io_class=deployment_disk_io_tag, prom_exporters=deployment_prom_exporter_tags) | asg_tag_list }}"
How do I make this pass linting?
You need to use YAML Folding Scalars, > without quotes. Then add 'block chomping' 'strip'docs indicator to remove the trailing newline. The below example will work correctly, with each newline translating to a space. Adding quotes will break it e.g.
tags: >-
{{ deployment_id | resource_tags('asg', base_resource_tags, deployment=deployment_id,
deployment_env=deployment_env, deployment_name=deployment_name, Purpose=deployment_purpose_tag,
cpu_utilization=deployment_cpu_utilization_tag, disk_io_class=deployment_disk_io_tag,
prom_exporters=deployment_prom_exporter_tags) | asg_tag_list }}
Under my Playbook , i want to run a shell command which is the following :
for STACK in stackone stacktwo;do docker stack ps --format "table {{.ID}}\t{{.Name}}\t{{.CurrentState}}\t{{.Error}}\t{{.Node}}" $STACK | (read -r; printf "%s\n" "$REPLY"; sort -k 2|grep srcd |grep -v Shutdown ); done;
As you can see my shell command is quite complex , so when i am running it like this , it fails throwing always syntax errors
- name : Check running services
shell: for STACK in stackone srcd-pilote;do docker stack ps --format "table {{.ID}}\t{{.Name}}\t{{.CurrentState}}\t{{.Error}}\t{{.Node}}" $STACK | (read -r; printf "%s\n" "$REPLY"; sort -k 2|grep srcd |grep -v Shutdown ); done;
register: result
I see there is many characters which need to be passed as strings such as {{.Name}} and |grep ...
So i ve tried this :
- name : Check running services
shell: "for STACK in srcd-current stacktwo;do docker stack ps --format 'table {{'"{{.ID}}"'}}\t{{'"{{.Name}}"'}}\t{{'"{{.CurrentState}}"'}}\t{{'"{{.Error}}"'}}\t{{'"{{.Node}}"'}}' $STACK | (read -r; printf '%s\n' '$REPLY'; sort -k 2'"{{|}}"'grep srcd '"{{|}}"'grep -v Shutdown ); done; "
register: result
But it stills failing .
Suggesstions ??
I had to use a shell for loop in one of my ansible tasks. I use a Literal Block Scalar '|' to do so. Try something like this:
- name : Check running services
shell: |
for STACK in stackone srcd-pilote;do
docker stack ps --format "table {{ '{{' }}.ID{{ '}}' }}\t{{ '{{' }}.Name{{ '}}' }}\t{{ '{{' }}.CurrentState{{ '}}' }}\t{{ '{{' }}.Error{{ '}}' }}\t{{ '{{' }}.Node{{ '}}' }}" $STACK | (read -r; printf "%s\n" "$REPLY"; sort -k 2|grep srcd |grep -v Shutdown );
done;
More doc on the scalar can be found on the Ansible doc about YAMLÂ syntax.
You should also escape the {{ and }} from the jinja2, by using {{ '{{' }} and {{ '}}' }}.
Note 1: Here is a nice sed command to do so:
s/{{\([^}]*\)}}/{{ '{{' }}\1{{ '}}' }}/g
Note 2:
You should review the command to split it in several lines to improve the readability (with \ at the end of line or using the > scalar).
Ansible-Lint rules recommend line shorter than 160 characters (rule E204).
I have to use Ansible modules in order to edit the /etc/ssh/sshd_config file - every time I create a new user I want to append it at these two lines:
AllowUsers root osadmin <new_user>
AllowGroups root staff <new_group>
At this moment I'm using the shell module to execute a sed command but would like to use lineinfile, if possible
- shell: "sed -i '/^Allow/ s/$/ {{ user_name }}/' /etc/ssh/sshd_config"
Any suggestions would be sincerely appreciated.
The replace module will replace all instances of a regular expression pattern within a file. Write a task to match the AllowUsers line and replace it with the original line appended with the user name. To ensure the task is idempotent, a negative lookahead assertion in the regular expression checks if the user name already appears in the line. For example:
- name: Add user to AllowUsers
replace:
backup: yes
dest: /etc/ssh/sshd_config
regexp: '^(AllowUsers(?!.*\b{{ user_name }}\b).*)$'
replace: '\1 {{ user_name }}'
You could do it in a single play with a newline, but I think it's cleaner to use two lineinfile plays for this.
- hosts: '127.0.0.1'
vars:
usernames:
- larry
- curly
- moe
usergroups:
- stooges
- admins
tasks:
- lineinfile:
dest: /etc/ssh/sshd_config
regexp: '^AllowUsers'
line: "AllowUsers {{usernames | join(' ')}}"
- lineinfile:
dest: /etc/ssh/sshd_config
regexp: '^AllowGroups'
line: "AllowGroups {{usergroups | join(' ')}}"
Note that groups is a reserved word so don't use that as a variable name.
The selected answer assumes that the complete list of users is available at runtime, while the most popular answer can fail when there is a dash in the username, because \b interprets it as a word boundary. The following solution assumes that the playbook cannot regenerate the complete list of usernames from scratch, and tries and handle the corner case of dashes:
name: add a user to the list of AllowUsers if not present
lineinfile:
path: /etc/ssh/sshd_config
backrefs: yes
backup: yes
regexp: "^AllowUsers((?:(?:\s+\S+(?!\S))(?<!\s{{ username }}))+\s*?)(\n?)$"
line: "AllowUsers\1 {{ username }}\2"
validate: /usr/sbin/sshd -t -f %s
As a bonus, I threw in sshd_config backup and verification.
How the (interesting part of the) regular expression works:
--------------------------+----------------------------------------------------
( |
--------------------------+----------------------------------------------------
(?: | This group is not captured
--------------------------+----------------------------------------------------
(?:\s+\S+(?!\S)) | Matches any sequence of whitespace characters fol-
| lowed by any sequence of non-whitespace characters,
| that is to say a leading space and a username. The
| negative look-ahead at the end prevents a "catast-
| rophic backtracking". Also, this group is not cap-
| tured.
--------------------------+----------------------------------------------------
(?<!\s{{ username }}) | Applies a negative look-behind on the username, so
| that if the username found by the previous expres-
| sion matches, the regular expression fails. The
| match on a leading whitespace character ensures
| that the comparison is made on the complete string.
--------------------------+----------------------------------------------------
)+ | Groups the detection of a username and its negative
| look-behind together. The "+" quantifier is used
| here on the assumption that the file already cont-
| ains at least one username, but "*" could be used
| for a more relaxed matching.
--------------------------+----------------------------------------------------
\s*? | Matches any trailing whitespace. The match is lazy
| in order to detect the newline character later on.
--------------------------+----------------------------------------------------
) | Captures the whole text after "AllowUsers" (this
| will be \1).
--------------------------+----------------------------------------------------
(\n?) | Captures either a newline character or an empty
| string (this will be \2).
--------------------------+----------------------------------------------------
If the regular expression matches, it means that the line exists and that it does not contain {{ username }}, so we append it.
If the regular expression does not match, it means that either the line does not exist or that it contains {{ username }}, and we do nothing.
I had the same problem. I needed add user to sudoers group, let's say 'testuser' to line:
User_Alias SOMEADMIN = smoeuser1, someuser2, someuser3
This worked well for me:
- name: add testuser to end of line
lineinfile:
dest: /etc/sudoers.d/somegroup
state: present
regexp: '^(User_Alias(.*)$)'
backrefs: yes
line: '\1, testuser'
The point is that if I had '^User_Alias(..)$'* in regexp and not '^(User_Alias(..)$)'* it didn't work and whole line was replaced. With () arround searched text the result was OK:
User_Alias SOMEADMIN = smoeuser1, someuser2, someuser3, testuser
So then anything can work in line:, included ansible variables like "{{ usernames | join(', ') }}"
This worked for me
- name: Add Group to AllowGroups
lineinfile:
dest=/etc/ssh/sshd_config
backup=True
backrefs=True
state=present
regexp='^(AllowGroups(?!.*\b{{ groupname }}\b).*)$'
line='\1 {{ groupname }}'