how to build up terraform local_file dynamically - amazon-ec2

I want to create few EC2 instances and grab their private_ips into a ec2.ini style file for further use with Ansible.
resource "local_file" "ec2_id" {
count = var.instance_count
content = "${aws_instance.instance[count.index].private_ip} ansible_ssh_user=ec2-user\n"
filename = "ec2.ini"
}
This always prints the private_ip of latest EC2 instance created.
Any idea how to solve this.
Update:-
data "template_file" "hehe" {
count = var.instance_count
template = "${element(aws_instance.instance.*.private_ip, count.index)} ansible_ssh_user=ec2-user subnetmask=${element(split("/", data.aws_subnet.selected-subnet-id.cidr_block),1)}\n"
}
resource "local_file" "ec2_id" {
count = var.instance_count
content = "${element(data.template_file.hehe.*.rendered, count.index)}"
filename = "ec2.ini"
}
does not work. gives me the last created instance private_ip.

When you use count inside a resource you are asking Terraform to create multiple instances of that resource. However, in your case you didn't include count.index in the filename argument and so all of your instances are competing to overwrite the same filename ec2.ini, and so only one of them can "win".
It sounds like your goal is to create only one file that contains all of the IP addresses. This is very close to one of the examples that are in the Terraform String Templates documentation at the time I write this, which we can adapt to your goal like this:
resource "local_file" "ec2_iini" {
filename = "ec2.ini"
content = <<-EOT
%{ for ip in aws_instance.instance.*.private_ip ~}
${ip} ansible_ssh_user=ec2-user
%{ endfor ~}
EOT
}
In the above example the local_file resource itself does not have count set, because our goal is to create only one file. Instead, we use Terraform's template for directive to repeat a string template once per instance, gathering the result as a single string which local_file can then use as its content argument.
I used the "heredoc" style of string literal here because I think it makes the for directive easier to read by splitting it over multiple lines. The - in <<-EOT causes Terraform to look at all of the lines between the opening <<-EOT and the closing EOT and find the smallest number of leading spaces those lines have in common, which it will then strip off when rendering. That means that you can have the template indented in your configuration but avoid those indentations appearing in the rendered string, which should look something like this:
10.1.0.34 ansible_ssh_user=ec2-user
10.1.0.5 ansible_ssh_user=ec2-user
10.1.0.92 ansible_ssh_user=ec2-user
The ~ markers on the end of the two %{ ... } directives instruct Terraform's template engine to ignore the newline and whitespace after them, so we can wrap the template over multiple lines without introducing additional newlines into the result. The only line of the template that generates a trailing newline here is the middle line containing the IP address interpolation and the ansible_ssh_user portion, so the result ends up having only one newline per entry as seems to be intended here.

Related

How to have ruby conditionally check if variables exist in a string?

So I have a string from a rendered template that looks like
"Dear {{user_name}},\r\n\r\nThank you for your purchase. If you have any questions, we are happy to help.\r\n\r\n\r\n{{company_name}}\r\n{{company_phone_number}}\r\n"
All those variables like {{user_name}} are optional and do not need to be included but I want to check that if they are, they have {{ in front of the variable name. I am using liquid to parse and render the template and couldn't get it to catch if the user only uses 1 (or no) opening brackets. I was only able to catch the proper number of closing brackets. So I wrote a method to check that if these variables exist, they have the correct opening brackets. It only works, however, if all those variables are found.
here is my method:
def validate_opening_brackets?(template)
text = %w(user_name company_name company_phone_number)
text.all? do |variable|
next unless template.include? variable
template.include? "{{#{variable}"
end
end
It works, but only if all variables are present. If, for example, the template created by the user does not include user_name, then it will return false. I've also done this loop using each, and creating a variable outside of the block that I assign false if the conditions are not met. I would really, however, like to get this to work using the all? method, as I can just return a boolean and it's cleaner.
If the question is about how to rewrite the all? block to make it return true if all present variable names have two brackets before them and false otherwise then you could use something like this:
def validate_opening_brackets?(template)
variables = %w(user_name company_name company_phone_number)
variables.all? do |variable|
!template.include?(variable) || template.include?("{{#{variable}")
end
end
TL;DR
There are multiple ways to do this, but the easiest way I can think of is to simply prefix/postfix a regular expression with the escaped characters used by Mustache/Liquid, and using alternation to check for each of your variable names within the template variable characters (e.g. double curly braces). You can then use String#scan and then return a Boolean from Enumerable#any? based on the contents of the Array returned by from #scan.
This works with your posted example, but there may certainly be other use cases where you need a more complex solution. YMMV.
Example Code
This solution escapes the leading and trailing { and } characters to avoid having them treated as special characters, and then interpolates the variable names with | for alternation. It returns a Boolean depending on whether templated variables are found.
def template_string_has_interpolations? str
var_names = %w[user_name company_name company_phone_number]
regexp = /\{\{#{var_names.join ?|}\}\}/
str.scan(regexp).any?
end
Tested Examples
template_string_has_interpolations? "Dear {{user_name}},\r\n\r\nThank you for your purchase. If you have any questions, we are happy to help.\r\n\r\n\r\n{{company_name}}\r\n{{company_phone_number}}\r\n"
#=> true
template_string_has_interpolations? "Dear Customer,\r\n\r\nThank you for your purchase. If you have any questions, we are happy to help.\r\n\r\n\r\nCompany, Inc.\r\n(555) 555-5555\r\n"
#=> false

Terraform: Need to pass a bash script to 3 different Launch templates's userdata

I am trying to pass to 3 AWS launch template userdata a bash script. This script calls other scripts from Github depending on a specific variable. Since each launch template must call different scripts what is the best way to accomplish it. I am currently trying to configure a data source template_file but I can´t find a way to do what I need.
This is a piece of the bash script where I put a variable that need to change its value depending on which launch template is being built every time:
#------------------------------------------------------------------------------------------
# Define here scripts (separated with 1 space) that will be executed on first run:
AMI_SCRIPTS="ami_base_lynis.sh ${ami_script}"
#------------------------------------------------------------------------------------------
download_and_run_scripts
This is the template file data source:
data "template_file" "AMIs"{
template = "${file("../AMIs/s1_aws_userdata.sh")}"
vars = {
ami = var.dci_appserver_ami
}
}
And this is the user data attribute:
user_data_base64 = base64encode(data.template_file.AMIs.rendered)
This is not working for me as it will replace the variable has the same value for all 3 launch templates. How can I assign each time a different value?
The syntax you used for user_data_base64 tells me that you're using Terraform v0.12 or later, so you should no longer use template_file as shown in the template_file documentation:
In Terraform 0.12 and later, the templatefile function offers a built-in mechanism for rendering a template from a file. Use that function instead, unless you are using Terraform 0.11 or earlier.
Because of that, I'm going to answer using the templatefile function instead.
Inside each of your launch template resource blocks, you can call templatefile with different values for the template variables in order to get a different result each time:
resource "aws_launch_template" "example1" {
# ...
user_data = base64encode(templatefile("${path.module}/../AMIs/s1_aws_userdata.sh", {
ami = var.dci_appserver_ami
ami_script = "script1.sh"
}))
}
resource "aws_launch_template" "example2" {
# ...
user_data = base64encode(templatefile("${path.module}/../AMIs/s1_aws_userdata.sh", {
ami = var.dci_appserver_ami
ami_script = "script2.sh"
}))
}
resource "aws_launch_template" "example3" {
# ...
user_data = base64encode(templatefile("${path.module}/../AMIs/s1_aws_userdata.sh", {
ami = var.dci_appserver_ami
ami_script = "script3.sh"
}))
}
You could in principle factor out constructing the templates into a local value if you want to do it more systematically, but since your question didn't include any indication that you are doing anything special with the launch templates here I've just written the simplest possible approach, where each launch template has its own template-rendering expression.

How to use get_object in ruby for AWS?

I am very new to ruby. I am able to connect to AWS S3 using ruby. I am using following code
filePath = '/TMEventLogs/stable/DeviceWiFi/20160803/1.0/20160803063600-2f9aa901-2ce7-4932-aafd-f7286cdb9871.csv'
s3.get_object({bucket: "analyticspoc", key:"TMEventLogs/stable/DeviceWiFi/20160803/1.0/"}, target:filePath ) do |chunk|
puts "1"
end
In above code s3 is client. "analyticspoc" is root bucket. My path to csv file is as follows All Buckets /analyticspoc/TMEventLogs/stable/DeviceWiFi/20160803/1.0/20160803063600-2f9aa901-2ce7-4932-aafd-f7286cdb9871.csv.
I have tried above code. I above code I was getting error Error getting objects: [Aws::S3::Errors::NoSuchKey] - The specified key does not exist. Using above code I want to read the contents of a file. How to do that ? Please tell me what is the mistake in above code
Got the answer. You can use list_objects for accessing array of file names in chunk(1000 at a time) where as get_object is used for accessing the content of a single file as follows
BUCKET = "analyticspoc"
path = "TMEventLogs/stable/DeviceWiFi/20160803/1.0/"
s3.list_objects(bucket:BUCKET, prefix: path).each do |response|
contents = response.contents
end
file_name = "TMEventLogs/stable/DeviceWiFi/20160803/1.0/012121212121"
response = s3.get_object(bucket: BUCKET, key: file_name)
As far as I can tell you're passing in the arguments incorrectly. It should be a single options hash according to the documentation for get_object:
s3.get_object(
bucket: "analyticspoc",
key: "TMEventLogs/stable/DeviceWiFi/20160803/1.0/",
target: filePath
) do |chunk|
puts "1"
end
I believe it was trying to use your hash as a string key which is obviously not going to work.
With Ruby the curly braces { } are only necessary in method calls if additional arguments follow that need to be in another hash or are non-hash in nature. This makes the syntax a lot less ugly in most cases where options are deliberately last, and sometimes first and last by virtue of being the only argument.

How to add a namespace to existing xml file

I want to open this file and get all elements that start with us-gaap.
ftp://ftp.sec.gov/edgar/data/916789/0001558370-15-001143.txt
To get elements I tried like this:
str = '<html><body><us-gaap:foo>foo</us-gaap:foo></body></html>'
doc = Nokogiri::XML(File.read(str))
doc.xpath('//us-gaap:*')
Nokogiri::XML::XPath::SyntaxError: Undefined namespace prefix: //us-gaap:*
from /Users/ironsand/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/searchable.rb:165:in `evaluate'
doc.namespaces returns {}, so I think I have to add namespace us-gaap.
There are some questions about "adding namespace with Nokogiri", but it looks like about how to create a new XML document, not how to add a namespace to existing documents.
How can I add a namespace to existing document?
I know I can remove the namespace by Nokogiri::XML::Document#remove_namespaces!, but I don't want to use it because it removes also necesarry information.
You have asked an XY Problem. You think that the problem is that you need to add a missing namespace; the real problem is that the file you're trying to parse is not valid XML.
require 'nokogiri'
doc = Nokogiri.XML( IO.read('0001558370-15-001143.txt') )
doc.errors.length
#=> 5716
For example, the <ACCEPTANCE-DATETIME> 'element' opened on line 3 is never closed, and on line 16 there is a raw ampersand in the text:
STANDARD INDUSTRIAL CLASSIFICATION: ELECTRIC HOUSEWARES & FANS [3634]
which ought to be escaped as an entity.
However, the document has valid XML fragments within it! In particular, there is one XML document that defines xmlns:us-gaap namespace, from lines 27243-49312. Let's extract just that, using only the knowledge that the root element defines the namespace we want, and the assumptions that no element with the same name is nested within the document, and that the root element does not have an unescaped > character in any attribute. (These assumptions are valid for this file, but may not be valid for every XML file.)
txt = IO.read('0001558370-15-001143.txt')
gaap_finder = %r{(<(\w+) [^>]+xmlns:us-gaap=.+?</\2>)}m
txt.scan(gaap_finder) do |xml,_|
doc = Nokogiri.XML( xml )
gaaps = doc.xpath('//us-gaap:*')
p gaaps.length
#=> 569
end
The code above handles the case where there may be more than one XML document in the txt file, though in this case there is only one.
Decoded, the gaap_finder regex says this:
%r{...}m — this is a regular expression (that allows slashes in it, unescaped) with "multiline mode", where a period will match newline characters
(...) — capture everything we find
< — start with a literal "less-than" symbol
(\w+) — find one or more word characters (the tag name), and save them
— the word characters must be followed by a space (important to avoid capturing the <xsd:xbrl ...> element in this file)
[^>]+ — followed by one or more characters that is NOT a "greater-than" symbol (to ensure that we stay in the same element that we started in)
xmlns:us-gaap\s*= — followed by this literal namespace declaration (which may have whitespace separating it from the equals sign)
.+? — followed by anything (as little as possible)...
</\2> — ...up until you see a closing tag with the same name as what we captured for the name of the starting tag
Because of the way scan works when the regex has capturing groups, each result is a two-element array, where the first element is the entire captured XML and the second element is the name of the tag that we captured (which we "discard" by assigning it to the _ variable).
If you want to be less magic about your capturing, the text file format appears to always wrap each XML document in <XBRL>...</XBRL>. So, you could do this to process every XML file (there are seven, five of which do not happen to have any us-gaap namespaces):
txt = IO.read('0001558370-15-001143.txt')
xbrls = %r{(?<=<XBRL>).+?(?=</XBRL>)}m # find text inside <XBRL>…</XBRL>
txt.scan(xbrls) do |xml|
doc = Nokogiri.XML( xml )
if doc.namespaces["xmlns:us-gaap"]
gaaps = doc.xpath('//us-gaap:*')
p gaaps.length
end
end
#=> 569
#=> 0 (for the XML Schema document that defines the namespace)
I couldn't figure out how to update an existing doc with a new namespace, but since Nokogiri will recognize namespaces on the root element, and those namespaces are, syntactically, just attributes, you can update the document with a new namespace declaration, serialize the doc to a string, and re-parse it:
str = '<html><body><us-gaap:foo>foo</us-gaap:foo></body></html>'
doc_without_ns = Nokogiri::XML(str)
doc_without_ns.root['xmlns:us-gaap'] = 'http://your/actual/ns/here'
doc = Nokogiri::XML(doc_without_ns.to_xml)
doc.xpath("//us-gaap:*")
# Returns [#<Nokogiri::XML::Element:0x3ff375583f9c name="foo" namespace=#<Nokogiri::XML::Namespace:0x3ff375583f24 prefix="us-gaap" href="http://your/actual/ns/here"> children=[#<Nokogiri::XML::Text:0x3ff375583768 "foo">]>]

Replacing scan by gsub in Ruby: how to allow code in gsub block?

I am parsing a Wiki text from an XML dump, for a string named 'section' which includes templates in double braces, including some arguments, which I want to reorganize.
This has an example named TextTerm:
section="Sample of a text with a first template {{TextTerm|arg1a|arg2a|arg3a...}} and then a second {{TextTerm|arg1b|arg2b|arg3b...}} etc."
I can use scan and a regex to get each template and work on it on a loop using:
section.scan(/\{\{(TextTerm)\|(.*?)\|(.*?)\}\}/i).each { |item| puts "1=" + item[1] # arg1a etc.}
And, I have been able to extract the database of the first argument of the template.
Now I also want to replace the name of the template "NewTextTerm" and reorganize its arguments by placing the second argument in place of the first.
Can I do it in the same loop? For example by changing scan by a gsub(rgexp){ block}:
section.gsub!(/\{\{(TextTerm)\|(.*?)\|(.*?)\}\}/) { |item| '{{NewTextTerm|\2|\1}}'}
I get:
"Sample of a text with a first template {{NewTextTerm|\\2|\\1}} and then a second {{NewTextTerm|\\2|\\1}} etc."
meaning that the arguments of the regexp are not recognized. Even if it worked, I would like to have some place within the gsub block to work on the arguments. For example, I can't have a puts in the gsub block similar to the scan().each block but only a string to be substituted.
Any ideas are welcome.
PS: Some editing: braces and "section= added", code is complete.
When you have the replacement as a string argument, you can use '\1', etc. like this:
string.gsub!(regex, '...\1...\2...')
When you have the replacement as a block, you can use "#$1", etc. like this:
string.gsub!(regex){"...#$1...#$2..."}
You are mixing the uses. Stick to either one.
Yes, changing the quote by a double quote isn't enough, #$1 is the answer. Here is the complete code:
section="Sample of a text with a first template {{TextTerm|arg1a|arg2a|arg3a...}} and then a second {{TextTerm|arg1b|arg2b|arg3b...}} etc."
section.gsub(/\{\{(TextTerm)\|(.*?)\|(.*?)\}\}/) { |item| "{{New#$1|#$3|#$2}}"}
"Sample of a text with a first template {{NewTextTerm|arg2a|arg3a...|arg1a}} and then a second {{NewTextTerm|arg2b|arg3b...|arg1b}} etc."
Thus, it works. Thanks.
But now I have to replace the string, by a "function" returning the changed string:
def stringreturn(arg1,arg2,arg3) strr = "{{New"+arg1 + arg3 +arg2 + "}}"; return strr ; end
and
section.gsub(/\{\{(TextTerm)\|(.*?)\|(.*?)\}\}/) { |item| stringreturn("#$1","|#$2","|#$3") }
will return:
"Sample of a text with a first template {{NewTextTerm|arg2a|arg3a...|arg1a}} and then a second {{NewTextTerm|arg2b|arg3b...|arg1b}} etc."
Thanks to all!
There is probably a better way to manipulate arguments in MediaWiki templates using Ruby.

Resources