I am writing a custom type for Puppet and use the following code to copy a module file specified by a puppet url to the user's home directory:
def generate
if self[:source]
uri = URI.parse(self[:source])
path = File.join(Etc.getpwnam(self[:user])[:dir], File.basename(uri.path))
file_opts = {}
file_opts[:name] = File.join(Etc.getpwnam(self[:user])[:dir], File.basename(uri.path))
file_opts[:ensure] = self[:ensure] == :absent ? :absent : :file
file_opts[:source] = self[:source]
file_opts[:owner] = self[:user]
self[:source] = path
Puppet::Type.type(:file).new(file_opts)
end
end
Things are working fine so far. The resource is added to the catalog and created on the agent side. But I have a problem...
How can I specify that this additional file resource must be created before the actual type gets executed? Unfortunatley, I cannot find an example which shows how to specify a dependency on an optional resource that is defined in a generate method.
Related
I'm trying to create a Terraform module that will build my JS lambdas, zip them and deploy them. This however proves to be problematic
resource "null_resource" "build_lambda" {
count = length(var.lambdas)
provisioner "local-exec" {
command = "mkdir tmp"
working_dir = path.root
}
provisioner "local-exec" {
command = var.lambdas[count.index].code.build_command
working_dir = var.lambdas[count.index].code.working_dir
}
}
data "archive_file" "lambda_zip" {
count = length(var.lambdas)
type = "zip"
source_dir = var.lambdas[count.index].code.working_dir
output_path = "${path.root}/tmp/${count.index}.zip"
depends_on = [
null_resource.build_lambda
]
}
/*******************************************************
* Lambda definition
*******************************************************/
resource "aws_lambda_function" "lambda" {
count = length(var.lambdas)
filename = data.archive_file.lambda_zip[count.index].output_path
source_code_hash = filebase64sha256(data.archive_file.lambda_zip[count.index].output_path)
function_name = "${var.application_name}-${var.lambdas[count.index].name}"
description = var.lambdas[count.index].description
handler = var.lambdas[count.index].handler
runtime = var.lambdas[count.index].runtime
role = aws_iam_role.iam_for_lambda.arn
memory_size = var.lambdas[count.index].memory_size
depends_on = [aws_iam_role_policy_attachment.lambda_logs, aws_cloudwatch_log_group.log_group, data.archive_file.lambda_zip]
}
The property source_code_hash = filebase64sha256(data.archive_file.lambda_zip[count.index].output_path) , although technically not obligatory, is necessary, or the existing lambda will never get overriden as Terraform will think that it is still the same version of lambda and will skip the deployment altogether. Unfortunately it looks like the method filebase64sha256 is evaluated before the creation of any resource. This means that there is no zip for the hash calculation and so I get the error
Error: Error in function call
on modules\api-gateway-lambda\main.tf line 35, in resource "aws_lambda_function" "lambda":
35: source_code_hash = filebase64sha256(data.archive_file.lambda_zip[count.index].output_path)
|----------------
| count.index is 0
| data.archive_file.lambda_zip is tuple with 1 element
Call to function "filebase64sha256" failed: no file exists at tmp\0.zip.
If i manually place a zip in the right location, I can see that the whole thing starts working and the zip eventually gets overriden by a new one, but the hash in this case must come from the previous zip.
What is the right way to execute the whole thing in the right order?
The archive_file data source has its own output_base64sha256 attribute which can give you that same result without asking Terraform to read a file that doesn't exist yet:
source_code_hash = data.archive_file.lambda_zip[count.index].output_base64sha256
The data source will populate this at the same time it creates the file, and because your lambda function depends on the data source it will therefore always be available before the lambda function configuration is evaluated.
I am attempting to add the follow url capability but can't seem to get it to work. I need to crawl all the pages. There are around 108 pages of the job listings. Thank you.
import scrapy
class JobItem(scrapy.Item):
# Data structure to store the title, company name and location of the job
title = scrapy.Field()
company = scrapy.Field()
location = scrapy.Field()
class PythonDocumentationSpider(scrapy.Spider):
name = 'pydoc'
start_urls = ['https://stackoverflow.com/jobs?med=site-ui&ref=jobs-tab']
def parse(self, response):
for follow_href in response.xpath('//h2[#class="fs-body2 job-details__spaced mb4"]/a/#href'):
follow_url = response.urljoin(follow_href.extract())
yield scrapy.Request(follow_url, callback=self.parse_page_title)
for a_el in response.xpath('//div[#class="-job-summary"]'):
section = JobItem()
section['title'] = a_el.xpath('.//a[#class="s-link s-link__visited job-link"]/text()').extract()[0]
span_texts = a_el.xpath('.//div[#class="fc-black-700 fs-body1 -company"]/span/text()').extract()
section['company'] = span_texts[0]
section['location'] = span_texts[1]
print(section['location'])
#print(type(section))
yield section
I am attempting to get the following url capability to work with my code and then be able to crawl the pages and store job postings in csv file.
.extract() return a list. In most cases you'll need to use .get() or .extract_first() instead if you don't need a list.
First you need to rewrite this part:
for follow_href in response.xpath('//h2[#class="fs-body2 job-details__spaced mb4"]/a/#href').getall(): # or .extract()
follow_url = response.urljoin(follow_href)
yield scrapy.Request(follow_url, callback=self.parse_page_title)
I checked the whole azure-storage-blob gem and didn't find any way to get the URI for a blob. Is there some way to construct it correctly and in a generic way that will work for any other blob in any region?
I used S3 SDK before and I'm well grounded in S3 but new to Azure.
There is a protected method called blob_uri that looks like this:
def blob_uri(container_name, blob_name, query = {}, options = {})
if container_name.nil? || container_name.empty?
path = blob_name
else
path = ::File.join(container_name, blob_name)
end
options = { encode: true }.merge(options)
generate_uri(path, query, options)
end
So you could take the short cut of:
blob_client = Azure::Storage::Blob::BlobService.create(storage_account_name: 'XXX' , storage_access_key: 'XXX')
blob_client.send(:blob_uri, container_name,blob_name)
However, the actual URI is simply:
https://[storage_account_name].blob.core.windows.net/container/[container[s]]/[blob file name]
So since you have to know the blob name and the container to access to blob.
File.join(blob_client.host,container,blob_name)
Is the URI to the blob
I've worked with the script outlined in the following answer:
Rally APIs: How to copy Test Folder and member Test Cases
and it's handy, but what I really want to do is to move an entire Test Folder into a different project. This is next to impossible through the Rally User Interface. According to Rally Support, the only way to do this in the UI is:
Un-assign the Test Cases from their current Test Folder
Setup a Custom Grid app on your dashboard
Use the Custom Grid bulk edit to update the Project of the Test Cases
Lastly use the Custom Grid bulk edit to update the Test Folder - now that you're in the target Project, of the Test Cases
Even though the above process is clunky, it is easier now than it used to be before the advent of the bulk edit within the Custom Grids. Before you had to go through and edit each Test Case one-by-one which was very manual and slow.
However, we have several thousand Test Cases we need to move, and the Custom Grid has a fatal flaw for us. It will only show the first 200 records in a query. So we would have to manually change our grid query in a step wise manner to accomplish the move we need. This is barely better than editing Test Cases one-by-one. Is there a way to move a Test Folder with Test Cases from one Project to another, using a script? Please tell me there is.
The following script will perform this task - it will move all Test Cases from a Source Test Folder identified by FormattedID, to a Target Test Folder, also identified by FormattedID. The Source Test Folder and Target Test Folder can be in different Projects (although they must be within the same Workspace). Like the Copy script referenced in the question, the Target Test Folder must exist, i.e. the script will not create a Test Folder for you if the Target is not found.
For those needing to install and configure the Ruby REST Toolkit, links are here:
Developer Portal: Rally REST API for Ruby
Github
# Copyright 2002-2012 Rally Software Development Corp. All Rights Reserved.
require 'rally_api'
$my_base_url = "https://rally1.rallydev.com/slm"
$my_username = "user#company.com"
$my_password = "password"
$my_workspace = "My Workspace"
$my_project = "My Project"
$wsapi_version = "1.39"
# Test Folders
$source_test_folder_formatted_id = "TF8"
$target_test_folder_formatted_id = "TF11"
#==================== Make a connection to Rally ====================
config = {:base_url => $my_base_url}
config[:username] = $my_username
config[:password] = $my_password
config[:workspace] = $my_workspace
config[:project] = $my_project
config[:version] = $wsapi_version
#rally = RallyAPI::RallyRestJson.new(config)
begin
# Lookup source Test Folder
source_test_folder_query = RallyAPI::RallyQuery.new()
source_test_folder_query.type = :testfolder
source_test_folder_query.fetch = true
source_test_folder_query.query_string = "(FormattedID = \"" + $source_test_folder_formatted_id + "\")"
source_test_folder_result = #rally.find(source_test_folder_query)
# Lookup Target Test Folder
target_test_folder_query = RallyAPI::RallyQuery.new()
target_test_folder_query.type = :testfolder
target_test_folder_query.fetch = true
target_test_folder_query.query_string = "(FormattedID = \"" + $target_test_folder_formatted_id + "\")"
target_test_folder_result = #rally.find(target_test_folder_query)
if source_test_folder_result.total_result_count == 0
puts "Source Test Folder: " + $source_test_folder_formatted_id + "not found. Exiting."
exit
end
if target_test_folder_result.total_result_count == 0
puts "Target Test Folder: " + $target_test_folder_formatted_id + "not found. Target must exist before moving."
exit
end
source_test_folder = source_test_folder_result.first()
target_test_folder = target_test_folder_result.first()
# Populate full object for both Source and Target Test Folders
full_source_test_folder = source_test_folder.read
full_target_test_folder = target_test_folder.read
# Grab collection of Source Test Cases
source_test_cases = source_test_folder["TestCases"]
# Loop through Source Test Cases and Move to Target
source_test_cases.each do |source_test_case|
begin
test_case_to_update = source_test_case.read
source_test_case_formatted_id = test_case_to_update["FormattedID"]
target_project = full_target_test_folder["Project"]
target_project_full_object = target_project.read
target_project_name = target_project_full_object["Name"]
source_project = full_source_test_folder["Project"]
source_project_full_object = source_project.read
source_project_name = source_project_full_object["Name"]
puts "Source Project Name: #{source_project_name}"
puts "Target Project Name: #{target_project_name}"
# Test if the source project and target project are the same
source_target_proj_match = source_project_name.eql?(target_project_name)
# If the target Test Folder is in a different Project, we have to do some homework first:
# "un-Test Folder" the project
# Assign the Test Case to the Target Project
# Assign the Test Case to the Target Test Folder
if !source_target_proj_match then
fields = {}
fields["TestFolder"] = ""
test_case_updated = #rally.update(:testcase, test_case_to_update.ObjectID, fields) #by ObjectID
puts "Test Case #{source_test_case_formatted_id} successfully dissociated from: #{$source_test_folder_formatted_id}"
# Get full object on Target Project and assign Test Case to Target Project
fields = {}
fields["Project"] = target_project_full_object
test_case_updated = #rally.update(:testcase, test_case_to_update.ObjectID, fields) #by ObjectID
puts "Test Case #{source_test_case_formatted_id} successfully assigned to Project: #{target_project_name}"
end
# Change the Test Folder attribute on the Test Case
fields = {}
fields["TestFolder"] = target_test_folder
test_case_updated = #rally.update(:testcase, test_case_to_update.ObjectID, fields) #by ObjectID
puts "Test Case #{source_test_case_formatted_id} successfully moved to #{$target_test_folder_formatted_id}"
rescue => ex
puts "Test Case #{source_test_case_formatted_id} not updated due to error"
puts ex
end
end
end
I'm currently experimenting with creating my own gem in Ruby. The gem requires some static resources (say an icon in ICO format). Where do I put such resources within my gem directory tree and how to I access them from code?
Also, parts of my extension are native C code and I would like the C-parts to have access to the resources too.
You can put resources anywhere you want, except in the lib directory. Since it will be will be part of Ruby's load path, the only files that should be there are the ones that you want people to require.
For example, I usually store translated text in the i18n/ directory. For icons, I'd just put them in resources/icons/.
As for how to access these resources... I ran into this problem enough that I wrote a little gem just to avoid repetition.
Basically, I was doing this all the time:
def Your::Gem.root
# Current file is /home/you/code/your/lib/your/gem.rb
File.expand_path '../..', File.dirname(__FILE__)
end
Your::Gem.root
# => /home/you/code/your/
I wrapped this up into a nice DSL, added some additional convenience stuff and ended up with this:
class Your::Gem < Jewel::Gem
root '../..'
end
root = Your::Gem.root
# => /home/you/code/your/
# No more joins!
path = root.resources.icons 'your.ico'
# => /home/you/code/your/resources/icons/your.ico
As for accessing your resources in C, path is just a Pathname. You can pass it to a C function as a string, open the file and just do what you need to do. You can even return an object to the Ruby world:
VALUE your_ico_new(VALUE klass, VALUE path) {
char * ico_file = NULL;
struct your_ico * ico = NULL;
ico_file = StringValueCStr(path);
ico = your_ico_load_from_file(ico_file); /* Implement this */
return Data_Wrap_Struct(your_ico_class, your_ico_mark, your_ico_free, ico);
}
Now you can access it from Ruby:
ico = Your::Ico.new path