Create a file descriptor in ruby - ruby

I am writing a script will perform various tasks with DSV or positional files. These tasks varies and are like creating an DB table for the file, or creating a shell script for parsing it.
As I have idealized my script would receive a "descriptor" as input to perform its tasks. It then would parse this descriptor and perform its tasks accordingly.
I came up with some ideas on how to specify the descriptor file, but didn't really manage to get something robust - probably due my inexperience in ruby.
It seems though, the best way to parse the descriptor would be using ruby language itself and then somehow catch parsing exceptions to turn into something more relevant to the context.
Example:
The file I will be reading looks like (myfile.dsv):
jhon,12343535,27/04/1984
dave,53245265,30/03/1977
...
Descriptor file myfile.des contains:
FILE_TYPE = "DSV"
DSV_SEPARATOR = ","
FIELDS = [
name => [:pos => 0, :type => "string"],
phone => [:pos => 1, :type => "number"],
birthdate => [:pos => 2, :type => "date", :mask = "dd/mm/yyyy"]
]
And the usage should be:
ruby script.rb myfile.des --task GenerateTable
So the program script.rb should load and parse the descriptor myfile.des and perform whatever tasks accordingly.
Any ideas on how to perform this?

Use YAML
Instead of rolling your own, use YAML from the standard library.
Sample YAML File
Name your file something like descriptor.yml, and fill it with:
---
:file_type: DSV
:dsv_separator: ","
:fields:
:name:
:pos: 0
:type: string
:phone:
:pos: 1
:type: number
:birthdate:
:pos: 2
:type: date
:mask: dd/mm/yyyy
Loading YAML
You can read your configuration back in with:
require 'yaml'
settings = YAML.load_file 'descriptor.yml'
This will return a settings Hash like:
{:file_type=>"DSV",
:dsv_separator=>",",
:fields=>
{:name=>{:pos=>0, :type=>"string"},
:phone=>{:pos=>1, :type=>"number"},
:birthdate=>{:pos=>2, :type=>"date", :mask=>"dd/mm/yyyy"}}}
which you can then access as needed to configure your application.

Related

How to save a parsed Gherkin into a Feature File (Ruby)

I need to programatically modify feature files of cucumber.
I have parsed a feature file using gherkin's gem 'gherkin/parser'.
The problem I find is that after parsing, I end up with a hash with the following data as example:
{:type=>:GherkinDocument, :feature=>{:type=>:Feature, :tags=>[], :location=>{:line=>1, :column=>1}, :language=>"en", :keyword=>"Feature", :name=>"MyFeature", :description=>" As an user\n I want to test a feature", :children=>[{:type=>:Scenario, :tags=>[{:type=>:Tag, :location=>{:line=>5, :column=>3}, :name=>"#MyTag"}], :location=>{:line=>6, :column=>3}, :keyword=>"Scenario", :name=>"My scenario", :steps=>[{:type=>:Step, :location=>{:line=>7, :column=>5}, :keyword=>"Given ", :text=>"I start the app"}, {:type=>:Step, :location=>{:line=>8, :column=>5}, :keyword=>"And ", :text=>"I generate a test user"}, {:type=>:Step, :location=>{:line=>9, :column=>5}, :keyword=>"And ", :text=>"I finish the flow"}]}]}, :comments=>[]}
is it possible to convert this GherkinDocument generated by the parser to a plain text feature file to save it? What method or gem should I use to get
According to the docs, you would use the Ruby Gherkin::Pickles::Compiler:
require 'gherkin/parser'
require 'gherkin/pickles/compiler'
parser = Gherkin::Parser.new
gherkin_document = parser.parse("Feature: ...")
# Make changes to gherkin_document
pickles = Gherkin::Pickles::Compiler.new.compile(gherkin_document)

Logstash can`t handle all input plugins

I use Logstash input http_handler to collect metrics from different endpoints.
For each endpoint I have separate config file with "input" plugin like:
input { http_poller {urls => { server_1 => { url => 'http://10.200.3.1:8809/metrics' } } request_timeout => 5 tags => 'TL.QA.proxy-service' interval => 60 metadata_target => 'http_poller_metadata' type => 'tl_qa_http_metrics'}}
I have ~1000 such files in one directory.
When I start Logstash I specify directory to read all those files, like:
./bin/logstash -f /opt/logstash-5.6.2/configs/
When I had small amount of files (~100) it works pretty good. But now looks like Logstash doesn't have enough time to read all files and it doesn't collect data from all endpoints.
Can you please advise how I can improve it?

File.new command for Ruby to upload mp3 file to soundcloud

I'm now trying to upload a mp3 file to Soundcloud. Here I'm bogged down to the use of File.new command in Ruby.
I send a request and a passing parameter looks like the below.
Parameters: {..."mp3_1"=>#<ActionDispatch::Http::UploadedFile:0x007ff24d5e3ea8 #tempfile=#<Tempfile:/var/folders/kk/y_wprlln2qv6mzylj03g14x00000gn/T/RackMultipart20160316-21426-14vu8x1.mp3>, #original_filename="datasecurity.mp3", #content_type="audio/mp3", #headers="Content-Disposition: form-data; name=\"mp3_1\"; filename=\"datasecurity.mp3\"\r\nContent-Type: audio/mp3\r\n">}
Then, I write File.new command with the potentail file name and params[:mp3_1] like the below.
client = Soundcloud.new(:access_token => 'XXX')
track = client.post('/tracks', :track => {
:title => 'This is my sound',
:asset_data => File.new("file name",params[:mp3_1])
})
Now I get an error saying:
no implicit conversion of ActionDispatch::Http::UploadedFile into String
The paperclip function works ( storing file to the storage directly has been what I've done ) but this file.new doesn't allow me to move forward. If I can get any help, I really appreciate that (:
Best
you already have a file, no need to create a new one with File.new
have a closer look to your dump :
#tempfile=#<Tempfile:/var/folders/...../RackMultipart20160316-21426-14vu8x1.mp3
this is a file, you may use it directly in your call
client.post('/tracks', :track => {
:title => 'This is my sound',
:asset_data => params[:mp3_1].tempfile)
})

Where is the ruby function 'powershell' defined?

I am using the msutter DSC module for puppet. While reading through the source code, I come across code like this (in dsc_configuration_provider.rb):
def create
Puppet.debug "\n" + ps_script_content('set')
output = powershell(ps_script_content('set'))
Puppet.debug output
end
What file defines the powershell function or method? Is it a ruby builtin? A puppet builtin? Inherited from a class? I know that it is being used to send text to powershell as a command and gathering results, but I need to see the source code to understand how to improve its error logging for my purposes, because certain powershell errors are being swallowed and no warnings are being printed to the Puppet log.
These lines in file dsc_provider_helpers.rb may be relevant:
provider.commands :powershell =>
if File.exists?("#{ENV['SYSTEMROOT']}\\sysnative\\WindowsPowershell\\v1.0\\powershell.exe")
"#{ENV['SYSTEMROOT']}\\sysnative\\WindowsPowershell\\v1.0\\powershell.exe"
elsif File.exists?("#{ENV['SYSTEMROOT']}\\system32\\WindowsPowershell\\v1.0\\powershell.exe")
"#{ENV['SYSTEMROOT']}\\system32\\WindowsPowershell\\v1.0\\powershell.exe"
else
'powershell.exe'
end
Surely this defines where the Powershell executable is located, but gives no indication how it is called and how its return value is derived. Are stdout and stderr combined? Am I given the text output or just the error code? etc.
This is core Puppet logic. When a provider has a command, like
commands :powershell => some binary
That is hooked up as a function powershell(*args).
You can see it with other providers like Chocolatey:
commands :chocolatey => chocolatey_command
def self.chocolatey_command
if Puppet::Util::Platform.windows?
# must determine how to get to params in ruby
#default_location = $chocolatey::params::install_location || ENV['ALLUSERSPROFILE'] + '\chocolatey'
chocopath = ENV['ChocolateyInstall'] ||
('C:\Chocolatey' if File.directory?('C:\Chocolatey')) ||
('C:\ProgramData\chocolatey' if File.directory?('C:\ProgramData\chocolatey')) ||
"#{ENV['ALLUSERSPROFILE']}\chocolatey"
chocopath += '\bin\choco.exe'
else
chocopath = 'choco.exe'
end
chocopath
end
Then other locations can just call chocolatey like a function with args:
chocolatey(*args)

Create XML based on puppet templates with correct custom values?

In a database config setup I have 20 something XML files with the database connection information.
They have all exactly the same content except the connection information that is different. I have to use these XML files for five different environments, so instead of updating them manually I was hoping there was a good way to update their content with puppet, as I use this to set up the rest of the environment:
<config-property-setting name="DatabaseName"></config-property-setting>
<config-property-setting name="Password"></config-property-setting>
<config-property-setting name="UserName"></config-property-setting>
<config-property-setting name="ServerName"></config-property-setting>
<config-property-setting name="DriverType"></config-property-setting>
<config-property-setting name="MaxStatements"></config-property-setting>
<config-property-setting name="NetworkProtocol"></config-property-setting>
<config-property-setting name="PortNumber"></config-property-setting>
<config-property-setting name="LoginTimeout"></config-property-setting>
These are the values needed to be changed for each XML file.
Is there a good way to have a config file where the values for these files are entered and then, through templates, are pushed out as the correct configured and named XML files?
The most elegant way to do that in pure puppet is probably by using Hiera and an Erb template.
First of all, make sure that your hiera.yaml contains a hierarchy appropriate for your use case. For example, if the distinct environments are in distinct domains (but it might be more common to use the environment variable) :
:hierarchy:
- "%{module_name}/%{::fqdn}"
- "%{module_name}/%{::domain}"
- "%{module_name}/global"
- "global"
Then you can store the various configuration values just as #mudasobwa suggested, for example in hieradata/xmlmodule/domain1.internal :
:properties:
:DatabaseName : 'name'
:Password : 'pwd'
:UserName : 'user'
:ServerName : 'server'
:DriverType : 'mysql'
:MaxStatements : 30
:NetworkProtocol : 'udp'
:PortNumber : 1234
:LoginTimeout : 60
Then it's as simple as :
$properties = hiera('properties')
file { '/where/the/xml/fileS/goes.xml':
content => template('template.xml.erb');
}
And you get the values you need in the properties hash :
<config-property-setting name="DatabaseName">
<%=properties['DatabaseName']%>
</config-property-setting>
<config-property-setting name="Password">
<%=properties['Password']%>
</config-property-setting>
...
I would use pure ruby for that. First of all, create YAML config file:
:properties:
:DatabaseName : 'name'
:Password : 'pwd'
:UserName : 'user'
:ServerName : 'server'
:DriverType : 'mysql'
:MaxStatements : 30
:NetworkProtocol : 'udp'
:PortNumber : 1234
:LoginTimeout : 60
Then use this ruby code to produce what you need:
require 'yaml'
cfg = YAML.load_file 'cfg.yml'
xml = cfg[:properties].map { |k, v|
"<config-property-setting name='#{k}'>#{v}</config-property-setting>"
}.join("\n")
And, finally, inject these values into xml config templates (there must be a placeholder within xml configs to denote the place for proper values, e. g. if you want to insert them right after the root node <root>):
xml_config_files.each { |file|
File.write(file, File.read(f).gsub /<root>/m, "<root>\n#{xml}\n")
}
Hope it helps.

Resources