I am new to ruby and was scratching my head for something that seems pretty trivial. I was trying to search for particular string in file and replace that value with the new value . Here is what I have:
file = open('file.txt') { |f| f.grep(/release/) }
The contents of the file file.txt is as follows:
develop: 0.0.0
release: 0.0.4
test: 0.0.0
I am getting the output of the above line in array as follows :
["release: 0.0.4\n"]
How do I change just the value of this line in above file to the new version, something like this :
develop: 0.0.0
release: 0.0.5
test: 0.0.0
Based on above suggestions from #tadman, I was able to get it working as follows:
require 'yaml'
file = YAML.load_file('file')
file["release"] = "#{new_version}"
file.open('file', 'w') { |f| YAML.dump(file, f) }
Related
I am building an open source project using Ruby for testing HTTP services: https://github.com/Comcast/http-blackbox-test-tool
I want to be able to reference environment variables in my test-plan.yaml file. I could use ERB, however I don't want to support embedding any random Ruby code and ERB syntax is odd for non-rubyists, I just want to access environment variables using the commonly used Unix style ${ENV_VAR} syntax.
e.g.
order-lunch-app-health:
request:
url: ${ORDER_APP_URL}
headers:
content-type: 'application/text'
method: get
expectedResponse:
statusCode: 200
maxRetryCount: 5
All examples I have found for Ruby use ERB. Does anyone have a suggestion on the best way to deal with this? I an open to using another tool to preprocess the YAML and then send that to the Ruby application.
I believe something like this should work under most circumstances:
require 'yaml'
def load_yaml(file)
content = File.read file
content.gsub! /\${([^}]+)}/ do
ENV[$1]
end
YAML.load content
end
p load_yaml 'sample.yml'
As opposed to my original answer, this is both simpler and handles undefined ENV variables well.
Try with this YAML:
# sample.yml
path: ${PATH}
home: ${HOME}
error: ${NO_SUCH_VAR}
Original Answer (left here for reference)
There are several ways to do it. If you want to allow your users to use the ${VAR} syntax, then perhaps one way would be to first convert these variables to Ruby string substitution format %{VAR} and then evaluate all environment variables together.
Here is a rough proof of concept:
require 'yaml'
# Transform environments to a hash of { symbol: value }
env_hash = ENV.to_h.transform_keys(&:to_sym)
# Load the file and convert ${ANYTHING} to %{ANYTHING}
content = File.read 'sample.yml'
content.gsub! /\${([^}]+)}/, "%{\\1}"
# Use Ruby string substitution to replace %{VARS}
content %= env_hash
# Done
yaml = YAML.load content
p yaml
Use it with this sample.yml for instance:
# sample.yml
path: ${PATH}
home: ${HOME}
There are many ways this can be improved upon of course.
Preprocessing is easy, and I recommend you use a YAML loaderd/dumper
based solution, as the replacement might require quotes around the
replacement scalar. (E.g. you substitute the string true, if that
were not quoted, the resulting YAML would be read as a boolean).
Assuming your "source" is in input.yaml and your env. variable
ORDER_APP_URL set to https://some.site/and/url. And the following
script in expand.py:
import sys
import os
from pathlib import Path
import ruamel.yaml
def substenv(d, env):
if isinstance(d, dict):
for k, v in d.items():
if isinstance(v, str) and '${' in v:
d[k] = v.replace('${', '{').format(**env)
else:
substenv(v, env)
elif isinstance(d, list):
for idx, item in enumerate(d):
if isinstance(v, str) and '${' in v:
d[idx] = item.replace('${', '{').format(**env)
else:
substenv(item, env)
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(Path(sys.argv[1]))
substenv(data, os.environ)
yaml.dump(data, Path(sys.argv[2]))
You can then do:
python expand.py input.yaml output.yaml
which writes output.yaml:
order-lunch-app-health:
request:
url: https://some.site/and/url
headers:
content-type: 'application/text'
method: get
expectedResponse:
statusCode: 200
maxRetryCount: 5
Please note that the spurious quotes around 'application/text' are preserved, as would be any comments
in the original file.
Quotes around the substituted URL are not necessary, but the would have been added if they were.
The substenv routine recursively traverses the loaded data, and substitutes even if the substitution is in mid-scalar, and if there are more than substitution in one scalar. You can "tighten" the test:
if isinstance(v, str) and '${' in v:
if that would match too many strings loaded from YAML.
I need to remove the tags field from each of the methods in my OpenAPI spec.
The spec must be in YAML format, as converting to JSON causes issues later on when publishing.
I couldn't find a ready tool for that, and my programming skills are insufficient. I tried Python with ruamel.yaml, but could not achieve anything.
I'm open to any suggestions how to approach this - a repo with a ready tool somewhere, a hint on what to try in Python... I'm out of my own ideas.
Maybe a regex that catches all cases all instances of tags so I can do a search and replace with Python, replacing them with nothing? Empty lines don't seem to break the publishing engine.
Here's a sample YAML piece (I know this is not a proper spec, just want to show where in the YAML tags sits)
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint
get:
tags:
-
tag1
-
tag3
#rest of GET definition
post:
tags:
- tag2
/anotherEndpoint
post:
tags:
- tag1
I need to get rid of all tags arrays entirely (not just make them empty)
I am not sure why you couldn't achieve anything with Python + ruamel.yaml. Assuing your spec
is in a file input.yaml:
import sys
from pathlib import Path
import ruamel.yaml
in_file = Path('input.yaml')
out_file = Path('output.yaml')
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(in_file)
# if you only have the three instances of 'tags', you can hard-code them:
# del data['paths']['/endpoint']['get']['tags']
# del data['paths']['/endpoint']['post']['tags']
# del data['paths']['/anotherEndpoint']['post']['tags']
# generic recursive removal of any key names 'tags' in the datastructure:
def rm_tags(d):
if isinstance(d, dict):
if 'tags' in d:
del d['tags']
for k in d:
rm_tags(d[k])
elif isinstance(d, list):
for item in d:
rm_tags(item)
rm_tags(data)
yaml.dump(data, out_file)
which gives as output.yaml:
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint:
get: {}
post: {}
/anotherEndpoint:
post: {}
You can write back data to input.yaml if you need that.
Please note that normally the comment #rest of GET definition would be preserved, but
not now as it is associated during loading with the key before it and that key gets deleted.
I'm trying to send data to Elasticsearch using logagent but while there doesn't seem to be any error sending the data, the index isn't being created in ELK. I'm trying to find the index by creating a new index pattern via the Kibana GUI but the index does not seem to exist. This is my logagent.conf right now:
input:
# bro-start:
# module: command
# # store BRO logs in /tmp/bro in JSON format
# command: mkdir /tmp/bro; cd /tmp/bro; /usr/local/bro/bin/bro -i eth0 -e 'redef LogAscii::use_json=T;'
# sourceName: bro
# restart: 1
# read the BRO logs from the file system ...
files:
- '/usr/local/bro/logs/current/*.log'
parser:
json:
enabled: true
transform: !!js/function >
function (sourceName, parsed, config) {
var src = sourceName.split('/')
// generate Elasticsearch _type out of the log file sourceName
// e.g. "dns" from /tmp/bro/dns.log
if (src && src[src.length-1]) {
parsed._type = src[src.length-1].replace(/\.log/g,'')
}
// store log file path in each doc
parsed.logSource = sourceName
// convert Bro timestamps to JavaScript timestamps
if (parsed.ts) {
parsed['#timestamp'] = new Date(parsed.ts * 1000)
}
}
output:
stdout: false
elasticsearch:
module: elasticsearch
url: http://10.10.10.10:9200
index: bro_logs
Maybe I have to create the index mappings manually? I don't know.
Thank you for any advice or insight!
I found out that there actually was an error . I was trying to send some authentication via a field called "auth" but that doesn't exist. I can do url: https://USERNAME:PASSWORD#10.10.10.10:9200 though.
I have a YAML file like that:
---
name: dummy
version: 0.2.0
title: dummy
summary: dummy
Now I tried to get the version number:
config = YAML.load_file('Index.yml')
oldversion = config[0]['version']
Why do it get the following error:
NoMethodError: undefined method `[]' for nil:NilClass
Try with this:
config = YAML.load_file('Index.yml')
oldversion = config['version']
With config[0] you are indexing a sequence/array, and at the top level of your yaml file you have a mapping, not a sequence, so leave that out: oldversion = config['version'] or change your YAML file to:
---
- name: dummy
version: 0.2.0
title: dummy
summary: dummy
if you eventually want a list of such objects (with name, version, etc.) in your configuration.
I am using a custom Ruby function in Puppet to read a string of text from a file. I am than comparing whatever version is read against a list of known versions to determine which config file I should use for that particular server. The problem is that when I compare the read version to my list of known versions, none of them match.
I printed out the variable to the screen, and it looked fine. I then added a '-' to the beginning and the end and this time, the following was printed
-2.2#012-
Does anyone know what this is and how it could be removed?
Here is my process.
A script that handles the installation of an app
sudo echo "2.2" > /opt/version
My ruby function
if FileTest.exists?("/opt/version")
Facter.add("app_version") do
setcode do
version = File.open('/opt/version', &:readline)
version
end
end
end
My puppet manifest
if versioncmp( $app_version, '2.2') == 0 {
notice("===> Installing 2.2 Configs")
} elsif versioncmp ($app_version, '2.3') == 0 {
notice("===> Installing 2.3 Configs")
} else {
notice("===> No version match. Continuing on.")
}
}
File.readline includes the line termination (in your case, "\n"). chomp will get rid of the line termination:
version = File.open('/opt/version', &:readline).chomp
When debugging and you want to see what's really in a variable, use p instead of puts. p will escape unprintable characters so you can see them:
puts "2.2\n" # => 2.2
#
p "2.2\n" # => "2.2\n"