Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I have a YAML file:
DATA_SVC_GEN_CONF_ATTR:
-
-
-
name: validationCodeNum
value: 1
-
name: validationCodeSymbol
value: "val1"
-
name: sql
value: Test
-
name: sqlByGridArea
value: Test1
I want to replace the value of the key sql from "Test" to "Select * from table".
What I am trying is:
asset.getAttributes.getList.each{|a|
if a.getName == "sqlByGridArea"
a.setValue "Select * from table"
(attrib_defs).map!{|a| a.map{|k,v| "#{k}=#{v}"}.join(',')}
end
asset.with_attributes attrib_defs
Here's how to test whether your YAML is valid, and how I generate YAML for use in files or elsewhere:
require 'yaml'
str = <<EOT
DATA_SVC_GEN_CONF_ATTR:
-
-
-
name: validationCodeNum
value: 1
-
name: validationCodeSymbol
value: "val1"
-
name: sql
value: Test
-
name: sqlByGridArea
value: Test1
EOT
data = YAML.load(str)
# => {"DATA_SVC_GEN_CONF_ATTR"=>
# [[[{"name"=>"validationCodeNum", "value"=>1},
# {"name"=>"validationCodeSymbol", "value"=>"val1"},
# {"name"=>"sql", "value"=>"Test"},
# {"name"=>"sqlByGridArea", "value"=>"Test1"}]]]}
YAML would complain if it couldn't parse the data. If it could and the data wasn't what you expect the output help you diagnose the problem.
When I want to create a YAML file, especially one that's complex, I start with the actual data structure and let YAML generate the serialized version of it:
data = {
'DATA_SVC_GEN_CONF_ATTR' =>
[
[
[
{
'name' => 'validationCodeNum',
'value' => 1
},
{
'name' => 'validationCodeSymbol',
'value' => 'val1'
},
{
'name' => 'sql',
'value' => 'Test'
},
{
'name' => 'sqlByGridArea',
'value' => 'Test1'
}
]
]
]
}
puts data.to_yaml
# >> ---
# >> DATA_SVC_GEN_CONF_ATTR:
# >> - - - name: validationCodeNum
# >> value: 1
# >> - name: validationCodeSymbol
# >> value: val1
# >> - name: sql
# >> value: Test
# >> - name: sqlByGridArea
# >> value: Test1
Moving on... To change elements in a YAML file you can treat the file as text, read it line-by-line, read it by slurping it, or have YAML open it and change the resulting object then have YAML rewrite it.
Slurping files is OK if you're sure the data will fit in memory and you can easily find the section you want to change. Reading it line-by-line often makes it easier to find specific text because you're dealing with individual lines, not the whole file.
Using YAML is the easiest I think as it reduces the change of a badly written regex doing the wrong thing.
data = <<EOT
---
DATA_SVC_GEN_CONF_ATTR:
- - - name: validationCodeNum
value: 1
- name: validationCodeSymbol
value: val1
- name: sql
value: Test
- name: sqlByGridArea
value: Test1
EOT
Pretending that data was the content of a YAML file on disk, it'd be loaded and parsed using YAML.load_file. In this example I'm loading as a string so I have to use load:
object = YAML.load(data)
The data is now parsed so it can be manipulated easily:
object['DATA_SVC_GEN_CONF_ATTR'].first.first[2]['name'] = "Select * from table"
Then it can be written back out:
puts object.to_yaml # => nil
# >> ---
# >> DATA_SVC_GEN_CONF_ATTR:
# >> - - - name: validationCodeNum
# >> value: 1
# >> - name: validationCodeSymbol
# >> value: val1
# >> - name: Select * from table
# >> value: Test
# >> - name: sqlByGridArea
# >> value: Test1
If this is a mission-critical (production) file, you'd want to do the appropriate rename, write, delete steps to protect the data in-case of errors, but that's the gist of it.
Your data structure is very questionable though. You're using a nested array-of-array-of-array just to hold your hashes, which is not very common or easily understood. Instead it should be reduced to a single array:
data = <<EOT
---
DATA_SVC_GEN_CONF_ATTR:
- name: validationCodeNum
value: 1
- name: validationCodeSymbol
value: val1
- name: sql
value: Test
- name: sqlByGridArea
value: Test1
EOT
object = YAML.load(data)
# => {"DATA_SVC_GEN_CONF_ATTR"=>
# [{"name"=>"validationCodeNum", "value"=>1},
# {"name"=>"validationCodeSymbol", "value"=>"val1"},
# {"name"=>"sql", "value"=>"Test"},
# {"name"=>"sqlByGridArea", "value"=>"Test1"}]}
object['DATA_SVC_GEN_CONF_ATTR'][2]['name'] = "Select * from table"
Related
I have a array in ruby named array, I aded value into yaml file, but after in file.yml, it remove me %YAML 1.1, so I won't
yaml_string = File.read "file.yaml"
data = YAML.load yaml_string
array.each do |value|
data["title"] <<"- "+value+"\n"
end
output = YAML.dump data
File.write("file.yaml", output)
before execution, the header is present, but after execution it remove it (%YAML 1.1) and all lines comment with #, so I won't
I think something like this is what you're trying to do.
I'm assuming your yaml array of titles matches your array object.
Otherwise you could just use something like Enum#with_index if you just want to map the number of the yaml array to the text.
require 'psych'
filename = "sample_yaml.yml"
array = [0, 1, 2, 3]
if File.exists?(filename)
puts "File exists. :) Parsing the yaml file."
yaml = Psych.load_file(filename)
array.each do |value|
yaml[value]["title"] << " - #{value}" # find the title that matches the index number of array
end
else
raise ArgumentError, "bad file name"
end
puts "Outputting to reformatted yaml file"
File.open("reformatted_file.yaml", 'wb') {|f| f.write "%YAML 1.1\n" + Psych.dump(yaml)}
assuming yaml file like such
---
- title: zero
- title: one
- title: two
- title: three
Outputs
---
- title: zero - 0
- title: one - 1
- title: two - 2
- title: three - 3
I'm facing a problem that I couldn't find a working solution yet.
I have my YAML config file for the environment, let's call it development.yml.
This file is used to create the hash that should be updated:
data = YAML.load_file(File.join(Rails.root, 'config', 'environments', 'development.yml'))
What I'm trying to accomplish is something along these lines. Let's suppose we have an element of the sort
data['server']['test']['user']
data['server']['test']['password']
What I want to have is:
data['server']['test']['user'] = #{Server.Test.User}
data['server']['test']['password'] = #{Server.Test.Password}
The idea is to create a placeholder for each value that is the key mapping for that value dynamically, going until the last level of the hash and replacing the value with the mapping to this value, concatenating the keys.
Sorry, it doesn't solve my problem. The location data['server']['test']['user'] will be built dynamically, via a loop that will go through a nested Hash. The only way I found to do it was to append to the string the key for the current iteration of the Hash. At the end, I have a string like "data['server']['test']['name']", which I was thinking on converting to a variable data['server']['test']['name'] and then assigning to this variable the value #{Server.Test.Name}. Reading my question I'm not sure if this is clear, I hope this helps to clarify it.
Input sample:
api: 'active'
server:
test:
user: 'test'
password: 'passwordfortest'
prod:
user: 'nottest'
password: 'morecomplicatedthantest'
In this case, the final result should be to update this yml file in this way:
api: #{Api}
server:
test:
user: #{Server.Test.User}
password: #{Server.Test.Password}
prod:
user: #{Server.Prod.User}
password: #{Server.Prod.Password}
It sounds silly, but I couldn't figure out a way to do it.
I am posting another answer now since I realize what the question is all about.
Use Iteraptor gem:
require 'iteraptor'
require 'yaml'
# or load from file
yaml = <<-YAML.squish
api: 'active'
server:
test:
user: 'test'
password: 'passwordfortest'
prod:
user: 'nottest'
password: 'morecomplicatedthantest'
YAML
mapped =
yaml.iteraptor.map(full_parent: true) do |parent, (k, _)|
v = parent.map(&:capitalize).join('.')
[k, "\#{#{v}}"]
end
puts YAML.dump(mapped)
#⇒ ---
# api: "#{Api}"
# server:
# test:
# user: "#{Server.Test.User}"
# password: "#{Server.Test.Password}"
# prod:
# user: "#{Server.Prod.User}"
# password: "#{Server.Prod.Password}"
puts YAML.dump(mapped).delete('"')
#⇒ ---
# api: #{Api}
# server:
# test:
# user: #{Server.Test.User}
# password: #{Server.Test.Password}
# prod:
# user: #{Server.Prod.User}
# password: #{Server.Prod.Password}
Use String#%:
input = %|
data['server']['host']['name'] = %{server_host}
data['server']['host']['user'] = %{server_host_user}
data['server']['host']['password'] = %{server_host_password}
|
puts (
input % {server_host: "Foo",
server_host_user: "Bar",
server_host_password: "Baz"})
#⇒ data['server']['host']['name'] = Foo
# data['server']['host']['user'] = Bar
# data['server']['host']['password'] = Baz
You can not add key-value pair to a string.
data['server']['host'] # => which results in a string
Option 1:
You can either save Server.Host as host name in the hash
data['server']['host']['name'] = "#{Server.Host}"
data['server']['host']['user'] = "#{Server.Host.User}"
data['server']['host']['password'] = "#{Server.Host.Password}"
Option 2:
You can construct the hash in a single step with Host as key.
data['server']['host'] = { "#{Server.Host}" => {
'user' => "#{Server.Host.User}",
'password' => "#{Server.Host.Password}"
}
}
I want to scrape the list of offers of a given product from amazon.com with the quantity in stoke for each offer.
To find this last information (quantity) I need to add that offer to cart, than edit the cart with the quantity 999. and than get the quantity from the next page.
take for example this product (http://www.amazon.com/gp/offer-listing/B00DW58ENU/ref=olp_tab_all&startIndex=1) the button Add to Cart is a form with a single submit. I can find the form with the code
offer_form = agent.page.forms_with(action: /item-dispatch/)[0]
##<Mechanize::Form
# {name nil}
# {method "POST"}
#{action "/gp/item-dispatch/ref=olp_atc_new_1/181-7026511-7466349"}
# {fields
# [hidden:0x1717018 type: hidden name: session-id value: 181-7026511-7466349]
# [hidden:0x1716a8c type: hidden name: qid value: ]
# [hidden:0x1716514 type: hidden name: sr value: ]
# [hidden:0x1715eac type: hidden name: signInToHUC value: 0]
# [hidden:0x17155d8 type: hidden name: metric-asin.B00NHQFA1I value: 1]
# [hidden:0x1714e80 type: hidden name: registryItemID.1 value: ]
# [hidden:0x1714a0c type: hidden name: registryID.1 value: ]
# [hidden:0x1714598 type: hidden name: quantity.1 value: 1]
# [hidden:0x1714138 type: hidden name: offeringID.1 value: RVm%2FgzxznRorTyxf%2F8fiGjVFjfScvgO1JJBElusLb7hLttElaCwmvhKe7NSGkE1LBMGmkM3oodMhTTBnKT%2FCP%2FnFeT7SBoLZdnRfmVwRFa0N7AHRTVnphw%3D%3D]
# [hidden:0x1707938 type: hidden name: isAddon value: 0]}
# {radiobuttons}
# {checkboxes}
# {file_uploads}
# {buttons
# [submit:0x17072e4 type: submit name: submit.addToCart value: Add to #cart]}>
page = offer_form.submit
##<Mechanize::Page
# {url #<URI::HTTP http://www.amazon.com/gp/item-dispatch/ref=olp_atc_new_1>}
# {meta_refresh}
# {title nil}
# {iframes}
# {frames}
# {links}
# {forms}>
I am wondering why I got this empty page as a result.
I though that maybe this is because the action is different than the real one found when I open the page using the browser (Chrome of Firefox).
but even if I change the offer_form.action to be like that found on the browser. It does not change the result, and I still get an empty page.
I have a data file about 1000000 lines, 300+ Mb. I want to convert it to a JSON file.
JSON.generate and Hash.merge can help generate JSON, but I need to wait for the program to generate one whole hash. That costs too much time.
I want to write JSON to file incrementally.
Here is my code:
require 'yajl/json_gem'
my_hash = {}
fd1 = File.open("foo.json", "w")
fd2 = File.open("foo.log")
fd2.each_line do | line |
l = fd2.lineno
remote_addr = line.split(" ")[0]
time_local = line.split("]")[0].split("[")[1]
item = {l => {:remote_addr => remote_addr, :time_local => time_local}}
# {
# 1: {
# remote_addr: "1.2.3.4",
# time_local: ""
# },
# 2: {
# ...
# },
# ...
# }
my_hash.merge!(item)
end
fd2.close
fd1.puts JSON.generate(my_hash)
fd1.close
Here is my data:
// access.log
1.2.3.4 - - [02/Apr/2014:03:23:06 +0800] "GET /index" 200 1 "http://foo" "Mozilla/5.0" "-"
Any ideas? Thanks in advance.
--
edit:
Actually, I'd better convert raw data like this:
[
{
id: "1",
remote_addr: "1.2.3.4",
time_local: "02/Apr/2014:03:23:06 +0800"
},
{
...
},
{
...
}
]
This is basic code to generate a valid JSON output based on the sample line:
require 'json'
lines = [
'1.2.3.4 - - [02/Apr/2014:03:23:06 +0800] "GET /index" 200 1 "http://foo" "Mozilla/5.0" "-"',
'1.2.3.5 - - [03/Apr/2014:03:23:06 +0800] "GET /index" 200 1 "http://foo" "Mozilla/5.0" "-"'
]
lines.each_with_index do |line, l|
puts '{' if l == 0
remote_addr, time_local = /^(\S+) .+ \[(.+)\]/.match(line).captures
print '"%d":{"remote_addr":"%s","time_local":"%s"}' % [l + 1, remote_addr, time_local]
puts ',' if l + 1 < lines.size
end
puts "\n}"
# >> {
# >> "1":{"remote_addr":"1.2.3.4","time_local":"02/Apr/2014:03:23:06 +0800"},
# >> "2":{"remote_addr":"1.2.3.5","time_local":"03/Apr/2014:03:23:06 +0800"}
# >> }
You'll need to convert the code to read and write files, or maybe read a single file and redirect the output to capture it. You'll also need to figure out how to determine the number of lines in a file to make it possible to find the places to output the commas. It's easy, and I know examples of doing that exist here on Stack Overflow.
In your original code you need to use the block form for open, rather than assigning to a variable and then explicitly closing the files; Using the block form of open is the Ruby-way. Also, use File.foreach to read through a file line-by-line.
#theTinMan posted a good answer, I will just add that a little more generic solution for a list of JSON object is to construct the list yourself, but let JSON generate each item:
fd1.puts '{'
fd1 = File.open("foo.json", "w")
fd2 = File.open("foo.log")
first_line = true
fd2.each_line do | line |
fd1.puts(',') unless first_line
first_line = false
l = fd2.lineno
remote_addr = line.split(" ")[0]
time_local = line.split("]")[0].split("[")[1]
fd1.print "\"#{l}\": #{JSON.generate(:remote_addr => remote_addr, :time_local => time_local)}"
end
fd1.puts "\n}"
The to_yaml method produces nice YAML output, but I would like to include comment lines before some of the elements. Is there a way to do so?
For example, I would like to produce:
# hostname or IP address of client
client: host4.example.com
# hostname or IP address of server
server: 192.168.222.222
From something similar to:
{
:client => 'host4.example.com',
:server => '192.168.222.222',
}.to_yaml
... but am not sure if the YAML module even has a way to accomplish.
UPDATE: I ended up not using the solution which used regexes to insert the comments, since it required the separation of the data from the comments. The easiest and most understandable solution for me is:
require 'yaml'
source = <<SOURCE
# hostname or IP address of client
client: host4.example.com
# hostname or IP address of server
server: 192.168.222.222
SOURCE
conf = YAML::load(source)
puts source
The benefit to me is that nothing is repeated (for example, 'client:' is only specified once), the data and comments are together, the source can be outputted as YAML, and the data structure (available in conf) is available for use.
You can do a string replace on all the insertions:
require 'yaml'
source = {
:client => 'host4.example.com',
:server => '192.168.222.222',
}.to_yaml
substitution_list = {
/:client:/ => "# hostname or IP address of client\n:client:",
/:server:/ => "# hostname or IP address of server\n:server:"
}
substitution_list.each do |pattern, replacement|
source.gsub!(pattern, replacement)
end
puts source
output:
---
# hostname or IP address of client
:client: host4.example.com
# hostname or IP address of server
:server: 192.168.222.222
Something like this:
my_hash = {a: 444}
y=YAML::Stream.new()
y.add(my_hash)
y.emit("# this is a comment")
Of course, you will need to walk the input hash yourself and either add() or emit() as needed.
You could look at the source of the to_yaml method for a quick start.
This isn't perfect (no mid-Array support, for example), but it works for my needs.
def commentify_yaml(db)
ret = []
db.to_yaml(line_width: -1).each_line do |l|
if l.match(/^\s*:c\d+:/)
l = l.sub(/:c(\d+)?:/, '#').
sub(/(^\s*# )["']/, '\1').
sub(/["']\s*$/, '').
gsub(/''(\S+?)''/, "'\\1'").
gsub(/(\S)''/, "\\1'")
end
ret << l.chomp
end
ret * "\n"
end
Example usage.
commentify_yaml(
{
c1: 'Comment line 1',
c2: 'Comment line 2',
'hash_1' => {
c1: 'Foo',
c2: 'Bar',
'key_1' => "Hello!",
},
'baz' => qux,
c3: 'Keep up-numbering the comments in the same hash',
'array_1' => [
1,
2,
3
]
}
)
==>
# Comment line 1
# Comment line 2
hash_1:
# Foo
# Bar
key_1: "Hello!"
baz: "Value of qux"
# Keep up-numbering the comments in the same hash
array_1:
- 1
- 2
- 3
(Note: Syck does not indent arrays they way it arguably should.)