I have some code which returns the error and i cannot understand why. I am abit of a newbie with Ruby but feel I am getting there:
line 27: NoMethodError "each" for NilClass
The code I am using is below:
require 'rubygems'
require 'nokogiri'
require 'sqlite3'
FIELDS = [['cityselect', 'VARCHAR'],['match', 'VARCHAR'], ['num_phone', 'NUMERIC'], ['name', 'VARCHAR'],['address', 'VARCHAR'] ]
DIV_ID = "#dgrSearch"
FILE_O = File.open('hold-data/directory-tel.txt', 'w')
FILE_O.puts( FIELDS.map{|f| f[0]}.join("\t") )
DB_NAME = "hold-data/directory-tel.sqlite"
File.delete(DB_NAME) if File.exists?DB_NAME
DATAB = SQLite3:Database.new( DB_NAME )
TABLE = "records_telephone"
DB_INSERT_STATEMENT = "INSERT into #{TABLE} values
(#{FIELDS.map{'?'}.join(',')})"
DATAB.execute "CREATE TABLE #{TABLE}(#{FIELDS.map{|f| "`#{f[0]}` #{f[1]}"}.join(', ')});"
FIELDS.each do |fn|
DATAB.execute "CREATE INDEX #{fn[2]} ON #{TABLE}(#{fn[0]})" unless fn[2].nil?
end
Dir.glob("hold-data/pages/*.html").reject{|f| f =~ /All match/}.each do |fname|
meta_stuff = File.basename(fname, '.html').split('--')
page = Nokogiri::HTML(open(fname))
page.css("#{DIV_ID} tr")[1..-2].each do |tr| # this is line #27
data_tds = tr.css('td').map{ |td|
td.text.gsub(/[$,](?=\d)/, '').gsub(/\302\240|\s/, ' ').strip
}
row_data = meta_stuff + data_tds
FILE_O.puts( data_row.join("\t"))
DATAB.execute(DB_INSERT_STATEMENT, row_data)
end
end
FILE_O.close
Can anybody see what I have done wrong?
It would be useful next time if you tell us which line the error occurs on.
From the looks of it theres only 1 line here where the expression your calling each on might return nil:
page.css("#{DIV_ID} tr")[1..-2].each do
Remember the_array[1..-2] will return nil if the_array is empty.
So page.css("#{DIV_ID} tr") might return an empty array hence the error.
If this behavior is not expiected I would consider invesigating why this is happening OR maybe check whether the array is empty before calling [1..-2] on it (see below). Just a suggestion.
if page.css("#{DIV_ID} tr").empty?
#empty array
else
#not empty
end
Related
I am getting an error when executing my test.
Failure/Error: expect(industry_sic_code).to include page.sic_code
TypeError:
no implicit conversion of Array into String
# ./spec/os/bal/company/company_filter_clean_harbors_industries_stub.rb:62:in `block (2 levels) in <top (required)>'
The Method:
def sic_code
subtables = #b.table(:class => 'industry-codes').tables(:class => 'industry-code-table')
subtables.each do |subtable|
if subtable.tbody.h4.text == "US SIC 1987:"
subtable.tr.next_siblings.each do |tr|
codes = tr.cell
puts codes.text.to_s
end
end
end
end
The Test:
it 'Given I search for a random Clean Harbors Industry' do
#Pick a random clean industry from the file
data = CSV.foreach(file_path, headers: true).map{ |row| row.to_h }
random = data.sample
random_industry = random["Class"]
industry_sic_code = random["SIC Code"]
end
it 'Then the result has the expected SIC code' do
page = DetailPage.new(#b)
page.view
expect(industry_sic_code).to include page.sic_code
end
I have tried to implicitly change each variable to a string but it still complain about the array issue.
When I include some puts statments, I get some really wonky responses. The method itself returns the expected result.
When I used the method in the test I end up with the code gibberish below.
here are the sic codes from the method
5511
Here are the codes from the test
#<Watir::Table:0x00007fa3cb23f020>
#<Watir::Table:0x00007fa3cb23ee40>
#<Watir::Table:0x00007fa3cb23ec88>
#<Watir::Table:0x00007fa3cb23ead0>
#<Watir::Table:0x00007fa3cb23e918>
#<Watir::Table:0x00007fa3cb23e738>
#<Watir::Table:0x00007fa3cb23e580>
Your sic_code method returns subtables array, that's why you have this error. It doesn't matter that the method puts something, every method in ruby implicitly returns result of its last line, in your case it is subtables.each do ... end, so you have an array.
You need to explicitly return needed value. Not sure if I correctly understood what are you doing in your code, but try something like this:
def sic_code
subtables = #b.table(:class => 'industry-codes').tables(:class => 'industry-code-table')
result = [] # you need to collect result somewhere to return it later
subtables.each do |subtable|
if subtable.tbody.h4.text == "US SIC 1987:"
subtable.tr.next_siblings.each do |tr|
codes = tr.cell
result << codes.text.to_s
end
end
end
result.join(', ')
end
I have defined a custom function currently based on the very simple example here: https://docs.puppet.com/guides/custom_functions.html
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
File.open(filename, 'a') {|fd| fd.puts hash_to_be_transformed }
end
end
This kinda works. I can call it like this:
$my_hash = { key => "value1" , key2 => "value2" }
notify{ "new hash!! $my_hash" :}
transform_service_hash('/var/tmp/blah',$my_hash)
and the file displays:
mgt21 ~ # cat /var/tmp/blah
keyvalue1key2value2
But, if I try to access elements of the hash, nothing changes:
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
element1 = hash_to_be_transformed["key"]
File.open(filename, 'a') {|fd| fd.puts element1 }
end
end
The above block outputs the exact same data to /var/tmp/blah.
And, interestingly, if I remove the filename pass and define it statically in the module:
$my_hash = { key => "value1" , key2 => "value2" }
notify{ "new hash!! $my_hash. element1 is: $my_hash.key" :}
transform_service_hash($my_hash)
and
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
hash_to_be_transformed = args[0]
element1 = hash_to_be_transformed["key"]
File.open('/var/tmp/blah2', 'a') {|fd| fd.puts element1 }
end
end
I get the following error: "Error 400 on SERVER: can't convert Hash into String" with a line reference pointing to "transform_service_hash($my_hash)"
I am new to both puppet and ruby...so I'm unsure I am not passing the element properly, if I am not receiving it properly, or if it something that puppet cannot handle. Please note that I am using version 3.8 of puppet and 1.8.7 of ruby.
Thanks for any help. I've been banging my head against this, and google hasn't been forthcoming yet.
---Edit to clarify my goals (I also edited my code a bit for specificity): I am attempting to pass a hash into a custom ruby function within puppet. The "test" hash has two elements: one string and one array. It is defined as such:
$my_hash = { key => "value1" , key2 => ['array_value1', 'array_value2'] }
$my_display_element=$my_hash["key2"][0]
notify{ "new hash!! $my_hash. the first value of the array stored in element2 is: $my_display_element" :}
transform_service_hash('/var/tmp/blah',$my_hash)
The function appears like so:
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
element1 = args[1]["key"]
element2 = args[1]["key2"][0]
#element1 = hash_to_be_transformed["key"]
#element2 = hash_to_be_transformed["key2"][0]
File.open(filename, 'a') {|fd| fd.puts "hash_to_be_transformed: #{hash_to_be_transformed}\n" }
File.open(filename, 'a') {|fd| fd.puts "element1: #{element1}\n" }
File.open(filename, 'a') {|fd| fd.puts "element2: #{element2}\n" }
end
end
For now, I just want to be able to see that I am able to access elements within the passed hash like a hash. So I'd love for the output file to look like:
hash_to_be_transformed: keyvalue1key2array_value1array_value2
element1: value1
element2: array_value1
However, in the output file, I see:
mgt21 ~ # cat /var/tmp/blah
keyvalue1key2array_value1array_value2
Clearly, something is off here as my text is not being added and the full hash is just printed out just once and seemingly in string form.
I believe that this may be related to the error that I get when I don't pass in a file name (see above). I think that my hash is getting interpreted (or passed) as a string and, as such, I am unable to access the elements. Unfortunately, I still have been unable to verify this or figure out why it might be happening.
---Edit2 based on Matt's answer below.
I decided to simplify my code to isolate this "can't convert Hash into String error". I also made his suggested changes to remove the ambiguity from my key declarations.
$my_hash = { 'key' => "value1" , 'key2' => ['array_value1', 'array_value2'] }
$my_display_element=$my_hash["key2"][0]
notify{ "new hash!! $my_hash. the first value of the array stored in element2 is: $my_display_element" :}
transform_service_hash($my_hash)
and
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
hash_to_be_transformed = args[0]
element1 = args[0]['key']
element2 = args[0]['key2'][0]
File.open('/var/tmp/blah', 'a') {|fd| fd.puts "hash_to_be_transformed: #{hash_to_be_transformed}\n" }
File.open('/var/tmp/blah', 'a') {|fd| fd.puts "element1: #{element1}\n" }
File.open('/var/tmp/blah', 'a') {|fd| fd.puts "element2: #{element2}\n" }
end
end
But, I still end up with the same "Hash to String error". It is worth noting that I also tried simplifying my hash to:
$my_hash = { 'key' => "value1" , 'key2' => "value2" }
and I still get the "Hash to String error".
I quickly took your custom parser function and converted it into pure ruby like the following:
hash = { 'key' => 'value1', 'key2' => %w(array_value1 array_value2) }
def newfunction(filename, a_hash)
element1 = a_hash['key']
element2 = a_hash['key2'][0]
File.open(filename, 'a') do |fd|
fd.puts "hash_to_be_transformed: #{a_hash}"
fd.puts "element1: #{element1}"
fd.puts "element2: #{element2}"
end
end
newfunction('foo.txt', hash)
This results in the output text file like the following:
hash_to_be_transformed: {"key"=>"value1", "key2"=>["array_value1", "array_value2"]}
element1: value1
element2: array_value1
This seems to confirm my initial suspicion about what is going wrong here. Your hash in Puppet of:
$my_hash = { key => "value1" , key2 => ['array_value1', 'array_value2'] }
has keys of implicit/ambiguous types. In the ruby code I used to test, I explicitly established them as strings. This also correlates strongly with these lines in your code failing:
element1 = args[1]["key"]
element2 = args[1]["key2"][0]
and your error message of:
Error 400 on SERVER: can't convert Hash into String
because you are specifying in your ruby code that you expect the keys to be string. Changing your hash in Puppet to:
$my_hash = { 'key' => "value1" , 'key2' => "value2" }
should fix this.
On an unrelated note, I recommend the use of linters to help you learn these languages. Puppet-Lint, Rubocop, and Reek will all help point out suboptimal and messy parts of your code to help you learn the new languages.
On a related note, you may want to put something like this at the top of your custom parser function:
raise(Puppet::ParseError, 'newfunction expects two arguments') if args.length != 2
After much gnashing of teeth (and some very helpful pointers from #MattSchuchard), I realized that none of the changes to my function were going into effect. One needs to restart the puppetmaster service after each change to a custom function: docs.puppet.com/guides/custom_functions.html (appropriately under "Gotchas").
Once I started restarting this service after each change to the function, my hash was able to be parsed properly:
from the .pp file:
$filename = "/var/tmp/test"
$my_hash = { 'key' => "value1" , 'key2' => ["M\'lady\n*doffs cap*", 'array_value2'] }
transform_service_hash($filename, $my_hash)
from the ruby file:
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
array_val = hash_to_be_transformed['key2'][0]
File.open(filename, 'a') {|fd| fd.puts "#{array_val}\n" }
end
end
and output:
mgt21 tmp # cat test
M'lady
*doffs cap*
Sorry, but I didn't find the documentation enlightening at all. Basically, I am trying to iterate through a where some options are not valid. The ones I want have 'class="active"'. Can I do that with mechanize? Here's what I have so far:
class Scraper
def init
mech = Mechanize.new
page = mech.get('url')
#Now go through the <select> to get product numbers for the different flavors
form = page.form_with(:id => 'twister')
select = form.field_with(:name => 'dropdown_selected_flavor_name')
select.options.each do |o|
if (o.text != "")
value = o
end
productNumber = trim_pn(value.to_s[2..12])
puts productNumber
end
end
#Checks validity of product number and removes excess characters if necessary
def trim_pn(pn)
if (pn[0] == ",")
pn = pn[1..-1]
end
return pn
end
end
p = Scraper.new
p.init
All that does is grabs the product number and removes some extra info that I don't want. I thought replacing the .each do with this:
select.options_with(:class => 'active').each do |o|
if (o.text != "")
value = o
end
end
But that throws "undefined method 'dom_class' for Mechanize:Form:Option blah blah." Is there are different way I should be approaching this?
I would like to use the Google bigquery gem (https://rubygems.org/gems/bigquery) to create an Array of table names. So far, this is what I have written:
require 'json'
bqRepsonse = bq.tables('myDataSet')
bqRepsonseCleaned = bqRepsonse.to_s.gsub("=>", ":")
data = JSON.parse(bqRepsonseCleaned)
tableListing = []
data["tableID"]["type"].each do |item|
case item["type"]
when 'TABLE'
bqTableList << item["tableId"]
else
end
end
If I print bqResponse, I get this result:
[{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableA",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableA"}, "type"=>"TABLE"},
{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableB",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableB"}, "type"=>"TABLE"},
{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableC",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableC"}, "type"=>"TABLE"},
{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableD",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableD"}, "type"=>"TABLE"}]
And running the code throws and error
`[]': no implicit conversion of String into Integer (TypeError)
Not sure where to correct this. My desired outcome is:
tableListing = ["TableA","TableB","TableC","TableD"]
Thanks in advance for your advice.
Try this:
require 'json'
string = '[{"kind": "bigquery#table", "id": "curious-idea-532:dataset_test_4.TableA", "tableReference" : {"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableA"}, "type":"TABLE"}, {"kind":"bigquery#table", "id":"curious-idea-532:dataset_test_4.TableB", "tableReference":{"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableB"}, "type":"TABLE"}, {"kind":"bigquery#table", "id":"curious-idea-532:dataset_test_4.TableC", "tableReference":{"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableC"}, "type":"TABLE"}, {"kind":"bigquery#table", "id":"curious-idea-532:dataset_test_4.TableD", "tableReference":{"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableD"}, "type":"TABLE"}]'
data = JSON.parse(string)
tableListing = []
# Here we are iterating over the data instead of its child element
data.each do |item|
case item["type"]
when 'TABLE'
tableListing << item["tableReference"]["tableId"]
else
end
end
puts tableListing
my environment: ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]
The thing is, I can make ruby return the parameters sent over GET. but when i'm trying to use them as arguements to my methods in if/else, ruby wont return anything and I end up with a blank page.
ph and pm return correctly:
http://127.0.0.1/cgi-bin/test.rb?hostname=node00.abit.dk&macadd=23:14:41:51:63
returns:
node00.abit.dk 23:14:41:51:63
Connection to the database (MySQL) works fine
When I test the method newHostName it outputs correctly:
puts newHostName
returns (which is correct)
node25.abit.dk
the code:
#!/usr/bin/ruby
require 'cgi'
require 'sequel'
require 'socket'
require 'timeout'
DB = Sequel.connect(:adapter=>'mysql', :host=>'localhost', :database=>'nodes', :user=>'nodeuser', :password=>'...')
#cgi-part to work
#takes 2 parameters:
#hostname & macadd
cgi = CGI.new
puts cgi.header
p = cgi.params
ph = p['hostname']
pm = p['macadd']
def nodeLookup(hostnameargv)
hostname = DB[:basenode]
h = hostname[:hostname => hostnameargv]
h1 = h[:hostname]
h2 = h[:macadd]
ary = [h1, h2]
return ary
end
def lastHostName()
#TODO: replace with correct sequel-code and NOT raw SQL
DB.fetch("SELECT hostname FROM basenode ORDER BY id DESC LIMIT 1") do |row|
return row[:hostname]
end
end
def newHostName()
org = lastHostName
#Need this 'hack' to make ruby grep for the number
#nodename e.g 'node01.abit.dk'
var1 = org[4]
var2 = org[5]
var3 = var1 + var2
sum = var3.to_i + 1
#puts sum
sum = "node" + sum.to_s + ".abit.dk"
return sum
end
def insertNewNode(newhost, newmac)
newnode = DB[:basenode]
newnode.insert(:hostname => newhost, :macadd => newmac)
return "#{newnode.count}"
end
#puts ph
#puts pm
#puts newHostName
cgi.out() do
cgi.html do
begin
if ph == "node00.abit.dk"
puts newHostName
else
puts nodeLookup(ph)
end
end
end
end
I feel like im missing something here. Any help is very much appreciated!
//M00kaw
What about modify last lines of your code as followed? CGI HTML generation methods take a block and yield the return value of the block as their content. So you should make newHostName or nodeLookup(ph) as the return value of the block passed to cgi.html(), rather than puts sth, which prints the content to your terminal and return nil. That's why cgi.html() got an empty string (nil.to_s).
#puts newHostName
cgi.out() do
cgi.html do
if ph == "node00.abit.dk"
newHostName
else
nodeLookup(ph)
end
end
end
p.s. It's conventional to indent your ruby code with 2 spaces :-)