I would like to use the Google bigquery gem (https://rubygems.org/gems/bigquery) to create an Array of table names. So far, this is what I have written:
require 'json'
bqRepsonse = bq.tables('myDataSet')
bqRepsonseCleaned = bqRepsonse.to_s.gsub("=>", ":")
data = JSON.parse(bqRepsonseCleaned)
tableListing = []
data["tableID"]["type"].each do |item|
case item["type"]
when 'TABLE'
bqTableList << item["tableId"]
else
end
end
If I print bqResponse, I get this result:
[{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableA",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableA"}, "type"=>"TABLE"},
{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableB",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableB"}, "type"=>"TABLE"},
{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableC",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableC"}, "type"=>"TABLE"},
{"kind"=>"bigquery#table",
"id"=>"curious-idea-532:dataset_test_4.TableD",
"tableReference"=>{"projectId"=>"curious-idea-532",
"datasetId"=>"dataset_test_4", "tableId"=>"TableD"}, "type"=>"TABLE"}]
And running the code throws and error
`[]': no implicit conversion of String into Integer (TypeError)
Not sure where to correct this. My desired outcome is:
tableListing = ["TableA","TableB","TableC","TableD"]
Thanks in advance for your advice.
Try this:
require 'json'
string = '[{"kind": "bigquery#table", "id": "curious-idea-532:dataset_test_4.TableA", "tableReference" : {"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableA"}, "type":"TABLE"}, {"kind":"bigquery#table", "id":"curious-idea-532:dataset_test_4.TableB", "tableReference":{"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableB"}, "type":"TABLE"}, {"kind":"bigquery#table", "id":"curious-idea-532:dataset_test_4.TableC", "tableReference":{"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableC"}, "type":"TABLE"}, {"kind":"bigquery#table", "id":"curious-idea-532:dataset_test_4.TableD", "tableReference":{"projectId":"curious-idea-532", "datasetId":"dataset_test_4", "tableId":"TableD"}, "type":"TABLE"}]'
data = JSON.parse(string)
tableListing = []
# Here we are iterating over the data instead of its child element
data.each do |item|
case item["type"]
when 'TABLE'
tableListing << item["tableReference"]["tableId"]
else
end
end
puts tableListing
Related
I am trying to write e ruby script that query an elasticsearch database and build a report on it. I am using jbuilder to build the query string like this:
require 'elasticsearch'
require 'date'
require 'jbuilder'
client = Elasticsearch::Client.new log: true, host: 'x.x.x.x', request_timeout: 10
filter_conditions = {}
filter_conditions['must'] = []
filter_conditions['should'] = []
filter_conditions['must'] << Jbuilder.encode do |json|
json.term do
json._type 'httpry-log'
end
end
filter_conditions['must'] << Jbuilder.encode do |json|
json.range do
json.set! '#timestamp' do
_now = DateTime.now
json.gte (_now - 1.00/24).strftime('%Q').to_i
json.lte _now.strftime('%Q').to_i
json.format 'epoch_millis'
end
end
end
query = Jbuilder.encode do |json|
json.size 10
json.query do
json.bool do
json.must do
json.array!(filter_conditions['must'])
end
end
end
end
puts query
But here is the result I get for the query:
{"size":10,"query":{"bool":{"must":["{\"term\":{\"_type\":\"httpry-log\"}}","{\"range\":{\"#timestamp\":{\"gte\":1477919154057,\"lte\":1477922754057,\"format\":\"epoch_millis\"}}}"]}}}
How to get the unscaped version of the inner array inside the main json output?
Thanks in advance,
Assuming that the query returns the same object and key each time:
a = your_query_hash
# The following map! will parse your string array into an array in ruby
a[:query][:bool][:must].map! { |arr| JSON.parse(arr) }
If the query does not return the same object and key each time, I'd suggest writing a recursive method that parses each value.
I have a query string that looks as follows:
http://localhost:3000/events?appointment_practices%5B10%5D=Injury&appointment_practices%5B18%5D=Immigration&appointment_practices%5B8%5D=Bankruptcy
appointment_practices is actually a hash I inserted into the query string during a redirect:
appointment_practices = practices.reduce({}) do |acc, practice|
acc[practice.id] = practice.class.name
acc
end
redirect_to events_path(appointment_practices: appointment_practices)
Now I want to parse that query string. When I tried to parse it with decode_www_form, it returns an array with a nil element:
[nil]
This is the code that is giving me the nil element:
#http_refer = #_env['HTTP_REFERER']
begin
uri = URI.parse #http_refer
practices = Hash[URI::decode_www_form(uri.query)].values_at('appointment_practices')
puts "practices: #{practices}"
rescue StandardError
end
I am trying to extract the hash. For example, in appointment_practices%5B10%5D=Injury, the id is 10 and the practice is Injury.
What other options do I have besides regex?
You can use Rack::Utils.parse_nested_query:
require 'uri'
require 'rack'
uri = URI.parse('http://localhost:3000/events?appointment_practices%5B10%5D=Injury&appointment_practices%5B18%5D=Immigration&appointment_practices%5B8%5D=Bankruptcy')
Rack::Utils.parse_nested_query(uri.query)
#=> {"appointment_practices"=>{"10"=>"Injury", "18"=>"Immigration", "8"=>"Bankruptcy"}}
Given the following CSV file, how would you remove all rows that contain the word 'true' in the column 'foo'?
Date,foo,bar
2014/10/31,true,derp
2014/10/31,false,derp
I have a working solution, however it requires making a secondary CSV object csv_no_foo
#csv = CSV.read(#csvfile, headers: true) #http://bit.ly/1mSlqfA
#headers = CSV.open(#csvfile,'r', :headers => true).read.headers
# Make a new CSV
#csv_no_foo = CSV.new(#headers)
#csv.each do |row|
# puts row[5]
if row[#headersHash['foo']] == 'false'
#csv_no_foo.add_row(row)
else
puts "not pushing row #{row}"
end
end
Ideally, I would just remove the offending row from the CSV like so:
...
if row[#headersHash['foo']] == 'false'
#csv.delete(true) #Doesn't work
...
Looking at the ruby documentation, it looks like the row class has a delete_if function. I'm confused on the syntax that that function requires. Is there a way to remove the row without making a new csv object?
http://ruby-doc.org/stdlib-1.9.2/libdoc/csv/rdoc/CSV/Row.html#method-i-each
You should be able to use CSV::Table#delete_if, but you need to use CSV::table instead of CSV::read, because the former will give you a CSV::Table object, whereas the latter results in an Array of Arrays. Be aware that this setting will also convert the headers to symbols.
table = CSV.table(#csvfile)
table.delete_if do |row|
row[:foo] == 'true'
end
File.open(#csvfile, 'w') do |f|
f.write(table.to_csv)
end
You might want to filter rows in a ruby manner:
require 'csv'
csv = CSV.parse(File.read(#csvfile), {
:col_sep => ",",
:headers => true
}
).collect { |item| item[:foo] != 'true' }
Hope it help.
This is my json data from our server.
{"id":3,"status":"Fix","severity":"Error","severityCode":2,"state":"Existing","c
ode":"MLK.MUST","message":"Memory leak. Dynamic memory stored in \u0027new_cat\u
0027 allocated through function \u0027malloc\u0027 at line 11 is lost at line 31
","file":"/data001/tools/test/test.c","method":"main","owner":"unowned","taxonom
yName":"C and C++","url":"http://xxx.xxx.xxx.xxx:8088/review/insight-review.html
#goto:project\u003dTest,pid\u003d3"}
If I convert this to xml, I can find the data below from the xml file.
<taxonomyName>C and C++</taxonomyName>
However, If I export xml from my web server menu directly, the file includes data such like
<taxonomies>
<taxonomy name="C and C++" metaInf=""/>
</taxonomies>
I think that this different result comes out because I generated wrong json data.
I used the following code to get json data from my server.
url_string = 'http://xxx.xxx.xxx.xxx:8088/review/api?action=search&user=wbt&project=Test<oken='
uri = URI(url_string)
res = Net::HTTP.post_form(uri, 'q' => 'ruby')
Could you let me know which of my codes cause this problem?
Thanks alot.
UPDATE
I modified my script below. And I can get the result as I expect
my_xml.each_line do |line|
next if line.start_with?("<?xml") or line.eql?("<hash>") or line.eql?("</hash>")
if line.include?("<taxonomyName>")
issues.puts " <taxonomies>"
issues.puts " <taxonomy name=\"#{results["taxonomyName"]}\"/>"
issues.puts " </taxonomies>"
else
issues.puts line
end
end
require 'json'
json_hash = JSON.parse(<<'END_OF_JSON')
{
"owner":"unowned",
"taxonomyName":"C and C++",
"taxonomyColor":"red",
"url":"hello.com",
"ownerAge":"20"
}
END_OF_JSON
tag_attrs = Hash.new { |hash, key| hash[key] = "" }
json_hash.keys.each do |key|
capital_index = key.index(/[A-Z]/)
if capital_index
tag_name = key[0...capital_index]
attr_name = key[capital_index..-1].downcase
tag_attrs[tag_name] << %Q{ #{attr_name}="#{json_hash[key]}"}
else
tag_attrs[key] #Because of the way the tag_attrs Hash was created, this creates an entry in the tag_attrs Hash.
end
end
tag_attrs.each do |tag_name, all_attrs|
puts "<#{tag_name}#{all_attrs}>#{json_hash[tag_name]}</#{tag_name}>"
end
--output:--
<owner age="20">unowned</owner>
<taxonomy name="C and C++" color="red"></taxonomy>
<url>hello.com</url>
I have some code which returns the error and i cannot understand why. I am abit of a newbie with Ruby but feel I am getting there:
line 27: NoMethodError "each" for NilClass
The code I am using is below:
require 'rubygems'
require 'nokogiri'
require 'sqlite3'
FIELDS = [['cityselect', 'VARCHAR'],['match', 'VARCHAR'], ['num_phone', 'NUMERIC'], ['name', 'VARCHAR'],['address', 'VARCHAR'] ]
DIV_ID = "#dgrSearch"
FILE_O = File.open('hold-data/directory-tel.txt', 'w')
FILE_O.puts( FIELDS.map{|f| f[0]}.join("\t") )
DB_NAME = "hold-data/directory-tel.sqlite"
File.delete(DB_NAME) if File.exists?DB_NAME
DATAB = SQLite3:Database.new( DB_NAME )
TABLE = "records_telephone"
DB_INSERT_STATEMENT = "INSERT into #{TABLE} values
(#{FIELDS.map{'?'}.join(',')})"
DATAB.execute "CREATE TABLE #{TABLE}(#{FIELDS.map{|f| "`#{f[0]}` #{f[1]}"}.join(', ')});"
FIELDS.each do |fn|
DATAB.execute "CREATE INDEX #{fn[2]} ON #{TABLE}(#{fn[0]})" unless fn[2].nil?
end
Dir.glob("hold-data/pages/*.html").reject{|f| f =~ /All match/}.each do |fname|
meta_stuff = File.basename(fname, '.html').split('--')
page = Nokogiri::HTML(open(fname))
page.css("#{DIV_ID} tr")[1..-2].each do |tr| # this is line #27
data_tds = tr.css('td').map{ |td|
td.text.gsub(/[$,](?=\d)/, '').gsub(/\302\240|\s/, ' ').strip
}
row_data = meta_stuff + data_tds
FILE_O.puts( data_row.join("\t"))
DATAB.execute(DB_INSERT_STATEMENT, row_data)
end
end
FILE_O.close
Can anybody see what I have done wrong?
It would be useful next time if you tell us which line the error occurs on.
From the looks of it theres only 1 line here where the expression your calling each on might return nil:
page.css("#{DIV_ID} tr")[1..-2].each do
Remember the_array[1..-2] will return nil if the_array is empty.
So page.css("#{DIV_ID} tr") might return an empty array hence the error.
If this behavior is not expiected I would consider invesigating why this is happening OR maybe check whether the array is empty before calling [1..-2] on it (see below). Just a suggestion.
if page.css("#{DIV_ID} tr").empty?
#empty array
else
#not empty
end