Access csv data from htttp request in ruby - ruby

I'm trying to access the csv data, which I recive if I make a http-request.
I don't save it to a csv file, so I save it to the variable.
Let's say this is the response I get, how can I print food?
uuid,event_id,category
12,1,food
13,2,cars
And this is the part of the ruby code which is important.
That's something I found, but it was originally used with a file, so it doesn't work.
csvdata = request(action,parameter)
#data_hash = {}
CSV.foreach(csvdata) do |row|
uuid, event_id, category = row
#data_hash[uuid] = event_id
end
Do I really need files for that or is there a easy way I can access the values?
Update
CSV.parse(csvdata,data = Hash.new) do |row|
puts data
end
The hash should look like this so I can use the column names
{"uuid" => "12,13", "event_id" => "323,3243", "category" => "food,cars"}

csv_data = Hash.new{|k, v| k[v] = []}
CSV.parse(csv_string, headers: true) do |row|
row.each{|k, v| csv_data[k] << v}
end
csv_data = Hash[csv_data.map{|k, v| [k, v.join(",")]}]
Update after specification Requested output.

Try this:
csvdata = request(action,parameter)
#data_hash = {}
CSV.parse(csvdata, headers: true) do |row|
#data_hash[row['uuid']] = row['event_id']
end
#data_hash
# => {"12"=>"1", "13"=>"2"}
When you parse a CSV, the seconds parameter (data = Hash.new in your code) is actually an options parameter. You can see the available options here:
:headers
If set to :first_row or true, the initial row of the CSV file will be treated as a row of headers. If set to an Array, the contents will be used as the headers. If set to a String, the String is run through a call of ::parse_line with the same :col_sep, :row_sep, and :quote_char as this instance to produce an Array of headers. This setting causes #shift to return rows as CSV::Row objects instead of Arrays and #read to return CSV::Table objects instead of an Array of Arrays.
When passing headers: true - values are parsed into a Row object, where they can be accessed by name.

Related

Filter unique values from a tsv file

I have a tsv file that has four columns. I'm having difficulty isolating the first column of the file (UUID), so I can strip out the 'UUID=' from each element, and also filter from unique values.
What am I doing wrong in my code? I've been pretty stuck on figuring this out. Thank you in advance!
Here's the link to the file, and my code below.
https://drive.google.com/file/d/1mGaK3n3YCrzrwOgSo5QQZ62FXDKJ3nZ8/view?usp=sharing
require "csv"
log_file = CSV.foreach("output_file.tsv",{:col_sep => "\t", :headers => true}) do |row|
uuid = row["UUID"]
ip = row["IP"]
time = row["TIME"]
ua = row["UA"]
uuid = uuid.drop(1)
ip = ip.drop(1)
time = time.drop(1)
ua = ua.drop(1)
uuid = uuid.map { |element|
element = element[5..-1]}
unique_logins = uuid.uniq
puts uuid.uniq.length
Probably you're confused a bit and think that CSV.foreach reads the whole column, but it actually reads your file row by row. That's why no need to drop(1).
This is the minimal code, which collects uuids from the file and prints the number of those uuids and then prints the number of unique uuids
require "csv"
uuids = []
log_file = CSV.foreach("output_file.tsv",{:col_sep => "\t", :headers => true}) do |row|
uuids << row["UUID"]
end
uuids = uuids.map { |element| element = element[5..-1]}
p uuids.length
unique_logins = uuids.uniq
p unique_logins.length
If your file isn't that big, you could also just read the entire file in at once, and then use the returned CSV::Table to read the entire column out and operate on that:
require 'csv'
tsv = CSV.read("output_file.tsv", col_sep: "\t", headers: true)
uuids = tsv['UUID'].map { |uuid| uuid[/\AUUID=(.+)\z/, 1] }.uniq
# => ["e9fc3b6e6641e69fb8cfbdfac48709ae", "f296020354e8c913454f62732d0e3dc4",
# "0300481b1e495e3c919b5214dda7b26c", "9ccc4096ed1d11d1b4c9e57ca1192176",
# "c0580eeb3f98d9c3fe232fc48694bf8e", "25ee63a754b9d4590b69b9ab2a4668cd",
# "aa61387f01797a839ca6f55daeb69b30", "9c7f37f5c187f662eaf7d0df83ac8804"]

How to access block parameters using Object.send

I'm trying to run the following code:
class RentLimit < ActiveRecord::Base
def self.load_data
rows = CSV.open("csvs/income_limits_2011_to_2015.csv").read
rows.shift
rows.each do |county, yr, date, _50pct_1br, _50pct_2br, _50pct_3br, _50pct_4br, _60pct_1br, _60pct_2br, _60pct_3br, _60pct_4br|
[50, 60].each do |ami|
[1, 2, 3, 4].each do |br|
r = new
r.county = county
r.state = "SC"
r.year = yr
r.effective_date = Date.parse(date)
r.pct_ami = ami
r.br = br
r.max_rent = self.send("_#{ami}pct_#{br}br".to_sym)
r.save
end#of brs
end# of amis
end# of rows
end
end
but am getting this error message when trying to run it:
NoMethodError: undefined method `_50pct_1br' for #<Class:0x007fe942ce3b18>
The send method isn't able to access those block parameters inside of the scope. Is there any way to give access to block parameters to send? If not, how else might I dynamically access block parameters?
How do I use send or its equivalent to access block parameters in Ruby?
This is much easier if you tell CSV.open what your column names are. It looks like your CSV file might have a header row that you're skipping with rows.shift, in which case you shouldn't skip it, and use the headers: true option. Then you can access each field by name with row["field_name"] or, in your case, row["_#{ami}pct_#{br}br"]:
CSV_PATH = "csvs/income_limits_2011_to_2015.csv"
DEFAULT_STATE = "SC"
def self.load_data
CSV.open(CSV_PATH, 'r', headers: true) do |csv|
csv.each do |row|
max_rent = row["_#{ami}pct_#{br}br"]
create(
county: row["county"],
state: DEFAULT_STATE,
year: row["yr"],
effective_date: Date.parse(row["date"]),
pct_ami: ami,
br: br,
max_rent: max_rent,
)
end
end
end
Note that I used CSV.open with a block to ensure that the file is closed after it's been read, which your original code wasn't doing. I also used create instead of new; ... save, since the latter is needlessly verbose.
If you're skipping the first row for some other reason, or you want to use field names other than those in the header row, you can set the options return_headers: false, headers: names, where names is an array of names, e.g.:
CSV_HEADERS = %w[
county yr date _50pct_1br _50pct_2br _50pct_3br _50pct_4br
_60pct_1br _60pct_2br _60pct_3br _60pct_4br
].freeze
def self.load_data
CSV.open(CSV_PATH, 'r', return_headers: false, headers: CSV_HEADERS) do |csv|
# ...
end
end
Finally, since some of your attributes are the same for every object created, I'd move those out of the loop:
def self.load_data
base_attrs = { state: DEFAULT_STATE, pct_ami: ami, br: br }
CSV.open(CSV_PATH, 'r', headers: true) do |csv|
csv.each do |row|
create(base_attrs.merge(
county: row["county"],
year: row["yr"],
effective_date: row["date"],
max_rent: row["_#{ami}pct_#{br}br"]
))
end
end
end

How to remove a row from a CSV with Ruby

Given the following CSV file, how would you remove all rows that contain the word 'true' in the column 'foo'?
Date,foo,bar
2014/10/31,true,derp
2014/10/31,false,derp
I have a working solution, however it requires making a secondary CSV object csv_no_foo
#csv = CSV.read(#csvfile, headers: true) #http://bit.ly/1mSlqfA
#headers = CSV.open(#csvfile,'r', :headers => true).read.headers
# Make a new CSV
#csv_no_foo = CSV.new(#headers)
#csv.each do |row|
# puts row[5]
if row[#headersHash['foo']] == 'false'
#csv_no_foo.add_row(row)
else
puts "not pushing row #{row}"
end
end
Ideally, I would just remove the offending row from the CSV like so:
...
if row[#headersHash['foo']] == 'false'
#csv.delete(true) #Doesn't work
...
Looking at the ruby documentation, it looks like the row class has a delete_if function. I'm confused on the syntax that that function requires. Is there a way to remove the row without making a new csv object?
http://ruby-doc.org/stdlib-1.9.2/libdoc/csv/rdoc/CSV/Row.html#method-i-each
You should be able to use CSV::Table#delete_if, but you need to use CSV::table instead of CSV::read, because the former will give you a CSV::Table object, whereas the latter results in an Array of Arrays. Be aware that this setting will also convert the headers to symbols.
table = CSV.table(#csvfile)
table.delete_if do |row|
row[:foo] == 'true'
end
File.open(#csvfile, 'w') do |f|
f.write(table.to_csv)
end
You might want to filter rows in a ruby manner:
require 'csv'
csv = CSV.parse(File.read(#csvfile), {
:col_sep => ",",
:headers => true
}
).collect { |item| item[:foo] != 'true' }
Hope it help.

Reading every line in a CSV and using it to query an API

I have the following Ruby code:
require 'octokit.rb'
require 'csv.rb'
CSV.foreach("actors.csv") do |row|
CSV.open("node_attributes.csv", "wb") do |csv|
csv << [Octokit.user "userid"]
end
end
I have a csv called actors.csv where every row has one entry - a string with a userid.
I want to go through all the rows, and for each row do Octokit.user "userid", and then store the output from each query on a separate row in a CSV - node_attributes.csv.
My code does not seem to do this? How can I modify it to make this work?
require 'csv'
DOC = 'actors.csv'
DOD = 'new_output.csv'
holder = CSV.read(DOC)
You can navigate it by calling
holder[0][0]
=> data in the array
holder[1][0]
=> moar data in array
make sense?
#make this a loop
profile = []
profile[0] = holder[0][0]
profile[1] = holder[1][0]
profile[2] = 'whatever it is you want to store in the new cell'
CSV.open(DOD, "a") do |data|
data << profile.map
end
#end the loop here
That last bit of code will print whatever you want into a new csv file

Ruby - Builder - Trying to convert CSV rows as data sets for constructing several XML's

Here's what I'm trying to accomplish. I need to have a single CSV with headers and several rows. I'm iterating through the headers and storing then and then associating the row data to the header. I need to be able to iterate through each of the rows in the CSV to use for constructing an XML's data. The constructed XML is then dumped as a .xml file and the program starts on the next row in the CSV. Each row has a column that provides the name of the XML file.
Here's what I've got so far.
Read in the data from the CSV file. Collect the header and row data.
def get_rows
raw_data = CSV.read('test.csv', {:skip_blanks => false, :headers => true})
data = []
raw_data.each { |row| data << row}
return build_header(data, raw_data)
end
take the header and row data and marry them up.
def build_header(data, raw_data)
(0..(data.length - 1)).each do |ri|
h = {}
raw_data.headers.each_with_index do |v, i|
h[v] = data[ri].fields[i]
end
return build_ostruct(h)
end
end
take the hash h and make an ostruct of it.
def build_ostruct(h)
x = OpenStruct.new(h)
uniq = x.tc_name
y = uniq_name.to_s + ".xml"
#marshal dump for debugging
x.marshal_dump.each{ |k,v| puts "#{k} => #{v}" }
return xml_builder(x, y)
end
Below this I'm taking the new ostruct "x" and calling the column headers from the CSV to #populate the XML nodes
For example: x.column1, x.column2, x.column3
Now the part I'm getting hung up on is getting the ostruct to receive the new row of data per iteration run. The objective is to have the ostruct populate with each row from the CSV. Currently the hash is displaying the proper data set and my XML is populating as expected but only with the first row of data. How do I get this to iterate through all the rows and populate the ostruct with the data per iteration so I can create a bulk set of XML's?
Thanks in advance for any and all help!
Something like this should work:
require 'csv'
require 'nokogiri'
CSV.foreach('test.csv', :headers => true) do |row|
builder = Nokogiri::XML::Builder.new do |xml|
xml.root do |root|
row.each do |k, v|
root.send k, v
end
end
end
File.open("#{row['tc_name']}.xml", 'w'){|f| f << builder.to_xml}
end
you are calling return in build_header, which ends the call. you need to collect your results in some way without immediately returning the first one, so that build_header can run for the entire set of rows.

Resources