I have a csv file that has 7000+ records that I process/manipulate and export to a new csv file. I have no issues doing that and everything works as expected.
I would like to change the process to where it breaks the output into multiple files. So instead of writing all 7000+ rows to the new csv file it would write the first 1000 rows to newexport1.csv and the next 1000 rows to newexport2.csv until it reaches the end of the data.
Is there an easy way to do this with CSV in Ruby 1.9?
My current write method:
CSV.open("#{PATH_TO_EXPORT_FILE}/newexport.csv", "w+", :col_sep => '|', :headers => true) do |f|
export_rows.each do |row|
f << row
The short answer is "no". You'll want to adjust your current code to split up the set and then dump each subset to a different file. This ought to be pretty close:
export_rows.each_slice(1000).with_index do |rows, idx|
CSV.open("#{PATH_TO_EXPORT_FILE}/newexport-#{idx.to_s}.csv", "w+", :col_sep => '|', :headers => true) do |f|
rows.each { |row| f << row }
end
end
Yes, there is.
It's embedded in Ruby 1.9
Check this link
To read:
CSV.foreach("path/to/file.csv") do |row|
# manipulate the content
end
To write:
CSV.open("path/to/file.csv", "wb") do |csv|
csv << ["row", "of", "CSV", "data"]
csv << ["another", "row"]
# something else
end
I think that you'll need to combine one inside the other.
FasterCSV is the standard CSV library since ruby 1.9, you can find a lot of example code in the examples folder:
https://github.com/JEG2/faster_csv/tree/master/examples
For the example code to work, you should change:
require "faster_csv"
for
require "csv"
Related
I've been trying to use Ruby to create a CSV file from json data. I was able to create the file, but I need to add a few headers. I tried following suggestions and answers from similar questions posted here on Stack Overflow, but I keep getting errors. Can anyone give me some pointers?
Here's my code.
require 'csv'
require 'json'
CSV.open("your_csv.csv", "w") do |csv|
JSON.parse(File.open("tojson.txt").read).each do |hash|
csv << hash.values
#csv.each { |line| line['New_header'] = line[0].to_i + line[1].to_i }
end
end
And here is the error I'm getting:
Anyone have any suggestions?
This is not how you add headers to a csv file. When you generate csv content, a header row is just a regular row. And should be generated as such. Example:
CSV.open("your_csv.csv", "w") do |csv|
csv << ['new_header', 'value1', 'value2'] # the headers
JSON.parse(File.open("tojson.txt").read).each do |hash|
row = [generate, values, for, headers, above]
csv << row
end
end
You don't have a #csv variable. You have a csv one.
I want to parse two CSV files of the MaxMind GeoIP2 database, do some joining based on a column and merge the result into one output file.
I used standard CSV ruby library, it is very slow. I think it tries to load all the file in memory.
block_file = File.read(block_path)
block_csv = CSV.parse(block_file, :headers => true)
location_file = File.read(location_path)
location_csv = CSV.parse(location_file, :headers => true)
CSV.open(output_path, "wb",
:write_headers=> true,
:headers => ["geoname_id","Y","Z"] ) do |csv|
block_csv.each do |block_row|
puts "#{block_row['geoname_id']}"
location_csv.each do |location_row|
if (block_row['geoname_id'] === location_row['geoname_id'])
puts " match :"
csv << [block_row['geoname_id'],block_row['Y'],block_row['Z']]
break location_row
end
end
end
Is there another ruby library that support processing in chuncks ?
block_csv is 800MB and location_csv is 100MB.
Just use CSV.open(block_path, 'r', :headers => true).each do |line| instead of File.read and CSV.parse. It will parse the file line by line.
In your current version, you explicitly tell it to read all the file with File.read and then to parse the whole file as a string with CSV.parse. So it does exactly what you have told.
I'm writing a program, that creates a csv-File. And I have a problem right at the beginning.
So, my code is
def create_csv
destfile = Rails.root.join("public", "reports", "statistic_csv#{id}.csv")
csv_string = FasterCSV.generate do |out|
out << ["row", "of", "CSV", "data"]
end
FasterCSV.open(destfile, "w") do |csv|
csv << csv_string
end
end
I thought, I will get 4 columns in the output file, smth like this row|of|csv|data. But what I get is "row,of,CSV,data" in one cell A1. How can i solve the Problem? Thanks in advance!
PS. I use ruby 1.8.7 and FasterCSV 1.5.5
You are encoding the CSV string twice. This should work:
def create_csv
destfile = Rails.root.join("public", "reports", "statistic_csv#{id}.csv")
FasterCSV.open(destfile, "wb") do |csv|
csv << ["row", "of", "CSV", "data"]
end
end
You can also specify a custom column separator:
FasterCSV.open(destfile, "wb", { :col_sep => "|" }) do |csv|
# ...
end
I presume you're opening this in Excel. Excel may not be detecting the file as a CSV file. Try importing the data into an excel workbook as opposed to opening the file in Excel.
Looking at the documentation for the CSV library of Ruby, I'm pretty sure this is possible and easy.
I simply need to delete the first three columns of a CSV file using Ruby but I haven't had any success getting it run.
csv_table = CSV.read(file_path_in, :headers => true)
csv_table.delete("header_name")
csv_table.to_csv # => The new CSV in string format
Check the CSV::Table documentation: http://ruby-doc.org/stdlib-1.9.2/libdoc/csv/rdoc/CSV/Table.html
csv_table = CSV.read("../path/to/file.csv", :headers => true)
keep = ["x", "y", "z"]
new_csv_table = csv_table.by_col!.delete_if do |column_name,column_values|
!keep.include? column_name
end
new_csv_table.to_csv
What about:
require 'csv'
File.open("resfile.csv","w+") do |f|
CSV.foreach("file.csv") do |row|
f.puts(row[3..-1].join(","))
end
end
I have built on a few of the questions (really liked what #fguillen did with CSV::Table) here but just made it a bit simpler to drop it into an existing project, target a file and make a quick change.
Have added byebug cause ... yes. Then also retained the headers from the original file (assuming they exist for anyone wanting to use this snippet).
The file is overwritten each time in case you want to test/tinker.
require 'csv'
require 'byebug'
in_file = './db/data/inbox/change__to_file_name.csv'
out_file = in_file + ".out"
target_col = "change_to_column_name"
csv_table = CSV.read(in_file, headers: true)
csv_table.delete(target_col)
CSV.open(out_file, 'w+', force_quotes: true) do |csv|
csv << csv_table.headers
csv_table.each_with_index do |row|
csv << row
end
end
I'm trying to query a table, fetch all records, and save the result as a CSV file.
This is what I've done so far:
require 'OCI8'
conn = OCI8.new('scott','tiger','020')
file = File.open('output.csv','w') do |f|
conn.exec('select * from emp') do |e|
f.write log.join(',')
end
end
.. And while it does generate a CSV file, the problem is that all records get saved onto a single line. How can I put the data such that each record goes onto a new line ?
Well, you can use f.puts instead of f.write there, but I'd recommend you take a look at CSV module:
http://ruby-doc.org/stdlib/libdoc/csv/rdoc/index.html
outfile = File.open('csvout', 'wb')
CSV::Writer.generate(outfile) do |csv|
csv << ['c1', nil, '', '"', "\r\n", 'c2']
...
end
outfile.close
PS: Actually, there is another CSV library called FasterCSV, which became CSV in standard library in Ruby 1.9. But in general, any should be better than writing it yourself.