I'm writing a test for one of my classes which has the following constructor:
def initialize(filepath)
#transactions = []
File.open(filepath).each do |line|
next if $. == 1
elements = line.split(/\t/).map { |e| e.strip }
transaction = Transaction.new(elements[0], Integer(1))
#transactions << transaction
end
end
I'd like to test this by using a fake file, not a fixture. So I wrote the following spec:
it "should read a file and create transactions" do
filepath = "path/to/file"
mock_file = double(File)
expect(File).to receive(:open).with(filepath).and_return(mock_file)
expect(mock_file).to receive(:each).with(no_args()).and_yield("phrase\tvalue\n").and_yield("yo\t2\n")
filereader = FileReader.new(filepath)
filereader.transactions.should_not be_nil
end
Unfortunately this fails because I'm relying on $. to equal 1 and increment on every line and for some reason that doesn't happen during the test. How can I ensure that it does?
Global variables make code hard to test. You could use each_with_index:
File.open(filepath) do |file|
file.each_with_index do |line, index|
next if index == 0 # zero based
# ...
end
end
But it looks like you're parsing a CSV file with a header line. Therefore I'd use Ruby's CSV library:
require 'csv'
CSV.foreach(filepath, col_sep: "\t", headers: true, converters: :numeric) do |row|
#transactions << Transaction.new(row['phrase'], row['value'])
end
You can (and should) use IO#each_line together with Enumerable#each_with_index which will look like:
File.open(filepath).each_line.each_with_index do |line, i|
next if i == 1
# …
end
Or you can drop the first line, and work with others:
File.open(filepath).each_line.drop(1).each do |line|
# …
end
If you don't want to mess around with mocking File for each test you can try FakeFS which implements an in memory file system based on StringIO that will clean up automatically after your tests.
This way your test's don't need to change if your implementation changes.
require 'fakefs/spec_helpers'
describe "FileReader" do
include FakeFS::SpecHelpers
def stub_file file, content
FileUtils.mkdir_p File.dirname(file)
File.open( file, 'w' ){|f| f.write( content ); }
end
it "should read a file and create transactions" do
file_path = "path/to/file"
stub_file file_path, "phrase\tvalue\nyo\t2\n"
filereader = FileReader.new(file_path)
expect( filereader.transactions ).to_not be_nil
end
end
Be warned: this is an implementation of most of the file access in Ruby, passing it back onto the original method where possible. If you are doing anything advanced with files you may start running into bugs in the FakeFS implementation. I got stuck with some binary file byte read/write operations which weren't implemented in FakeFS quite how Ruby implemented them.
I'm generating some CSV output using Ruby's built-in CSV. Everything works fine, but the customer wants the name field in the output to have wrapping double-quotes so the output looks like the input file. For instance, the input looks something like this:
1,1.1.1.1,"Firstname Lastname",more,fields
2,2.2.2.2,"Firstname Lastname, Jr.",more,fields
CSV's output, which is correct, looks like:
1,1.1.1.1,Firstname Lastname,more,fields
2,2.2.2.2,"Firstname Lastname, Jr.",more,fields
I know CSV is doing the right thing by not double-quoting the third field just because it has embedded blanks, and wrapping the field with double-quotes when it has the embedded comma. What I'd like to do, to help the customer feel warm and fuzzy, is tell CSV to always double-quote the third field.
I tried wrapping the field in double-quotes in my to_a method, which creates a "Firstname Lastname" field being passed to CSV, but CSV laughed at my puny-human attempt and output """Firstname Lastname""". That is the correct thing to do because it's escaping the double-quotes, so that didn't work.
Then I tried setting CSV's :force_quotes => true in the open method, which output double-quotes wrapping all fields as expected, but the customer didn't like that, which I expected also. So, that didn't work either.
I've looked through the Table and Row docs and nothing appeared to give me access to the "generate a String field" method, or a way to set a "for field n always use quoting" flag.
I'm about to dive into the source to see if there's some super-secret tweaks, or if there's a way to monkey-patch CSV and bend it to do my will, but wondered if anyone had some special knowledge or had run into this before.
And, yes, I know I could roll my own CSV output, but I prefer to not reinvent well-tested wheels. And, I'm also aware of FasterCSV; That's now part of Ruby 1.9.2, which I'm using, so explicitly using FasterCSV buys me nothing special. Also, I'm not using Rails and have no intention of rewriting it in Rails, so unless you have a cute way of implementing it using a small subset of Rails, don't bother. I'll downvote any recommendations to use any of those ways just because you didn't bother to read this far.
Well, there's a way to do it but it wasn't as clean as I'd hoped the CSV code could allow.
I had to subclass CSV, then override the CSV::Row.<<= method and add another method forced_quote_fields= to make it possible to define the fields I want to force-quoting on, plus pull two lambdas from other methods. At least it works for what I want:
require 'csv'
class MyCSV < CSV
def <<(row)
# make sure headers have been assigned
if header_row? and [Array, String].include? #use_headers.class
parse_headers # won't read data for Array or String
self << #headers if #write_headers
end
# handle CSV::Row objects and Hashes
row = case row
when self.class::Row then row.fields
when Hash then #headers.map { |header| row[header] }
else row
end
#headers = row if header_row?
#lineno += 1
#do_quote ||= lambda do |field|
field = String(field)
encoded_quote = #quote_char.encode(field.encoding)
encoded_quote +
field.gsub(encoded_quote, encoded_quote * 2) +
encoded_quote
end
#quotable_chars ||= encode_str("\r\n", #col_sep, #quote_char)
#forced_quote_fields ||= []
#my_quote_lambda ||= lambda do |field, index|
if field.nil? # represent +nil+ fields as empty unquoted fields
""
else
field = String(field) # Stringify fields
# represent empty fields as empty quoted fields
if (
field.empty? or
field.count(#quotable_chars).nonzero? or
#forced_quote_fields.include?(index)
)
#do_quote.call(field)
else
field # unquoted field
end
end
end
output = row.map.with_index(&#my_quote_lambda).join(#col_sep) + #row_sep # quote and separate
if (
#io.is_a?(StringIO) and
output.encoding != raw_encoding and
(compatible_encoding = Encoding.compatible?(#io.string, output))
)
#io = StringIO.new(#io.string.force_encoding(compatible_encoding))
#io.seek(0, IO::SEEK_END)
end
#io << output
self # for chaining
end
alias_method :add_row, :<<
alias_method :puts, :<<
def forced_quote_fields=(indexes=[])
#forced_quote_fields = indexes
end
end
That's the code. Calling it:
data = [
%w[1 2 3],
[ 2, 'two too', 3 ],
[ 3, 'two, too', 3 ]
]
quote_fields = [1]
puts "Ruby version: #{ RUBY_VERSION }"
puts "Quoting fields: #{ quote_fields.join(', ') }", "\n"
csv = MyCSV.generate do |_csv|
_csv.forced_quote_fields = quote_fields
data.each do |d|
_csv << d
end
end
puts csv
results in:
# >> Ruby version: 1.9.2
# >> Quoting fields: 1
# >>
# >> 1,"2",3
# >> 2,"two too",3
# >> 3,"two, too",3
This post is old, but I can't believe no one thought of this.
Why not do:
csv = CSV.generate :quote_char => "\0" do |csv|
where \0 is a null character, then just add quotes to each field where they are needed:
csv << [product.upc, "\"" + product.name + "\"" # ...
Then at the end you can do a
csv.gsub!(/\0/, '')
I doubt if this will help the customer feeling warm and fuzzy after all this time, but this seems to work:
require 'csv'
#prepare a lambda which converts field with index 2
quote_col2 = lambda do |field, fieldinfo|
# fieldinfo has a line- ,header- and index-method
if fieldinfo.index == 2 && !field.start_with?('"') then
'"' + field + '"'
else
field
end
end
# specify above lambda as one of the converters
csv = CSV.read("test1.csv", :converters => [quote_col2])
p csv
# => [["aaa", "bbb", "\"ccc\"", "ddd"], ["fff", "ggg", "\"hhh\"", "iii"]]
File.open("test1.txt","w"){|out| csv.each{|line|out.puts line.join(",")}}
CSV has a force_quotes option that will force it to quote all fields (it may not have been there when you posted this originally). I realize this isn't exactly what you were proposing, but it's less monkey patching.
2.1.0 :008 > puts CSV.generate_line [1,'1.1.1.1','Firstname Lastname','more','fields']
1,1.1.1.1,Firstname Lastname,more,fields
2.1.0 :009 > puts CSV.generate_line [1,'1.1.1.1','Firstname Lastname','more','fields'], force_quotes: true
"1","1.1.1.1","Firstname Lastname","more","fields"
The drawback is that the first integer value ends up listed as a string, which changes things when you import into Excel.
It's been a long time, but since the CSV library has been patched, this might help someone if they're now facing this issue:
require 'csv'
# puts CSV::VERSION # this should be 3.1.9+
headers = ['id', 'ip', 'name', 'foo', 'bar']
data = [
[1, '1.1.1.1','Firstname Lastname','more','fields'],
[2, '2.2.2.2','Firstname Lastname, Jr.','more','fields']
]
quoter = Proc.new do |field, field_meta|
# the index starts at zero, that's why the third field would be 2:
field = '"' + field + '"' if field_meta.index == 2 && fields_meta.index > 1
field = '"' + field + '"' if field.is_a?(String) && field.include?(',')
# ^ CSV format needs to escape fields containing comma(s): ,
field
end
file = CSV.generate(headers: true, quote_char: '', write_converters: quoter) do |csv|
csv << headers
data.each { |row| csv << row }
end
puts file
the output would be:
id,ip,name,foo,bar
1,1.1.1.1,"Firstname Lastname",more,fields
2,2.2.2.2,"Firstname Lastname, Jr.",more,fields
It doesn't look like there's any way to do this with the existing CSV implementation short of monkey-patching/rewriting it.
However, assuming you have full control over the source data, you could do this:
Append a custom string including a comma (i.e. one that would never be naturally found in the data) to the end of the field in question for each row; maybe something like "FORCE_COMMAS,".
Generate the CSV output.
Now that you have CSV output with quotes on every row for your field, remove the custom string: csv.gsub!(/FORCE_COMMAS,/, "")
Customer feels warm and fuzzy.
CSV has changed a bit in Ruby 2.1 as mentioned by #jwadsack, however here's an working version of #the-tin-man's MyCSV. Bit modified, you set the forced_quote_fields via options.
MyCSV.generate(forced_quote_fields: [1]) do |_csv|...
The modified code
require 'csv'
class MyCSV < CSV
def <<(row)
# make sure headers have been assigned
if header_row? and [Array, String].include? #use_headers.class
parse_headers # won't read data for Array or String
self << #headers if #write_headers
end
# handle CSV::Row objects and Hashes
row = case row
when self.class::Row then row.fields
when Hash then #headers.map { |header| row[header] }
else row
end
#headers = row if header_row?
#lineno += 1
output = row.map.with_index(&#quote).join(#col_sep) + #row_sep # quote and separate
if #io.is_a?(StringIO) and
output.encoding != (encoding = raw_encoding)
if #force_encoding
output = output.encode(encoding)
elsif (compatible_encoding = Encoding.compatible?(#io.string, output))
#io.set_encoding(compatible_encoding)
#io.seek(0, IO::SEEK_END)
end
end
#io << output
self # for chaining
end
def init_separators(options)
# store the selected separators
#col_sep = options.delete(:col_sep).to_s.encode(#encoding)
#row_sep = options.delete(:row_sep) # encode after resolving :auto
#quote_char = options.delete(:quote_char).to_s.encode(#encoding)
#forced_quote_fields = options.delete(:forced_quote_fields) || []
if #quote_char.length != 1
raise ArgumentError, ":quote_char has to be a single character String"
end
#
# automatically discover row separator when requested
# (not fully encoding safe)
#
if #row_sep == :auto
if [ARGF, STDIN, STDOUT, STDERR].include?(#io) or
(defined?(Zlib) and #io.class == Zlib::GzipWriter)
#row_sep = $INPUT_RECORD_SEPARATOR
else
begin
#
# remember where we were (pos() will raise an exception if #io is pipe
# or not opened for reading)
#
saved_pos = #io.pos
while #row_sep == :auto
#
# if we run out of data, it's probably a single line
# (ensure will set default value)
#
break unless sample = #io.gets(nil, 1024)
# extend sample if we're unsure of the line ending
if sample.end_with? encode_str("\r")
sample << (#io.gets(nil, 1) || "")
end
# try to find a standard separator
if sample =~ encode_re("\r\n?|\n")
#row_sep = $&
break
end
end
# tricky seek() clone to work around GzipReader's lack of seek()
#io.rewind
# reset back to the remembered position
while saved_pos > 1024 # avoid loading a lot of data into memory
#io.read(1024)
saved_pos -= 1024
end
#io.read(saved_pos) if saved_pos.nonzero?
rescue IOError # not opened for reading
# do nothing: ensure will set default
rescue NoMethodError # Zlib::GzipWriter doesn't have some IO methods
# do nothing: ensure will set default
rescue SystemCallError # pipe
# do nothing: ensure will set default
ensure
#
# set default if we failed to detect
# (stream not opened for reading, a pipe, or a single line of data)
#
#row_sep = $INPUT_RECORD_SEPARATOR if #row_sep == :auto
end
end
end
#row_sep = #row_sep.to_s.encode(#encoding)
# establish quoting rules
#force_quotes = options.delete(:force_quotes)
do_quote = lambda do |field|
field = String(field)
encoded_quote = #quote_char.encode(field.encoding)
encoded_quote +
field.gsub(encoded_quote, encoded_quote * 2) +
encoded_quote
end
quotable_chars = encode_str("\r\n", #col_sep, #quote_char)
#quote = if #force_quotes
do_quote
else
lambda do |field, index|
if field.nil? # represent +nil+ fields as empty unquoted fields
""
else
field = String(field) # Stringify fields
# represent empty fields as empty quoted fields
if field.empty? or
field.count(quotable_chars).nonzero? or
#forced_quote_fields.include?(index)
do_quote.call(field)
else
field # unquoted field
end
end
end
end
end
end
I want to obtain the information of students in class c79363 ONLY.
The following file(users.txt) contains the user id's
c79363::7117:dputnam,gliao01,hmccon01,crober06,cpurce01,cdavid03,dlevin01,jsmith88
d79363::7117:dputn,gliao0,hmcc01,crob06,cpur01,cdad03,dlen01,jsmh88
f79363::7117:dpnam,gli01,hmcn01,ober06,crce01,cdav03,dln01,jith88
FILENAME=user_info.txt
The other one contains specific information about a user like in this format
jsmith88:*:4185:208:jsmith113:/students/jsmith88:/usr/bin/bash
userd:*:4185:208:jsmith113:/students/jsmith88:/usr/bin/bash
gliao01:*:4185:208:jsmith113:/students/jsmith88:/usr/bin/bash
Here was my solution but was slow! I want to optimize the speed.
pseudo code
I read the file using File.readlines(users.txt) ->
I used split(/,/) -> I then pop array until i had an array with the following values
dputnam,gliao01,hmccon01,crober06,cpurce01,cdavid03,dlevin01,jsmith88
I then continue to read user_info.txt with File.readlines(user_info.txt)
I split(/:/) i have USER ARRAY
Finally I compared the first entry USER ARRAY with my users in class c79363.
user_info = {}
File.foreach("user_info.txt") {|line| user_info[line[/^[^:]+/]] = line.chomp}
class_name = "c79363"
File.foreach("users.txt") do |line|
next unless line[/^[^:]+/] == class_name
line[/[^:]+$/].rstrip.split(/,/).each do |user|
puts user_info[user] if user_info.has_key?(user)
end
end
user_info = {}
File.readlines("users_info.txt").each do |line|
user_info[line.split(/:/,2)[0]] = line.chomp
end
class_name = "c79363"
File.readlines("users.txt").each do |line|
line=line.strip.split(/:/)
if line[0] == class_name then
line[-1].split(",").each {|u| puts user_info[u] if user_info[u] }
end
end
I'm new to Ruby - I'm having troubles on every step...
Imagine a Ruby script main.rb and a lot of unknown script files script1.rb ... scriptN.rb.
Each scriptX.rb contains unique module with one procedure needs to be executed:
Module X
def some_procedure(i)
puts "{#i} Module X procedure executed successfully!"
end
end
All I need is to:
iterate over all files in current directory
if current file has name like /^script.*?\.rb$/
then load it and execute some_procedure
How can I do it in main.rb ?
Thank you in advance!
Choose from these great answers in SO on loading the files: Best way to require all files from a directory in ruby?
Then in your files, just have them execute on load, rather than on a method call.
The problem might be that, when a file is required, it doesn't return the list of modules (or, in general, constants) which it defines. So, unless you don't know which module a script has defined, you will not know where to pass your some_procedure message.
As a workaround, you may try getting the list of defined constants before and after the script was required, find a difference, i.e. list of constants during require, and iterate through all of them, checking which one implements the method you need.
First, we need to put some restriction:
Every file script_my1.rb will have the module named Script_my1. I.e. first letter capitalized, all other letters - lowercase.
Create two files script_my1.rb and script_my2.rb as follows:
---script_my1.rb:
module Script_my1
#value = 0
def self.some_procedure(i)
puts "#{i} my1 executed!"
#value = i
end
def self.another_procedure()
return #value
end
end
---script_my2.rb:
module Script_my2
#value = 0
def self.some_procedure(i)
puts "#{i} my2 executed!"
#value = i
end
def self.another_procedure()
return #value
end
end
Now the main script, that loads and executes some_procedure() in each module, and then another_procedure().
Please notice, that each module can have separated variables with the same name #value.
Moreover, I think every module can be executed in a separate thread and have access to global variables, but I have not tested it yet.
---main.rb:
# Load all files from the current directory
# with name like script_xxx.rb
i = 1
result = nil
Dir['./script_*.rb'].each { |f|
next if File.directory?(f)
require (f)
moduleName = f[2,f.length].rpartition('.rb')[0].capitalize
eval ( "#{moduleName}.some_procedure(%d)" % i )
eval ( "result = #{moduleName}.another_procedure()" )
puts result
i = i + 1
}
Output of this program is:
1 my1 executed!
1
2 my2 executed!
2
That is all!
Some improvement to previous solution can be made. If we want to avoid special naming, we can use global hash to store procedure's names. Each loaded script_xx.rb file would register it's own procedures in this global hash.
Please notice, that in this case we make two cycles:
first we load all files script_xx.b
every file while loading will register it's procedures in $global_procs array.
then iterate over all entries in $global_procs to execute all registered procedures via eval()
Hope, this is a more 'ruby-like' solution!
---script_my1.rb
module My1
#value = 0
def self.some_procedure(i)
puts "#{i} my1 executed!"
#value = i
end
def self.another_procedure()
return #value
end
end
$global_procs << { 'module' => 'My1',
'some_procedure' => 'My1.some_procedure',
'another_procedure' => 'My1.another_procedure' }
---script_my2.rb
module MMM2
#value = 0
def self.some_procedure(i)
puts "#{i} MMM2 executed!"
#value = i
end
def self.another_procedure()
return #value
end
end
$global_procs << { 'module' => 'MMM2',
'some_procedure' => 'MMM2.some_procedure',
'another_procedure' => 'MMM2.another_procedure' }
---main.rb
# Create global array for holding module's info
$global_procs = []
Dir['./script_*.rb'].each { |f|
next if File.directory?(f)
require (f)
}
i = 1
result = nil
$global_procs.each { |p|
puts "Module name: " + p['module']
eval(p['some_procedure']+'(i)')
result = eval(p['another_procedure']+'()')
puts result
i = i + 1
}
I am parsing a text file and want to be able to extend the sets of tokens that can be recognized easily. Currently I have the following:
if line =~ /!DOCTYPE/
puts "token doctype " + line[0,20]
#ast[:doctype] << line
elsif line =~ /<html/
puts "token main HTML start " + line[0,20]
html_scanner_off = false
elsif line =~ /<head/ and not html_scanner_off
puts "token HTML header starts " + line[0,20]
html_header_scanner_on = true
elsif line =~ /<title/
puts "token HTML title " + line[0,20]
#ast[:HTML_header_title] << line
end
Is there a way to write this with a yield block, e.g. something like:
scanLine("title", :HTML_header_title, line)
?
Don't parse HTML with regexes.
That aside, there are several ways to do what you're talking about. One:
class Parser
class Token
attr_reader :name, :pattern, :block
def initialize(name, pattern, block)
#name = name
#pattern = pattern
#block = block
end
def process(line)
#block.call(self, line)
end
end
def initialize
#tokens = []
end
def scanLine(line)
#tokens.find {|t| line =~ t.pattern}.process(line)
end
def addToken(name, pattern, &block)
#tokens << Token.new(name, pattern, block)
end
end
p = Parser.new
p.addToken("title", /<title/) {|token, line| puts "token #{token.name}: #{line}"}
p.scanLine('<title>This is the title</title>')
This has some limitations (like not checking for duplicate tokens), but works:
$ ruby parser.rb
token title: <title>This is the title</title>
$
If you're intending to parse HTML content, you might want to use one of the HTML parsers like nokogiri (http://nokogiri.org/) or Hpricot (http://hpricot.com/) which are really high-quality. A roll-your-own approach will probably take longer to perfect than figuring out how to use one of these parsers.
On the other hand, if you're dealing with something that's not quite HTML, and can't be parsed that way, then you'll need to roll your own somehow. There's a few Ruby parser frameworks out there that may help, but for simple tasks where performance isn't a critical factor, you can get by with a pile of regexps like you have here.