I know there's lots of similar questions but I haven't found a solution yet. I'm trying to use the CSV parsing library with Ruby 1.9.1 but I keep getting:
/usr/lib/ruby/1.9.1/csv.rb:1925:in `block (2 levels) in shift': Illegal quoting in line 1. (CSV::MalformedCSVError)
My CSV files were created in Windows 7 but it's Ubuntu 12.04 that I'm using to run the Ruby script, which looks like this:
require 'csv'
CSV.foreach('out.csv', :col_sep => ';') do |row|
puts row
end
Nothing complicated, just a test, so I assumed it must be the Windows control characters causing problems. Vim shows up this:
"Part 1";;;;^M
;;;;^M
;;;;^M
Failure to Lodge Income Tax Return(s);;;;^M
NAME;ADDRESS;OCCUPATION;"NO OF CHARGES";"FINE/PENALTY £"^M
some name;"some,address";Bookkeeper;3;1,250.00^M
some name;"some,address";Haulier;1;600.00^M
some name;"some,address";Scaffolding Hire;1;250.00^M
some name;"some,address";Farmer;2;500.00^M
some name;"some,address";Builder;2;3000.00
I've tried removing those control characters for carraige returns that Windows added (^M), but %s/^V^M//g and %s/^M//g result in no pattern found. If I run %s/\r//g then the ^M characters are removed, but the same error still persists when I run the Ruby script. I've also tried running set ffs=unix,dos but it has no effect. Thanks.
Update:
If I remove the double quotes around the Part 1 on the first line, then the script prints out what it should and then throws a new error: Unquoted fields do not allow \r or \n (line 10). If I then remove the \r characters, the script runs fine.
I understand that I would have to remove the \r characters, but why will it only work if I unquote the first value?
The problem causing the Illegal quoting error was due to a Byte-Order-Mark (BOM) at the very beginning of the file. It didn't show up in editors, but the Ruby CSV lib was choking on it unless :encoding => 'bom|utf-8' was set.
Once that was fixed, I still needed to remove all the '^M' characters by running %s/\r//g in vim. And everything was working fine after that.
Related
This is killing me. I have a config file, "myconfig.cfg", with the following content:
SOME_VAR=2
echo "I LOVE THIS"
Then I have a script that I'm trying to run, that sources the config file in order to use the settings in there as variables. I can print them out fine, but when I try to put one into a numeric variable for use in something like a "seq " command, I get this weird "invalid arithmetic operator" error.
Here's the script:
#!/bin/bash
source ./myconfig.cfg
echo "SOME_VAR=${SOME_VAR}"
let someVarNum=${SOME_VAR}
echo "someVarNum=${someVarNum}"
And here's the output:
I LOVE THIS
SOME_VAR=2
")syntax error: invalid arithmetic operator (error token is "
someVarNum=
I've tried countless things that theoretically shouldn't make a difference, and, surprise, they don't. I simply can't figure it out. If I simply take the line "SOME_VAR=2" and put it directly into the script, everything's fine. I'm guessing I'll have to read in the config file line by line, split the strings by "=", and find+create the variables I want to use manually.
The error is precisely as indicated in a comment by #TomFenech. The first line (and possibly all the lines) in myconfig.cfg is terminated with a Windows CR-LF line ending. Bash considers CR to be an ordinary character (not whitespace), so it will set SOME_VAR to the two character string 2CR. (CR is the character with hex code 0x0D. You could see that if you display the file with a hex-dumper: hd myconfig.cfg.)
The let command performs arithmetic on numbers. It also considers the CR to be an ordinary character, but it is neither a digit nor an operator so it complains. Unfortunately, it does not make any attempt to sanitize the display of the character in the error message, so the carriage return is displayed between the two " symbols. Consequently, the end of the error message overwrites the beginning.
Don't create Unix files with a Windows text editor. Or use a utility like dos2unix to fix them once you copy them to the Unix machine.
The new Mac OS update moved the system Ruby up to 2.0, which is great, but now I'm seeing errors in a lot of my scripts that I don't know how to fix. Specifically, I had code that called for files using mdfind and then read them, like this:
files = %x{mdfind -onlyin /Users/Username/Dropbox/Tasks 'kMDItemContentModificationDate >= "$time.today(-1)"'}
files.each do |file|
Now I'm getting an error that says
undefined method `each' for #<String:0x007f83521865c8> (NoMethodError)"
It seems as if each now needs a qualifier. I tried each_line but that yielded additional errors down the line. Is there a simple replacement for this that I'm overlooking?
Ruby 1.8 used to have String#each which was doing implicit splitting.
each(separator=$/) {|substr| block } => str
Splits str using the supplied parameter as the record separator ($/ by default), passing each substring in turn to the supplied block. If a zero-length record separator is supplied, the string is split into paragraphs delimited by multiple successive newlines.
Explicit splitting should work in modern rubies, I believe.
files.split($/).each do |file|
Where $/ is newline char. You can use explicit char, since your script is not portable anyway.
files.split("\n").each do |file|
Update
or you can just use an alias of now-extinct each
files.each_line do |file|
Running d:\ruby\test.rb is always successful.
Running the copy of this file path which is at d:\программирование\test.rb fails, apparently because it contains non-ASCII, Cyrillic in this case, symbols:
No such file or directory - D:\... (Errno::ENOENT)
What should I do to make it work?
I'm using Ruby 1.9 and Windows.
If you are creating your path like:
"d:\программирование\test.rb"
Then Ruby is treating the "\t" character as if is is escaped: It is converting \t into a tab before passing the filename to any routine. That character is illegal in a filename. Well, maybe not illegal, but a real pain to deal with and not what you expect.
Instead, use:
'd:\программирование\test.rb'
Or, better yet, let Ruby do the lifting and reverse your backslashes when you define the name. Ruby should do the right thing and convert them on the fly for you:
"d:/программирование/test.rb"
When I was developing my project on my local file I had this line in code which worked correctly:
#json = Location.qty_of_deliv_from(params[:from_rc])
.qty_of_deliv_to(params[:to_rc])
When I deployd with passenger I get a syntax error on this line which goes avay if I have all the code in the same line:
#json = Location.qty_of_deliv_from(params[:from_rc]).qty_of_deliv_to(params[:to_rc])
Is this a known issue?
Perhaps your server's ruby version is different and parses differently?
In any case, in Ruby, when writing multiline code you typically want to make sure your lines to be wrapped are syntactically incomplete, so as not to confuse the parser, e.g. by using a hanging dot, instead.
Location.qty_of_deliv_from(params[:form_rc]).
qty_of_deliv_to(params[:to_rc])
Or you can use the backslash to escape the new line:
Location.qty_of_deliv_from(params[:form_rc]) \
.qty_of_deliv_to(params[:to_rc])
I'm using Ruby's CSV library to parse some CSV. I have a seemingly well-formed CSV file that I created by exporting an Excel file as CSV.
However CSV.open(filename, 'r') causes a CSV::IllegalFormatError.
There are no rogue commas or quotation marks in the file, nor anything else that I can see that might cause problems.
I suspect the problem could be to do with line endings. I am able to parse data entered manually via a text editor (Aquamacs). It is just when I try with data exported from Excel (for OS X) that problems occur. When I open up the exported CSV in vim, all the text appears on one line, with ^M appearing between lines.
From the docs, it seems that you can provide open with a row separator; however I am unsure what it should be in this case.
Try: CSV.open('filename', 'r', ?,, ?\r)
As cantlin notes, for Ruby 2 it's:
CSV.new('file.csv', 'r', :col_sep => ?,, :row_sep => ?\r)
I'm pretty sure these will DTRT for you. You can also "fix" the file itself (in which case keep the old open) with the following vim command: :%s/\r/\r/g
Yes, I know that command looks like a total no-op, but it will work.
Stripping \r characters seemed to work for me
CSV.parse(File.read('filename').gsub(/\r/, '')) do |row|
...
end
Another option is to open the CSV file or the original spreadsheet in Excel and save it as "Windows Comma Separated" rather than "Comma Separated Values". This will output the file with line endings that FasterCSV is able to understand.
"""
When I open up the exported CSV in vim, all the text appears on one line, with ^M appearing between lines.
From the docs, it seems that you can provide open with a row separator; however I am unsure what it should be in this case.
"""
Read back a sentence ... ^M means keyboard Ctrl-M aka '\x0D' (M is the 13th letter of the ASCII alphabet; 0x0D == 13) aka ASCII CR (carriage return) aka '\r' ... IOW what Macs used to use as a line terminator before OS X.
It seems newer versions of the CSV parser and/or any component it uses read DOS/Windows line endings without issues. Mac OS X's stock one (not sure the version) was not cutting it, installed Ruby 2.0.0 and it parsed the file just fine, without the special arguments...
I had similar problem. I got an error:
"error_message"=>"Illegal quoting in line 1.", "error_class"=>"CSV::MalformedCSVError"
The problem was the file had Windows line endings, which are of course other than Unix. What helped me was defining row_sep: "\r\n":
CSV.open(path, 'w', headers: :first_row, col_sep: ';', quote_char: '"', row_sep: "\r\n")