Writing an iterpreter in Ruby, not able to consume whitespace properly - ruby

Question

So I am following the "Writing an interpreter book" and implementing it in Ruby instead of Go. I am able to scan tokens like ; =, +, etc but it seems to behave differently when I have identifiers like let, 10, etc in my input string. Tried to hunt this bug this whole week but in vain so I thought a fresh pair of eyes might be able to catch it.
Here is an overview.
The codebase is very small and most of the logic resides in lib/lexer/lexer.rb
The class Lexer maintains the following state a cursor for the current character in the input string, a cursor for the next character and the current character in the input string
Lexer has the following methods
read_char which sets the data members to appropriate values
read_indentifier which is used to extract all the characters belonging to strings that are not reserved keywords but identifiers and call read_char before returning
read_number same as read_identifier but for numbers
consume_whitespace to skip over spaces, newlines, etc
next_token used to match the current character with the appropriate case and return its Token object defined in lib/token/token.rb and call read_char to increment the cursors before returning
require_relative '../token/token'
def is_letter(ch) #basically decides syntax acceptable for variable names
#puts ch.class
'a' <= ch && ch <= 'z' || 'A' <= ch && ch <= 'Z' || ch == '_'
end
def is_digit(ch) #checks if digit
'0' <= ch && ch <= '9'
end
class Lexer
def initialize(input)
#input = input
#position = 0
#readPosition = 0
#ch =''
read_char
end
def read_char
#puts caller[0]
#ch = #readPosition >= #input.length ? '' : #input[#readPosition]
#position = #readPosition
#readPosition += 1
#puts "INSIDE READ_CHAR #{#position} #{#readPosition} #{#ch}"
end
# SUPPOSED TO BE A LOOP WAS JUST A CONDITION. NOW FIXED.
def consume_whitespace
while #ch == ' ' || #ch =='\t' || #ch == '\n' || #ch == '\r' do
read_char
end
end
def read_identifier
pos = #position
#puts "RI: char #{#ch} pos #{pos} position #{#position}"
while is_letter(#ch) do
#puts #ch
read_char
end
puts "METHOD read_identifier: char #{#ch} pos #{pos} position #{#position}\n"
#input[pos..#position-1]
end
def read_number
pos = #position
#puts "RN: char #{#ch} pos #{pos} position #{#position}"
while is_digit(#ch) do
read_char
end
puts "METHOD read_number: char #{#ch} pos #{pos} position #{#position}\n"
#input[pos..#position-1]
end
def next_token
#puts #ch, #ch.class
#puts "\nX=X=X=X=X=X=X=X=X=: #{#ch}, #{#ch.ord}, X=X=X=X=X=X=X=X=X=\n"
tok = nil
consume_whitespace
tok =
case #ch
when '=' then Token.new(ASSIGN, #ch)
when '+' then Token.new(PLUS, #ch)
when '-' then Token.new(MINUS, #ch)
when '/' then Token.new(DIVIDE, #ch)
when '*' then Token.new(MULTIPLY, #ch)
when '%' then Token.new(MODULO, #ch)
#when '==' then Token.new(EQUAL_TO, #ch)
when '>' then Token.new(GREATER_THAN, #ch)
when '<' then Token.new(LESS_THAN, #ch)
#when '!=' then Token.new(UNEQUAL_TO, #ch)
#when '&&' then Token.new(AND, #ch)
#when '||' then Token.new(OR, #ch)
when '!' then Token.new(NOT, #ch)
when ',' then Token.new(COMMA, #ch)
when ';' then Token.new(SEMICOLON, #ch)
when '?' then Token.new(QUESTION, #ch)
when '(' then Token.new(LPAREN, #ch)
when ')' then Token.new(RPAREN, #ch)
when '[' then Token.new(LSQUARE, #ch)
when ']' then Token.new(RSQUARE, #ch)
when '{' then Token.new(LCURLY, #ch)
when '}' then Token.new(RCURLY, #ch)
else
#puts 'hello from next_token', #ch.ord
# STATE WAS BEING MUTATED NOW FIXED
puts "letter #{#ch}"
puts "letter ascii #{#ch.ord}"
#puts "isletter "
if is_letter(#ch)
literal = read_identifier
Token.new(look_up_ident(literal), literal)
elsif is_digit(#ch)
Token.new(INT, read_number)
else
Token.new(ILLEGAL, "ILLEGAL")
end
end
read_char
return tok
end
end
Now the rake test failures weren't helpful in debugging so I decided to do it simply by writing a main.rb script which would import and run my lexer and a sprinkled a lot of putss throughout the codebase
This is my main.rb
require_relative 'lib/lexer/lexer'
lex = Lexer.new('five = 5;
ten = 10;')
i = 1
while i <= 8
tok = lex.next_token
puts "\nIN_MAIN: #{tok.type} ==> #{tok.literal}\n\n"
i=i+1
end
This is the output of ruby main.rb
letter f
letter ascii 102
METHOD read_identifier: char pos 0 position 4
IN_MAIN: IDENTIFIER ==> five
IN_MAIN: = ==> =
letter 5
letter ascii 53
METHOD read_number: char ; pos 7 position 8
IN_MAIN: INT ==> 5
letter
letter ascii 10
IN_MAIN: ILLEGAL ==> ILLEGAL
letter t
letter ascii 116
METHOD read_identifier: char pos 27 position 30
IN_MAIN: IDENTIFIER ==> ten
IN_MAIN: = ==> =
letter 1
letter ascii 49
METHOD read_number: char ; pos 33 position 35
IN_MAIN: INT ==> 10
letter
Traceback (most recent call last):
2: from main.rb:8:in `<main>'
1: from /home/palash25/gundoochy/lib/lexer/lexer.rb:89:in `next_token'
/home/palash25/gundoochy/lib/lexer/lexer.rb:89:in `ord': empty string (ArgumentError)
We can ignore the last line because I am not able to handle how to return an object for EOF right now but here is the gist of what is happening before that
The lexer is able to scan the tokens correctly till five = 5 after that it skips over the next immediate character which was ; and does return a token object for that and instead returns a token object of ILLEGAL type for the \n that is right after ; (I even printed out the ascii values of the character to know for sure it was the \n returning and ILLEGAL)
This should not have happened since consume_whitespace is supposed to skip over all kinds of whitespace but it still didn't for newlines anyway after that we are able to scan the next line that is ten = 10 but the last semicolon is nowhere to be seen in the output just like the first one
If I use an input string without any identifier or number it works perfectly fine.
Here is the link to the full codebase https://gitlab.com/palash25/gundoochy

Answer 1

In your Lexer code (which you should include in your original question), you have the following method:
def consume_whitespace
while #ch == ' ' || #ch =='\t' || #ch == '\n' || #ch == '\r' do
read_char
end
end
Here, you attempt to specify various whitespace characters. However, since you have named them with single quotes, the escape sequences with the backslash are not applied. Instead, you consume a literal backslash followed by an t, n, or r character.
If you use double quotes here, the characters in your source code are interpreted as tab, newline, or carriage return characters respectively:
def consume_whitespace
while #ch == ' ' || #ch == "\t" || #ch == "\n" || #ch == "\r" do
read_char
end
end