Ruby HTMLish tokenizer - ruby

Question

I'm looking for a resource for tokenizing HTMLish markup. I'm creating a markup language that is a lot like (but isn't) HTML. All I want is something that can parse it up into tags, text, comments, etc. I don't need the tokens to be arranged into a tree structure or checked if they're valid tags or whatever - I'll do that myself.
So, for example, if given this string:
hello <x> dude <whatever></x>
it would return an array something like this:
hello
<x>
dude
<whatever>
</x>
It could also return objects representing those strings. Either would be cool.
I've looked into Nokogiri and Oga, but they seem to just want to parse and tree HTML. Suggestions?

Answer 1

If you're willing to do much of the validation yourself, could a regular expression work? Something like:
html = 'hello <x> dude <whatever></x>'
html.split(/(<[^<>]+>)/)
#=> ["hello ", "<x>", " dude ", "<whatever>", "", "</x>"]
Otherwise, I wonder: could your markup be XMLish rather than HTMLish? For example, do you need to support void elements like <whatever>, or would it be enough to support self-closing tags like <whatever />? That is, are you committed to supporting markup like hello <x> dude <whatever></x>, or would supporting hello <x> dude <whatever /></x> (with the self-closing <whatever />) be enough?
If self-closing tags are enough, it sounds like an XML parser could do the trick. Even if the parser builds a tree, you can usually flatten that into an array.
If you need custom void elements, you may need to find an HTML parser that supports those. I don't know any offhand, but it should be possible to modify Oga to do that. You could also modify Oga to support flattening a tree into an array. Something like:
module Oga
module XML
# Redefine the list of void elements.
remove_const :HTML_VOID_ELEMENTS
const_set :HTML_VOID_ELEMENTS, Whitelist.new(%w{
whatever
})
class TokenGenerator < Generator
def initialize(*args)
super
#tokens = []
end
%i[
on_element on_text on_cdata on_comment on_xml_declaration
on_processing_instruction on_doctype on_document
after_element
].each do |method|
define_method method do |content, output|
token = super(content, '')
#tokens << token if token
super(content, output)
end
end
def to_tokens
#tokens = []
to_xml
#tokens
end
end
end
end
html = Oga.parse_html('hello <x> dude <whatever></x>')
Oga::XML::TokenGenerator.new(html).to_tokens
=> ["hello ", "<x>", " dude ", "<whatever>", "</x>"]