Regex a regexed match in 1 search? Other minor regex questions - ruby

Question

I have an email that has some html code that I'm looking to regex. I'm using a gmail gem to read my emails and using nokogiri fails when reading through gmail. Thus I'm looking for a regex solution
What I'd like to do is to scan for the section that is labeled important title and then look at the unordered list within that section, capturing the urls. The html code that is labeled important title is provided below.
I wasn't sure how to do this so I thought the proper way to do it, was to regex for the section called important title and capture everything up to the end of the unordered list. Then within this match, subsequently find the links.
To find the links, I used this regex which works fine: (?:")([^"]*)(?:" )
To capture the section called important title however, I wanted to simply use the following regex (?:important title).*(?:<\/ul>). From my understanding that would look for important title then as many characters as possible, followed by </ul>. However from the below, it only captures </h3>. The new line character is causing it to stop. Which is one of my questions: why is . which is supposed to capture all characters, not capturing a new line character? If that's by design, I don't need more than a simply 'its by design'...
So assuming it's by design, I then tried (?:important title)((.|\s)*)(?:<\/ul>) and that's giving me 2 matches for some reason. The first matches the entire code that I need, stopping at </ul> and the second match is literally just a blank string. I don't get why that's the case...
Finally my last and most important question is, do I need to do 2 regexes to get the links? Or is there a way to combine both regexes so that my "link regex" only searches within my "section regex"?
<h3>the important title </h3>
<ul>
<li><a href="http://www.link.com/23232=
.32434" target="_blank">first link»</a></li>
<li><a href="http://www.link.com/234234468=
.059400" target="_blank">second link »</a></li>
<li><a href="http://www.link.com/287=
.059400" target="_blank">third link»</a></li>
<li><a href="http://www.link.com/4234501=
.059400" target="_blank">fourth link»</a></li>
<li><a href="http://www.link.com/34517=
.059400" target="_blank">5th link»</a></li>
</ul>

Answer 1

An example with nokogiri:
# encoding: utf-8
require 'nokogiri'
html_doc = '''
<h3>the important title </h3>
<ul>
<li>first link»</li>
<li>second link »</li>
<li>third link»</li>
<li>fourth link»</li>
<li>5th link»</li>
</ul>
'''
doc = Nokogiri::HTML.parse(html_doc)
doc.search('//h3[text()="the important title "]/following-sibling::ul[1]/li/a/#href').each do |link|
puts link.content
end
The regex way use the anchor \G that matches the position at the end of the precedent match, since this anchor is initialized to the start of the string at the begining, you must add (?!\A) (not a the start of the string) to forbid this case, and only allow the first match with the second entry point.
To be more readable, all the pattern use the extended mode (or verbose mode, or comment mode, or free-spacing mode...) that allows comments inside the pattern and where spaces are ignored. This mode can be set or unset inline with (?x) and (?-x)
pattern = Regexp.new('
# entry points
(?:
\G (?!\A) # contiguous to the precedent match
|
<h3> \s* (?-x)the important title(?x) \s* </h3> \s* <ul> \s*
)
<li>
<a \s+ href=" (?<url> [^"]* ) " [^>]* >
(?<txt> (?> [^<]+ | <(?!/a>) )* )
\s* </a> \s* </li> \s*', Regexp::EXTENDED | Regexp::IGNORECASE)
html_doc.scan(pattern) do |url, txt|
puts "\nurl: #{url}\ntxt: #{txt}"
end
The first match uses the second entry point: <h3> \s* (?-x)the important title(?x) \s* </h3> \s* <ul> \s* and all next matches use the second: \G (?!\A)
After the last match, since there is no more contiguous li tags (there is only a closing ul tag), the pattern fails. To succeed again the regex engine will find a new second entry point.

Answer 2

I have html that I'm looking to regex.
Use the nokogiri gem: http://nokogiri.org/
It's the defacto standard for searching html. Ignore the requirements that are listed--they are out of date.
require 'nokogiri'
require 'open-uri'
#doc = Nokogiri::HTML(open('http://www.some_site.com'))
html_doc = Nokogiri::HTML(<<'END_OF_HTML')
<h3>not important</h3>
<ul>
<li>first link»</li>
<li>second link »</li>
</ul>
<h3>the important title </h3>
<ul>
<li>first link</li>
<li>second link</li>
<li>third link</li>
<li>fourth link</li>
<li>5th link</li>
</ul>
END_OF_HTML
a_tags = html_doc.xpath(
'//h3[text()="the important title "]/following-sibling::ul[1]//a'
)
a_tags.each do |tag|
puts tag.content
puts tag['href']
end
--output:--
first link
http://www.link.com/23232=.32434
second link
http://www.link.com/234234468=.059400
third link
http://www.link.com/287=.059400
fourth link
http://www.link.com/4234501=.059400
5th link
http://www.link.com/34517=.059400