Scraping separate sets of items from a website with Ruby & Nokogiri - ruby

Question

I'm working on a project for school and I'm having issues with finding the correct CSS selectors within the website's HTML in order to pull in the data I'm looking for. This is also my very first time with web scraping & I'm fairly new to Ruby as well so I apologize if this is a silly question.
I have successfully parsed the first set of data (although I'm sure there are better ways to do this, my method IS working but even feedback on this is welcome):
The website is platinumgod.co.uk for reference.
The HTML I scraped for the first part is as follows (along with the first item listed as an example):
<div class="repentanceitems-container">
<h2>
"Repentance Items "
<span class="rep-item-ttl">(169)</span>
</h2>
<li class="textbox" data-tid="42.5" data-cid="42" data-sid="263">
<a
<div onclick class="item reb-itm-new re-itm263"></div>
<span>
<p class="item-title">Clear Rune</p>
<p class="r-itemid">ItemID: 263</p>
<p class="pickup">"Rune mimic"</p>
<p class="quality">Quality: 2</p>
<p>"When used, copies the effect of the Rune or Soul stone you are holding (like the Blank Card)"</p>
<p>Drops a random rune on the floor when picked up</p>
<p>The recharge time of this item depends on the Rune/Soul Stone held:</p>
<p>1 room: Soul of Lazarus</p>
<p>2 rooms: Rune of Ansuz, Rune of Berkano, Rune of Hagalaz, Soul of Cain</p>
<p>3 rooms: Rune of Algiz, Blank Rune, Soul of Magdalene, Soul of Judas, Soul of ???, Soul of the Lost</p>
<p>4 rooms: Rune of Ehwaz, Rune of Perthro, Black Rune, Soul of Isaac, Soul of Eve, Soul of Eden, Soul of the Forgotten, Soul of Jacob and Esau</p>
<p>6 rooms: Rune of Dagaz, Soul of Samson, Soul of Azazel, Soul of Apollyon, Soul of Bethany</p>
<p>12 rooms: Rune of Jera, Soul of Lilith, Soul of the Keeper</p>
<ul>
<p>Type: Active</p>
<p>Recharge time: Varies</p>
<p>Item Pool: Secret Room, Crane Game</p>
</ul>
<p class="tags">* Secret Room</p>
</span>
</a>
</li>
This is just an example of one item in the Repentance Items category, so this is my code to parse all the information for each item in that category:
# Repentance Items
repentance_items = []
html.at(".repentanceitems-container").css("li.textbox").each do |item |
item_name = item.css("a span p.item-title").text
item_id = item.css("a span p.r-itemid").text.sub(/^ItemID: /, "")
pickup_text = item.css("a span p.pickup").text.gsub("\"", "")
quality = item.css("a span p.quality").text.sub(/^Quality: /, "")
use = item.css(".quality ~ p:not(.tags)").map { |row| row.text }
item_type = item.css("a span ul")
item.css("a span ul").each.map do |child|
item_type = child.css("p")[0].text.sub(/^Type: /, "")
if child.css("p")[1].text.match "Recharge time"
recharge_time = child.css("p")[1].text.sub(/^Recharge time: /, "")
item_pool = child.css("p")[2].text.sub(/^Item Pool: /, "").gsub(/,\s*$/m, "").split(", ")
else
recharge_time = "N/A"
item_pool = child.css("p")[1].text.sub(/^Item Pool: /, "").gsub(/,\s*$/m, "").split(", ")
end
repentance_items << {name: item_name, item_id: item_id, pickup_text: pickup_text, quality: quality, use: use, item_type: item_type, recharge_time: recharge_time, item_pool: item_pool}
end
end
The problem I'm facing is when I try to scrape the next category, which is Repentance Item Trinkets, I'm not sure what the CSS selectors should be in order to get this information because a lot of the same classes are used as in the Repentance Items HTML & so I just get the same items I did before. The HTML for the trinkets is the following (along with the first item listed as an example):
<div class="repentanceitems-container">
<h2>
"Repentance Trinkets "
<span class="a-item-ttl">(61)</span>
</h2>
<li class="textbox" data-tid="1000" data-cid="804" data-sid="10129">
<a
<div onclick class="item rep-item rep-trink rep-junxx129"></div>
<span>
<p class="item-title">Jawbreaker</p>
<p class="r-itemid">TrinketID: 129</p>
<p class="pickup">"Don't chew on it"</p>
<p>Tears have a chance to become a tooth, dealing x3.2 damage, similar to Tough Love</p>
<p>The chance to fire a tooth with this trinket is affected by your Luck stat</p>
<p>At +0 luck you have ~12% chance for this effect to activate</p>
<p>At +9 luck every tear you fire will be a tooth</p>
<p class="tags">*, </p>
</span>
</a>
</li>
I'm not sure where to begin in order to select only these items. If I go by the same selectors used in the first part of my code, it obviously just re-pulls in the Repentance Items & not the Trinkets.
Hopefully I've explained this well enough but please feel free to ask me more questions & I'll do my best to explain better.
Thank you all so much in advance for helping me!

Answer 1

Maybe you could start by breaking your first selector line in 2 parts: one to capture the container, and then another one to look for items.
This could look like this (not tested):
repentance_items = []
repentance_trinklets = []
html.at(".repentanceitems-container").each do |container|
# Check to know in what category you are, so in which table to add the results, something like:
repentance_target = if container.css('h2').text =~ /items/i
repentance_items
else
repentance_trinklets
end
css("li.textbox").each do |item|
# your current logic
# affectation in the correct results array
repentance_target << ...
end
end
In the end the two arrays should be populated with the correct items
This is a bit drafty, but I hope that helps, let me know if something is not clear