Scrapy: Fetch HTML class as string and/or int? - xpath

Question

For educative purposes I've been creating a spider to fetch data from a HTML-based marketplace. I managed to fetch all the text based data I need, however, I need to fetch the itemID, the data-mins-elapsed, the item quality and item trait. These items are not text based, but they are HTML classes and such.
The ItemID is a unique ID for each item. In the HTML code of the website it can be found under: (I need the number "319842588", this number is unique to each item)
<tr class="cursor-pointer" data-on-click-link="/pc/Trade/Detail/319842588" data-on-click-link-action="NewWindow" data-toggle="tooltip" data-original-title="" title="">
The data-mins-elapsed keeps track of when the item has been posted. This number will change everytime you refresh the webpage as time goes by. It can be found under: (I need the number "3", this number will change constantly)
<td class="bold hidden-xs" data-mins-elapsed="3">Now</td>
The itemquality is the quality of a certain item. In the HTML code of the website it can be found under: (I need the "superior", the quality is unique to each item)
<img class="trade-item-icon item-quality-superior"
alt="Icon"
src="/Content/icons/bow.png"
data-trait="Infused"
/>
The itemtrait is the trait of a certain item. In the HTML code of the website it can be found under: (I need the "Infused", the trait is unique to each item)
<img class="trade-item-icon item-quality-superior"
alt="Icon"
src="/Content/icons/bow.png"
data-trait="Infused"
/>
How do I build a XPATH or something similar to fetch these numbers?
Website link: https://eu.tamrieltradecentre.com/pc/Trade/SearchResult?SearchType=Sell&ItemID=10052&ItemNamePattern=Briarheart+Bow&IsChampionPoint=true&LevelMin=160&LevelMax=&ItemCategory1ID=&ItemCategory2ID=&ItemCategory3ID=&ItemQualityID=&ItemTraitID=3&PriceMin=&PriceMax=25000
Part of the relevant HTML code
This includes the HTML for each product, each product is listed in a TR with tha class "cursor-pointer"
<table class="trade-list-table max-width">
<thead>
...
</thead>
<tr class="cursor-pointer" data-on-click-link="/pc/Trade/Detail/319836098"
data-on-click-link-action="NewWindow" data-toggle="tooltip">
<td>
<img class="trade-item-icon item-quality-superior"alt="Icon"
src="/Content/icons/bow.png"data-trait="Infused"/>
<div class="item-quality-superior">
Briarheart Bow
</div>
<div>
Level:
<img class="small-icon" src="/Content/icons/championPoint.png" />
160
</div>
</td>
<td class="hidden-xs">
...
</td>
<td class="hidden-xs">
...
</td>
<td class="gold-amount bold">
...
</td>
<td class="bold hidden-xs" data-mins-elapsed="15"></td>
</tr>
Spider file
# -*- coding: utf-8 -*-
import scrapy
import os
import csv
class TTCSpider(scrapy.Spider):
name = "ttc_spider"
allowed_domains = ["eu.tamrieltradecentre.com"]
start_urls = ['https://eu.tamrieltradecentre.com/pc/Trade/SearchResult?ItemID=10052&SearchType=Sell&ItemNamePattern=Briarheart+Bow&ItemCategory1ID=&ItemCategory2ID=&ItemCategory3ID=&ItemTraitID=3&ItemQualityID=&IsChampionPoint=true&IsChampionPoint=false&LevelMin=160&LevelMax=&MasterWritVoucherMin=&MasterWritVoucherMax=&AmountMin=&AmountMax=&PriceMin=&PriceMax=25000']
def start_requests(self):
"""Read keywords from keywords file amd construct the search URL"""
with open(os.path.join(os.path.dirname(__file__), "../resources/keywords.csv")) as search_keywords:
for keyword in csv.DictReader(search_keywords):
search_text=keyword["keyword"]
url="https://eu.tamrieltradecentre.com/pc/Trade/{0}".format(search_text)
# The meta is used to send our search text into the parser as metadata
yield scrapy.Request(url, callback = self.parse, meta = {"search_text": search_text})
def parse(self, response):
containers = response.css('.cursor-pointer')
for container in containers:
#Defining the XPAths
XPATH_ITEM_NAME = ".//td[1]//div[1]//text()"
XPATH_ITEM_LEVEL = ".//td[1]//div[2]//text()"
XPATH_ITEM_LOCATION = ".//td[3]//div[1]//text()"
XPATH_ITEM_TRADER = ".//td[3]//div[2]//text()"
XPATH_ITEM_PRICE = ".//td[4]//text()[2]"
XPATH_ITEM_QUANTITY = ".//td[4]//text()[4]"
XPATH_ITEM_LASTSEEN = "Help me plis :3"
XPATH_ITEM_ITEMID = "Help me plis :3"
XPATH_ITEM_QUALITY = "Help me plis :3"
XPATH_ITEM_TRAIT = "Help me plis :3"
#Extracting from list
raw_item_name = container.xpath(XPATH_ITEM_NAME).extract()
raw_item_level = container.xpath(XPATH_ITEM_LEVEL).extract()
raw_item_location = container.xpath(XPATH_ITEM_LOCATION).extract()
raw_item_trader = container.xpath(XPATH_ITEM_TRADER).extract()
raw_item_price = container.xpath(XPATH_ITEM_PRICE).extract()
raw_item_quantity = container.xpath(XPATH_ITEM_QUANTITY).extract()
raw_item_lastseen = container.xpath(XPATH_ITEM_LASTSEEN).extract()
raw_item_itemid = container.xpath(XPATH_ITEM_ITEMID).extract()
raw_item_quality = container.xpath(XPATH_ITEM_QUALITY).extract()
raw_item_trait = container.xpath(XPATH_ITEM_TRAIT).extract()
#Cleaning the data
item_name = ''.join(raw_item_name).strip() if raw_item_name else None
item_level = ''.join(raw_item_level).replace('Level:','').strip() if raw_item_level else None
item_location = ''.join(raw_item_location).strip() if raw_item_location else None
item_trader = ''.join(raw_item_trader).strip() if raw_item_trader else None
item_price = ''.join(raw_item_price).strip() if raw_item_price else None
item_quantity = ''.join(raw_item_quantity).strip() if raw_item_quantity else None
item_lastseen = ''.join(raw_item_lastseen).strip() if raw_item_lastseen else None
item_itemid = ''.join(raw_item_itemid).strip() if raw_item_itemid else None
item_quality = ''.join(raw_item_quality).strip() if raw_item_quality else None
item_trait = ''.join(raw_item_trait).strip() if raw_item_trait else None
yield {
'item_name':item_name,
'item_level':item_level,
'item_location':item_location,
'item_trader':item_trader,
'item_price':item_price,
'item_quantity':item_quantity,
'item_lastseen':item_lastseen,
'item_itemid':item_itemid,
'item_quality':item_quality,
'item_trait':item_trait,
}

Answer 1

You can use built-in .re_first() to match regular expression for ItemID:
ItemID = container.xpath('./#data-on-click-link').re_first(r'(\d+)$') # same code for ItemQuality
ItemTrait = container.xpath('.//img[#data-trait]/#data-trait').get()

Answer 2

First of all you shouldn't be asking such questions, a simple google search should suffice. Nonetheless all you need is way to access data available in the attributes of a HTML Node. The way is using # as a prefix to attribute name. e.g: for accessing class attribute you would use div/#class.
For your problem I could suggest a XPath for one of your item, you should be able to take on from that.
XPATH_ITEM_LASTSEEN = ".//td[4]/#data-mins-elapsed"
Also, for getting 319842588 out of data-on-click-link="/pc/Trade/Detail/319842588", you can use XPATH similar to above in addition to python's inbuilt functions like replace() or split() to get the desired data. for example:
suppose you have -
x = "/pc/Trade/Detail/319842588"
# you could do something like
x = x.replace('/pc/Trade/Detail/','') OR x = x.split('/')[-1]
Hope that helps.
Cheers!!