Why does xpath inside a selector loop still return a list in the tutorial - xpath

Question

I am learning scrapy with the tutorial: http://doc.scrapy.org/en/1.0/intro/tutorial.html
When I run the following example script in the tutorial. I found that even though it was already looping through the selector list, the tile I got from sel.xpath('a/text()').extract() was still a list, which contained one string. Like [u'Python 3 Object Oriented Programming'] rather than u'Python 3 Object Oriented Programming'. In a later example the list is assigned to item as item['title'] = sel.xpath('a/text()').extract(), which I think is not logically correct.
import scrapy
class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
]
def parse(self, response):
for sel in response.xpath('//ul/li'):
title = sel.xpath('a/text()').extract()
link = sel.xpath('a/#href').extract()
desc = sel.xpath('text()').extract()
print title, link, desc
However if I use the following code:
import scrapy
class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/",
]
def parse(self, response):
for href in response.css("ul.directory.dir-col > li > a::attr('href')"):
link = href.extract()
print(link)
the link is a string rather than a list.
Is this a bug or intended?

Answer 1

.xpath().extract() and .css().extract() return a list because .xpath() and .css() return SelectorList objects.
See https://parsel.readthedocs.org/en/v1.0.1/usage.html#parsel.selector.SelectorList.extract
(SelectorList) .extract():
Call the .extract() method for each element is this list and return their results flattened, as a list of unicode strings.
.extract_first() is what you are looking for (which is poorly documented)
Taken from http://doc.scrapy.org/en/latest/topics/selectors.html :
If you want to extract only first matched element, you can call the selector .extract_first()
>>> response.xpath('//div[#id="images"]/a/text()').extract_first()
u'Name: My image 1 '
In your other example:
def parse(self, response):
for href in response.css("ul.directory.dir-col > li > a::attr('href')"):
link = href.extract()
print(link)
each href in the loop will be a Selector object. Calling .extract() on it will get you a single Unicode string back:
$ scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/"
2016-02-26 12:11:36 [scrapy] INFO: Scrapy 1.0.5 started (bot: scrapybot)
(...)
In [1]: response.css("ul.directory.dir-col > li > a::attr('href')")
Out[1]:
[<Selector xpath=u"descendant-or-self::ul[#class and contains(concat(' ', normalize-space(#class), ' '), ' directory ') and (#class and contains(concat(' ', normalize-space(#class), ' '), ' dir-col '))]/li/a/#href" data=u'/Computers/Programming/Languages/Python/'>,
<Selector xpath=u"descendant-or-self::ul[#class and contains(concat(' ', normalize-space(#class), ' '), ' directory ') and (#class and contains(concat(' ', normalize-space(#class), ' '), ' dir-col '))]/li/a/#href" data=u'/Computers/Programming/Languages/Python/'>,
...
<Selector xpath=u"descendant-or-self::ul[#class and contains(concat(' ', normalize-space(#class), ' '), ' directory ') and (#class and contains(concat(' ', normalize-space(#class), ' '), ' dir-col '))]/li/a/#href" data=u'/Computers/Programming/Languages/Python/'>]
so .css() on the response returns a SelectorList:
In [2]: type(response.css("ul.directory.dir-col > li > a::attr('href')"))
Out[2]: scrapy.selector.unified.SelectorList
Looping on that object gives you Selector instances:
In [5]: for href in response.css("ul.directory.dir-col > li > a::attr('href')"):
...: print href
...:
<Selector xpath=u"descendant-or-self::ul[#class and contains(concat(' ', normalize-space(#class), ' '), ' directory ') and (#class and contains(concat(' ', normalize-space(#class), ' '), ' dir-col '))]/li/a/#href" data=u'/Computers/Programming/Languages/Python/'>
<Selector xpath=u"descendant-or-self::ul[#class and contains(concat(' ', normalize-space(#class), ' '), ' directory ') and (#class and contains(concat(' ', normalize-space(#class), ' '), ' dir-col '))]/li/a/#href" data=u'/Computers/Programming/Languages/Python/'>
(...)
<Selector xpath=u"descendant-or-self::ul[#class and contains(concat(' ', normalize-space(#class), ' '), ' directory ') and (#class and contains(concat(' ', normalize-space(#class), ' '), ' dir-col '))]/li/a/#href" data=u'/Computers/Programming/Languages/Python/'>
And calling .extract() gives you a single Unicode string:
In [6]: for href in response.css("ul.directory.dir-col > li > a::attr('href')"):
print type(href.extract())
...:
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
Note: .extract() on Selector is wrongly documented as returning a list of strings. I'll open an issue on parsel (which is the same as Scrapy selectors, and used under the hood in scrapy 1.1+)