Using Request Header and Body information for Ajax query with scrapy - ajax

Question

I am scraping the following web site https://www2.woolworthsonline.com.au
My first step is to get a list of product categories, which I can do.
Then I need to get sub-categories. The sub-categories seem to be dynamically generated. I have set up what I thought to be the right header and body information for the request and a small test call-back to see if the request is working correctly. My code never gets to the callback, so presumably there is something wrong with the request
def parse(self, response):
# Get links to the categories
categories_links = response.xpath('//a[#class="navigation-link"]').re(r'href="(\S+)"')
for link in categories_links:
# Generate and yield a request for the sub-categories
get_request_headers = dict()
get_request_headers['Accept'] = 'application/xml, text/xml, */*; q=0.01'
get_request_headers['Accept-Encoding'] = 'gzip, deflate, sdch'
get_request_headers['Accept-Language'] = 'en-US,en;q=0.8'
get_request_headers['Connection'] = 'keep-alive'
get_request_body = urllib.urlencode(
{'_mode' : 'ajax',
'_ajaxsource' : 'navigation-panel',
'_referrer' : link,
'_bannerViews' : '6064',
'_' : '1429428492880'}
)
url_link = 'https://www2.woolworthsonline.com.au'+link
yield Request(url=url_link, callback=self.subcategories, headers = get_request_headers, method='POST', body = get_request_body, meta={'category_link' : link} )
return
def subcategories(self, response):
print "sub-categories test: ", response.url
return

Answer 1

Try this out,
BASE_URL = 'https://www2.woolworthsonline.com.au'
def parse(self, response):
categories = response.xpath(
'//a[#class="navigation-link"]/#href').extract()
for link in categories:
yield Request(url=self.BASE_URL + link, callback=parse_subcategory)
def parse_subcategory(self, response):
sub_categories = response.xpath(
'//li[#class="navigation-node navigation-category selected"]/ul/li/span/a/#href').extract()
for link in sub_categories:
yield Request(url=self.BASE_URL + link, callback=parse_products)
def parse_products(self, response):
# here you will get the list of products based on the subcategory
# extract the product details here
First we extract the category list from the start-url in parse function
Then give request to extracted category-url ( we have to provide BASE_URL along with the extracted category-url, since that urls are given as relative urls not the absolute one)
In the callback function (here parse_subcategory) we can get the response of each category-url
Likewise we are doing the same thing for sub-categories also
Finally extract the product details