Scrapy ajax POST request not working, though working in Postman - ajax

Question

I am implementing a scrapy spider to crawl a website that contains real estate offers. The site contains a telephone number to the real estate agent, which can be retreived be an ajax post request. The request yielded by scrapy returns an error from the server, while the same request sent from Postman returns the desired data.
Here's the site URL: https://www.otodom.pl/oferta/piekne-mieszkanie-na-mokotowie-do-wynajecia-ID3ezHA.html
I recorded the request using Network tab in chrome's dev tools. The url of the ajax request is: enter link description here The data needed to send the request is the CSRFtoken contained in the page's source, which changes periodically. In Postman giving only the CSRFtoken as form-data gives an expected answer.
This is how I construct the request in scrapy:
token_input = response.xpath('//script[contains(./text(), "csrf")]/text()').extract_first()
csrf_token = token_input[23:-4]
offerID_input = response.xpath('//link[#rel="canonical"]/#href').extract_first()
offerID = (offerID_input[:-5])[-7:]
form_data = {'CSRFToken' : csrf_token}
request_to_send = scrapy.Request(url='https://www.otodom.pl/ajax/misc/contact/phone/3ezHA/', headers = {"Content-Type" : "application/x-www-form-urlencoded"}, method="POST", body=urllib.urlencode(form_data), callback = self.get_phone)
yield request_to_send
Unfortunately, I get an error, though everything should be ok. Does anybody have any idea what might be the problem? Is is maybe connected with encoding? The site uses utf-8.

Answer 1

You can find the token in page source:
<script type="text/javascript">
var csrfToken = '0ec80a520930fb2006e4a3e5a4beb9f7e0d6f0de264d15f9c87b572a9b33df0a';
</script>
And you can get it quite easily with this regular expression:
re.findall("csrfToken = '(.+?)'", response.body)
To get the whole thing you can use scrapy's FormRequest which can make a correct post request for you:
def parse(self, response):
token = re.findall("csrfToken = '(.+?)'", response.body)[0]
yield FormRequest('https://www.otodom.pl/ajax/misc/contact/phone/3ezHA/',
formdata={'CSRFToken': token},
callback=self.parse_phone)
def parse_phone(self, response):
print(response.body)
#'{"value":"515 174 616"}'
You can debug your scrapy requests by insersting inspect_response call and looking into request object:
def parse_phone(self, response):
from scrapy.shell import inspect_response
inspect_response(response, self)
# shell opens up here and spider is put on pause
# now check `request.body` and `request.headers`, match those to what you see in your browser