Unable to iterate over multiple pages while web scraping - for-loop

Question

I am trying to scrape
https://www.maybank.co.id/others/locate-us?Keyword=&LocType=branch&LocSubType=all
to obtain branch name and address for all bank branches. There are 44 pages I need to scrape for which the url doesn't change. I cant iterate over the pages.
for page_no in range(1,45):
payload='page='+str(page_no)+'&PageSize=9&id=%7B5066AC98-FE40-407A-B4FE-03C814BED5F5%7D&keyword=&LocType=branch&LocSubType=all'
response = requests.post(url, data=payload)
page = requests.post(url,data=payload)
print('Page',page_no)
for i in soup.find_all('div',class_="col-md-4 col-sm-6 col-xs-12 property-item"):
Branch=i.find_all('h3') if i.find_all('h3') else ''
Address=i.find_all('p') if i.find_all('p') else ''
for j in Address:
j = re.sub(r'<(.*?)>', '', str(j))
j = j.strip()
Address_list.append(j)
for k in Branch:
k=re.sub(r'<(.*?)>', '', str(k))
Branch_list.append(k)
Can someone suggest should be done here?

Answer 1

You should use the API to get what you need.
Try this:
from urllib.parse import urlencode
import requests
from bs4 import BeautifulSoup
api_url = "https://www.maybank.co.id/api/sitecore/MapsLocation/MapsLocationListPaging?"
payload = {
"page": "44",
"id": "{5066AC98-FE40-407A-B4FE-03C814BED5F5}",
"keyword": "",
"LocType": "branch",
"LocSubType": "all",
}
headers = {
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36",
"x-requested-with": "XMLHttpRequest",
}
for page in range(1, 45):
payload["PageSize"] = page
page = requests.get(f"{api_url}{urlencode(payload)}", headers).text
soup = BeautifulSoup(page, "html.parser").find("div", {"class": "col-md-4 col-sm-6 col-xs-12 property-item"})
branch_data = [
soup.find("h3").getText(strip=True),
[p.getText(strip=True) for p in soup.find_all("p")],
soup.find("a")["href"],
]
print(branch_data)
Output:
['KC MANADO', ['Jl. Kawasan Mega Mas Jl. Pierre Tendean Boulevard Blok I C1 No. 24,25,26 dan Blok I C2 No. 27,28,29 Manado', 'Closed until 03.30 PM0431 - 860543'], '/others/locate-us/locate-us-detail?id=337&loctype=Branch&locsubtype=']
['KC SUNSET ROAD, DPS', ['Jl. Sunset Road No 811, Kuta - Badung, Bali', 'Closed until 03.30 PM0361 - 3003811'], '/others/locate-us/locate-us-detail?id=294&loctype=Branch&locsubtype=']
['KCP BSB CITY', ['Ruko Taman Niaga Bukit Semarang Baru (BSB) Blok E No. 3A, Semarang', 'Closed until 03.30 PM(024) 76670611'], '/others/locate-us/locate-us-detail?id=217&loctype=Branch&locsubtype=']
['KCP GRAHA IRAMA', ['Jl. HR Rasuna Said Kav. 1-2 Ground Floor Blok B Jakarta Selatan', 'Closed until 03.30 PM021-5261330-4'], '/others/locate-us/locate-us-detail?id=111&loctype=Branch&locsubtype=']
['KCP KLP. GADING BULEVARD II', ['Jl. Raya Boulevard I-3 no. 4, Jakarta', 'Closed until 03.30 PM021 - 4515253'], '/others/locate-us/locate-us-detail?id=199&loctype=Branch&locsubtype=']
['KCP PALM SPRING BATAM CENTER', ['Komplek Palm Spring BTC Blok D1 No. 10, Batam Centre', 'Closed until 03.30 PM0778 - 6053070'], '/others/locate-us/locate-us-detail?id=26&loctype=Branch&locsubtype=']
and so on...