Handling pagination and infinite scroll

The idea

Sites rarely show everything at once. Listings are split up in one of three ways:

1. Numbered pages with a Next link (?page=2).
2. A Load more button that appends items to the page.
3. Infinite scroll: new items load as you scroll down.

Each needs a loop with a clear stop condition. Getting the stop condition wrong is the classic bug: either you miss the last page, or the loop never ends.

1. Following the Next link

The shop's product list is split over several pages, with a "Next" link at the bottom of each. The most reliable way to collect everything is to do what a person would: scrape the page, click Next, and repeat until there's no Next link left:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/products")

    names = []
    while True:
        names += page.locator(".product-card .title").all_inner_texts()
        next_link = page.locator("a.next")
        if next_link.count() == 0:          # on the last page, "Next" is not a link
            break
        next_link.click()
    print(len(names), names[0], "...", names[-1])
    print(page.url)
    browser.close()

36 Galaxy A55 5G ... Fitbit Inspire 3
https://nusantara-gadget.test/products?page=3

The stop condition comes from the page itself: on the last page, Next is a disabled <span>, not an <a>. Following the site's own links is the most robust approach, because it works however the URLs are built.

A trap with URL patterns

Looping over ?page=1, 2, 3... yourself is tempting. But what happens after the last page?

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    for n in (3, 4, 50):
        page.goto(f"https://nusantara-gadget.test/products?page={n}")
        print(n, "->", page.locator(".result-count").inner_text(), "|", page.locator(".product-card .title").first.inner_text())
    browser.close()

3 -> Showing 12 of 36 products (page 3 of 3) | K380 Keyboard
4 -> Showing 12 of 36 products (page 3 of 3) | K380 Keyboard
50 -> Showing 12 of 36 products (page 3 of 3) | K380 Keyboard

This site quietly shows the last page again for any page number that's too high. A loop that stops "when a page has no products" would run forever, collecting duplicates. Sites behave differently (some return 404, some an empty list, some the last page), so read the page's own information: here, "page 3 of 3".

import re

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/products?category=audio")
    total_pages = int(re.search(r"of (\d+)\)", page.locator(".result-count").inner_text()).group(1))
    print("audio pages:", total_pages)
    browser.close()

audio pages: 1

2. Load more buttons

The catalog shows 12 products and a Load more button that fetches 12 more each time and disappears when there's nothing left:

from playwright.sync_api import expect

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/catalog")
    cards = page.locator(".product-card")
    cards.first.wait_for()
    load_more = page.get_by_role("button", name="Load more")

    clicks = 0
    while load_more.is_visible():
        before = cards.count()
        load_more.click()
        cards.nth(before).wait_for()       # wait for the first NEW card
        clicks += 1
    print(clicks, "clicks,", cards.count(), "products")
    browser.close()

2 clicks, 36 products

The key line is cards.nth(before).wait_for(): after clicking, wait until there's a card at position before (the first new one). Without it, the loop would check the button again before the new items arrived.

3. Infinite scroll

The "New arrivals" feed loads 12 products, then more each time you scroll near the bottom, and shows "You've seen everything." at the end:

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/feed")
    cards = page.locator(".product-card")
    cards.first.wait_for()
    end_message = page.locator("#feed-end")

    scrolls = 0
    while not end_message.is_visible():
        before = cards.count()
        page.mouse.wheel(0, 10_000)        # scroll down, like a user
        cards.nth(before).wait_for()
        scrolls += 1
    ids = [card.get_attribute("data-id") for card in cards.all()]
    print(scrolls, "scrolls,", len(ids), "products,", len(set(ids)), "unique")
    browser.close()

2 scrolls, 36 products, 36 unique

The same shape as Load more: act, wait for new items, check the stop condition. Other ways to scroll: page.keyboard.press("End"), or locator.last.scroll_into_view_if_needed().

Not every feed has an end message. Then stop when a scroll produces nothing new within a timeout (catch the TimeoutError from wait_for), and always add a maximum number of iterations so a misbehaving page can't trap your scraper.

Why this matters for clean code

Write each strategy once, as a function (scrape_all_pages(page), load_all(page)), with the stop condition and the safety limit visible at the top of the loop. When a site switches from pages to infinite scroll (it happens), you swap one function and the rest of the scraper doesn't notice.

Recap

• Numbered pages: follow the Next link until it isn't a link.
• URL loops: learn how the site handles out-of-range pages; use its "page X of Y" information.
• Load more: click, cards.nth(before).wait_for(), repeat while the button is visible.
• Infinite scroll: page.mouse.wheel(0, 10_000), wait for new items, stop at the end marker or when nothing new loads.
• Always cap the iterations and de-duplicate by id.