Waiting for elements and content (auto-waiting)

The idea

On dynamic pages, content arrives after the page has "loaded": a spinner shows, JavaScript fetches data, cards appear half a second later. A scraper that reads too early sees nothing. The classic fix is time.sleep(5), which is slow when the page is fast and broken when the page is slow. Playwright has a better answer: wait for the thing you need, and most of the time it does that for you.

The problem

The shop's catalog page loads its products a moment after the page itself appears. To a person this is invisible: you see a spinner for a split second, then the products. Here's what a script sees if it counts the products immediately:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/catalog")
    print(page.locator(".product-card").count())
    print(page.locator(".loading").inner_text())
    browser.close()

0
Loading products…

goto returned as soon as the page loaded, but the catalog's cards come from a background request that finishes a moment later. count() reports what's there right now: nothing yet.

Auto-waiting

Actions that need an element (inner_text, get_attribute, click, fill...) wait for it automatically, up to 30 seconds by default:

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/catalog")
    print(page.locator(".product-card .title").first.inner_text())   # waits for the first card
    print(page.locator(".product-card").count())                     # now they're all there
    browser.close()

Galaxy A55 5G
12

The rule to remember:

So before using a "returns immediately" method on dynamic content, wait for the content first.

Waiting explicitly

The flash deals page shows a spinner for 2.5 seconds before its deals appear. Three ways to wait:

import time
from playwright.sync_api import expect

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/deals")
    start = time.time()

    page.locator(".deal").first.wait_for()               # 1. wait for an element to be visible
    print(f"deals appeared after {time.time() - start:.1f}s")

    page.wait_for_selector(".spinner", state="detached")  # 2. wait for something to disappear
    expect(page.locator(".deal")).to_have_count(5)        # 3. wait for an exact condition
    print(page.locator(".deal .deal-name").all_inner_texts())
    browser.close()

deals appeared after 2.5s
['Redmi Note 13 Pro', 'Galaxy Buds FE', 'Soundcore Life Q30', '65W GaN Charger', 'Redmi Watch 4']

1. locator.wait_for() waits until the element is visible (state="attached", "hidden" or "detached" for other conditions).
2. page.wait_for_selector(selector, state=...) does the same with a selector string.
3. expect(locator).to_have_count(n) (and to_have_text, to_be_visible...) keeps retrying until the condition holds or 5 seconds pass. Handy when you know exactly what "done" looks like.

The page took 2.5 seconds, and the script waited 2.5 seconds: no more, no less. A sleep(5) would have wasted half the time; a sleep(2) would have read an empty page.

When content never arrives

Waits have a timeout. If it passes, Playwright raises TimeoutError, which you can catch:

from playwright.sync_api import TimeoutError as PlaywrightTimeout

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/deals")
    try:
        page.locator(".deal").first.wait_for(timeout=1000)   # milliseconds
        print("deals found")
    except PlaywrightTimeout:
        print("no deals within 1 second")
    browser.close()

no deals within 1 second

Timeouts are in milliseconds. Set them per call (timeout=...), or for the whole page with page.set_default_timeout(10_000). A shorter timeout makes a broken page fail fast instead of hanging for 30 seconds; pick something comfortably above the page's normal loading time.

(Importing it as PlaywrightTimeout avoids confusing it with Python's built-in TimeoutError.)

What about wait_for_timeout and networkidle?

• page.wait_for_timeout(2000) is a fixed sleep. It's fine for demonstrations and for deliberately slowing down (being polite to a server), but never as a way to wait for content.
• page.goto(url, wait_until="networkidle") waits until no requests have happened for 500 ms. It works on simple pages but is slow, and pages with polling, analytics or live updates may never become idle. Waiting for an element is more precise.

Why this matters for clean code

Fixed sleeps are the number-one source of flaky scrapers: they pass on your fast laptop and fail on a slow server, or the reverse. Waiting on explicit conditions makes the code say what it means ("wait until the deals are shown") and makes it as fast as the site allows. That's both more readable and more reliable.

Recap

• Dynamic content appears after goto returns; count() may be 0 at first.
• Actions (inner_text, click, get_attribute...) auto-wait up to 30 s.
• locator.wait_for(state=...), page.wait_for_selector(...), expect(locator).to_have_count(...) wait explicitly.
• Timeouts are in milliseconds; catch playwright.sync_api.TimeoutError.
• Avoid fixed sleeps and over-reliance on networkidle.