Navigating pages and taking screenshots

The idea

A scraper is a browser on autopilot: it goes to pages, follows links, and occasionally needs to show you what it saw. This module covers moving around a site reliably (including noticing when a page failed to load) and screenshots, your best debugging tool.

goto returns a response

When a scraper visits hundreds of product pages, some of them won't exist anymore. A person would notice the "Not found" message straight away, but a script won't unless you check. goto hands you the server's response, so you can:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    response = page.goto("https://nusantara-gadget.test/product/3")
    print(response.status, response.ok)
    print(page.url)
    print(page.title())

    missing = page.goto("https://nusantara-gadget.test/product/999")
    print(missing.status, missing.ok)
    print(page.title())
    browser.close()

200 True
https://nusantara-gadget.test/product/3
iPhone 15 — Nusantara Gadget
404 False
Not found

page.goto returns the main document's response. The browser happily shows a "Not found" page, so without checking the status your scraper would carry on and scrape an error page as if it were a product. Check response.ok (status below 400) or response.status after every important navigation.

Following links and moving around

Real users click links; so can your scraper. Clicking a link navigates, and page.url tells you where you ended up:

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/")
    page.get_by_role("link", name="Laptops").click()
    print(page.url)
    print(page.title())

    page.go_back()
    print(page.url)
    page.reload()
    print(page.title())
    browser.close()

https://nusantara-gadget.test/products?category=laptops
Products — page 1 — Nusantara Gadget
https://nusantara-gadget.test/
Nusantara Gadget — Official Gadget Store

• click() on a link navigates and waits for the new page.
• go_back() and reload() work like the browser buttons (go_forward() exists too).
• page.url always holds the current address, including any query string.

(get_by_role finds elements the way a person describes them, "the link called Laptops". Module 4 covers it properly.)

Relative links

Links in HTML are often relative (/product/3, ?page=2). page.goto resolves them against the current page, just like a browser:

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/products")
    page.goto("/products?page=2")
    print(page.url)
    browser.close()

https://nusantara-gadget.test/products?page=2

When you build URLs yourself in Python, urllib.parse.urljoin(page.url, href) does the same.

Screenshots

When a scraper misbehaves, the fastest question to answer is "what did the page actually look like?":

import os

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/products")
    page.screenshot(path="products.png")                       # the visible window
    page.screenshot(path="products_full.png", full_page=True)  # the whole scrollable page
    page.locator(".product-card").first.screenshot(path="first_card.png")   # one element
    browser.close()

print(sorted(f for f in os.listdir(".") if f.endswith(".png")))

['first_card.png', 'products.png', 'products_full.png']

• path= saves a PNG; without it, screenshot() returns the image bytes.
• full_page=True captures the entire page, not just the visible part.
• locator(...).screenshot() captures a single element.

A useful habit: in your scraper's error handling, save a screenshot named after the failing URL. When something breaks at 3 a.m., the picture usually explains it in seconds.

wait_until: when is a page "loaded"?

goto waits for the load event by default: the HTML, CSS, scripts and images are in. It does not wait for data that JavaScript fetches afterwards (the catalog from module 1). Options:

For JavaScript-loaded content, it's better to wait for the specific element you need, which module 6 covers.

Why this matters for clean code

Silent failures are the worst kind: a scraper that collects 500 "Product not found" pages reports success and corrupts your dataset. Checking statuses and failing loudly (or logging and skipping) keeps bad data out. Screenshots on failure make the problems you can't prevent quick to diagnose.

Recap

• response = page.goto(url); check response.ok / response.status.
• Navigate by clicking links or with goto; relative URLs resolve against the current page.
• page.url, go_back(), go_forward(), reload().
• page.screenshot(path=..., full_page=True) and locator.screenshot(path=...).
• goto waits for load; wait for specific elements when content arrives via JavaScript.