How pages load; static vs dynamic (why requests isn't enough)

The idea

Web scraping means collecting data from websites with code instead of copying it by hand: competitor prices, product reviews, job listings, stock levels. For a data analyst, it's a way to get data nobody will send you as a CSV.

Before writing a scraper, you need to understand how a page actually arrives in your browser, because modern sites don't send you the page you see. They send you a starting point, and JavaScript builds the rest. That single fact decides which tool you need.

What happens when you open a URL

When you type https://nusantara-gadget.test/products and press Enter, roughly this happens:

1. Your browser sends an HTTP request to the server: "GET /products, please".
2. The server sends back a response: a status code (200 means OK) and a body of HTML.
3. The browser reads the HTML and finds more things to download: CSS for styling, images, and JavaScript files.
4. The browser runs the JavaScript. On many modern sites, that JavaScript sends more requests in the background, usually to an API that returns raw data as JSON, and then inserts that data into the page.
5. You finally see the products.

Steps 4 and 5 are where scrapers get caught out.

Static pages

On a static (or server-rendered) page, the data is already inside the first HTML response. Our practice shop, Nusantara Gadget, has one: /products. You can fetch it with the requests library, which sends a single HTTP request and gives you the response:

import requests

response = requests.get("https://nusantara-gadget.test/products")
html = response.text
print(response.status_code)
print(html.count('class="product-card"'))

200
12

Twelve product cards, right there in the HTML. For a page like this, requests plus an HTML parser would be enough.

Dynamic pages

Now the /catalog page. In a browser it shows the same products, but look at what requests receives:

response = requests.get("https://nusantara-gadget.test/catalog")
html = response.text
print(html.count('class="product-card"'))
print("Loading products" in html)

0
True

Zero products, only a "Loading products…" placeholder. The products are added later by JavaScript, and requests never runs JavaScript. It downloads the HTML and stops at step 2. This is a dynamic page (also called client-side rendered), and it's how most large shops, marketplaces and social sites work today.

This is the most common beginner frustration in scraping: "The data is right there on my screen, but my scraper sees nothing." Now you know why.

Where did the data come from?

The JavaScript fetched it from an API. On this site the catalog loads /api/products?page=1&per_page=12, and you can request that URL directly:

data = requests.get("https://nusantara-gadget.test/api/products?page=1&per_page=12").json()
print(data["total"], "products in total")
print(data["items"][0]["name"], data["items"][0]["price"])

36 products in total
Galaxy A55 5G 5999000

.json() turns the JSON response into Python dictionaries and lists. Clean, structured data with no HTML to untangle. Finding these background APIs is often the best scraping strategy of all, and you'll learn to capture them properly with Playwright's network interception in the Advanced level.

Enter Playwright: a real browser you control with code

Sometimes you can't call the API directly: it needs tokens set by JavaScript, cookies from a login, or the site blocks plain HTTP clients. Then you need a real browser that loads the page exactly like a person would, JavaScript and all. That's what Playwright gives you: Chromium, Firefox or WebKit, controlled from Python.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/catalog")
    page.wait_for_selector(".product-card")
    print(page.locator(".product-card").count())
    browser.close()

12

The browser ran the page's JavaScript, the products appeared, and Playwright counted them. Each line is explained properly in the next modules; for now, notice the shape: launch a browser, open a page, go to a URL, wait for the content, read it.

On your own computer you install it with two commands, one for the Python package and one for the browsers it controls:

uv add playwright

uv run playwright install chromium

How to tell which kind of page you have

Before writing any code, check the site in your normal browser:

• View the page source (Ctrl+U, or Cmd+Option+U on a Mac). This shows the raw HTML the server sent, the same thing requests sees. If your data is in there, the page is static for your purposes.
• Open developer tools (F12), go to the Network tab, filter by Fetch/XHR, and reload. Requests returning JSON are the background APIs. Click one and look at the Response tab.
• Search the page source for a value you can see on screen, such as a price. Not there? It's loaded dynamically.

Why this matters for clean code

A scraper is only as reliable as your understanding of the page. If you write browser automation for a page that's static, your scraper is slow and fragile for no reason. If you parse HTML when a clean JSON API exists, you'll break every time the site changes its layout. Spending five minutes in developer tools first leads to simpler, faster, more robust code.

Recap

• A page loads in stages: HTML first, then CSS, images and JavaScript, then JavaScript may fetch more data from an API.
• Static pages have the data in the first HTML response. Dynamic pages add it later with JavaScript.
• requests gets only the first response and never runs JavaScript.
• Many dynamic pages load JSON from an API, which you can often request directly.
• Playwright drives a real browser, so it sees what a person sees.
• Check View Source and the Network tab before choosing a tool.