Extracting text and attributes

The idea

Finding elements is half the job; the other half is reading them into clean Python values. A product card shows Rp 5.999.000 and ★ 4.6 (1243 reviews), but your analysis needs 5999000 and 4.6. This module covers the reading methods and the habit that matters most: turning each repeated element into a tidy dictionary.

Text: inner_text vs text_content

Once a locator points at the right element, you want what's in it, usually its text. Playwright has two ways to read text, and they don't always give the same answer:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/product/1")

    print(page.locator(".product-title").inner_text())
    print(page.locator(".price").inner_text())
    print(page.locator(".rating").inner_text())
    print(page.locator(".specs tr").all_inner_texts())
    browser.close()

Galaxy A55 5G
Rp 5.999.000
★ 4.6 · 1243 reviews · 3200 sold
['Brand Samsung', 'Category Phones', 'Warranty 1 year official warranty', 'SKU NG-0001']

• inner_text() returns the text as displayed: hidden parts skipped, whitespace tidied. Use it by default.
• text_content() returns the raw text of every node, including hidden ones and all the whitespace from the HTML.
• all_inner_texts() returns a list, one string per match.

Attributes: links, images and hidden data

Much useful information isn't visible text at all:

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/products")
    card = page.locator(".product-card").first

    print(card.get_attribute("data-id"), card.get_attribute("data-category"))
    print(card.locator(".title a").get_attribute("href"))
    print(card.locator("img").get_attribute("src"), "|", card.locator("img").get_attribute("alt"))
    print(card.locator(".price").get_attribute("data-price"))
    browser.close()

1 phones
/product/1
/img/products/1.jpg | Galaxy A55 5G
5999000

Notice data-price="5999000": many sites put the clean value in a data-* attribute and the formatted one in the text. When it's there, read the attribute and skip the parsing. Attributes are always strings (or None if missing), so convert them: int(...), float(...).

Links come back as written in the HTML, often relative (/product/1). Make them absolute with urljoin:

from urllib.parse import urljoin

print(urljoin("https://nusantara-gadget.test/products?page=2", "/product/1"))

https://nusantara-gadget.test/product/1

Parsing displayed text

When the clean value isn't in an attribute, parse the text. Keep each parser small and testable:

import re


def parse_rupiah(text):
    """'Rp 5.999.000' -> 5999000"""
    return int(re.sub(r"[^\d]", "", text))


def parse_rating(text):
    """'★ 4.6 (1243 reviews)' -> (4.6, 1243)"""
    rating, reviews = re.search(r"([\d.]+)\D+(\d+)", text).groups()
    return float(rating), int(reviews)


print(parse_rupiah("Rp 5.999.000"))
print(parse_rating("★ 4.6 (1243 reviews)"))

5999000
(4.6, 1243)

Indonesian formatting uses . as the thousands separator, so float("5.999.000") fails and float("5.999") would be wrong by a factor of 1,000. Stripping everything that isn't a digit is the safe approach for whole-rupiah prices.

One record per element

The standard pattern: loop over the repeated element with .all(), and read each field relative to that element:

def parse_card(card):
    return {
        "id": int(card.get_attribute("data-id")),
        "name": card.locator(".title").inner_text(),
        "brand": card.locator(".brand").inner_text(),
        "price": int(card.locator(".price").get_attribute("data-price")),
        "rating": float(card.locator(".rating").get_attribute("data-rating")),
        "in_stock": card.locator(".in-stock").count() > 0,
        "url": urljoin("https://nusantara-gadget.test", card.locator(".title a").get_attribute("href")),
    }


with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://nusantara-gadget.test/products")
    products = [parse_card(card) for card in page.locator(".product-card").all()]
    browser.close()

print(len(products))
print(products[1])

12
{'id': 2, 'name': 'Redmi Note 13 Pro', 'brand': 'Xiaomi', 'price': 4299000, 'rating': 4.5, 'in_stock': False, 'url': 'https://nusantara-gadget.test/product/2'}

Why per-card rather than collecting all names, then all prices? Parallel lists drift out of step the moment one card lacks a field (a missing rating would shift every rating after it to the wrong product). Reading each card as a unit keeps every field attached to its product.

Why this matters for clean code

parse_card is a pure, single-purpose function: it takes one element and returns one record. You can test it on a single card, reuse it on every page that shows cards (products, search, catalog), and fix it in one place when the markup changes. The scraping loop stays a one-liner.

Recap

• inner_text() (displayed text), text_content() (raw), all_inner_texts() (list).
• get_attribute(name) for href, src, alt and data-*; returns a string or None.
• Make relative links absolute with urljoin.
• Parse formatted text with small functions (re.sub(r"[^\d]", "", text) for rupiah).
• Loop for card in locator.all() and read fields relative to each card into a dict.