Handling missing data (NaN, fillna, dropna)

The idea

Real data has gaps: a customer skipped the city field, a sensor went offline, a price wasn't entered yet. pandas marks a missing value as NaN ("not a number"), or None/NaT in some types. Missing data isn't a nuisance to delete; it's information. Why is it missing, and what should your analysis do about it? Every choice (fill, drop, keep and flag) changes your numbers, so it should be a deliberate one.

import pandas as pd
import numpy as np

customers = pd.DataFrame({
    "name": ["Ayu", "Budi", "Citra", "Dimas", "Eka", "Fajar"],
    "city": ["Jakarta", "Bandung", None, "Jakarta", np.nan, "Medan"],
    "orders": [3, np.nan, 5, 1, 2, np.nan],
    "spent": [1245000, 1145000, np.nan, 550000, 410000, 0],
})
print(customers)

    name     city  orders      spent
0    Ayu  Jakarta     3.0  1245000.0
1   Budi  Bandung     NaN  1145000.0
2  Citra     None     5.0        NaN
3  Dimas  Jakarta     1.0   550000.0
4    Eka      NaN     2.0   410000.0
5  Fajar    Medan     NaN        0.0

None and np.nan both show as missing. Notice orders turned into floats (3.0): a NumPy integer column can't hold NaN, so pandas upgrades it to float.

Finding missing values

This module uses a small customers table with a few gaps in it. Before you can decide what to do about missing values, you need to know where they are and how many there are. .isna() marks every missing cell, and adding up those marks gives a count per column:

print(customers.isna().sum())
print(customers[customers["city"].isna()])

name      0
city      2
orders    2
spent     1
dtype: int64
    name  city  orders     spent
2  Citra  None     5.0       NaN
4    Eka   NaN     2.0  410000.0

.isna() (or its twin .isnull()) gives True where values are missing; .notna() the opposite. Sum it to count per column, or use it as a mask to look at the affected rows. Looking at the rows matters: maybe all the missing cities come from one sign-up form, which tells you something about the cause.

How missing values affect calculations

Here's why missing values need a decision rather than being ignored. pandas' summary methods quietly skip missing values, so the answer you get depends on what you think the gaps mean:

print(customers["orders"].sum())
print(customers["orders"].mean())
print(customers["orders"].count(), len(customers))

11.0
2.75
4 6

The mean is 11 / 4 = 2.75, calculated over the 4 known values, not all 6 customers. That's often reasonable, but not always: if a missing orders really means "zero orders", the true mean is 11 / 6 ≈ 1.83. Same data, two different answers, depending on what missing means. That's why you have to decide.

Also notice spent has a 0 for Fajar and a NaN for Citra. Those are different facts: Fajar spent nothing; for Citra, we don't know.

Option 1: drop

The simplest response is to remove the rows (or columns) that have missing values. It's also the easiest to overdo, so look at how many rows each version leaves you with:

print(len(customers.dropna()))
print(len(customers.dropna(subset=["city"])))
print(customers.dropna(axis="columns").columns.tolist())

2
4
['name']

• dropna() removes rows with any missing value: here only 2 of 6 survive, which is usually far too aggressive.
• dropna(subset=["city"]) only looks at the columns you name. This is the version you'll use most.
• dropna(axis="columns") drops columns with missing values instead.

Drop when the missing rows can't be used for the question at hand (you can't map a customer with no city) and when there are few of them. Dropping 40% of a dataset can quietly bias your results.

Option 2: fill

The other option is to keep the rows and put a sensible value in each gap. What counts as sensible depends on the column: a missing city can become "Unknown", and a missing order count might really mean zero:

filled = customers.fillna({"city": "Unknown", "orders": 0})
print(filled[["name", "city", "orders"]])

    name     city  orders
0    Ayu  Jakarta     3.0
1   Budi  Bandung     0.0
2  Citra  Unknown     5.0
3  Dimas  Jakarta     1.0
4    Eka  Unknown     2.0
5  Fajar    Medan     0.0

fillna with a dictionary fills each column with its own value. Common choices:

median_spent = customers["spent"].median()
customers["spent_filled"] = customers["spent"].fillna(median_spent)
print(median_spent, customers["spent_filled"].tolist())

550000.0 [1245000.0, 1145000.0, 550000.0, 550000.0, 410000.0, 0.0]

Filling with the median keeps the row, but it invents data. It also makes the column look more certain than it is. Use it knowingly, and usually alongside a flag.

Forward fill for time series

For values recorded over time, like daily stock counts, there's often a better guess than a fixed number. If someone forgot to count the stock on Tuesday, the best guess is usually Monday's number, because stock doesn't change unless something happens:

stock = pd.Series([24, np.nan, np.nan, 18, np.nan], index=["Mon", "Tue", "Wed", "Thu", "Fri"])
print(stock.ffill().tolist())

[24.0, 24.0, 24.0, 18.0, 18.0]

.ffill() carries the last known value forward: if nobody recorded stock on Tuesday, it was probably still 24. .bfill() fills backwards. .interpolate() estimates values in between.

Option 3: keep and flag

Often the best move is to keep the missing value and record that it was missing, so later steps (and readers) can see it:

customers["city_missing"] = customers["city"].isna()
print(customers[["name", "city", "city_missing"]])

    name     city  city_missing
0    Ayu  Jakarta         False
1   Budi  Bandung         False
2  Citra     None          True
3  Dimas  Jakarta         False
4    Eka      NaN          True
5  Fajar    Medan         False

A flag column lets you report "4 of 6 customers have a known city" honestly, and lets a model or a dashboard treat unknowns separately.

Empty strings and fake missing values

Not every gap arrives as NaN. Exports are full of empty strings, "N/A", "-", "none" and 0 used as "unknown". Convert them to real missing values first, so the tools above work:

raw = pd.Series(["Jakarta", "", "N/A", " ", "Bandung", "-"])
clean = raw.str.strip().replace({"": np.nan, "N/A": np.nan, "-": np.nan})
print(clean.isna().sum())

4

The best time to do this is when reading the file: pd.read_csv(..., na_values=["N/A", "-"]), as you saw in module 2.

Why this matters for clean code

Make missing-data decisions explicit and early, in one cleaning step, with a comment explaining why: "missing orders means zero orders: the field is only filled when an order exists". Avoid scattering fillna(0) calls through an analysis, where each one silently changes the meaning of the data. And never let NaN reach a report unexplained.

Recap

• NaN marks missing values; integer columns with NaN become floats.
• .isna().sum() counts per column; use .isna() as a mask to inspect rows. Never compare with == np.nan.
• sum, mean, count skip missing values: decide what missing means first.
• dropna(subset=[...]) drops rows missing key fields; plain dropna() is usually too aggressive.
• fillna({...}) fills per column: a label, 0, the median, or ffill() for time series.
• Flag missingness with a boolean column when it matters.
• Convert fake missing values ("", "N/A", "-") to NaN first.