DataLens

Boolean and fancy indexing

Filter arrays with conditions, combine conditions safely, replace values that match, and pick elements by a list of positions.

Level: Foundations · 9 min read

Practise this module

The questions for this lesson run in your browser and check your answer straight away.

The idea

Slicing picks values by position. Real questions pick values by condition: days with more than 60 orders, products under Rp 100.000, sensor readings that are clearly broken. NumPy does this with boolean masks: an array of True/False, one per element, that says which ones to keep.

This is exactly how pandas filtering works under the hood: df[df["price"] > 100] is a boolean mask too.

Masks

Say the café wants to know which days were busy, meaning more than 60 orders. Compare the whole array with 60, and instead of a single True or False, you get one answer per day. That array of answers is called a mask:

import numpy as np

daily_orders = np.array([42, 38, 51, 47, 60, 75, 68, 44, 39, 55, 49, 63, 80, 71])
busy = daily_orders > 60
print(busy)
[False False False False False  True  True False False False False  True
  True  True]

Put the mask inside square brackets, and you get only the values where it's True:

print(daily_orders[busy])
print(daily_orders[daily_orders < 40])
[75 68 63 80 71]
[38 39]

Usually you write the condition directly inside the brackets, as in the second line. Read it as "daily orders where daily orders are under 40".

Counting and measuring with masks

True counts as 1 and False as 0, so sum counts matches and mean gives the share:

print((daily_orders > 60).sum())     # how many busy days
print((daily_orders > 60).mean())    # share of busy days
5
0.35714285714285715

.any() and .all() answer yes/no questions: "was there any day under 40?", "were all days above 30?"

print((daily_orders < 40).any(), (daily_orders > 30).all())
True True

Combining conditions: & | ~

To combine masks, use & (and), | (or) and ~ (not), and wrap each condition in parentheses:

weekend = np.array([False, False, False, False, False, True, True] * 2)
print(daily_orders[(daily_orders > 60) & ~weekend])   # busy weekdays
[63]

Why not and / or? Python's and asks "is this whole thing true?", and an array of many booleans has no single answer:

print(daily_orders > 40 and daily_orders < 60)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

And why the parentheses? & binds more tightly than >, so daily_orders > 40 & daily_orders < 60 is read as daily_orders > (40 & daily_orders) < 60, which is nonsense. Parentheses around every condition avoid the problem entirely.

Key idea

In NumPy (and pandas), combine conditions with &, |, ~, and put every condition in parentheses: (a > 40) & (a < 60).

Changing values that match

A mask on the left of = changes only the matching elements. Sensor-style data often has impossible readings; here two days were logged as negative by a broken import:

orders = np.array([42, -1, 51, 47, -1, 75, 68])
orders[orders < 0] = 0
print(orders)
[42  0 51 47  0 75 68]

np.where: choose per element

np.where(condition, if_true, if_false) builds a new array, picking from one of two values (or arrays) element by element. It's the vectorized version of an if/else:

labels = np.where(daily_orders > 60, "busy", "normal")
print(labels)

bonus = np.where(daily_orders > 60, daily_orders * 0.1, 0)
print(bonus)
['normal' 'normal' 'normal' 'normal' 'normal' 'busy' 'busy' 'normal'
 'normal' 'normal' 'normal' 'busy' 'busy' 'busy']
[0.  0.  0.  0.  0.  7.5 6.8 0.  0.  0.  0.  6.3 8.  7.1]

Fancy indexing: a list of positions

Pass a list (or array) of indexes to pick several elements at once, in any order:

products = np.array(["French Press", "Kettle", "Arabica", "Robusta", "Toraja"])
prices = np.array([185, 320, 95, 65, 110])
print(prices[[1, 4]])
print(products[[4, 0, 2]])
[320 110]
['Toraja' 'French Press' 'Arabica']

This shines with np.argsort, which returns the positions that would sort an array. Use them to sort one array by another:

order = np.argsort(prices)[::-1]     # positions from most to least expensive
print(order)
print(products[order])
print(prices[order])
[1 0 4 2 3]
['Kettle' 'French Press' 'Toraja' 'Arabica' 'Robusta']
[320 185 110  95  65]

Because products and prices are indexed with the same positions, the names stay matched to their prices. That's the "sort by another column" trick, NumPy style.

Masks in 2-D

Masks work on grids too. The result is flattened to 1-D, since the matching values don't form a rectangle:

units = np.array([
    [120, 135, 150, 160],
    [ 80,  85,  90,  88],
    [ 95,  92, 105, 110],
])
print(units[units > 100])
[120 135 150 160 105 110]

To keep whole rows, build a 1-D mask over the rows, for example stores whose first month was above 90:

print(units[units[:, 0] > 90])
[[120 135 150 160]
 [ 95  92 105 110]]

Why this matters for clean code

A named mask turns a filter into a sentence: busy = daily_orders > 60, then daily_orders[busy & ~weekend]. It's faster than a loop with if, and it reads like the business question. The same habit carries straight into pandas, where almost every filter is a mask.

Recap

Practise this module

The questions for this lesson run in your browser and check your answer straight away.

Next: Element-wise operations and broadcasting →