explainer
pandas: select, align, and check a table of robot readings
Learn pandas Series and DataFrames through robot readings. Compare loc and iloc, inspect label alignment, handle missing values, and check grouped summaries and joins.
What you will learn
- Predict the shape and index of label-based and positional selections.
- Explain missing results from aligned arithmetic and missing measurements.
- Prepare a small table with explicit types, grouped summaries, and validated joins.
Before you start
- Basic Python lists, dictionaries, and function calls
pandas helps you inspect and transform labeled data in Python. Its labels determine how you select rows, combine columns, and align measurements. Learning those rules helps prevent a plausible-looking table from pairing the wrong values.
This lesson uses four synthetic robot readings. You will select observations, add corrections, and summarize the results. The Python example was verified with Python 3.12.14 and pandas 3.0.5.
Give each observation a label
A Series is a one-dimensional object with an index and a dtype. A DataFrame is a two-dimensional table with row and column labels. Its columns can have different dtypes. The pandas data-structures guide introduces both objects.
Our DataFrame, named df, contains:
| Index label | robot | range_m | battery |
|---|---|---|---|
| 101 | rover | 1.2 | 90 |
| 205 | scout | 2.4 | 75 |
| 310 | rover | missing | 60 |
| 450 | scout | 3.6 | 85 |
Each row represents one observation. range_m uses metres, and battery uses percent. The index labels identify observations; their positions are 0, 1, 2, 3 from top to bottom.
This table has shape (4, 3): four rows and three data columns. The index adds labels without adding a data column. The tensor lesson offers an optional introduction to axes and shape.
df["range_m"] returns a Series with shape (4,). df[["range_m"]] passes a list of column names and returns a DataFrame with shape (4, 1). Both keep the original row labels.
Choose labels or positions
Use .loc for labels and .iloc for integer positions. In this table, df.loc[205] selects the observation labeled 205. df.iloc[0] selects the first observation, whose label is 101.
Selecting one row this way returns a Series. Its index contains the original column names: robot, range_m, and battery. Selecting both a single row and column, as in df.loc[205, "range_m"], returns the scalar 2.4.
Slice boundaries differ:
df.loc[205:310, ["range_m"]]includes both existing endpoint labels, giving rows 205 and 310.df.iloc[1:2, [1]]excludes stop position 2, giving only row 205.
An explicit missing label raises KeyError; an explicit out-of-range position raises IndexError. Positional slices can extend beyond the available rows without raising that error. The indexing guide documents these rules.
Sorting or filtering can change positions while preserving labels. Use the accessor that expresses your intended operation. The examples here use unique numeric labels; duplicate labels and unsorted label slices need additional care.
Inspect selection and alignment
Start with the label slice. Its output has shape (2, 1), index 205, 310, and one missing measurement. Switch to the position slice and inspect the removed endpoint.
Choose “Filter by range” and move the cutoff. At 1.5 m, rows 205 and 450 remain. At 5.0 m, the result has no rows but still has two selected columns, giving shape (0, 2).
The browser uses JavaScript transformations to illustrate this finite menu of operations. It does not run Python, accept arbitrary pandas expressions, or reproduce the full pandas type system. The runnable example below checks the same core behavior in pandas itself.
Work through aligned arithmetic
Set readings = df["range_m"]. Suppose a separate offsets Series contains corrections in this order: 450: 0.4, 101: 0.1, 900: 0.5.
The expression readings + offsets matches values by index label. For these different indexes, pandas forms their sorted union. It does not pair the first reading with the first correction.
| Index | Reading | Offset | Result |
|---|---|---|---|
| 101 | 1.2 | 0.1 | 1.3 |
| 205 | 2.4 | no label | missing |
| 310 | missing | no label | missing |
| 450 | 3.6 | 0.4 | 4.0 |
| 900 | no label | 0.5 | missing |
For label 101, the calculation is 1.2 + 0.1 = 1.3 m. Label 205 lacks a correction. Label 900 lacks a reading, so neither can produce an ordinary sum.
Switch the explorer to “Same labels, reordered.” The correction for 205 is now −0.2, giving 2.4 − 0.2 = 2.2 m. The reading at 310 remains missing even though its correction exists.
When both Series already have the same index in the same order, arithmetic preserves that order. Labels also guide assignment from a Series into a DataFrame column. The data-structures guide explains automatic alignment.
Handle missing data deliberately
A missing measurement carries different information from zero. Zero metres is a measurement; a missing range records that a usable value is unavailable. Decide what that absence means before dropping or replacing it.
The example explicitly uses nullable Float64 for ranges and Int64 for battery readings. These dtypes support pd.NA. Their names differ from NumPy's lowercase float64 and int64, so specify the intended dtype carefully.
Use isna() or notna() to detect missing values. Different dtypes can use different missing markers, including NaN, NaT, or pd.NA. The missing-data guide explains their behavior.
A nullable comparison such as df["range_m"] > 1.5 can contain pd.NA. During Boolean row selection, that missing mask entry does not select the row. To combine conditions, put each comparison in parentheses and use & for elementwise AND or | for OR.
For example, df.loc[(df["range_m"] > 1.5) & (df["battery"] >= 80)] selects only label 450. Both conditions apply to the same labeled observations.
Summarize readings by robot
groupby collects rows with the same key so you can calculate a statistic for each group. Use df.groupby("robot", dropna=False)["range_m"].mean() to calculate each robot's mean known range.
The rover group contains 1.2 and a missing value, so its mean is 1.2 m. The scout group contains 2.4 and 3.6, so its mean is (2.4 + 3.6) / 2 = 3.0 m. The mean skips missing measurements by default.
Check how many measurements support each result. Grouped size() counts rows; grouped count() on range_m counts nonmissing values. Rover has two rows but only one known range.
By default, groupby omits missing group keys. dropna=False keeps a group for them. That option concerns a missing robot name, which differs from a missing range inside a named group. The groupby guide covers these choices.
Check the keys before joining
Suppose a lookup table assigns one team to each robot. Many reading rows can share a robot name, while the lookup should contain that name once. Express that expectation with merge(..., on="robot", how="left", validate="many_to_one", indicator=True).
The validation checks uniqueness on the lookup side. A duplicate robot key there raises a merge error. Without the check, duplicate keys can multiply reading rows.
Uniqueness does not prove that every reading found a match. Inspect the added _merge column for left_only rows. The Python example asserts that every row has value both.
Check missing join keys too: pandas can match null keys to each other, unlike a typical SQL join. A column-based merge also replaces the original index, so the example first uses reset_index() to preserve reading_id as a column. The merging guide describes key validation and match indicators.
Use pd.concat when stacking batches of rows. Preserve observation IDs before requesting a fresh positional index. These checks belong in the data preparation stage shown in the machine learning fields map.
Update data under pandas 3.0
Copy-on-Write is the default and only mode in pandas 3.0. A derived pandas object behaves independently when you modify it. Editing a selected column or table updates that object without silently updating its source.
To change df, assign through df directly, using a single operation such as df.loc[205, "battery"] = 70. Chained assignment cannot update the original under Copy-on-Write. Assigning other = df still creates another name for the same object. The Copy-on-Write guide explains the distinction.
pandas 3.0 also infers a dedicated str dtype for ordinary string data. That default uses NaN for missing values. The explicitly requested string dtype in this example uses pd.NA; those two spellings have different missing-value behavior. The pandas 3.0 release notes document the string default.
Prefer column operations and built-in reductions for this work. Measure your own pipeline before making performance claims. When text becomes a model input, the NLP lesson builds on the same need to preserve row identity.
Run the example in pandas
This example runs with pandas 3.0.5; install that version in a Python environment to reproduce the output exactly. It uses explicit dtypes so the missing-value representation is deliberate.
import pandas as pd
df = pd.DataFrame(
{"robot": ["rover", "scout", "rover", "scout"],
"range_m": [1.2, 2.4, None, 3.6],
"battery": [90, 75, 60, 85]},
index=pd.Index([101, 205, 310, 450], name="reading_id"),
).astype({"robot": "string", "range_m": "Float64", "battery": "Int64"})
print(f"pandas {pd.__version__}")
print("loc labels:", df.loc[205:310].index.tolist())
print("iloc labels:", df.iloc[1:2].index.tolist())
filtered = df.loc[df["range_m"] > 1.5, ["robot", "range_m"]]
print("Filtered labels:", filtered.index.tolist())
offsets = pd.Series([0.4, 0.1, 0.5], index=[450, 101, 900], dtype="Float64")
aligned = df["range_m"] + offsets
print("Aligned:", [(label, "NA" if pd.isna(value) else round(float(value), 1))
for label, value in aligned.items()])
means = df.groupby("robot", dropna=False)["range_m"].mean()
print("Mean by robot:", {str(robot): float(value) for robot, value in means.items()})
teams = pd.DataFrame({"robot": ["rover", "scout"], "team": ["A", "B"]})
joined = df.reset_index().merge(
teams, on="robot", how="left", validate="many_to_one", indicator=True
)
assert joined["_merge"].eq("both").all()
print("Join rows:", len(joined), "all matched:", bool(joined["_merge"].eq("both").all()))
subset = df[["range_m"]]
subset.loc[101, "range_m"] = 9.9
print("Original range after subset edit:", df.loc[101, "range_m"])
Expected output:
pandas 3.0.5
loc labels: [205, 310]
iloc labels: [205]
Filtered labels: [205, 450]
Aligned: [(101, 1.3), (205, 'NA'), (310, 'NA'), (450, 4.0), (900, 'NA')]
Mean by robot: {'rover': 1.2, 'scout': 3.0}
Join rows: 4 all matched: True
Original range after subset edit: 1.2
The final line checks Copy-on-Write: the subset holds 9.9 at label 101, while df retains 1.2. You can now pass checked numeric columns into linear regression or random forests, with training and evaluation rows kept separate.
Try it yourself
Exercise 1. What happens with df.loc[0] and df.iloc[0] on this table? Give the result type, name, and index for the successful selection.
Show solution: distinguish a label from a position
df.loc[0] raises KeyError because no row has label 0. df.iloc[0] returns the first row as a Series named 101.
The Series index is robot, range_m, battery, and its values are rover, 1.2, 90. Its shape is (3,).
Exercise 2. Rover has ranges 1.2 and missing. What mean does pandas report? What mean would replacing the missing value with zero produce, and what assumption would that make?
Show solution: count the known measurements
The default mean skips the missing range and reports 1.2 m, based on one known measurement. Replacing the missing value with zero gives (1.2 + 0) / 2 = 0.6 m.
That replacement treats the absent measurement as an observed zero-metre range. Confirm that meaning from the data source before using it in a summary or a model.
Sources and further study
- pandas: introduction to data structures, for Series, DataFrames, and label alignment.
- pandas: indexing and selecting data, for loc, iloc, slices, and Boolean selection.
- pandas: working with missing data, for missing markers and nullable dtypes.
- pandas: groupby, for aggregation, counts, and missing group keys.
- pandas: merging and joining, for key validation, indicators, and null-key behavior.
- pandas: Copy-on-Write, for assignment behavior in pandas 3.0.
- pandas 3.0 release notes, for the inferred string dtype and migration changes.