pandas Series
& DataFrame

Four parts, and they're all separate objects

A DataFrame is not one thing. It's an index, a set of column labels, the values underneath, and a dtype for each column — and each of those can be inspected and replaced independently.

import pandas as pd

s = pd.Series([412, 388, 501],
              index=["r-01", "r-02", "r-03"],
              name="hours")

s.values     # the numpy array underneath
s.index      # Index(['r-01','r-02','r-03'])
s.name       # 'hours' — becomes the column name
s.dtype      # int64

df = pd.DataFrame({
    "site":   ["lisbon", "porto"],
    "hours":  [412, 388],
}, index=["r-01", "r-02"])

df.index       # row labels
df.columns     # column labels — also an Index!
df.dtypes      # one dtype per column
df.shape       # (2, 2)
df.info()      # the first thing to run on new data

The label layer, and what it buys you

The index is not a column. It's a separate structure sitting alongside the data, and it does four jobs.

# moving between index and column
df = df.set_index("robot_id")        # column becomes the index
df = df.reset_index()                # index becomes a column again
df = df.reset_index(drop=True)      # …or just throw it away
df = df.rename_axis("robot")         # name the index itself
df = df.sort_index()                 # sort by label, not by value

# checks worth running before you trust a join or a lookup
df.index.is_unique                   # duplicates cause silent row multiplication
df.index.has_duplicates
df.index.is_monotonic_increasing     # required for some slice operations

Six ways in, and the rules that separate them

# chained assignment: the classic pandas trap
df[df["errors"] > 5]["flag"] = True       # ← may silently do nothing

# the first [] may return a copy, so the second [] writes to
# a temporary that is then discarded. one accessor, not two:
df.loc[df["errors"] > 5, "flag"] = True    # ← correct

# pandas 2.x offers Copy-on-Write as an opt-in and 3.0 makes it
# the default, turning "sometimes silently works" into
# "reliably never works" — better, but write it right anyway.
pd.options.mode.copy_on_write = True

The behaviour that makes pandas pandas

When you combine two labeled objects, pandas does not line them up by position. It lines them up by label, takes the union of what it finds, and fills the gaps with NaN. Toggle labels in or out of each Series and watch the result reorganize itself.

# alignment is usually protecting you
a = df["hours"].sort_values()          # reordered — labels travel with values
b = df["errors"]

a - b                                # still correct! matched on label
a.to_numpy() - b.to_numpy()          # WRONG — positional, ignores the sort

# explicit alignment, when you want to see it happen
a, b = a.align(b, join="inner")      # or "outer", "left", "right"
s.reindex(["r-01", "r-09"], fill_value=0)

Note the third line. Reaching for .to_numpy() to "make it simpler" is how you discard the protection and get a silently wrong answer instead of a visible NaN.

What's actually in the column

Each column has one dtype, and it determines speed, memory, and which operations exist. df.dtypes should be the second thing you look at, right after df.shape.

df = df.astype({"site": "category", "hours": "int32"})
df["ts"] = pd.to_datetime(df["ts"], errors="coerce", utc=True)
df["n"]  = pd.to_numeric(df["n"], errors="coerce")

df.memory_usage(deep=True).sum() / 1e6     # MB, honestly counted
df.select_dtypes("object").columns          # the suspects

Several kinds of nothing

# finding it
df.isna().sum()                      # nulls per column — run this early, always
df.isna().mean().sort_values()       # as a proportion, which is more informative

# removing it
df.dropna()                          # any null in the row → gone. brutal
df.dropna(subset=["hours"])         # only where it matters
df.dropna(thresh=3)                  # keep rows with ≥3 non-null values

# filling it
df["hours"].fillna(df["hours"].median())
df["site"].fillna("unknown")
df.sort_index().ffill()              # carry forward — sort first, or it's nonsense
df.interpolate()                     # numeric only, assumes an ordering

# the habit worth forming: record that it was missing
df["hours_missing"] = df["hours"].isna()
df["hours"] = df["hours"].fillna(df["hours"].median())

Wide and long

The same data can be laid out with one row per entity and many columns, or one row per observation with the variable name in a column. Most analysis wants long; most humans want wide; you will convert between them constantly.

merge, join, concat

out = left.merge(
    right,
    on="robot_id",          # or left_on / right_on
    how="left",              # inner, left, right, outer, cross
    validate="one_to_one",   # ← use this
    indicator=True,         # adds a _merge column
    suffixes=("_a", "_b"),
)

# concat: stacking, not joining
pd.concat([df1, df2])                       # rows, aligning on columns
pd.concat([df1, df2], axis=1)               # columns, aligning on index ←!
pd.concat([df1, df2], ignore_index=True)     # drop the old indexes
pd.concat({"a": df1, "b": df2}, names=["src"])  # tag the source

# build a list and concat ONCE — never concat inside a loop
frames = [transform(p) for p in paths]
all_data = pd.concat(frames, ignore_index=True)

Split, apply, combine

Partition the rows by some key, run a computation within each partition, and reassemble. It's the single most useful operation in the library, and the four ways to finish it do genuinely different things.

# named aggregation — readable, and names the output columns for you
summary = (df.groupby("site", as_index=False, observed=True)
             .agg(total_hours=("hours", "sum"),
                  mean_uptime=("uptime", "mean"),
                  n_robots=("hours", "size"),
                  worst=("errors", "max")))

# transform: the group's mean, on every row, without a merge
df["site_mean"] = df.groupby("site")["uptime"].transform("mean")
df["vs_site"]   = df["uptime"] - df["site_mean"]

# filter: keep only sites with more than two robots
df.groupby("site").filter(lambda g: len(g) > 2)

# two keys → a MultiIndex result
df.groupby(["site", "shift"])["errors"].sum().unstack(fill_value=0)

What a DatetimeIndex unlocks

Time is the case pandas was originally built for — the library came out of finance — and it shows. Put timestamps in the index and a set of operations becomes available that have no equivalent elsewhere.

df = df.set_index(pd.to_datetime(df["ts"], utc=True)).sort_index()

# partial string slicing — reads like what it does
df.loc["2026-03"]                  # all of March
df.loc["2026-03-01":"2026-03-15"]   # inclusive, as label slices always are

# resample: groupby, but the groups are time bins
df.resample("D")["errors"].sum()      # daily totals
df.resample("W").agg({"hours": "sum", "uptime": "mean"})

# rolling and expanding windows
df["uptime_7d"] = df["uptime"].rolling("7D").mean()
df["cumulative"] = df["errors"].expanding().sum()

# shifting — the basis of every lag feature
df["prev"]   = df["errors"].shift(1)
df["change"] = df["errors"].diff()
df["growth"] = df["errors"].pct_change()

# the .dt accessor works on any datetime column, index or not
df["hour"]    = df.index.hour
df["weekday"] = df.index.day_name()

Writing pandas that reads well and runs fast

# method chaining: one expression, top to bottom, nothing mutated
clean = (
    pd.read_csv("telemetry.csv")
      .rename(columns=str.lower)
      .astype({"site": "category"})
      .assign(
          ts=lambda d: pd.to_datetime(d["ts"], utc=True),
          error_rate=lambda d: d["errors"] / d["hours"],
      )
      .query("hours > 0")
      .sort_values("ts")
      .reset_index(drop=True)
      .pipe(add_custom_features)          # your own function, inline
)

Drills, and the ritual for new data

df.shape                    # how much
df.head(10)                 # what it looks like
df.dtypes                   # any surprise objects?
df.isna().mean()            # how much is missing, proportionally
df.describe(include="all")   # ranges, and impossible values
df.index.is_unique          # can I trust a lookup?
for c in df.select_dtypes(["object", "category"]):
    print(c, df[c].nunique(), df[c].unique()[:5])