The Machine Learning Checklist

1.1 Linear Algebra

• Scalars, vectors, matrices, tensors — the four object types and how dimensions are written and readexplainer ↗
• Vector addition and scalar multiplication — the two operations that define a vector spaceexplainer ↗
• Dot product / inner product — measures alignment between vectors; the single most common operation in MLexplainer ↗
• Vector norms (L0, L1, L2, L∞) — different ways to measure length, each producing different regularization behavior
• Unit vectors and normalization — scaling to length 1 so direction can be compared independently of magnitude
• Cosine similarity — angle-based similarity, the standard metric for embeddings
• Orthogonality and orthonormal bases — perpendicular directions that make decomposition clean
• Matrix multiplication — composition of linear maps; the primitive that GPUs are built to accelerate
• Matrix transpose — swapping rows and columns, and why Xᵀ appears everywhere
• Identity and inverse matrices — the multiplicative unit and undoing a transformation
• Determinant — signed volume scaling factor, and its role in invertibility
• Trace — sum of the diagonal, appears in many derivations and losses
• Rank and linear independence — how many genuinely distinct directions a matrix spans
• Null space and column space — what a matrix destroys and what it can produce
• Matrix as linear transformation — the geometric reading that makes everything else intuitive
• Eigenvalues and eigenvectors — directions preserved by a transformation, scaled by a factorexplainer ↗
• Eigendecomposition — rebuilding a matrix from its eigen-structure
• Singular Value Decomposition (SVD) — the universal decomposition; underlies PCA, pseudo-inverse, and low-rank approximation
• Low-rank approximation — compressing a matrix by keeping only the largest singular values
• Positive definite and semi-definite matrices — the curvature condition behind convexity and covariance
• Covariance matrices — how features vary together, and why they must be symmetric PSD
• QR decomposition — orthogonal factorization used for numerically stable least squares
• Cholesky decomposition — efficient factorization for PSD matrices, used in Gaussian sampling
• LU decomposition — factorization for solving linear systems
• Moore–Penrose pseudo-inverse — the least-squares solution when a true inverse doesn't exist
• Projections and least squares geometry — why the residual is orthogonal to the fitted subspace
• Matrix calculus identities — derivatives with respect to vectors and matrices, needed to derive any gradient
• Broadcasting rules — how NumPy and PyTorch align mismatched shapes, and the bugs this causes
• Einstein summation (einsum) — a single notation for expressing arbitrary tensor contractions

1.2 Calculus

• Limits and continuity — the formal groundwork for derivatives
• Derivatives and differentiation rules — product, quotient, and power rules
• The chain rule — the single most important rule in ML; backpropagation is nothing else
• Partial derivatives — rate of change with respect to one variable holding others fixed
• Gradients — the vector of partial derivatives, pointing in the direction of steepest ascent
• Directional derivatives — rate of change along an arbitrary direction
• Jacobian matrix — all first derivatives of a vector-valued function
• Hessian matrix — all second derivatives; encodes local curvature
• Taylor series expansion — local polynomial approximation, the basis of Newton's method
• Critical points, minima, maxima, saddle points — classifying stationary points via the Hessian
• Convex and non-convex functions — whether local optimality implies global optimality
• Integration and the fundamental theorem — needed for probability densities and expectations
• Multivariable integration — integrating over joint distributions
• Automatic differentiation — forward and reverse mode; how frameworks compute exact gradients without symbolic math
• Computational graphs — representing a computation as a DAG so gradients can flow backward through it

1.3 Probability

• Sample spaces, events, and axioms of probability — the formal setup
• Conditional probability — probability given that something else is knownexplainer ↗
• Independence and conditional independence — the assumption that makes most models tractable
• Bayes' theorem — inverting conditional probabilities; the backbone of probabilistic ML
• Prior, likelihood, posterior, evidence — the four terms of Bayesian inference and how they interact
• Random variables (discrete and continuous) — mapping outcomes to numbers
• Probability mass and density functions — the discrete and continuous descriptions of a distribution
• Cumulative distribution functions — probability accumulated up to a point
• Expectation — the probability-weighted average, and its linearity
• Variance and standard deviation — spread around the mean
• Covariance and correlation — joint variation, and why correlation is scale-free
• Moments and moment generating functions — summarizing distribution shape
• Joint, marginal, and conditional distributions — and how to move between them
• Bernoulli distribution — a single binary trial; the noise model behind logistic regression
• Binomial distribution — counts of successes across independent trials
• Categorical and multinomial distributions — the multi-class generalizations
• Poisson distribution — counts of rare events over an interval
• Uniform distribution — equal density over a range
• Gaussian / normal distribution — the default continuous distribution and why it appears everywhere
• Multivariate Gaussian — the joint version, parameterized by a mean vector and covariance matrix
• Exponential and gamma distributions — waiting times and positive-valued quantities
• Beta and Dirichlet distributions — distributions over probabilities; conjugate priors for Bernoulli and categorical
• Laplace distribution — heavier tails than Gaussian, the prior corresponding to L1 regularizationexplainer ↗
• Student's t-distribution — heavy-tailed, robust to outliers, and the basis of the t-test
• Conjugate priors — prior families that keep the posterior in the same family, making Bayesian updates analytic
• Law of large numbers — sample averages converge to the true mean
• Central limit theorem — why sums of independent things tend toward Gaussian
• Maximum Likelihood Estimation (MLE) — choosing parameters that make the observed data most probable
• Maximum A Posteriori (MAP) estimation — MLE with a prior; equivalent to regularized MLE
• Sampling methods (inverse transform, rejection, importance) — drawing from distributions you can't sample directly
• Markov chains and stationary distributions — memoryless stochastic processesexplainer ↗
• Monte Carlo methods — estimating quantities by simulation
• Markov Chain Monte Carlo (MCMC) — Metropolis–Hastings, Gibbs sampling, and Hamiltonian Monte Carlo

1.4 Statistics

• Population versus sample — the distinction that all inference rests on
• Descriptive statistics — mean, median, mode, quantiles, IQR, skewness, kurtosis
• Sampling methods and sampling bias — random, stratified, cluster, and how each fails
• Standard error — the variability of an estimate, not of the data
• Confidence intervals — interval estimates and what they do and don't mean
• Hypothesis testing framework — null and alternative hypotheses, test statistics, decision rules
• p-values — what they measure and the many ways they're misread
• Type I and Type II errors — false positives and false negatives, and the tradeoff between them
• Statistical power and sample size calculation — how much data you need to detect an effect
• t-tests — one-sample, two-sample, and paired comparisons of means
• ANOVA — comparing means across more than two groups
• Chi-squared tests — independence and goodness of fit for categorical data
• Non-parametric tests — Mann–Whitney, Wilcoxon, Kruskal–Wallis when distributional assumptions fail
• Multiple comparisons problem — why testing many hypotheses inflates false positives
• Bonferroni and Benjamini–Hochberg corrections — controlling family-wise error and false discovery rate
• Effect size — the magnitude of a difference, which p-values do not report
• Bootstrapping — resampling with replacement to estimate any statistic's uncertainty
• Jackknife resampling — leave-one-out resampling for bias and variance estimation
• Permutation tests — building a null distribution by shuffling labels
• Bias–variance decomposition — splitting expected error into three interpretable parts
• Estimator properties — unbiasedness, consistency, efficiency, sufficiency
• Correlation versus causation — the distinction that most misused analysis ignores
• Simpson's paradox — when aggregate and subgroup trends reverse
• Survivorship and selection bias — how the data you can see distorts what you conclude
• Frequentist versus Bayesian interpretation — two coherent philosophies of what probability means
• Experimental design — randomization, blocking, controls, and factorial designs

1.5 Optimization

• Objective, loss, and cost functions — naming conventions and what is actually being minimized
• Convexity and convex sets — the property that guarantees a unique global optimum
• Analytical optimization via derivatives — setting the gradient to zero and solving
• Gradient descent — the fundamental iterative algorithmexplainer ↗
• Learning rate and its effect — divergence, oscillation, and crawling
• Batch, stochastic, and mini-batch gradient descent — the three data-consumption regimes
• Momentum — accumulating velocity to accelerate through ravines
• Nesterov accelerated gradient — momentum with a look-ahead correction
• AdaGrad — per-parameter learning rates that decay with accumulated gradient
• RMSProp — AdaGrad with an exponential moving average, fixing its decay problem
• Adam and AdamW — momentum plus adaptive rates; AdamW fixes weight decay coupling
• Learning rate schedules — step decay, cosine annealing, warmup, cyclical rates, one-cycle
• Second-order methods — Newton's method, quasi-Newton, BFGS and L-BFGS
• Conjugate gradient — efficient iterative solving for large linear systems
• Coordinate descent — optimizing one parameter at a time; the standard Lasso solver
• Constrained optimization and Lagrange multipliers — optimizing subject to equality constraints
• KKT conditions — the generalization to inequality constraints, underlying SVMs
• Duality and the dual problem — reformulating a problem in terms of its constraints
• Proximal methods — handling non-differentiable penalties like L1
• Expectation–Maximization (EM) — alternating optimization for latent variable models
• Simulated annealing — probabilistic escape from local minima
• Genetic and evolutionary algorithms — population-based search without gradients
• Bayesian optimization — sample-efficient global optimization of expensive black-box functions
• Line search and trust region methods — controlling step size rigorously
• Saddle points in high dimensions — why they, not local minima, are the real obstacle in deep learning

1.6 Information Theory

• Information content / surprisal — how much a single outcome tells you
• Entropy — average uncertainty in a distribution
• Joint and conditional entropy — uncertainty in combinations and given knowledge
• Cross-entropy — the cost of encoding one distribution using another; the standard classification loss
• KL divergence — asymmetric distance between distributions, central to variational methods
• Jensen–Shannon divergence — the symmetric, bounded alternative
• Mutual information — how much knowing one variable reduces uncertainty about another
• Perplexity — exponentiated entropy; the standard language model metric
• Minimum description length — model selection framed as compression
• Channel capacity and coding theory basics — the origins of the field and its intuitions

1.7 Discrete Math, Algorithms & Numerical Computing

• Big-O notation and complexity analysis — reasoning about time and space cost
• Core data structures — arrays, hash maps, trees, heaps, graphs and their tradeoffs
• Sorting and searching algorithms — foundational, and directly relevant to retrieval
• Recursion and dynamic programming — the technique behind Viterbi, sequence alignment, and edit distance
• Graph algorithms — BFS, DFS, shortest paths, connected components, topological sort
• Greedy algorithms — locally optimal choices, and when they're globally correct
• Set theory and combinatorics — counting arguments used throughout probability
• Floating point representation — precision limits, and why 0.1 + 0.2 ≠ 0.3
• Numerical stability — catastrophic cancellation, overflow, underflow
• The log-sum-exp trick — computing softmax and log-likelihoods without overflowing
• Condition number — how sensitive a problem is to small input perturbations
• Vectorization — replacing loops with array operations for orders-of-magnitude speedups
• Sparse matrix representations — CSR, CSC, COO and when they pay off

2.1 Python

• Core syntax and control flow — the baseline
• Data types and mutability — lists, tuples, dicts, sets, and the aliasing bugs mutability causes
• Comprehensions and generators — concise iteration and lazy evaluation for large data
• Functions, args/kwargs, and scope — including the mutable default argument trap
• Object-oriented programming — classes, inheritance, composition, dunder methods
• Decorators — wrapping functions; how framework APIs are built
• Context managers — the `with` statement for resource handling
• Error handling and custom exceptions — failing loudly and informatively
• Type hints and static checking — annotations, mypy, pyright
• Modules, packages, and imports — structuring a project so it's importable
• Virtual environments — venv, conda, uv, poetry for reproducible dependencies
• Package management and dependency pinning — pip, requirements files, lockfiles
• Standard library essentials — itertools, collections, functools, pathlib, json, re, datetime
• Multiprocessing and threading — the GIL and when each actually helps
• Async programming — asyncio for I/O-bound workloads like API calls
• Profiling and optimization — cProfile, line_profiler, memory_profiler
• Testing — pytest, fixtures, parametrization, mocking
• Logging — structured logging instead of print statements

2.2 Numerical & Data Libraries

• NumPy arrays and dtypes — the foundation everything else is built on
• NumPy indexing, slicing, and fancy indexing — selecting data without loops
• NumPy broadcasting — implicit shape alignment and its rules
• NumPy random number generation — the modern Generator API and seeding for reproducibility
• pandas Series and DataFrame — labeled tabular dataexplainer ↗
• pandas indexing (loc, iloc, boolean masks) — the three ways to select rows and columns
• pandas groupby and aggregation — split-apply-combine
• pandas merge, join, and concat — combining tables, and the cardinality bugs that follow
• pandas reshaping — pivot, melt, stack, unstack
• pandas time series functionality — resampling, rolling windows, date offsets
• Handling missing data in pandas — fillna, dropna, interpolate, and the NaN semantics
• Polars — the faster, lazily-evaluated modern alternative to pandas
• SciPy — optimization, statistics, sparse matrices, signal processing, distance metrics
• Dask — parallel and out-of-core computation with a pandas-like API
• Apache Spark / PySpark — distributed data processing at cluster scale
• Apache Arrow and Parquet — columnar in-memory and on-disk formats

2.3 ML Frameworks

• scikit-learn estimator API — fit, predict, transform and the consistency that makes it composable
• scikit-learn pipelines — chaining preprocessing and models so cross-validation stays honest
• scikit-learn ColumnTransformer — different preprocessing for different column types
• PyTorch tensors and autograd — the dominant research framework
• PyTorch nn.Module and model definition — building networks compositionally
• PyTorch DataLoader and Dataset — efficient batching, shuffling, and parallel loading
• PyTorch training loop — forward, loss, backward, step, and zero_grad
• PyTorch Lightning — removing boilerplate and standardizing training structure
• TensorFlow and Keras — the alternative ecosystem, still common in production
• JAX — functional transformations, jit, grad, vmap, pmap
• ONNX — cross-framework model interchange for deployment
• XGBoost, LightGBM, CatBoost APIs — the gradient boosting libraries that win most tabular competitions
• Hugging Face transformers — pretrained models and pipelines
• Hugging Face datasets and tokenizers — data loading and fast tokenization
• statsmodels — classical statistical modeling with proper inference tables

2.4 Visualization

• matplotlib fundamentals — figures, axes, and the object-oriented interface
• seaborn — statistical plots with sensible defaults
• plotly — interactive charts for exploration and dashboards
• altair — declarative grammar-of-graphics plotting
• Plot selection — choosing the right chart for the question being asked
• Visualization for model diagnostics — residual plots, learning curves, confusion matrices, calibration curves
• Dashboarding — Streamlit, Gradio, Dash for shareable interfaces

2.5 Data Access & Engineering

• SQL SELECT, WHERE, ORDER BY, LIMIT — the basics
• SQL JOINs — inner, left, right, full, cross, and self-joins
• SQL aggregation and GROUP BY / HAVING — summarizing at the database
• SQL window functions — running totals, rankings, lags, and partitioned aggregates
• SQL CTEs and subqueries — structuring complex queries readably
• Query optimization and indexing — why a query is slow and what to do about it
• Database types — relational, document, key-value, columnar, vector, graph
• Data warehouses and lakes — Snowflake, BigQuery, Redshift, Delta Lake, Iceberg
• ETL and ELT pipelines — extracting, transforming, and loading data on a schedule
• Workflow orchestration — Airflow, Prefect, Dagster
• dbt — version-controlled, tested SQL transformations
• Streaming data — Kafka, event-driven pipelines, windowed aggregation
• APIs and web scraping — REST, pagination, rate limiting, robots.txt and the ethics of collection
• File formats — CSV, JSON, JSONL, Parquet, Avro, HDF5 and their tradeoffs

2.6 Engineering Practice

• Command line proficiency — bash, pipes, grep, sed, awk, find, ssh
• Git fundamentals — commit, branch, merge, rebase, resolving conflicts
• Git collaboration — pull requests, code review, branching strategies
• Jupyter notebooks — and their well-documented reproducibility problems
• IDE proficiency — VS Code or PyCharm, debugger, breakpoints, refactoring tools
• Docker — containerizing an environment so it runs identically elsewhere
• Docker Compose — multi-container local development stacks
• Kubernetes basics — orchestration for scaled deployment
• CI/CD — GitHub Actions or similar, running tests and builds automatically
• Cloud fundamentals — compute, storage, IAM on AWS, GCP, or Azure
• GPU environments — CUDA, drivers, memory management, and the version-mismatch tax
• Code quality tooling — ruff, black, isort, pre-commit hooks
• Configuration management — Hydra, OmegaConf, or plain YAML for experiment configs
• Reproducibility practice — seeding, environment capture, deterministic ops, artifact hashing

3.1 Acquisition & Understanding

• Problem framing — translating a business or research question into an ML task, or deciding it isn't one
• Defining the target variable — often the highest-leverage and most-neglected decision in a project
• Data collection strategies — logging, surveys, scraping, purchase, synthetic generation
• Data documentation — datasheets, data dictionaries, provenance records
• Exploratory data analysis (EDA) — systematically looking at the data before modeling it
• Univariate analysis — distributions, ranges, and outliers per feature
• Bivariate and multivariate analysis — relationships, correlations, and interactions
• Target distribution analysis — class balance, skew, and whether the target needs transformation
• Data profiling tools — ydata-profiling, sweetviz for fast first passes

3.2 Cleaning & Preparation

• Missing data mechanisms — MCAR, MAR, MNAR and why the distinction changes the correct fix
• Deletion strategies — listwise and pairwise deletion, and the bias they introduce
• Simple imputation — mean, median, mode, constant fill
• Advanced imputation — KNN imputation, iterative/MICE, model-based imputation
• Missingness indicators — adding a binary flag because absence often carries signal
• Outlier detection — z-score, IQR, isolation forest, domain rules
• Outlier treatment — removal, winsorizing, capping, transformation, robust models
• Duplicate detection and removal — exact and fuzzy matching
• Data type correction and parsing — dates, numerics stored as strings, mixed types
• String cleaning and standardization — whitespace, casing, encoding, unicode normalization
• Entity resolution and record linkage — deciding when two records describe the same thing
• Handling inconsistent categories — merging typos and near-duplicate labels
• Unit and currency normalization — a classic silent-error source

3.3 Feature Engineering

• Feature scaling: standardization — mean 0, standard deviation 1
• Feature scaling: min-max normalization — squeezing into a fixed range
• Robust scaling — using median and IQR so outliers don't dominate
• Log and power transforms — Box-Cox, Yeo-Johnson for skewed features
• Quantile transformation — forcing a feature into a uniform or normal distribution
• One-hot encoding — and the dummy variable trap
• Ordinal encoding — when categories have a genuine order
• Target encoding — replacing categories with target statistics, and the leakage it invites
• Frequency and count encoding — encoding by how often a category appears
• Hashing trick — fixed-width encoding for very high cardinality features
• Binning and discretization — equal-width, equal-frequency, and supervised binning
• Polynomial and interaction features — explicitly modeling combined effects
• Ratio and difference features — often more predictive than the raw quantities
• Date and time features — hour, day of week, month, holiday flags, elapsed time
• Cyclical encoding — sine/cosine pairs so December is close to January
• Aggregation features — group statistics joined back onto rows
• Text features — length, token counts, TF-IDF, embeddings
• Domain-driven features — the ones that actually win, and the reason domain expertise matters
• Automated feature engineering — featuretools, deep feature synthesis and their limits
• Feature selection: filter methods — correlation, chi-squared, mutual information, variance threshold
• Feature selection: wrapper methods — forward selection, backward elimination, recursive feature elimination
• Feature selection: embedded methods — L1 regularization, tree importances
• Feature stores — centralized, versioned feature definitions shared between training and serving

3.4 Splitting, Leakage & Imbalance

• Train / validation / test split — three sets and the distinct job each performs
• Random splitting — the default, and when it's wrong
• Stratified splitting — preserving class proportions across splits
• Group-aware splitting — keeping all records for a subject or customer in one split
• Time-based splitting — chronological splits for any temporal problem
• Data leakage — the single most common cause of models that work in notebooks and fail in production
• Target leakage — features that encode the answer or are only available after the fact
• Train-test contamination — preprocessing fitted on the full dataset before splitting
• Temporal leakage — using future information to predict the past
• Class imbalance: the problem — why accuracy stops being informative
• Random oversampling and undersampling — the simplest rebalancing methods
• SMOTE and variants — synthesizing minority examples, and the calibration damage it causes
• Class weighting — reweighting the loss instead of resampling the data
• Threshold moving — usually the first thing to try, and often sufficient on its own
• Data augmentation for tabular data — noise injection, mixup, synthetic generation
• Synthetic data generation — GANs, VAEs, and simulation for privacy or scarcity

3.5 Labeling & Data Quality

• Annotation guidelines — writing instructions precise enough for consistent labeling
• Inter-annotator agreement — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha
• Label noise — its effect on training and methods for detecting it
• Weak supervision — labeling functions and programmatic labeling, e.g. Snorkel
• Active learning — selecting the most informative examples to label next
• Crowdsourcing — platform mechanics, quality control, and the labor ethics involved
• Data versioning — DVC, LakeFS, or dataset hashes so experiments are reproducible
• Data validation — Great Expectations, pandera for schema and distribution checks in pipelines

• What machine learning is — fitting parameters from data rather than specifying rules
• Supervised learning — learning a mapping from inputs to labeled outputs
• Unsupervised learning — finding structure without labels
• Semi-supervised learning — a small labeled set plus a large unlabeled one
• Self-supervised learning — generating labels from the data's own structure; the engine behind modern pretraining
• Reinforcement learning — learning from interaction and delayed reward
• Online versus batch learning — updating continuously versus retraining periodically
• Parametric versus non-parametric models — fixed capacity versus capacity that grows with data
• Discriminative versus generative models — modeling P(y|x) versus modeling P(x,y)
• Inductive bias — the assumptions a model makes that let it generalize at all
• The generalization problem — performing well on data you've never seen
• Overfitting — fitting noise, recognized by a large train-test gap
• Underfitting — insufficient capacity, recognized by poor performance everywhere
• The bias–variance tradeoff — the central tension in model selection
• Model capacity and complexity — what determines where a model sits on that tradeoff
• Regularization — any modification intended to reduce generalization error
• L1 regularization — sparsity and implicit feature selection
• L2 regularization — smooth shrinkage and stability under correlated features
• Elastic net — combining both penalties
• Early stopping — halting training when validation performance turns
• Cross-validation — k-fold, stratified k-fold, leave-one-out, repeated k-fold
• Nested cross-validation — honest performance estimation when you also tune hyperparameters
• Time series cross-validation — expanding and rolling window splits
• Hyperparameters versus parameters — what you set versus what is learned
• Grid search — exhaustive search over a specified grid
• Random search — usually more efficient than grid search in high dimensions
• Bayesian hyperparameter optimization — Optuna, Hyperopt, and surrogate-model search
• Successive halving and Hyperband — allocating budget adaptively across configurations
• The curse of dimensionality — why distance and density stop being meaningful in high dimensions
• No free lunch theorem — no algorithm is best across all possible problems
• Occam's razor in model selection — preferring the simplest adequate model
• Ensemble learning — combining models to reduce variance or bias
• Bagging — bootstrap aggregation, training on resampled data in parallel
• Boosting — sequentially fitting models to the previous ones' errors
• Stacking and blending — training a meta-model on base model predictions
• Voting classifiers — hard and soft voting across diverse models
• Baselines — the naive comparison every project needs before claiming success
• Model selection criteria — AIC, BIC, and cross-validated performance
• Learning curves — performance against training set size, and diagnosing bias versus variance from them
• Validation curves — performance against a hyperparameter value

5.1 Linear Models

• Linear regression — predicting a continuous target as a weighted sum of featuresexplainer ↗
• Ordinary least squares — minimizing squared residuals, with a closed-form solution
• The normal equation — the exact algebraic solution and its O(p³) cost
• Gradient descent for linear regression — the iterative alternative that scales
• Assumptions of linear regression — linearity, independence, homoscedasticity, normality of errors, no multicollinearity
• Residual analysis — diagnosing model failure from the pattern of errors
• Polynomial regression — curved fits that remain linear in the parameters
• Ridge regression — L2-penalized least squares
• Lasso regression — L1-penalized least squares with automatic feature selection
• Elastic net regression — the combined penalty
• Logistic regression — predicting binary class probability via the sigmoidexplainer ↗
• The sigmoid function and the logit link — mapping the real line to (0,1) and back
• Odds, log-odds, and odds ratios — how logistic coefficients must actually be interpreted
• Log loss / binary cross-entropy — the objective, and why squared error fails here
• Maximum likelihood estimation for logistic regression — no closed form, hence iterative solvers
• Newton–Raphson and IRLS — the second-order solvers classical libraries use
• Perfect separation — when the likelihood has no finite optimum, and why regularization fixes it
• Decision threshold selection — a cost decision, separate from the model itself
• Softmax / multinomial logistic regression — the multi-class generalization
• One-vs-rest and one-vs-one — strategies for making binary classifiers multi-class
• Generalized linear models (GLMs) — the unifying framework of link functions and error families
• Poisson and negative binomial regression — modeling counts and overdispersed counts
• Ordinal logistic regression — for ordered categorical outcomes
• Quantile regression — modeling conditional quantiles instead of the mean
• Robust regression — Huber and RANSAC for outlier-heavy data
• Generalized additive models (GAMs) — smooth per-feature functions that stay interpretable
• Linear and quadratic discriminant analysis (LDA/QDA) — generative classifiers with Gaussian class densities

5.2 Tree-Based Models

• Decision trees — recursive splitting into regions
• Split criteria — Gini impurity, entropy/information gain, variance reduction
• Tree pruning — pre-pruning by depth or leaf size, cost-complexity post-pruning
• Advantages and failure modes of single trees — interpretability versus high variance
• Random forests — bagged trees with random feature subsets at each splitexplainer ↗
• Out-of-bag error estimation — free validation from the bootstrap process
• Extremely randomized trees (Extra Trees) — random split thresholds for extra variance reduction
• AdaBoost — reweighting misclassified examples across sequential weak learners
• Gradient boosting machines — fitting each new tree to the gradient of the loss
• XGBoost — regularized gradient boosting with second-order optimization
• LightGBM — leaf-wise growth and histogram binning for speed on large data
• CatBoost — ordered boosting and native categorical handling
• Key boosting hyperparameters — learning rate, depth, subsample, colsample, min child weight, n_estimators
• Early stopping in boosting — using a validation set to choose the number of trees
• Feature importance methods — split gain, permutation importance, and why the default is biased
• Monotonic constraints — enforcing known directional relationships for business acceptability
• Why gradient boosting dominates tabular ML — and the conditions under which it doesn't

5.3 Instance-Based & Kernel Methods

• k-Nearest Neighbors — classification and regression by local majority or average
• Distance metrics — Euclidean, Manhattan, Minkowski, cosine, Hamming, Mahalanobis
• Choosing k and weighting by distance — the bias-variance dial for kNN
• Curse of dimensionality for kNN — why it degrades badly as features multiply
• Efficient nearest neighbor search — KD-trees, ball trees, HNSW, IVF, product quantization
• Support Vector Machines — maximum margin classification
• Hard versus soft margin — the C parameter and tolerance for violations
• Support vectors — the small subset of points that determine the boundary
• The kernel trick — implicit high-dimensional mapping without explicit computation
• Common kernels — linear, polynomial, RBF/Gaussian, sigmoid
• Support vector regression — the epsilon-insensitive tube formulation
• SVM strengths and limits — powerful on small-to-medium data, poor scaling past ~100k rows

5.4 Probabilistic Classifiers

• Naive Bayes — Bayes' theorem with a strong conditional independence assumption
• Gaussian, multinomial, and Bernoulli naive Bayes — variants for different feature types
• Laplace / additive smoothing — handling zero-probability events
• Why naive Bayes works despite a false assumption — argmax survives badly calibrated probabilities
• Bayesian linear regression — posterior distributions over coefficients rather than point estimates
• Gaussian processes — non-parametric Bayesian regression with built-in uncertainty estimates
• Kernel selection for GPs — RBF, Matérn, periodic, and composing kernels

6.1 Clustering

• k-Means — partitioning into k clusters by minimizing within-cluster variance
• Lloyd's algorithm and k-means++ — the standard iteration and smart initialization
• Choosing k — elbow method, silhouette analysis, gap statistic
• k-Means assumptions and failure modes — spherical, equally-sized clusters and what breaks otherwise
• k-Medoids / PAM — using actual data points as centers, robust to outliers
• Hierarchical clustering — agglomerative and divisive strategies
• Linkage criteria — single, complete, average, Ward and their differing cluster shapes
• Dendrograms — reading the merge tree and cutting it at a chosen height
• DBSCAN — density-based clustering that finds arbitrary shapes and labels noise
• HDBSCAN — hierarchical density clustering that handles varying densities
• OPTICS — ordering points to extract clustering structure across density scales
• Gaussian Mixture Models — soft clustering with probabilistic assignments
• EM algorithm for GMMs — alternating between responsibilities and parameter updates
• Spectral clustering — clustering in the eigenspace of a similarity graph
• Mean shift — mode-seeking without specifying cluster count
• Affinity propagation — message passing to select exemplars
• Clustering evaluation: internal — silhouette score, Davies–Bouldin, Calinski–Harabasz
• Clustering evaluation: external — adjusted Rand index, normalized mutual information
• The fundamental difficulty of clustering — there is no ground truth, so validation is inherently partial

6.2 Dimensionality Reduction

• Why reduce dimensions — visualization, noise reduction, compute, and the curse of dimensionality
• Principal Component Analysis (PCA) — orthogonal directions of maximum variance
• PCA via SVD — the numerically stable computation
• Explained variance ratio — choosing how many components to keep
• PCA assumptions and limits — linearity, and sensitivity to scaling
• Kernel PCA — non-linear PCA via the kernel trick
• Incremental and randomized PCA — for data that doesn't fit in memory
• Independent Component Analysis (ICA) — separating statistically independent source signals
• Non-negative Matrix Factorization (NMF) — parts-based decomposition with non-negativity
• Factor analysis — modeling observed variables as noisy combinations of latent factors
• Truncated SVD / LSA — dimensionality reduction that works on sparse matrices
• Multidimensional scaling (MDS) — preserving pairwise distances in low dimensions
• Isomap and locally linear embedding — manifold learning via geodesic and local structure
• t-SNE — non-linear visualization preserving local neighborhoods
• t-SNE pitfalls — cluster sizes and distances between clusters are not meaningful
• UMAP — faster than t-SNE with better global structure preservation
• Autoencoders for dimensionality reduction — learned non-linear compression
• Random projection — Johnson–Lindenstrauss and dimension reduction that ignores the data

6.3 Anomaly Detection & Association

• Anomaly detection problem types — point, contextual, and collective anomalies
• Statistical methods — z-score, modified z-score, Grubbs' test
• Isolation Forest — isolating anomalies via random partitioning
• One-Class SVM — learning a boundary around normal data
• Local Outlier Factor — density relative to neighbors
• Elliptic envelope — robust covariance estimation for Gaussian data
• Autoencoder reconstruction error — flagging what the model can't reproduce
• Evaluating anomaly detection — the label scarcity problem and precision at k
• Association rule mining — support, confidence, lift
• Apriori algorithm — frequent itemset generation by pruning
• FP-Growth — a faster tree-based alternative
• Market basket analysis — the canonical retail application

7.1 Regression Metrics

• Mean Squared Error (MSE) — average squared error, dominated by large misses
• Root Mean Squared Error (RMSE) — MSE in the units of the target
• Mean Absolute Error (MAE) — average absolute error, robust to outliers
• Mean Absolute Percentage Error (MAPE) — relative error, undefined near zero
• Symmetric MAPE and weighted MAPE — attempts to fix MAPE's asymmetry
• R² / coefficient of determination — fraction of variance explained
• Adjusted R² — R² penalized for feature count
• Huber loss — quadratic near zero, linear in the tails
• Pinball loss — the objective for quantile predictions
• Comparing RMSE and MAE — a large gap between them indicates outliers

7.2 Classification Metrics

• Confusion matrix — the four-cell foundation everything else derives from
• Accuracy — and why it's misleading under class imbalance
• Precision — of what you flagged, how much was correct
• Recall / sensitivity / true positive rate — of what was real, how much you caught
• Specificity / true negative rate — the recall of the negative class
• F1 score — harmonic mean of precision and recall
• Fβ score — weighting recall over precision or vice versa
• Macro, micro, and weighted averaging — aggregating metrics across multiple classes
• ROC curve — recall against false positive rate across all thresholds
• ROC-AUC — threshold-free ranking quality, optimistic under imbalance
• Precision-recall curve — the more informative view when positives are rare
• PR-AUC / average precision — area under the PR curve, with the positive rate as baseline
• Matthews correlation coefficient — a balanced measure usable even with skewed classes
• Cohen's kappa — agreement corrected for chance
• Log loss — penalizing confident errors, sensitive to calibration
• Brier score — mean squared error of predicted probabilities
• Cost-sensitive evaluation — assigning real costs to each confusion matrix cell
• Threshold selection — choosing the operating point from a cost argument rather than habit

7.3 Calibration & Ranking

• Probability calibration — whether predicted probabilities match observed frequencies
• Reliability diagrams — the visual calibration check
• Expected calibration error — the scalar summary
• Platt scaling — fitting a logistic regression on model outputs to recalibrate
• Isotonic regression calibration — non-parametric recalibration, needs more data
• What breaks calibration — resampling, heavy regularization, prevalence shift
• Ranking metrics — MAP, MRR, NDCG, precision@k, recall@k
• Hit rate and coverage — practical retrieval and recommendation metrics

7.4 Validation Practice

• Hold-out validation — a single split, and when it's enough
• k-fold cross-validation — the standard approach
• Stratified k-fold — preserving class balance in each fold
• Group k-fold — preventing subject-level leakage across folds
• Leave-one-out cross-validation — maximal data use, high variance, expensive
• Repeated cross-validation — reducing split-dependent variance in the estimate
• Nested cross-validation — separating model selection from performance estimation
• Time series splits — expanding window, rolling window, purged and embargoed splits
• Statistical comparison of models — paired t-tests, McNemar's test, and their assumptions
• Confidence intervals on metrics — bootstrapping the test set to quantify uncertainty
• Multiple hypothesis testing in model selection — how repeated evaluation inflates apparent gains

8.1 Neural Network Fundamentals

• Biological inspiration and its limits — a useful metaphor, not an accurate model
• The perceptron — the single-neuron linear classifier and its inability to solve XOR
• Multi-layer perceptron (MLP) — stacked layers with non-linearities in between
• Neurons, weights, biases, and layers — the vocabulary
• Forward propagation — computing outputs layer by layer
• Why non-linear activations are required — stacked linear layers collapse to one linear layer
• Universal approximation theorem — what it promises and what it conspicuously doesn't
• Backpropagation — the chain rule applied efficiently through the computational graph
• Computational graphs and reverse-mode autodiff — how frameworks implement it
• Depth versus width — the tradeoffs in how to spend parameter budget

8.2 Activation Functions

• Sigmoid — historic, saturating, mostly relegated to binary output layers
• Tanh — zero-centered sigmoid, still saturating
• ReLU — the modern default; fast, sparse, and prone to dying units
• Leaky ReLU and Parametric ReLU — small negative slope to prevent dead neurons
• ELU and SELU — smoother negatives, with self-normalizing properties
• GELU — the smooth activation used in most transformers
• Swish / SiLU — self-gated activation found by architecture search
• GLU variants (SwiGLU, GeGLU) — gated feed-forward blocks in modern LLMs
• Softmax — turning logits into a probability distribution over classes
• Softplus and Mish — smooth alternatives with occasional advantages
• Choosing activations — practical defaults per architecture and layer position

8.3 Loss Functions

• Mean squared error — the standard regression loss
• Mean absolute error and Huber — robust regression losses
• Binary cross-entropy — the standard binary classification loss
• Categorical cross-entropy — the multi-class version
• Sparse categorical cross-entropy — the same thing with integer labels
• Focal loss — down-weighting easy examples for heavy class imbalance
• Hinge loss — the SVM objective, occasionally used in networks
• KL divergence loss — matching distributions, used in VAEs and distillation
• Contrastive loss — pulling similar pairs together and pushing dissimilar apart
• Triplet loss — anchor, positive, negative for metric learning
• InfoNCE / NT-Xent — the contrastive objectives behind modern self-supervised learning
• CTC loss — sequence alignment without frame-level labels, used in speech and OCR
• Dice and IoU losses — segmentation objectives that handle class imbalance
• Custom and composite losses — weighting multiple objectives, and the tuning that requires

8.4 Training Mechanics

• Weight initialization — why zeros fail, and the symmetry-breaking requirement
• Xavier/Glorot and He initialization — variance-preserving schemes matched to activation type
• The vanishing gradient problem — signal dying across depth
• The exploding gradient problem — and gradient clipping as the fix
• Batch normalization — normalizing activations per mini-batch to stabilize training
• Layer normalization — per-sample normalization, the transformer standard
• RMSNorm, group norm, instance norm — variants for specific architectures
• Dropout — randomly zeroing units to prevent co-adaptation
• DropConnect, DropPath, stochastic depth — structured dropout variants
• Weight decay — L2 regularization in the optimizer, and its AdamW correction
• Data augmentation as regularization — expanding the effective dataset
• Label smoothing — softening targets to reduce overconfidence
• Early stopping and checkpointing — saving the best model, not the last one
• Batch size effects — on generalization, memory, and gradient noise
• Gradient accumulation — simulating large batches on limited memory
• Learning rate warmup — avoiding early instability in large models
• Learning rate finder — empirically locating a usable range
• Mixed precision training — fp16/bf16 for speed and memory, with loss scaling
• Gradient checkpointing — trading compute for memory by recomputing activations
• Overfitting a single batch — the standard sanity check before any real training run
• Debugging training — loss not decreasing, NaNs, exploding metrics and their usual causes
• Reproducibility in deep learning — seeds, deterministic kernels, and why exactness is hard on GPUs

8.5 Scaling & Distribution

• Data parallelism — replicating the model across devices, splitting the batch
• Model parallelism — splitting a single model across devices
• Tensor and pipeline parallelism — splitting within and across layers
• Distributed Data Parallel (DDP) — the standard PyTorch multi-GPU approach
• ZeRO and FSDP — sharding optimizer state, gradients, and parameters
• Gradient synchronization and all-reduce — the communication primitive and its cost
• Scaling laws — predictable relationships between compute, data, parameters, and loss
• Chinchilla-optimal training — the compute-optimal data-to-parameter ratio
• Curriculum learning — ordering training data from easy to hard

9.1 Convolutional Networks

• The convolution operation — local, weight-shared filters over a grid
• Kernels, filters, and feature maps — what a convolutional layer actually produces
• Stride, padding, and dilation — controlling output size and receptive field
• Receptive field — how much of the input a given unit can see
• Pooling layers — max, average, and global pooling for downsampling
• 1D, 2D, and 3D convolutions — for sequences, images, and volumes
• Depthwise separable convolutions — factorizing convolution for efficiency
• Transposed convolution — learned upsampling for generation and segmentation
• LeNet and AlexNet — the origin and the breakthrough
• VGG — depth through stacked small kernels
• Inception / GoogLeNet — multi-scale processing in parallel branches
• ResNet and residual connections — the skip connection that made very deep networks trainable
• DenseNet — connecting every layer to every subsequent layer
• MobileNet and EfficientNet — architectures designed for constrained compute
• ConvNeXt — modernized CNNs designed to compete with vision transformers

9.2 Recurrent Networks

• Recurrent neural networks — hidden state carried across time steps
• Backpropagation through time — unrolling the recurrence to compute gradients
• Truncated BPTT — limiting unroll depth for tractability
• LSTM — input, forget, and output gates with a persistent cell state
• GRU — a simpler gated unit with comparable performance
• Bidirectional RNNs — reading a sequence in both directions
• Sequence-to-sequence models — encoder-decoder for variable-length input and output
• Teacher forcing — feeding ground truth during training, and the exposure bias it creates
• Attention in seq2seq — the fix for the fixed-size bottleneck
• Why RNNs were displaced — sequential computation prevents parallel training

9.3 Transformers

• Self-attention — every token attending to every other
• Query, key, value projections — the three roles each token plays
• Scaled dot-product attention — the scoring function and why it's divided by √d
• Multi-head attention — parallel attention subspaces capturing different relations
• Cross-attention — the decoder attending to encoder outputs
• Causal masking — preventing a position from seeing the future
• Positional encoding — sinusoidal, learned, relative, and rotary (RoPE)
• The feed-forward network block — per-position transformation holding most parameters
• Residual connections and layer norm placement — pre-norm versus post-norm
• The full transformer block — how the components compose
• Encoder-only architectures — BERT and its descendants for understanding tasks
• Decoder-only architectures — GPT-style models for generation
• Encoder-decoder architectures — T5 and BART for transformation tasks
• Quadratic attention cost — the fundamental constraint on context length
• Efficient attention — FlashAttention, sparse attention, sliding window, linear attention
• KV caching — reusing computed keys and values during autoregressive generation
• Grouped-query and multi-query attention — reducing KV cache memory
• Mixture of Experts (MoE) — routing tokens to a subset of parameters for sparse scaling
• State space models — Mamba and S4 as sub-quadratic sequence model alternatives

9.4 Generative Architectures

• Autoencoders — encode to a bottleneck and reconstruct
• Denoising and sparse autoencoders — variants that learn more useful representations
• Variational autoencoders (VAEs) — probabilistic latent spaces with the reparameterization trick
• The ELBO — the variational objective balancing reconstruction and KL terms
• Generative Adversarial Networks (GANs) — a generator and discriminator trained in opposition
• GAN training instability — mode collapse, non-convergence, and vanishing discriminator gradients
• DCGAN, WGAN, StyleGAN — the architectures that made GANs work
• Conditional GANs and pix2pix — controlled generation and image-to-image translation
• CycleGAN — unpaired translation via cycle consistency
• Diffusion models — iteratively denoising from pure noise
• Forward and reverse diffusion processes — the noise schedule and the learned reversal
• DDPM and DDIM sampling — stochastic and deterministic sampling procedures
• Latent diffusion — diffusing in a compressed latent space, as in Stable Diffusion
• Classifier-free guidance — trading diversity for prompt adherence
• Normalizing flows — invertible transformations with exact likelihoods
• Autoregressive image models — PixelRNN, PixelCNN, and image tokenization
• Comparing generative families — sample quality, diversity, likelihood, and sampling speed

9.5 Graph Neural Networks

• Graph representations — adjacency matrices, edge lists, node and edge features
• Message passing framework — aggregate from neighbors, update, repeat
• Graph Convolutional Networks (GCN) — spectral-inspired neighborhood averaging
• GraphSAGE — sampling neighbors for scalability to large graphs
• Graph Attention Networks (GAT) — learned attention weights over neighbors
• Graph pooling and readout — producing graph-level representations
• Node, edge, and graph-level tasks — classification, link prediction, property prediction
• Over-smoothing — why deep GNNs collapse node representations
• Node2vec and DeepWalk — random-walk graph embeddings
• Knowledge graph embeddings — TransE, RotatE and relational reasoning
• Applications — molecules, social networks, recommendations, fraud rings, traffic

10.1 The Field

• What NLP is — turning text into computable representations and backexplainer ↗
• Why language is hard — ambiguity, discreteness, compositionality, long tail, world knowledge
• Levels of linguistic analysis — phonology, morphology, syntax, semantics, pragmatics, discourse
• The historical arc — rules, then statistics, then embeddings, then transformers, then scale
• Zipf's law — the power-law word frequency distribution and its consequences
• Corpora and benchmarks — GLUE, SuperGLUE, SQuAD, MMLU, HELM and their limitations

10.2 Text Preprocessing

• Unicode normalization — NFC/NFKC and the invisible bugs it prevents
• Encoding handling — UTF-8, byte-level processing, mojibake
• Whitespace and control character cleaning — including zero-width character attacks
• Case folding — when to lowercase and when case carries signal
• Punctuation handling — removing it destroys sentence structure
• Stopword removal — appropriate for search, harmful for neural models
• Stemming — Porter, Snowball, Lancaster; fast, crude, produces non-words
• Lemmatization — dictionary-based normalization requiring POS context
• Sentence segmentation — harder than splitting on periods, and still needed for chunking
• Language identification — detecting which language a document is in
• Text deduplication — exact, near-duplicate, and MinHash/LSH approaches
• Spelling correction and normalization — edit distance, noisy channel models

10.3 Tokenization

• Why tokenization matters — it determines what the model can perceive and what it costs
• Character-level tokenization — tiny vocabulary, very long sequences
• Word-level tokenization — meaningful units, unbounded vocabulary, out-of-vocabulary problem
• Subword tokenization — the universal modern compromise
• Byte Pair Encoding (BPE) — iteratively merging the most frequent adjacent pair
• Byte-level BPE — operating on bytes so nothing is ever unrepresentable
• WordPiece — likelihood-driven merges, used by BERT
• SentencePiece — language-agnostic tokenization requiring no pre-splitting
• Unigram language model tokenization — probabilistic vocabulary pruning
• Vocabulary size tradeoffs — sequence length against embedding table size
• Special tokens — CLS, SEP, MASK, PAD, BOS, EOS and their roles
• Token counting and cost — context windows, API pricing, and generation latency
• The multilingual token tax — why non-English text costs more and performs worse
• Tokenization and arithmetic — inconsistent number splitting and its effect on math ability
• Character-level blindness — why models struggle to count letters in words

10.4 Text Representation

• One-hot encoding — the trivial baseline with no notion of similarity
• Bag of words — counting vocabulary occurrences and discarding order
• N-gram features — recovering limited local order
• TF-IDF — weighting by term frequency and inverse document frequency
• BM25 — the refined ranking function still used in production search
• Latent Semantic Analysis — SVD over the term-document matrix
• Latent Dirichlet Allocation — probabilistic topic modeling
• word2vec — skip-gram and CBOW with negative sampling
• GloVe — global co-occurrence matrix factorization
• fastText — subword-aware embeddings handling unseen words
• Word analogies and vector arithmetic — the geometry of learned semantics
• Bias in word embeddings — encoded social associations and debiasing attempts
• Contextual embeddings — ELMo, BERT and the polysemy fix
• Sentence and document embeddings — Sentence-BERT, SimCSE, and pooling strategies
• Cross-encoders versus bi-encoders — accuracy against scalability in similarity scoring

10.5 Language Modeling

• The next-token prediction objective — the chain rule of probability applied to text
• N-gram language models — counting-based prediction with a truncated history
• Smoothing techniques — Laplace, Good-Turing, Kneser-Ney for unseen n-grams
• Neural language models — replacing counts with learned representations
• Masked language modeling — BERT's bidirectional pretraining objective
• Causal language modeling — GPT's autoregressive objective
• Perplexity — the standard intrinsic evaluation metric and its comparability limits
• Decoding strategies: greedy — always taking the argmax, and why it's repetitive
• Beam search — keeping multiple hypotheses, standard for translation
• Temperature sampling — the confidence dial reshaping the output distribution
• Top-k sampling — restricting to the k most likely tokens
• Nucleus / top-p sampling — restricting to the smallest set covering probability mass p
• Repetition and presence penalties — discouraging loops
• Constrained decoding — forcing outputs to match a grammar or JSON schema
• Speculative decoding — using a small draft model to accelerate a large one

10.6 Core NLP Tasks

• Text classification — topic, intent, spam, and document categorization
• Sentiment analysis — document-level, sentence-level, and aspect-based
• Part-of-speech tagging — assigning grammatical categories
• Named entity recognition — identifying and typing spans of text
• BIO/IOB tagging schemes — how span labels are encoded for token classification
• Dependency parsing — grammatical relations between words
• Constituency parsing — hierarchical phrase structure
• Semantic role labeling — who did what to whom
• Coreference resolution — linking mentions that refer to the same entity
• Word sense disambiguation — choosing the correct meaning in context
• Relation extraction — identifying typed relationships between entities
• Entity linking — grounding mentions to a knowledge base
• Question answering: extractive — selecting a span from a provided passage
• Question answering: abstractive — generating an answer in new words
• Text summarization: extractive — selecting the most important existing sentences
• Text summarization: abstractive — generating a condensed version, prone to invention
• Machine translation — statistical, neural, and multilingual approaches
• Natural language inference — entailment, contradiction, neutral
• Text similarity and paraphrase detection — semantic equivalence scoring
• Keyword and keyphrase extraction — RAKE, YAKE, KeyBERT
• Topic modeling — LDA, NMF, BERTopic for discovering document themes
• Text generation — open-ended and conditional
• Structured extraction — turning unstructured text into JSON, tables, or database rows
• Grammatical error correction — detecting and fixing language errors
• Text-to-SQL — translating natural language into database queries

10.7 Information Retrieval

• The retrieval problem — ranking documents by relevance to a query
• Inverted indexes — the classical data structure behind fast text search
• Sparse retrieval — TF-IDF and BM25 scoring
• Dense retrieval — embedding queries and documents into a shared vector space
• Hybrid search — combining sparse and dense signals, usually beating either
• Approximate nearest neighbor search — HNSW, IVF, ScaNN, product quantization
• Vector databases — FAISS, Qdrant, Weaviate, Milvus, pgvector
• Reranking — cross-encoder rescoring of the top candidates
• Chunking strategies — fixed size, sentence-aware, semantic, hierarchical, and overlap
• Query expansion and rewriting — improving recall before retrieval
• Retrieval evaluation — recall@k, MRR, NDCG, and building a labeled retrieval set

10.8 Speech & Audio

• Audio representations — waveforms, spectrograms, mel-spectrograms, MFCCs
• Automatic speech recognition — audio to text
• CTC and attention-based ASR — the two dominant alignment approaches
• Whisper and modern end-to-end ASR — multilingual, robust, transformer-based
• Text-to-speech — Tacotron, VITS, neural vocoders
• Speaker diarization — determining who spoke when
• Voice activity detection — segmenting speech from silence and noise
• Audio classification — event detection, music tagging, acoustic scene analysis
• Speech-to-speech and voice cloning — and the impersonation risks involved

11.1 The Field

• What computer vision is — extracting structured meaning from pixels
• Why vision is hard — viewpoint, illumination, scale, occlusion, deformation, clutter, intra-class variation
• The historical arc — hand-crafted features, then CNNs, then vision transformers, then multimodal foundation models
• Benchmark datasets — MNIST, CIFAR, ImageNet, COCO, Open Images, ADE20K
• The ImageNet moment — why 2012 changed the field

11.2 Image Fundamentals

• Digital image representation — pixels, channels, bit depth, resolution
• Color spaces — RGB, BGR, HSV, LAB, grayscale and when each is useful
• Image file formats — JPEG, PNG, WebP, TIFF and lossy versus lossless tradeoffs
• Histograms and histogram equalization — analyzing and correcting intensity distribution
• Convolution and filtering — blurring, sharpening, and the origin of the CNN operation
• Edge detection — Sobel, Prewitt, Canny
• Corner and blob detection — Harris, FAST, DoG
• Morphological operations — erosion, dilation, opening, closing
• Thresholding and segmentation basics — Otsu, adaptive thresholding, watershed
• Geometric transformations — translation, rotation, scaling, affine, perspective warping
• Image pyramids — multi-scale representations
• Classical feature descriptors — SIFT, SURF, ORB, HOG
• Feature matching and homography — stitching, alignment, and RANSAC

11.3 Vision Data Handling

• Image preprocessing — resizing, cropping, normalization to dataset statistics
• Geometric augmentation — flips, rotations, crops, affine and elastic distortions
• Photometric augmentation — brightness, contrast, saturation, hue, noise, blur
• Advanced augmentation — Cutout, Mixup, CutMix, RandAugment, AutoAugment
• Augmentation libraries — albumentations, torchvision transforms, kornia
• Test-time augmentation — averaging predictions over transformed copies
• Handling variable image sizes — padding, resizing, and aspect-ratio-preserving strategies
• Efficient data loading — prefetching, caching, decoding on GPU, webdataset formats
• Annotation formats — COCO JSON, Pascal VOC XML, YOLO txt

11.4 Core Vision Tasks

• Image classification — assigning one or more labels to a whole image
• Multi-label classification — multiple simultaneously valid labels
• Fine-grained classification — distinguishing visually similar subcategories
• Object detection — locating and classifying multiple objects with bounding boxes
• Two-stage detectors — R-CNN, Fast R-CNN, Faster R-CNN with region proposals
• Single-stage detectors — YOLO family, SSD, RetinaNet for real-time detection
• Anchor-based versus anchor-free detection — FCOS, CenterNet and the simplification trend
• DETR and transformer detectors — set prediction with bipartite matching, no NMS
• Non-maximum suppression — removing duplicate overlapping detections
• Intersection over Union (IoU) — the overlap metric underpinning detection evaluation
• Mean Average Precision (mAP) — the standard detection metric and its variants
• Semantic segmentation — labeling every pixel with a class
• Instance segmentation — separating individual object instances at pixel level
• Panoptic segmentation — unifying semantic and instance segmentation
• U-Net — the encoder-decoder with skip connections, dominant in medical imaging
• Mask R-CNN — detection plus per-instance segmentation masks
• Segment Anything (SAM) — promptable, zero-shot segmentation foundation models
• Keypoint detection and pose estimation — locating joints and landmarks
• Face detection and recognition — and the substantial privacy and bias concerns
• Optical character recognition (OCR) — text detection and recognition in images
• Document understanding — layout analysis, table extraction, form parsing
• Image retrieval — finding visually or semantically similar images
• Depth estimation — monocular and stereo depth prediction
• Optical flow — per-pixel motion between frames
• 3D vision — point clouds, voxels, meshes, NeRF, Gaussian splatting
• Super-resolution — upscaling with learned detail
• Image denoising, deblurring, and inpainting — restoration tasks
• Anomaly detection in images — industrial defect detection with few or no defect examples

11.5 Vision Architectures & Training

• CNN architectures for vision — the ResNet/EfficientNet/ConvNeXt lineage
• Vision Transformers (ViT) — treating image patches as tokens
• Patch embedding — how an image becomes a sequence
• Swin Transformer — hierarchical windowed attention for dense prediction
• Hybrid CNN-transformer architectures — combining local and global inductive biases
• Transfer learning in vision — the default approach; pretrain on ImageNet and fine-tune
• Feature extraction versus full fine-tuning — freezing the backbone or not
• Self-supervised vision pretraining — SimCLR, MoCo, BYOL, DINO, MAE
• CLIP — contrastive image-text pretraining enabling zero-shot classification
• Object detection training tricks — focal loss, FPN, multi-scale training, mosaic augmentation
• Model efficiency for vision — pruning, quantization, distillation, edge deployment
• Video understanding — action recognition, temporal modeling, 3D convolutions, video transformers
• Object tracking — SORT, DeepSORT, ByteTrack for multi-object tracking over time

12.1 Foundations

• The RL problem setting — agent, environment, state, action, reward, policy
• Markov Decision Processes — the formal framework
• Return, discount factor, and horizon — how future reward is aggregated
• Value functions — state value V(s) and action value Q(s,a)
• The Bellman equations — recursive consistency conditions for value functions
• Policy versus value-based methods — two routes to a solution
• Exploration versus exploitation — the central dilemma
• Epsilon-greedy, softmax, and UCB exploration — practical exploration strategies
• On-policy versus off-policy learning — learning from your own actions or someone else's
• Model-based versus model-free RL — whether the agent learns environment dynamics
• Credit assignment problem — attributing delayed reward to earlier actions

12.2 Algorithms

• Multi-armed bandits — the simplest RL setting, with contextual variants
• Dynamic programming — policy iteration and value iteration with a known model
• Monte Carlo methods — learning from complete episodes
• Temporal difference learning — bootstrapping from incomplete episodes
• SARSA — on-policy TD control
• Q-learning — off-policy TD control, the classic algorithm
• Deep Q-Networks (DQN) — neural function approximation with replay and target networks
• DQN improvements — Double DQN, Dueling DQN, Prioritized Replay, Rainbow
• Policy gradient methods — directly optimizing the policy
• REINFORCE — the basic Monte Carlo policy gradient
• Actor-critic methods — combining value estimation with policy gradients
• A2C and A3C — synchronous and asynchronous advantage actor-critic
• Trust Region Policy Optimization (TRPO) — constrained policy updates
• Proximal Policy Optimization (PPO) — the practical workhorse, used in RLHF
• Deep Deterministic Policy Gradient (DDPG) — continuous action spaces
• TD3 and Soft Actor-Critic (SAC) — modern continuous control algorithms
• Monte Carlo Tree Search — planning by simulation, the core of AlphaGo
• AlphaZero and MuZero — self-play with learned models
• Offline / batch RL — learning from a fixed dataset without environment interaction
• Inverse RL and imitation learning — inferring reward from demonstrations
• Behavioral cloning — supervised learning from expert trajectories
• Reward shaping and reward hacking — the difficulty of specifying what you actually want
• RL environments — Gymnasium, MuJoCo, PettingZoo, Isaac Gym
• Sim-to-real transfer — domain randomization and the reality gap
• Multi-agent RL — cooperation, competition, and non-stationarity

• The recommendation problem — matching users to items at scale
• Content-based filtering — recommending items similar to what a user liked
• Collaborative filtering — using patterns across users
• User-based and item-based collaborative filtering — neighborhood methods
• Matrix factorization — latent factors for users and items
• Alternating Least Squares and SGD for MF — the two standard solvers
• Implicit versus explicit feedback — ratings against clicks, views, and dwell time
• Bayesian Personalized Ranking — pairwise ranking loss for implicit feedback
• Factorization machines — modeling feature interactions in sparse settings
• Neural collaborative filtering — replacing the dot product with a learned function
• Two-tower architectures — separate user and item encoders for scalable retrieval
• Wide and Deep, DeepFM, DLRM — production-scale recommendation architectures
• Sequential and session-based recommendation — modeling order with RNNs or transformers
• The cold start problem — new users, new items, and hybrid workarounds
• Candidate generation and ranking — the two-stage architecture used everywhere
• Diversity, novelty, and serendipity — objectives beyond raw accuracy
• Popularity bias and filter bubbles — feedback loops that narrow what users see
• Offline evaluation — precision@k, recall@k, NDCG, MAP and why offline gains often don't transfer
• Online evaluation — A/B testing, interleaving, and long-term metrics
• Position and presentation bias — why click data is not a clean relevance signal
• Counterfactual evaluation — inverse propensity scoring for logged bandit feedback

• Time series components — trend, seasonality, cyclicality, and residual noise
• Stationarity — why most classical methods require it
• Differencing and transformations — making a series stationary
• Augmented Dickey-Fuller and KPSS tests — testing for stationarity
• Autocorrelation and partial autocorrelation — ACF and PACF plots for model identification
• Time series decomposition — additive, multiplicative, and STL
• Moving averages and exponential smoothing — simple, double, and triple (Holt-Winters)
• ARIMA — autoregressive integrated moving average
• SARIMA and SARIMAX — seasonal and exogenous variable extensions
• Vector autoregression (VAR) — multivariate time series
• GARCH models — modeling time-varying volatility
• Prophet — decomposable additive forecasting designed for business series
• Feature engineering for time series — lags, rolling statistics, expanding windows, date parts
• Tree-based forecasting — reframing forecasting as tabular regression, often the strongest baseline
• Deep learning for time series — LSTMs, TCNs, N-BEATS, DeepAR
• Transformer forecasters — Informer, Autoformer, PatchTST, TimesNet
• Foundation models for time series — Chronos, TimeGPT and zero-shot forecasting
• Hierarchical forecasting and reconciliation — consistency across aggregation levels
• Probabilistic forecasting — prediction intervals and quantile forecasts
• Forecast evaluation metrics — MAE, RMSE, MAPE, sMAPE, MASE, pinball loss
• Backtesting — rolling-origin evaluation and walk-forward validation
• Multi-step forecasting strategies — recursive, direct, and multi-output
• Anomaly detection in time series — changepoints, seasonal decomposition residuals, forecasting error
• Survival analysis — Kaplan-Meier, Cox proportional hazards, and censored data
• Common pitfalls — random splits, look-ahead bias, ignoring holidays, and drift

15.1 Working With Large Models

• What an LLM is — a large transformer trained on next-token prediction, then aligned
• Pretraining — the expensive stage where language competence and world knowledge form
• Supervised fine-tuning — instruction-response pairs turning a continuer into an assistant
• RLHF — reward modeling and PPO to align with human preferences
• DPO and preference optimization variants — simpler alternatives to full RLHF
• Constitutional AI and RLAIF — using model feedback to reduce human labeling load
• Emergent capabilities — and the ongoing debate over whether they're real or metric artifacts
• Context windows — what fits, what it costs, and lost-in-the-middle effects
• Model selection — capability, latency, cost, context length, licensing, hosting

15.2 Prompting

• Zero-shot prompting — instruction only
• Few-shot prompting — in-context examples
• Chain-of-thought prompting — eliciting intermediate reasoning steps
• Self-consistency — sampling multiple reasoning paths and taking the majority
• ReAct — interleaving reasoning with tool calls
• System prompts and role framing — setting persistent behavior
• Output format specification — schemas, examples, and delimiters
• Prompt chaining and decomposition — breaking a task into reliable steps
• Prompt versioning and testing — treating prompts as code with a regression suite
• Prompt injection — the central security problem when data enters the prompt
• Prompt compression and caching — reducing cost on repeated context

15.3 Retrieval-Augmented Generation

• The RAG pattern — retrieve relevant context, then generate grounded in it
• Document ingestion and parsing — PDFs, HTML, tables, and the messiness of real sources
• Chunking strategy — size, overlap, semantic boundaries, hierarchical chunks
• Embedding model selection — dimension, domain fit, multilingual support, cost
• Vector store selection and indexing — HNSW parameters, filtering, metadata
• Hybrid retrieval — combining BM25 and dense search
• Reranking — cross-encoders to improve the final context set
• Query transformation — rewriting, decomposition, HyDE, multi-query
• Context assembly — ordering, deduplication, and fitting the budget
• Citation and attribution — grounding claims to sources
• RAG evaluation — retrieval recall, faithfulness, answer relevance, RAGAS-style metrics
• Graph RAG and structured retrieval — retrieving over knowledge graphs and databases
• Why RAG fails — usually retrieval, not generation

15.4 Fine-Tuning & Adaptation

• When to fine-tune versus prompt versus retrieve — the decision that saves the most money
• Full fine-tuning — updating all parameters, and its memory cost
• LoRA — low-rank adapters trained while the base model stays frozen
• QLoRA — LoRA on a quantized base model for single-GPU fine-tuning
• Adapter layers, prefix tuning, prompt tuning — other parameter-efficient methods
• Instruction dataset construction — quality, diversity, and format consistency
• Catastrophic forgetting — losing general capability while specializing
• Knowledge distillation — training a small model to imitate a large one
• Continued pretraining — domain adaptation on unlabeled in-domain text

15.5 Agents & Tools

• Tool and function calling — giving a model callable capabilities with typed schemas
• Agent loops — plan, act, observe, repeat, and bounding the iteration
• Multi-agent systems — specialization, delegation, and coordination overhead
• Memory — short-term context, long-term stores, and summarization strategies
• Error compounding — why multi-step reliability degrades geometrically
• Sandboxing and permissions — constraining what an agent can actually do
• Human-in-the-loop checkpoints — confirmation before irreversible actions
• Observability for agents — tracing, logging every tool call, and replayability

15.6 Evaluating Generative Systems

• Why generation evaluation is hard — many valid outputs and no reference list
• Reference-based metrics — BLEU, ROUGE, METEOR, chrF and their weak human correlation
• Embedding-based metrics — BERTScore, MoverScore
• LLM-as-judge — scalable evaluation, with position, length, and self-preference biases
• Pairwise comparison and Elo ranking — relative rather than absolute scoring
• Task-specific eval sets — the hundred hand-checked examples that beat any public benchmark
• Golden datasets and regression testing — catching quality drops before users do
• Benchmark contamination — public test sets leaking into training data
• Human evaluation protocols — rubrics, blind comparison, inter-rater agreement
• Red teaming — adversarial testing for harmful, unsafe, or leaked outputs
• Hallucination measurement — factuality checking and groundedness scoring

15.7 Serving & Optimization

• Inference optimization — KV caching, batching, continuous batching
• Quantization — INT8, INT4, GPTQ, AWQ, GGUF and the quality tradeoff
• Pruning and sparsity — removing weights with minimal quality loss
• Distillation for deployment — smaller student models for production
• Serving frameworks — vLLM, TGI, TensorRT-LLM, Ollama
• Streaming responses — token-by-token delivery and perceived latency
• Cost management — token accounting, caching, model routing, and cascades
• Latency budgets — time to first token versus total generation time
• Guardrails — input filtering, output validation, and refusal handling

• Bayesian inference — updating beliefs with evidenceexplainer ↗
• Prior selection — informative, weakly informative, and improper priors
• Posterior predictive distributions — predicting with full uncertainty propagated
• Conjugate models — analytically tractable Bayesian updates
• Markov Chain Monte Carlo — Metropolis-Hastings, Gibbs, Hamiltonian Monte Carlo, NUTS
• MCMC diagnostics — trace plots, R-hat, effective sample size, divergences
• Variational inference — approximating the posterior by optimization
• The evidence lower bound (ELBO) — the variational objective
• Probabilistic programming — PyMC, Stan, NumPyro, Pyro
• Hierarchical and multilevel models — partial pooling across groups
• Bayesian model comparison — Bayes factors, WAIC, LOO-CV
• Bayesian neural networks — distributions over weights
• Monte Carlo dropout — cheap approximate uncertainty from a standard network
• Deep ensembles — the simple, strong baseline for uncertainty estimation
• Aleatoric versus epistemic uncertainty — irreducible noise against reducible ignorance
• Conformal prediction — distribution-free prediction sets with coverage guarantees
• Hidden Markov Models — latent state sequence models with the Viterbi and forward-backward algorithms
• Kalman filters — recursive state estimation for linear Gaussian systems
• Particle filters — sequential Monte Carlo for non-linear state estimation
• Probabilistic graphical models — Bayesian networks and Markov random fields
• Belief propagation — inference by message passing on graphs

• Why correlation is not causation — confounding, selection, and reverse causality
• Potential outcomes framework — counterfactuals and the fundamental problem of causal inference
• Directed acyclic graphs (DAGs) — encoding causal assumptions explicitly
• Confounders, colliders, and mediators — and why controlling for the wrong one creates bias
• Backdoor and frontdoor criteria — identifying valid adjustment sets
• do-calculus — the formal rules for reasoning about interventions
• Randomized controlled trials — the gold standard and its practical limits
• A/B testing — design, sample size, sequential testing, and peeking problems
• Propensity score methods — matching, weighting, and stratification
• Inverse probability weighting — reweighting to simulate randomization
• Difference-in-differences — exploiting a policy change across groups over time
• Regression discontinuity — using a threshold as a natural experiment
• Instrumental variables — leveraging exogenous variation to handle unmeasured confounding
• Synthetic control — constructing a weighted comparison unit
• Uplift modeling / heterogeneous treatment effects — estimating who a treatment actually helps
• Causal forests and meta-learners — S-, T-, X-learners for treatment effect estimation
• Sensitivity analysis — how strong an unmeasured confounder would need to be
• Causal ML libraries — DoWhy, EconML, CausalML

• Transfer learning — reusing knowledge from a related task or domain
• Domain adaptation — handling a shift between training and deployment distributions
• Multi-task learning — shared representations across related objectives
• Few-shot learning — learning from a handful of examples
• One-shot and zero-shot learning — generalizing from one example or from descriptions alone
• Meta-learning — learning to learn; MAML, prototypical networks, matching networks
• Self-supervised learning — creating supervision from data structure alone
• Contrastive learning — SimCLR, MoCo, CLIP and the positive/negative pair framing
• Masked modeling — BERT and MAE style reconstruction objectives
• Semi-supervised learning — pseudo-labeling, consistency regularization, FixMatch
• Active learning — choosing which examples to label for maximum information gain
• Continual / lifelong learning — learning new tasks without forgetting old ones
• Curriculum learning — ordering training examples by difficulty
• Federated learning — training across decentralized data that never moves
• Multimodal learning — jointly modeling text, image, audio, and video
• Vision-language models — CLIP, BLIP, LLaVA, and multimodal LLMs
• Metric learning — learning embedding spaces where distance means similarity
• Neural architecture search — automating architecture design
• AutoML — automating the full pipeline, and where it helps and hurts

• Why interpretability matters — debugging, trust, regulation, and scientific understanding
• Intrinsic versus post-hoc interpretability — transparent models against explaining opaque ones
• Global versus local explanations — overall behavior against individual predictions
• Linear model coefficients — the most interpretable explanation, with scaling caveats
• Decision tree visualization — reading the rules directly
• Feature importance — impurity-based, permutation-based, and their biases
• Partial dependence plots — average marginal effect of a feature
• Individual conditional expectation (ICE) plots — per-instance versions revealing heterogeneity
• Accumulated local effects (ALE) — a correlation-robust alternative to PDPs
• LIME — local surrogate models around a single prediction
• SHAP — Shapley-value attributions with consistency guarantees
• SHAP variants — TreeSHAP, KernelSHAP, DeepSHAP and their cost tradeoffs
• Counterfactual explanations — the minimal change that would flip the outcome
• Anchors — high-precision rule-based explanations
• Saliency maps and gradient attribution — vanilla gradients, Integrated Gradients, SmoothGrad
• Grad-CAM — class-discriminative visual explanations for CNNs
• Attention visualization — and why attention weights are not reliable explanations
• Probing classifiers — testing what information a representation encodes
• Mechanistic interpretability — circuits, features, superposition, sparse autoencoders
• Concept-based explanations — TCAV and human-meaningful concept attribution
• Faithfulness versus plausibility — an explanation can be convincing and wrong
• Model cards and documentation — communicating capability and limitation honestly

• Sources of bias — historical, representation, measurement, aggregation, deployment
• Protected attributes and proxies — why dropping a variable doesn't remove its influence
• Fairness definitions — demographic parity, equalized odds, equal opportunity, calibration
• The impossibility results — why several intuitive fairness criteria cannot hold simultaneously
• Individual versus group fairness — treating similar individuals alike against parity across groups
• Bias measurement — disaggregated evaluation across subgroups
• Pre-processing mitigation — reweighting and resampling the training data
• In-processing mitigation — fairness constraints and adversarial debiasing
• Post-processing mitigation — group-specific thresholds and their legal complications
• Fairness toolkits — Fairlearn, AI Fairness 360, What-If Tool
• Algorithmic accountability — who is responsible when a model causes harm
• Transparency and the right to explanation — GDPR, the EU AI Act, sectoral regulation
• Consent and data provenance — whether the training data should have been collected
• Environmental cost — the energy and carbon footprint of large-scale training
• Labor considerations — annotation work, content moderation, and worker welfare
• Dual use and misuse — capabilities that serve both benign and harmful ends
• Automation and displacement — the socioeconomic effects of deployed systems
• AI safety — alignment, specification gaming, robustness, and interpretability as a safety tool
• Responsible release practices — staged rollout, access controls, model and system cards

• PII identification and handling — recognizing and protecting personal data
• Anonymization and pseudonymization — and why anonymization frequently fails
• k-anonymity, l-diversity, t-closeness — classical privacy guarantees and their limits
• Differential privacy — formal privacy with a quantifiable budget
• DP-SGD — training with gradient clipping and calibrated noise
• Federated learning for privacy — keeping raw data on device
• Secure multi-party computation and homomorphic encryption — computing on data you cannot see
• Membership inference attacks — determining whether a record was in the training set
• Model inversion and extraction attacks — reconstructing data or stealing a model through its API
• Training data memorization — verbatim reproduction and its privacy and copyright implications
• Adversarial examples — imperceptible perturbations that flip predictions
• Adversarial attack methods — FGSM, PGD, Carlini-Wagner
• Adversarial training and defenses — robustness at a cost to clean accuracy
• Data poisoning and backdoor attacks — corrupting training data to install hidden behavior
• Prompt injection and jailbreaking — the LLM-specific attack surface
• Supply chain security — untrusted model weights, pickle deserialization, dependency risk
• Regulatory compliance — GDPR, CCPA, HIPAA, the EU AI Act and sector-specific rules

22.1 Experimentation

• Experiment tracking — MLflow, Weights & Biases, Neptune, Comet
• Reproducible experiments — code, data, environment, and seed captured together
• Model registries — versioned, staged model artifacts with lineage
• Hyperparameter tuning at scale — Optuna, Ray Tune, distributed search
• Notebook to production — refactoring exploratory code into tested modules

22.2 Deployment

• Batch inference — scheduled scoring over large datasets
• Real-time inference — REST and gRPC APIs with latency requirements
• Streaming inference — scoring events as they arrive
• Edge and on-device deployment — mobile, embedded, browser, TFLite, Core ML, ONNX Runtime
• Model serialization — pickle, joblib, SavedModel, TorchScript, ONNX and their portability
• Serving frameworks — TorchServe, TF Serving, BentoML, Seldon, KServe, Triton
• API design for ML services — input validation, versioning, error handling, timeouts
• Containerization and orchestration — Docker images and Kubernetes deployment
• Autoscaling and load balancing — handling variable traffic economically
• Shadow deployment — running a new model alongside production without serving it
• Canary and blue-green deployment — gradual, reversible rollout
• A/B testing models in production — measuring the business metric, not the offline one
• Rollback procedures — the plan for when a deployment goes wrong

22.3 Monitoring & Maintenance

• Operational monitoring — latency, throughput, error rate, resource utilization
• Prediction monitoring — output distribution and confidence over time
• Data drift detection — PSI, KS test, KL divergence on input features
• Concept drift detection — when the input-output relationship itself changes
• Label delay and delayed feedback — evaluating when ground truth arrives late or never
• Performance monitoring — tracking live metrics against the offline estimate
• Training-serving skew — differences between training and inference preprocessing
• Alerting and on-call — thresholds that catch real problems without alert fatigue
• Retraining strategies — scheduled, triggered, or continuous
• Automated retraining pipelines — and the safeguards they require
• Model and data lineage — tracing any prediction back to the data and code that produced it
• Feature stores in production — consistent features across training and serving
• Cost monitoring — compute, storage, and inference spend per prediction
• Technical debt in ML systems — hidden feedback loops, entanglement, undeclared consumers
• Incident response for ML — debugging a model failure under time pressure
• Documentation — model cards, runbooks, architecture decision records

• CPU versus GPU versus TPU — what each is good at and why
• GPU architecture basics — SMs, warps, memory hierarchy, tensor cores
• CUDA fundamentals — kernels, streams, and the host-device boundary
• Memory bandwidth versus compute bound — diagnosing which limits your workload
• Arithmetic intensity and the roofline model — reasoning about achievable performance
• Precision formats — fp32, tf32, fp16, bf16, fp8, int8, int4
• Kernel fusion — reducing memory traffic by combining operations
• torch.compile and graph optimization — compiler-level speedups
• Profiling ML workloads — PyTorch profiler, Nsight, identifying the actual bottleneck
• Distributed training infrastructure — interconnects, NCCL, topology awareness
• Storage and I/O for training — keeping the GPU fed
• Cost-performance optimization — spot instances, right-sizing, scheduling

• Statistical learning theory — the formal framework for generalization
• Empirical risk minimization — what training actually optimizes
• PAC learning — probably approximately correct learnability
• VC dimension — a capacity measure and its generalization bounds
• Rademacher complexity — a data-dependent capacity measure
• Generalization bounds — and why classical bounds are vacuous for deep networks
• The bias-variance-noise decomposition — formally derived
• Double descent — why test error can fall again past the interpolation threshold
• The lottery ticket hypothesis — sparse subnetworks that train as well as the whole
• Neural tangent kernel — infinite-width networks as kernel methods
• Implicit regularization of SGD — why optimization choices affect which solution you find
• Expressivity versus learnability — what a model could represent against what it can be trained to find
• Universal approximation and its limits — existence proofs say nothing about efficiency
• Information bottleneck theory — compression as a lens on representation learning
• Computational learning complexity — what is provably hard to learn

• Reading research papers — abstract, figures, results, then method; and reading critically
• Reproducing papers — the fastest way to discover what papers omit
• Following the field — arXiv, conference proceedings, curated newsletters, without drowning
• Key conferences — NeurIPS, ICML, ICLR, CVPR, ACL, EMNLP, KDD
• Writing about your work — clear technical communication as a compounding skill
• Building a portfolio — end-to-end projects with real data and honest evaluation
• Kaggle and competitions — excellent for modeling craft, misleading about problem framing
• Open source contribution — reading production ML code written by other people
• Working with stakeholders — translating business problems into ML problems and back
• Scoping and estimating ML work — the inherent uncertainty and how to communicate it
• Knowing when not to use ML — rules, heuristics, or a SQL query are often the right answer
• Ethics in practice — raising concerns, refusing work, and documenting limitations