Logistic Regression — RoboticsFutures

01Anatomy

Two stages, one model

Logistic regression is a linear model with a translator bolted on the end. Stage one produces a number on the whole real line. Stage two converts that number into a probability.

Stage one — the linear score

z = β₀ + β₁x₁ + ⋯ + β_px_p

Identical to linear regression. This quantity is called the logit or the log-odds. It ranges from −∞ to +∞ and has no upper or lower bound.

Stage two — the sigmoid

σ(z) = ¹⁄_{1 + e^−z}

Also called the logistic function. Maps the whole real line into (0, 1), is symmetric about z = 0 where it returns exactly 0.5, and saturates at both ends.

Terms

y: Target. Binary — 0 or 1, negative or positive, no or yes. Which class you call "1" is your choice, and it flips the sign of every coefficient.
z: Logit / log-odds. The linear score before squashing.
p̂: Predicted probability that y = 1 given the features. The model's actual output.
β₀: Intercept. The log-odds when every feature is zero — the baseline rate.
βⱼ: Coefficient. Change in log-odds per one-unit increase in the feature. Not a change in probability.
threshold: Cutoff. The probability above which you declare class 1. Not part of the model — a separate decision you make afterwards.

The decision boundary is linear. You classify as 1 when p̂ > 0.5, which happens exactly when z > 0 — a straight line, plane, or hyperplane. The curve is in the probabilities, not the boundary.

Curved probabilities, straight boundary

With two features, the probability surface rises smoothly from one corner to the other — but every contour of constant probability is a straight line, and they are all parallel. The 0.5 contour is the decision boundary.

This is the model's central limitation and its central virtue: it can only carve the space with a straight cut, which is why it can be read as a simple weighted rule, and why it fails on data that needs a curved separation.

02Odds & log-odds

Why not just fit a straight line?

The obvious thing is to code the classes as 0 and 1 and run ordinary least squares. It half-works, and the ways it fails point directly at what the sigmoid is for.

Linear fit vs. sigmoid on the same binary outcome

It predicts impossible probabilities. A line has no ceiling. Push far enough along x and it confidently reports a 130% chance, or −20%. There is no coherent way to act on that.

The errors can't be Gaussian. For a fixed x, the outcome is either 0 or 1, so the residual takes exactly two values. Least squares' whole inferential apparatus assumes otherwise.

Variance isn't constant. A Bernoulli outcome has variance p(1−p) — largest at 0.5, near zero at the extremes. Heteroscedasticity is guaranteed, not accidental.

Outliers drag the boundary. A single far-out point in the correct class still pulls the line toward it, worsening classification for everyone else.

The chain: probability → odds → log-odds

The sigmoid isn't arbitrary. It's the inverse of a transformation that takes a bounded probability and stretches it onto the entire real line, which is precisely where a linear model is comfortable.

Probability	Odds p/(1−p)	Log-odds ln(odds)
0.01	0.010	−4.60
0.10	0.111	−2.20
0.25	0.333	−1.10
0.50	1.000	0.00
0.75	3.000	+1.10
0.90	9.000	+2.20
0.99	99.00	+4.60

logit(p) = ln(p / (1 − p))

Notice the symmetry: log-odds of 0 is a coin flip, and equal distances in either direction are mirror probabilities. Notice also how slowly it moves in the middle and how violently at the edges — going from 0.98 to 0.99 is the same distance in log-odds as going from 0.5 to 0.73.

ln(^p⁄_1−p) = β₀ + β₁x₁ + ⋯

Read the model this way and it is a linear regression — just on the log-odds scale rather than the outcome scale. Everything you know about linear models transfers, provided you remember which scale you're standing on.

03The objective

Log loss, and why not squared error

Linear regression minimizes squared error. Logistic regression maximizes the likelihood of the observed labels — equivalently, minimizes log loss, also called binary cross-entropy.

The loss

−¹⁄_n ∑ y ln p̂ + (1−y) ln(1−p̂)

Only one term survives per observation. If the true label is 1, you pay −ln p̂; if it's 0, you pay −ln(1−p̂). Either way, the cost is the negative log of the probability you assigned to what actually happened.

Confident and wrong is ruinous. Assign 0.999 to a class that turns out false and the loss goes to nearly 7; assign exactly 0 to something that happens and it is infinite. Log loss doesn't just want correct answers, it wants honest ones.

Cost paid vs. probability assigned, for each true label

Why not MSE here

It stops being convex. Squared error composed with the sigmoid produces a surface with local minima. Gradient descent can settle somewhere that isn't the answer.
The gradients vanish exactly when you need them. A badly wrong, highly confident prediction sits on the sigmoid's flat tail, so MSE's gradient goes to nearly zero and learning stalls. Log loss's gradient stays large.
It isn't the maximum-likelihood estimator. For Bernoulli outcomes, log loss is; squared error assumes the wrong noise model.

This is the same reason cross-entropy, not MSE, trains classification networks.

The elegant gradient

∇_β = ¹⁄_n X^⊤(p̂ − y)

The sigmoid's derivative and the log's derivative cancel almost completely, leaving prediction minus truth, weighted by the features — exactly the form of linear regression's gradient.

Two different models, two different losses, one update rule. That is not a coincidence: both are generalized linear models with their natural link function, and this cancellation is a general property of that family.

04Fitting

No closed form — only search

This is the sharpest practical break from linear regression. Setting the derivative to zero gives equations with β trapped inside an exponential, and no algebraic rearrangement frees it. The parameters must be found iteratively.

Method	How it works	Notes
Gradient descent	Step against ∇ repeatedly	Simple, scales to enormous data, needs a learning rate and scaled features
Newton–Raphson / IRLS	Uses the second derivative to jump straight toward the minimum	Converges in a handful of iterations; costs O(p³) per step. What statsmodels uses
L-BFGS	Approximates curvature without storing the full Hessian	scikit-learn's default solver — a good middle ground
liblinear	Coordinate descent	Strong on small data and L1 penalties
SAG / SAGA	Stochastic with variance reduction	Best on very large datasets; SAGA handles L1 and elastic net

The good news: log loss is convex in β. There is one minimum and no local traps, so every one of these solvers converges to the same answer given enough iterations. You are choosing speed, not correctness.

Perfect separation breaks it. If some combination of features splits the classes cleanly, the likelihood keeps improving as coefficients grow, and the optimum sits at infinity. You'll see enormous coefficients, absurd standard errors, and convergence warnings. Any regularization at all fixes it — which is why scikit-learn regularizes by default.

05Coefficients

Reading the weights

This is where most people go wrong. A logistic coefficient is not a change in probability. It is a change in log-odds, and the same coefficient produces a completely different probability shift depending on where you start.

Three ways to say the same thing

β = 0.7: A one-unit increase raises the log-odds by 0.7.
e^β = 2.01: Odds ratio. A one-unit increase roughly doubles the odds. This is the form to quote to a domain expert.
β/4 ≈ 0.18: Divide-by-four rule. Near p = 0.5, a one-unit increase moves the probability by at most about 18 percentage points. An upper bound, and only near the middle.

Odds are not probability. Doubling the odds from 1:100 takes you from 0.99% to 1.96%. Doubling them from 1:1 takes you from 50% to 67%. The same odds ratio, wildly different real-world impact.

Practical rules

Sign is unambiguous. Positive raises the probability, negative lowers it, everywhere. Only the magnitude is context-dependent.
Scale features before comparing magnitudes. Otherwise you're comparing the effect of one year against the effect of one dollar.
The intercept is the log-odds when all features are zero. Centre your features and it becomes the log-odds at the average case, which is usually meaningful.
Categorical coefficients are odds ratios relative to whichever level you dropped. Always state the reference level.
To report probability effects, compute average marginal effects — predict for everyone, nudge the feature, predict again, average the difference.

Statsmodels prints coefficients, standard errors, z-statistics and confidence intervals directly. Exponentiate the coefficient and its interval together to get the odds ratio with its interval.

06The threshold

The model gives probabilities. You choose the cutoff.

0.5 is a default, not a law, and it is very often the wrong choice. The threshold encodes how much you care about a false positive relative to a false negative — a business or clinical judgement, not a statistical one. Move the slider and watch the same model become a different classifier.

Threshold 0.50

Predicted probabilities by true class · 320 cases

Predicted 1

Predicted 0

Actual 1

—True positive

—False negative

Actual 0

—False positive

—True negative

Accuracy—

Precision—

Recall—

F1—

What the ROC curve actually is

Sweep the threshold from 1 down to 0 and plot recall against the false positive rate at every stop. The marker is your current cutoff. The area underneath — AUC — summarizes every threshold at once, so it measures the ranking the model produces rather than any particular decision.

An AUC of 0.5 is the diagonal: coin-flip ranking. It is also the reason AUC can look respectable on badly imbalanced data while the model is useless in production — the diagonal is a low bar when positives are rare.

Lower the threshold

Catch more positives, accept more false alarms. Right when a miss is expensive and a false alarm is cheap: disease screening, fraud pre-checks, safety alerts.

Raise the threshold

Only flag the confident cases. Right when acting is costly or intrusive: sending a technician, blocking a transaction, accusing someone of something.

Or don't threshold at all

Rank by probability and work down the list until resources run out. Often the most useful deployment — no cutoff needed, and it uses the full information the model produced.

07Evaluation

Metrics, and the ones that lie

Metric	Definition	When it's the right one
Accuracy	(TP+TN) / all	Balanced classes and symmetric costs only. Otherwise deeply misleading.
Precision	TP / (TP+FP)	Of everything you flagged, how much was real. Matters when acting on a false positive costs something.
Recall	TP / (TP+FN)	Of everything real, how much you caught. Matters when missing a positive is the expensive failure.
F1	Harmonic mean of the two	A single number when you need one and both errors matter roughly equally.
ROC-AUC	Area under recall vs. FPR	Threshold-free ranking quality. Optimistic under heavy imbalance.
PR-AUC	Area under precision vs. recall	The honest choice when positives are rare. Baseline is the positive rate, not 0.5.
Log loss	The training objective itself	Rewards well-calibrated probabilities, not just correct ordering.
Brier score	Mean squared error of the probabilities	Calibration plus discrimination in one number, gentler on confident mistakes than log loss.

The accuracy trap. If 1% of transactions are fraudulent, a model that answers "not fraud" every single time scores 99% accuracy. It has learned nothing. Whenever classes are imbalanced, report precision, recall and PR-AUC — and look at the confusion matrix itself.

Calibration — are the probabilities honest?

A model can rank perfectly and still lie about magnitudes. Calibration asks: of all the cases where the model said 70%, did roughly 70% turn out positive? Bucket the predictions, plot claimed against observed, and compare to the diagonal.

Reliability diagram

Logistic regression is usually well calibrated out of the box — a direct consequence of optimizing log loss, which is a proper scoring rule. This is one of its quiet advantages over models that need a calibration wrapper.

Three things break it: heavy regularization pulls probabilities toward the base rate; resampling to fix imbalance shifts every prediction upward; and training on a population with a different prevalence than production does the same.

Fix by recalibrating on a held-out set — Platt scaling or isotonic regression — or by correcting the intercept for the known prevalence shift.

08Regularization

Penalties, and scikit-learn's inverted dial

The same L1 and L2 penalties from linear regression apply, added to the log loss instead of the squared error, and for the same reasons: control variance, handle correlated features, keep coefficients from exploding.

L2 · Ridge

Shrinks smoothly, never to zero. The default in scikit-learn. Also the standard cure for perfect separation, since it forbids infinite coefficients.

L1 · Lasso

Zeroes out coefficients entirely, giving automatic feature selection and a model short enough to read. Needs the liblinear or saga solver.

Elastic net

Both, mixed by l1_ratio. Sparsity with stability under correlated features. Saga solver only.

The C parameter is inverted, and it catches everyone. scikit-learn parameterizes by C = 1/λ, so small C means strong regularization and large C means almost none. It defaults to C=1.0, which is real regularization — meaning scikit-learn's "plain" logistic regression is not the same model statsmodels fits, and their coefficients will differ. Set penalty=None if you want a true unpenalized fit.

Scale first, always. The penalty acts on raw coefficient magnitudes, so unscaled features are punished in proportion to their units. Since regularization is on by default here, this bites more often than it does in linear regression.

09Requirements

What it assumes — and what it doesn't

The assumption list is shorter than linear regression's. Normality of errors, constant variance, and a normally distributed outcome are all off the table, because the Bernoulli distribution already specifies the noise.

Requirement	Meaning	Detect	Fix
Linear in log-odds	Each feature's effect on the logit is a straight line	Box–Tidwell test; plot logit against the feature in bins	Add polynomial terms, splines, or bin the feature
Independence	Observations don't influence each other	Repeated measures per subject; clustered sampling	Mixed-effects or GEE models, clustered standard errors
No perfect separation	No feature combination splits classes cleanly	Huge coefficients, huge standard errors, convergence warnings	Regularize; Firth's penalized likelihood; drop the offending feature
No severe multicollinearity	Features aren't near-duplicates	VIF above ~5–10	Drop, combine, or use L2
Enough events	Roughly 10–20 events per feature	Count the rarer class, divide by feature count	Fewer features, regularization, or gather more data
Correct labels	The outcome means what you think	Domain review of how the label was constructed	Nothing technical fixes this one

Not required: normally distributed features, a linear relationship on the probability scale, equal class sizes, or homoscedasticity. Plenty of tutorials get this wrong.

10More classes

Beyond two outcomes

One-vs-rest

Train one binary model per class — this class against everything else — and take the highest score. Simple and parallelizable, but the probabilities come from separate models and won't sum to 1 without normalizing.

Multinomial · softmax

p_k = e^z_k ⁄ ∑_j e^z_j

One linear score per class, exponentiated and normalized so they sum to 1. A single joint fit with properly coupled probabilities. This is the generalization, and with two classes it reduces exactly to the sigmoid. It is also, precisely, the final layer of an image classifier.

Softmax carves the space into convex regions

Ordinal outcomes — mild, moderate, severe — deserve ordered logistic regression, which respects the ranking and fits a single slope with several thresholds. Treating an ordered outcome as unordered categories throws away real information.

11Imbalance

When one class is rare

Fraud, disease, churn, defects — the interesting class is usually the small one. Logistic regression handles imbalance better than its reputation suggests, provided you stop looking at accuracy.

Approaches, roughly in order of preference

Change the threshold, not the data. Usually sufficient. The model's ranking is often fine; only the cutoff was wrong.
Class weights. class_weight='balanced' reweights the loss so the rare class carries proportionally more. Cleaner than resampling and doesn't invent data.
Undersample the majority if you have abundant data and want faster training.
Oversample or SMOTE last. Synthetic minority points can help, but they fabricate structure and they always wreck calibration.

Everything above shifts your probabilities. Reweighting and resampling both change the effective base rate, so predictions come out systematically too high. If you need real probabilities rather than a ranking, recalibrate afterwards or correct the intercept by the known prevalence.

Sometimes rare is just rare. If the true probability of the event is 2%, a well-fitted model should mostly output small probabilities. That isn't a failure — it's the model being correct. The question is whether the ranking is useful, which PR-AUC will tell you.

12Where it sits

Its role in machine learning

Why it endures

The classification baseline. Same discipline as linear regression: if a boosted ensemble can't clearly beat it, the complexity isn't paying rent.
Calibrated by construction. The probabilities mean something without post-processing, which matters enormously when a human acts on them.
Auditable. Credit and clinical regulators frequently require a model whose reasoning can be written as a sentence per feature. This is that model.
The output layer of nearly every classifier. Sigmoid for binary, softmax for multiclass, cross-entropy loss — a neural network is a stack of feature extractors ending in exactly this.
Cheap and stable. Fast to train, trivial to serve, degrades gracefully.

When to move on

The boundary is genuinely curved and you can't hand-engineer the terms → gradient boosting, random forests, SVM with a kernel
Many high-order interactions → tree ensembles find them for you
Images, audio, text, sequences → neural networks
You need the probability of several correlated outcomes at once → multi-task or structured models
Very high dimension with few events → heavy regularization, or a fundamentally simpler feature set

The pairing worth remembering: logistic regression for the probability and the explanation, a boosted ensemble for the last few points of accuracy. Fit both. The gap between them tells you how much non-linearity actually exists in your problem.

13In code

The whole workflow, minimally

# scikit-learn — note that regularization is ON by default (C = 1/lambda)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.metrics import (classification_report, roc_auc_score,
                             average_precision_score, log_loss, confusion_matrix)
import numpy as np

# stratify so both splits keep the class balance
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

pipe = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=2000, class_weight="balanced"),
)
grid = GridSearchCV(
    pipe,
    {"logisticregression__C": np.logspace(-3, 3, 13)},
    cv=StratifiedKFold(5, shuffle=True, random_state=0),
    scoring="average_precision",   # PR-AUC: the honest choice when positives are rare
).fit(X_tr, y_tr)

model = grid.best_estimator_
proba = model.predict_proba(X_te)[:, 1]        # the actual output — keep it

print("ROC-AUC ", roc_auc_score(y_te, proba))
print("PR-AUC  ", average_precision_score(y_te, proba))
print("Log loss", log_loss(y_te, proba))

# choose the threshold deliberately, on validation data — not 0.5 by reflex
for t in [0.2, 0.35, 0.5, 0.65]:
    print(t, confusion_matrix(y_te, (proba >= t).astype(int)).ravel())

# odds ratios, on the scaled features
clf = model[-1]
for name, b in sorted(zip(feature_names, clf.coef_[0]), key=lambda t: -abs(t[1])):
    print(f"{name:>22}  beta={b:+.3f}  OR={np.exp(b):.2f}")

For p-values, confidence intervals and a proper inference table, use statsmodels.Logit — and remember it fits unpenalized by default, so its coefficients will not match scikit-learn's unless you disable the penalty there.

14Pitfalls

Mistakes that look like success

Reporting accuracy on imbalanced data. The single most common way a useless classifier gets shipped. Always show the confusion matrix.

Treating coefficients as probability changes. They're log-odds. Exponentiate for odds ratios, or compute marginal effects for probabilities.

Accepting 0.5 without thinking. The threshold is a cost decision. Choosing it by default means choosing equal costs by default, which is almost never true.

Forgetting sklearn regularizes. Coefficients silently shrink, and they won't match statsmodels or the literature.

Ignoring convergence warnings. They usually mean unscaled features, too few iterations, or perfect separation — the last of which invalidates every coefficient in the table.

Resampling then trusting the probabilities. SMOTE and friends destroy calibration. Recalibrate, or stick to threshold tuning.

Discarding predict_proba. Calling predict() throws away the model's most valuable output and silently hard-codes a 0.5 cutoff.

Reading it causally. An odds ratio from observational data is an association under whatever you controlled for. Same caution as linear regression, higher stakes, because these models are usually deployed on people.

15Learning path

An order that works

Step	Do this	You'll know it when
1	Plot the sigmoid by hand for several β₁ values	You can predict the curve's shape from the coefficients before plotting it
2	Convert between probability, odds and log-odds until it's automatic	You can state an odds ratio as a probability change at a given baseline
3	Implement log loss and its gradient in NumPy; fit by gradient descent	Your coefficients match sklearn with penalty=None
4	Fit MSE with a sigmoid instead and watch it stall or find a worse optimum	You can explain the vanishing-gradient argument without notes
5	Take an imbalanced dataset. Sweep the threshold; plot precision, recall and F1 against it	You can pick a threshold from a cost argument rather than habit
6	Plot a reliability diagram before and after applying SMOTE	You can see the calibration damage resampling causes
7	Construct perfectly separable data and try to fit it	You recognize the symptoms instantly and know regularization is the cure
8	Move to softmax regression, then to a one-layer neural network	You can see the network's final layer is this model

The one-paragraph summary

Logistic regression predicts the probability of a binary outcome by passing a linear score through the sigmoid, which makes the model linear in the log-odds. It is fitted by maximizing likelihood — minimizing log loss — which has no closed form and requires an iterative solver, though the loss is convex so the answer is unique. Coefficients are log-odds changes; exponentiate them for odds ratios. The model outputs a probability, and turning that into a class requires a threshold you choose from the relative cost of the two mistakes, not from habit. Evaluate with precision, recall and PR-AUC rather than accuracy whenever the classes are imbalanced, and check that the probabilities are calibrated as well as correctly ordered. It is the standard baseline for classification, the model of record wherever decisions must be explained, and the final layer of essentially every neural network classifier.