Two exponentials, back to back
Parameters
- μ
- Location. Simultaneously the mean, the median, and the mode. The distribution is perfectly symmetric about it.
- b
- Scale, sometimes called the diversity. Larger b means flatter and wider. Note it is not the standard deviation.
- σ = b√2
- The standard deviation. Variance is 2b², so to compare fairly against a Gaussian of standard deviation σ you need b = σ/√2.
- b = MAD
- The scale parameter is the mean absolute deviation from the centre — a much more direct interpretation than the Gaussian's σ.
Why "double exponential"
For x above μ it's an exponential decaying to the right. For x below μ it's the mirror image. Glue them at μ, halve the height so the total area is one, and you have it.
Equivalently: if E1 and E2 are independent exponentials with the same rate, then E1 − E2 is Laplace. Any process that is the difference of two waiting times lands here naturally.
The numbers, against the Gaussian
| Property | Laplace(μ, b) | Normal(μ, σ²) |
|---|---|---|
| density kernel | exp(−|x−μ| / b) | exp(−(x−μ)² / 2σ²) |
| mean | μ | μ |
| median | μ | μ |
| mode | μ (a corner) | μ (smooth) |
| variance | 2b² | σ² |
| peak height | 1 / 2b ≈ 0.354/σ | 1 / σ√(2π) ≈ 0.399/σ |
| skewness | 0 | 0 |
| excess kurtosis | 3 | 0 |
| mean abs deviation | b = σ/√2 | σ√(2/π) ≈ 0.798σ |
| entropy | 1 + ln(2b) | ½ ln(2πeσ²) |
| tail P(|X−μ| > t) | exp(−t / b) — exponential | 2(1 − Φ(t/σ)) — Gaussian |
| maximum entropy for | fixed mean absolute deviation | fixed variance |
| MLE of location | the median | the mean |
| differentiable at μ | no | yes |
Straight lines against a parabola
The log density is where the difference becomes structural rather than cosmetic. Take logs of both kernels and you get −|x−μ|/b against −(x−μ)²/2σ². One is linear in distance; the other is quadratic. So on a log axis the Laplace descends at a constant rate forever, while the Gaussian's descent steepens without limit.
| k | Laplace P(|X| > kσ) | Gaussian P(|Z| > k) | ratio | reading |
|---|---|---|---|---|
| 1 | 0.243 | 0.317 | 0.77× | Gaussian is more likely here |
| 1.65 | ≈0.096 | ≈0.099 | ≈1× | the crossover point |
| 2 | 0.059 | 0.0455 | 1.3× | Laplace overtakes |
| 3 | 0.0143 | 0.00270 | 5.3× | a clear gap |
| 4 | 0.00349 | 6.3 × 10⁻⁵ | 55× | different regimes |
| 5 | 8.5 × 10⁻⁴ | 5.7 × 10⁻⁷ | 1,480× | a once-in-a-lifetime Gaussian event is routine |
| 6 | 2.1 × 10⁻⁴ | 2.0 × 10⁻⁹ | 105,000× | the Gaussian has effectively stopped |
Where robustness comes from
Assume Gaussian noise and maximum likelihood hands you the sample mean. Assume Laplace noise and the very same procedure hands you the sample median. That substitution is the entire mathematical content of "robust estimation".
Minimizing squared distances gives the mean; minimizing absolute distances gives the median. Both are one-line calculus exercises, and together they explain why one estimator is dragged by outliers and the other isn't.
Scale estimate: the MLE for b is the mean absolute deviation from the fitted median, b̂ = (1/n)Σ|xᵢ − median|. Note it uses the median, not the mean — plugging in the mean is a common and quietly damaging error.
Why the corner produces exact zeros
A MAP estimate maximizes prior times likelihood, which after taking logs becomes a penalized least-squares problem. The penalty is just the negative log prior — so choosing a prior is choosing a regularizer.
Gaussian prior → Ridge
Smooth everywhere, with derivative 2β that vanishes as β approaches zero. The penalty stops pushing exactly when it would need to push hardest, so coefficients shrink toward zero and never arrive.
Laplace prior → Lasso
Derivative ±1 right up to zero, where it is undefined. The penalty keeps pushing with full force all the way in — so if the data's pull is weaker than λ, the coefficient is pinned at exactly zero.
The shrinkage operator
# the correspondence, made explicit # posterior ∝ likelihood × prior # −log posterior = −log likelihood − log prior # = RSS / (2σ²) + Σ|β_j| / b + const # minimize: RSS + λ Σ|β_j| with λ = 2σ²/b # so a TIGHT Laplace prior (small b) means STRONG regularization (large λ) # and the lasso path is a path through prior strengths, not just penalties from sklearn.linear_model import Lasso, Ridge Lasso(alpha=0.1) # MAP under a Laplace prior → some coefficients exactly 0 Ridge(alpha=0.1) # MAP under a Gaussian prior → all coefficients merely smaller
The full posterior version
The scale-mixture trick
A Laplace distribution can be written as a Gaussian whose variance is itself exponentially distributed. Marginalize the variance out and the Laplace reappears.
That representation is what makes the Bayesian lasso practical: conditional on the latent variances, everything is Gaussian and conjugate, so a Gibbs sampler alternates between drawing coefficients and drawing variances. Park and Casella's 2008 formulation is the standard reference.
What you gain, and what you don't
- Gain: credible intervals for every coefficient, which the lasso point estimate cannot give you — its sampling distribution is notoriously awkward.
- Gain: λ can be given a prior and inferred rather than cross-validated.
- Lose: exact sparsity. Nothing is ever precisely zero, so you need a thresholding rule to declare selection.
- Lose: speed. Coordinate descent fits a lasso in milliseconds; MCMC takes minutes.
Absolute-error loss, from the other direction
Section 05 put the Laplace on the coefficients. Put it on the residuals instead and you get a different and equally useful model.
| Noise assumption | Loss it implies | Estimator | Behaviour |
|---|---|---|---|
| Gaussian | squared error | OLS | Efficient if true, dragged badly by outliers |
| Laplace | absolute error | Least absolute deviations | Robust, fits the conditional median, no closed form |
| Asymmetric Laplace | pinball / check loss | Quantile regression | Fits any conditional quantile you name |
| Gaussian centre, Laplace tails | Huber loss | Huber regression | The practical compromise, and usually the right default |
| Student-t | a bounded influence loss | t-regression | Even more robust; extreme points get almost no vote |
The Laplace mechanism
The distribution's most consequential modern application isn't statistical at all. Differential privacy needs noise whose ratio of densities at two nearby points is bounded — and because the Laplace's log density is linear in distance, that ratio depends only on the gap between them. No other common distribution has that property so cleanly.
The two knobs
- Δ
- Sensitivity. The most the answer could change if one person's record were added or removed. For a simple count that's 1.
- ε
- Privacy budget. Small ε means strong privacy and loud noise. Large ε means the reverse. It is a ratio bound on how much any single person can influence the output distribution.
Drag ε down and watch the releases scatter until the true count is unrecoverable. That is the guarantee working as designed, not a defect.
Where it sits among the heavy-tailed distributions
| Distribution | Tail decay | Notes |
|---|---|---|
| Normal | exp(−x²) | The reference. All moments finite, tails effectively vanish past 5σ |
| Laplace | exp(−|x|) | All moments still finite, MGF exists. Robust but not extreme |
| Student-t (ν) | |x|−(ν+1) | Power law. Moments above ν don't exist. Tunable via ν |
| Cauchy | |x|−2 | t with ν = 1. No mean, no variance. The sample mean never converges |
| Pareto | x−α | One-sided. Wealth, city sizes, file sizes |
Asymmetric Laplace
Different scales either side of μ. The likelihood behind quantile regression and expectile methods.
Log-Laplace
The exponential of a Laplace variable. Positive-valued with a power-law tail — used for income and insurance losses.
Geometric stable
The Laplace is the geometric-stable analogue of the Gaussian: sums of a geometrically distributed number of iid terms converge to it.
Sampling, fitting, using
import numpy as np from scipy import stats # sampling — three equivalent routes x = np.random.laplace(loc=0, scale=b, size=n) # the direct way x = stats.laplace(loc=0, scale=b).rvs(n) u = np.random.uniform(-0.5, 0.5, n) # inverse CDF, by hand x = -b * np.sign(u) * np.log(1 - 2*np.abs(u)) x = np.random.exponential(b, n) - np.random.exponential(b, n) # difference of two # fitting — note the estimators are the median and the MAD-about-median mu_hat = np.median(x) b_hat = np.mean(np.abs(x - mu_hat)) loc, scale = stats.laplace.fit(x) # the same thing, done for you # matching a Gaussian's variance requires the √2 sigma = 1.0 b_matched = sigma / np.sqrt(2) # NOT b = sigma # which distribution actually fits? compare on the log scale, not the density for dist in (stats.norm, stats.laplace, stats.t): params = dist.fit(x) ll = dist.logpdf(x, *params).sum() k = len(params) print(dist.name, "AIC", 2*k - 2*ll) # a QQ plot against Laplace is the honest visual check stats.probplot(x, dist=stats.laplace, plot=plt)
Drills
| # | Do this | You'll know it when |
|---|---|---|
| 1 | Derive the variance 2b² by integration, and confirm b = σ/√2 | You stop confusing the scale parameter with the standard deviation |
| 2 | Sample 10,000 draws from a Laplace and a variance-matched Gaussian; count values past 4σ in each | The counts differ by roughly the factor the table predicts |
| 3 | Plot both empirical log densities and find the crossover | It lands near 1.65σ, and you can explain why the middle matters too |
| 4 | Prove that minimizing Σ|x−μ| gives the median | Robustness is a derivative argument in your head, not a slogan |
| 5 | Implement soft thresholding and verify it solves the 1D lasso | The dead zone is a formula, not a picture |
| 6 | Fit lasso and ridge on the same data across a λ path; plot both coefficient paths | You watch lasso coefficients hit zero and ridge coefficients never do |
| 7 | Fit OLS and LAD to data with one large outlier and compare the fitted lines | The vote-weighting argument becomes concrete |
| 8 | Implement the Laplace mechanism for a counting query and empirically check the density ratio bound | You see why the exponential tail is load-bearing |
| 9 | Fit Normal, Laplace and Student-t to a real dataset and compare by AIC and QQ plot | You pick a noise model from evidence rather than habit |
The one-paragraph summary
The Laplace distribution replaces the Gaussian's squared distance with an absolute one, making its density two exponentials joined at a corner. At matched variance it is more concentrated near the centre and substantially heavier beyond about 1.65σ — at five standard deviations it is roughly 1,500 times more likely than a Gaussian — while still having all moments finite, which places it between the Gaussian and genuinely power-law distributions. Because minimizing absolute rather than squared deviations gives the median rather than the mean, assuming Laplace noise yields robust estimators in which every observation carries equal weight regardless of distance. Used as a prior on coefficients its negative log becomes the L1 penalty, and because the derivative stays at ±1 all the way to the origin rather than vanishing, the resulting MAP estimate soft-thresholds: coefficients whose evidence is weaker than λ are set to exactly zero, which is why lasso selects and ridge merely shrinks. And because its log density is linear in distance, the ratio of densities at two nearby points is bounded by a constant — the property that makes it the noise distribution of choice for ε-differential privacy.