ECON 4309 · Quantitative Methods for Social Sciences · Reading companion

Matching, in about 15 minutes

A plain-language guide to Chapter 14 of The Effect by Nick Huntington-Klein. Read this first, play with the two simulators, and then use the reading guide at the end to decide which parts of the chapter to read closely.

Pairs with Lecture 7: Randomization and matchingBuilds on Lecture 5: do() and potential outcomesBook: Ch. 14~15 min

Note: This is the printable version. The interactive version, with the simulators, is on the course website.

1. The big idea in 60 seconds

The chapter opens with a job-training program. The people who could sign up were half men and half women, but the program happened to be advertised mostly to men, so the people who actually enrolled were 80% men. Men and women also had different job-finding rates to begin with. So gender sits on a back door: job outcome ← gender → training.

In the chapter's numbers, 72% of trainees found jobs versus 62.5% of non-trainees, a raw gap of 9.5 points. Matching builds a comparison group that looks like the trainees: it counts each untreated man more and each untreated woman less until the comparison group is also 80% men. Now the comparison group's job rate is 67%, and the gap shrinks to 5 points. The other 4.5 points came from the gender back door, not from training.

That's the whole chapter in miniature. Matching closes back doors by choosing (or weighting) untreated observations so they look like the treated ones on the variables that sit on back doors. Then you compare outcomes. The rest of the chapter is about the many choices hidden in "look like":

What does "similar" mean? Close on the variables themselves (distance matching) or equally likely to be treated (propensity score)?
Pick matches or weight everyone? Keep the best controls "in" and the rest "out," or give every control a weight that depends on how similar it is?
How bad a match will you accept? That's the caliper, and it brings a tradeoff between bias and precision.

2. Seven words you need

We'll use the mosquito-net example from class throughout. Households choose whether to use a bed net, and we want the effect of nets on malaria risk (0–100). The data are observational: richer, healthier households, and households in places with different nighttime temperatures, are more or less likely to use nets and have different malaria risk anyway.

Matching variablesThe variables you make treated and control look alike on. They should be the ones that close the back doors in your DAG.Nets: income, health, and nighttime temperature.

Distance matching"Similar" means close on the matching variables themselves. With several variables, the usual yardstick is the Mahalanobis distance, a straight-line distance after putting every variable on the same scale.A net user earning $950 is matched to a non-user earning $940 with similar health and temperature.

Propensity scoreThe probability of being treated given the matching variables, usually estimated with a logit. "Similar" means equally likely to be treated.A household with a 0.70 chance of using a net, whether or not it actually did.

Caliper(also called the bandwidth) The worst match you'll accept. A treated unit with no control inside the caliper gets dropped."Only match if incomes are within 0.1 standard deviations."

Common supportOverlap. For the kinds of treated units you have, there are untreated units like them too.If every rich household used a net, there are no rich non-users to compare against.

BalanceAfter matching, the treated and control groups have (nearly) the same distribution of each matching variable. That's how you check that the back doors really closed.Average income of net users ≈ average income of matched non-users.

Inverse probability weight (IPW)A weight built from the propensity score. Units that got the treatment they were unlikely to get count more.A household with a 0.10 chance of using a net that used one anyway gets weight 1/0.10 = 10.

3. Simulator: pick a caliper, check the balance

The data below are simulated, so we know the truth: using a net lowers malaria risk by exactly 10 points for every household (the same −10 built into the class example). Richer households are both more likely to use a net and less at risk anyway, so income is a confounder. To keep the picture simple, income is the only one here.

The simulator uses radius matching, one of the chapter's options: every net user is compared with the average of all non-users whose income falls inside the caliper. If none do, that net user is dropped.

Mosquito nets: income vs. malaria risk (simulated data)

Orange = used a net, blue = didn't. Faded blue points are non-users that aren't anyone's match. Hollow orange rings are net users dropped because no non-user is close enough.

Net user (kept)Net user dropped (no match)Non-user used as a matchNon-user not used

Caliper: ±0.10 SD of income (±$25)

Naive difference

—

Matched estimate

—

True effect

−10.0

Net users kept

—

Income gap before

—

Income gap after

—

Try 1: Look at the naive difference first. It's far more negative than −10, because net users are richer, and richer households have lower risk anyway. Then compare the income gap before and after matching.
Try 2: Slide the caliper all the way right (±2 SD), then "Repeat 200 times." Nobody gets dropped and the dots are tight, but they center well below −10, and the income gap after matching is large. Wide calipers let in bad matches, so the back door stays partly open. That's bias.
Try 3: Slide it all the way left, then repeat. Balance is nearly perfect and the dots center on −10, but most net users are dropped (watch the high-income rings) and the dots spread out. That's noise. Find a middle caliper where the dots are both centered and fairly tight.

The core tradeoff (the chapter comes back to it in almost every choice) Fewer but better matches mean less bias but less precision. More but worse matches mean more precision but more bias. The same tradeoff shows up in the caliper, in how many neighbors you take (one-to-one vs. k-nearest-neighbor), and in whether a control can be reused (with or without replacement). There's no single right setting, so report how many treated units you lost and check balance.

Dropping treated units changes who you're learning about In this simulator every household has the same −10 effect, so dropping the rich net users only costs precision. In real data the effect might differ for rich households. If a narrow caliper drops most of them, your estimate describes the net users who could be matched, not all net users. The chapter makes this point about coarsened exact matching, which often drops many treated observations.

4. Why this counts as causal

Here's a simplified version of the mosquito-net DAG from class. Income, health, and nighttime temperature each affect both net use and malaria risk, so each one opens a back door.

If we compare net users with non-users who have the same income, health, and temperature, those three variables no longer differ between the groups. A variable that doesn't vary between the groups can't create a difference between them, so those back doors are closed. What's left is the net → malaria arrow. This is the same back-door adjustment from Lecture 5. Matching is just a different way to do it than adding controls to a regression.

Every matching method is really a set of weights

The chapter's unifying idea: matching gives every observation a weight, then compares the weighted average outcome of the treated with the weighted average outcome of the controls. Selecting matches is the special case where controls get weight 1 ("in") or 0 ("out"). The job-training example used weights of 0.16 for untreated men and 0.04 for untreated women. The weights are chosen so the matching variables look the same in both groups.

Distance or propensity score?

Distance matchingFind controls whose matching variables are actually close to the treated unit's. It's intuitive and forces balance on each variable. The catch is the curse of dimensionality: the more variables you match on, the harder it is to find anyone close on all of them. The chapter's picture is two coworkers who look side by side on a floor plan but turn out to be 21 floors apart.

Propensity scoreCollapse all the matching variables into one number, the predicted probability of treatment, and treat units with similar scores as comparable. It handles many variables easily, but now you have to get the logit model right. The chapter recommends using the score for weighting (IPW), not for picking one-to-one matches.

Matching vs. regressionBoth rely on the same key assumption: your variables are enough to close every back door. Their other assumptions differ. Matching doesn't need a straight-line relationship between the controls and the outcome, and it makes it easy to choose which average effect you get (ATT, ATU, or ATE; see section 7). The chapter's view is that neither one is always better, and they can be combined (see "doubly robust" in 14.7.2).

5. What has to be true (and how to check it)

Assumption 1: Conditional independence (all back doors are closed)

This is matching's name for "no open back doors." Your matching variables have to cover every back door. If net users are also more careful about draining standing water and you never measured that, matching on income, health, and temperature won't fix it. Matching can only balance what you measured.

How to check it: you can't test it with the data. You defend it with your DAG and what you know about the setting. Also match only on variables that sit on back doors, not on things the treatment causes (mediators) or colliders.

Assumption 2: Common support (there's someone to match with)

Every kind of treated unit needs comparable untreated units. The chapter's example is an early-release program for prisoners where everyone with a behavior score of 8–10 is released early and nobody with 1–7 is. There are no un-released 9s to compare with the released 9s, so matching has nothing to work with. If only a handful of un-released 9s exist, they get huge weights and the estimate becomes noisy.

How to check it: plot the distribution of each matching variable (or of the propensity score) for treated and control together and look for regions where only one group shows up. When selecting matches, count how many treated units find no match. A few percent is fine; most of them is a failure. The usual fix is to trim to the range where both groups exist, which, again, changes who your estimate describes.

Simulator: when do the matches run out?

Each bar shows what share of a group has a given propensity score (chance of using a net). Net users are on top, non-users on the bottom. The data are simulated, so these are the true scores; in real data you'd estimate them with a logit. Red shading marks scores where there are net users but no non-users at all. Slide right to make net users more and more different from non-users.

How different are net users from non-users? 1.0

Net users with 5+ close matches

—

Largest IPW weight

—

Scores < .02 or > .98

—

Try 1: At 0 the two groups look identical (like a randomized experiment). Every net user has plenty of matches and no weight is large.
Try 2: Move to about 1.5. The groups overlap less and support is getting thin at the high end. Notice how fast the largest IPW weight climbs.
Try 3: Move to 3 or more. The right end of the net-user distribution (shaded red) has no non-users at all. Those net users can't be matched, and the few non-users with high scores get enormous IPW weights. Trimming would throw away a big chunk of the treated group.

Assumption 3: Balance (did the matching actually work?)

After matching, the treated and matched-control groups should look the same on every matching variable. Unlike conditional independence, you can check this, and if it fails you go back and change the matching (add variables, change the propensity model, tighten the caliper) and try again. The chapter treats this as a normal, repeated loop.

How to check it:

Balance table: means of each matching variable for treated vs. control, before and after matching, often with a standardized difference (the gap divided by a pooled standard deviation, so variables on different scales can be compared). You want the "after" gaps to be tiny.
Plot the whole distribution. The chapter warns that equal means aren't enough: two groups can have the same average income but very different shapes. Overlay the treated and control densities after matching. For propensity scores, this plot is almost required.

6. Matching vs. inverse probability weighting

Your IPW assignment uses the other big tool in the chapter. Inverse probability weighting doesn't pick matches at all. It estimates each unit's propensity score p and then weights every unit by the inverse of the probability of what actually happened to it:

Treated (used a net): weight = 1 / p · Untreated: weight = 1 / (1 − p)

In Lecture 7's words, IPW weights units by how "weird" they are. A household with a 0.9 chance of using a net that didn't use one looks a lot like the net users, so it gets a big weight (1/0.1 = 10) and stands in for them in the control group. A household with a 0.1 chance that didn't use one is unsurprising and gets a weight near 1.

	Matching (e.g., nearest neighbor)	Inverse probability weighting
What it does	Finds similar controls for each treated unit; unmatched units are dropped (weight 0).	Keeps everyone, with weights from the propensity score.
Main choices	Distance measure, number of matches, replacement, caliper.	Mostly just the propensity-score model (which variables, which terms).
Main danger	Throwing away lots of data; bad matches if the caliper is too wide.	Huge weights when scores are near 0 or 1, which make the estimate unstable.
Usual fix	Report how many units were dropped; check balance.	Trim scores near 0 or 1, or cap the largest weights (class used a cap); check balance with the weights applied.
In the class example	Mahalanobis matching without the matching weights gave about −12.7.	IPW gave about −10.1, close to the true −10.

The chapter's author says that when people ask him how to do matching, he usually suggests IPW with a propensity score or entropy balancing (a newer method that picks weights so the means of the matching variables come out exactly equal). He also notes that in large samples with a flexible propensity model, weighting is the most precise approach.

7. What you actually estimate: ATT, ATU, or ATE

Recall from Lecture 5: the ATT is the average effect for the treated, the ATU for the untreated, and the ATE for everyone, with ATE = (share treated × ATT) + (share untreated × ATU). One nice feature of matching is that you can choose which one you get:

ATTBuild controls that look like the treated (what we've done all along). With IPW: treated get weight 1, controls get p / (1 − p).Effect of nets on the households that use them.

ATUFlip it: find treated units that look like the untreated. With IPW: controls get weight 1, treated get (1 − p) / p.Effect nets would have on households that don't use them now. Useful for deciding whether to hand out more nets.

ATEDo both and average them. The standard 1/p and 1/(1 − p) IPW weights from class give the ATE.Effect of nets averaged over all households.

Check which one your software reportsThe chapter warns that packages have different defaults. Some give the ATE unless you ask otherwise, while MatchIt defaults to the ATT. Also, even the "ATT" from matching is weighted toward the most common kinds of treated units. In the chapter's example, the matched estimate (2.34) lands much closer to the men's effect (2) than the women's (4) because most of the treated are men.

8. Doing it in R (with AI help)

In this class, AI writes most of the code. Your job is the design and the checking. These are the steps a matching or IPW analysis should include and what to verify at each one (variable names follow the mosquito-net example):

Step	Typical R	What you check
1. Choose matching variables	`adjustmentSets(mosquito_dag)` (dagitty)	Are they all on back doors? None caused by the treatment?
2. Naive comparison	`lm(malaria_risk ~ net, data = nets)`	Just a baseline. Not causal.
3. Match	`m <- matchit(net ~ income + temperature + health, data = nets, method = "nearest", distance = "mahalanobis", replace = TRUE)`	`summary(m)`: how many treated were matched, how many controls used, how many discarded?
4. Check balance	`bal.tab(m, un = TRUE)`, `love.plot(m)` (cobalt)	Are standardized differences near zero after matching? Plot distributions too, not just means.
5. Estimate (matching)	`lm(malaria_risk ~ net, data = match.data(m), weights = weights)`	Did you use the `weights` column? With `replace = TRUE`, leaving it out overcounts reused controls.
6. Propensity score	`glm(net ~ income + temperature + health, family = binomial(link = "logit"))`, then `augment_columns(..., type.predict = "response")`	Are all scores between 0 and 1? (Without `"response"` you get log-odds.) Do the treated and control score distributions overlap?
7. IPW weights and estimate	`mutate(ipw = net_num / propensity + (1 - net_num) / (1 - propensity))`, then `lm(malaria_risk ~ net, weights = ipw)`; or `weightit(..., estimand = "ATE")` (WeightIt)	What is the largest weight? Does capping it (e.g., at 10) change the answer much? ATE or ATT?
8. Standard errors	The package's own matching SEs, or a bootstrap that redoes the whole weighting each time	Plain `lm` SEs ignore that the weights were estimated. The chapter says bootstrapping works for weighting but not for in/out match selection.

Don't stop at one numberHave the AI run the naive estimate, the matching estimate, and the IPW estimate side by side (e.g., modelsummary()), as the class example did. If they disagree a lot, figure out why. Check balance, overlap, the extreme weights, and how many units were dropped before you trust any of them.

Common mistakes

Thinking matching fixes unmeasured confounders. It only balances what you matched on.
Matching on a variable the treatment affects (a mediator) or a collider.
Not reporting how many treated units were dropped, then claiming the result applies to all of them.
Checking balance only on means, or not checking it at all.
Using the propensity score to pick one-to-one matches (the chapter advises weighting instead).
Forgetting type.predict = "response", so the "propensity scores" are actually log-odds.
Ignoring enormous IPW weights from scores near 0 or 1.
Calling the result "the ATE" when your method gave the ATT, or the reverse.

9. How to read the chapter

The chapter is long (over 20,000 words counting the code) and written partly for graduate students. Here's where to spend your time:

Section	What it covers
14.1 Another Way to Close Back Doors	What matching is, the job-training example, and matching vs. regression.	Read
14.2 Weighted Averages	Matching as weights; the 9.5 → 5 point worked example.	Read
14.3 Matching in Concept	The five big choices (distance vs. propensity, select vs. weight, how many matches, kernels and IPW, calipers) using credit-card data. The heart of the chapter. You can skim the kernel formula.	Read closely
14.4 / 14.4.1 Multiple Matching Variables	Mahalanobis distance, the curse of dimensionality, coarsened exact matching. Read the intuition and skip the matrix algebra and the code.	Skim
14.4.2 Entropy Balancing	A newer weighting method. Short.	Optional
14.5 Propensity Score Weighting	How to build the propensity model (logit, what to include, machine-learning options).	Skim
14.6 Assumptions for Matching	Conditional independence, common support (the early-release example), balance tables and density plots. Directly useful for your assignment.	Read closely
14.7.1 Estimating Mean Differences	Getting the estimate, and why standard errors are tricky (bootstrap).	Skim
14.7.2 Combining Matching and Regression	Matching weights inside a regression; doubly robust estimation.	Optional
14.8 Matching and Treatment Effects	Getting the ATT, ATU, or ATE from matching and from IPW weights.	Read