ECON 4309 · Quantitative Methods for Social Sciences · Reading companion

Difference-in-Differences, in about 15 minutes

A plain-language guide to Chapter 18 of The Effect by Nick Huntington-Klein. Read this first, play with the two simulators, and then use the reading guide at the end to decide which parts of the chapter to read closely.

Pairs with Lecture 8: Diff-in-diff I & IIBook: Ch. 18~15 min

Note: This is the printable version. The interactive version, with the simulators, is on the course website.

1. The big idea in 60 seconds

The chapter opens in 1850s London with John Snow, who suspected that cholera spread through dirty water, not "bad air." Several water companies drew from the Thames. Between 1849 and 1854, one of them, the Lambeth company, was required to move its intake upstream of the city's sewage. Other companies didn't move.

Looking only at Lambeth areas, cholera deaths fell from about 130 to 85 per 10,000. But maybe cholera was fading everywhere? So Snow's logic also looked at areas without Lambeth water, where deaths actually rose, from about 135 to 147. The effect of clean water is how much more the Lambeth areas changed than the other areas: roughly −57 deaths per 10,000.

That is difference-in-differences (DiD). In the chapter's words, the recipe has two steps:

Take the before-to-after change in each group (the first "difference"). This removes fixed differences between the groups.
Subtract the untreated group's change from the treated group's change (the second "difference"). The untreated group's change stands in for what would have happened to the treated group anyway.

DiD = (Treated_after − Treated_before) − (Control_after − Control_before)

2. Six words you need

We'll use the minimum-wage example from Lecture 8 throughout: in 1992, New Jersey raised its minimum wage from $4.25 to $5.05, and neighboring Pennsylvania did not. The outcome is the average number of jobs per fast-food restaurant.

Treated groupThe group that gets the policy, at some point in time.Minimum wage: New Jersey restaurants.

Control group(also "untreated" or "comparison" group) A group that never gets the policy during the study.Minimum wage: Pennsylvania restaurants.

Before / after(also "pre" and "post") The periods before and after the policy starts. You need both, for both groups.Restaurants surveyed before and after New Jersey's increase.

CounterfactualWhat would have happened to the treated group without the policy. We never see it, so DiD builds it from the control group's change.New Jersey's job count in 1992 if the wage had stayed at $4.25.

Parallel trendsThe key assumption: without the policy, the gap between the groups would have stayed the same. The groups can start at different levels, but they must move together.Without the increase, NJ jobs would have changed by the same amount as PA jobs.

ATTAverage treatment effect on the treated: the effect for the group that actually got the policy.The effect of the wage increase on New Jersey restaurants, not on restaurants everywhere.

3. Simulator: build the 2×2

Every basic DiD comes down to four averages: each group, before and after. Lecture 8 labels them A, B (control) and C, D (treated). Start with the lecture's New Jersey numbers, then switch to the two examples from the chapter, or move the sliders yourself.

Minimum wage: jobs per fast-food restaurant

The dashed orange line is the counterfactual: the treated group's starting point plus the control group's change. DiD is the gap between that line and what actually happened.

Treated (New Jersey)Control (Pennsylvania)Dashed = counterfactual · bracket = DiD

A · Control, before:

B · Control, after:

C · Treated, before:

D · Treated, after:

Treated change (D − C)

—

Control change (B − A)

—

DiD

—

Try 1: Use the NJ vs. PA numbers. A before/after comparison of New Jersey alone gives +0.59 jobs. Comparing NJ to PA only after the increase gives −0.14. Both are misleading. DiD uses all four numbers and gets about +2.75.
Try 2: Drag B (control, after) up. The dashed line rises and the DiD shrinks, one for one. Whatever the control group does is treated as "what would have happened anyway."
Try 3: Raise A and B by the same amount (move the whole blue line up). The DiD doesn't change. DiD doesn't care that the groups start at different levels; it only compares changes.

Why one difference isn't enough (Lecture 8: "two wrongs make a right")

Before vs. after in the treated group only: you can't tell the policy apart from everything else that changed over time, such as the economy or the season.
Treated vs. control after only: you can't tell the policy apart from differences the two groups always had.

Each comparison is "wrong" on its own. Subtracting one from the other cancels both problems, if parallel trends holds.

4. Why this counts as causal

Here is the chapter's causal diagram (Figure 18.2), simplified. Treatment is tangled up with two back doors: Time (things change between before and after) and Group (the treated group is different from the control group).

Adding a control group seems to make things worse, because now there's a second back door (Group). But having both groups lets us close both doors. Looking at each group's change over time (within variation) removes fixed differences between groups, which closes Group. Comparing those changes across the two groups removes whatever happened to everyone over time, which closes Time. What's left is the Treatment → Outcome arrow.

The same thing as a regression

Computing four averages by hand gets tedious, and you'll often want standard errors. Lecture 8's regression gives the same answer:

Y = α + β·Group + γ·Time + δ·(Group × Time) + ε

Group = 1 for the treated group, Time = 1 for the after period. The interaction Group × Time = 1 only for the treated group after treatment. Here is what each coefficient means in the 2×2:

	Before	After	Change (after − before)
Control	α	α + γ	γ
Treated	α + β	α + β + γ + δ	γ + δ
Treated − control	β	β + δ	δ = the DiD

β is the gap the groups always have, γ is the change that happens to everyone, and δ is the extra change in the treated group, which is the DiD. It's the hot-dog interaction from class: δ is the "extra" you get from combining "treated group" with "after."

5. Parallel trends: what has to be true (and how to check it)

The assumption

DiD uses the control group's change as a stand-in for what would have happened to the treated group. That only works if, without the treatment, the gap between the two groups would have stayed the same. The chapter puts it as simple accounting:

Treated group's change = effect of treatment + other changes in the treated group
Control group's change = other changes in the control group
DiD = effect of treatment + (other treated changes − other control changes)

DiD equals the true effect only if the "other changes" cancel out. Parallel trends is about a counterfactual, so you can never observe it directly.

Two ways it fails (the chapter's examples)

Say you want the effect of Chicago building new roads in 2018 on the popularity of its downtown restaurants.

Something else happens to the control group at the same time. You compare with Los Angeles, but in 2018 a wave of hyped new restaurants opened in downtown LA. DiD mixes "Chicago's roads" with "LA's new restaurants." The trends could have been perfectly parallel before 2018 and still break at the moment of treatment.
The groups were already drifting apart. You compare with New York, but New York's restaurant popularity had been falling for years. That trend would have kept widening the gap even without the roads, so DiD credits it to the roads.

Good signs in a control group (helpful, but none are proof): there's no reason it would suddenly change when treatment starts; it's similar to the treated group in general (for a Miami study, Atlanta or Tampa is more believable than Reykjavik); and its outcome was moving the same way as the treated group's before treatment.

How to check it (suggestive evidence, not proof)

1. Prior trends test. Plot both groups' averages over the periods before treatment. Is the gap roughly constant? You can also test it with a regression on pre-treatment data only, letting each group have its own time trend.

2. Placebo test. Use only pre-treatment data, pretend the treatment happened at a fake earlier date, and run your DiD. Lecture 8: "pretend the treatment happened earlier; if there's an effect, there's likely an underlying trend." A "significant effect" at a date when nothing happened is a red flag.

The chapter stresses that passing these checks does not prove parallel trends. It only means the data didn't give you a reason to doubt it. Failing them makes the assumption harder to believe.

Simulator: break parallel trends and watch the estimate drift

Simulated data (not from the book): 30 treated and 30 control units, followed for 10 periods. Treatment starts in period 1, so periods −4 to 0 are "before." The true effect is exactly 3.0 in every treated period.

Treated averageControl averageOrange dashed = what DiD assumes the treated group would have doneGreen dotted = what it really would have done (only knowable in a simulation)

Extra trend in the treated group: 0.0 per period

Shock to control group at treatment: 0.0

DiD estimate

—

True effect

3.0

Bias (estimate − truth)

—

Event-study view. Each dot is a DiD comparing one period with period 0 (the last period before treatment), with a 95% confidence interval. Dots before treatment are built-in placebo tests and should sit near zero.

Estimate with 95% CIHollow dot = reference period (0 by definition)Green dashed = true effect (3)

Try 1: Leave both sliders at 0. The DiD lands near 3, the pre-period dots hug zero, and the dashed and dotted lines sit on top of each other. Parallel trends holds.
Try 2: Slide the extra trend to 0.5. The treated group was already pulling away before treatment, so the DiD overshoots (about 5.5 instead of 3). Notice that the pre-period dots slope upward: the event study warns you.
Try 3: Set the trend back to 0 and move the control shock to +3 (like LA's new restaurants). The DiD falls toward 0, but the pre-period dots look perfect. Pre-trend checks can't catch something that only happens when treatment starts. That's why the chapter says they're not proof.

Parallel trends depends on how you measure the outcome If the gap is constant in dollars, it is not constant in percentage terms, and vice versa. The chapter's example: a control group going from 10 to 15 and a treated group from 20 to 25 have a constant gap of 10. In logs, the gap shrinks (0.69 to 0.51). So parallel trends can hold for y but fail for log(y). Decide which version you believe before you run anything.

6. What you actually estimate: the ATT

DiD compares the treated group with its own counterfactual, so it gives you the average treatment effect on the treated (ATT): the effect for the groups that actually got the policy.

When that's usefulIf the question is "did New Jersey's wage increase cost New Jersey jobs?", the ATT answers exactly that.

When it's a limitationThe control group is never treated, so we never see how it would respond. The NJ estimate doesn't automatically tell you what a wage increase would do in Pennsylvania, or in a very different state.

In the chapter's organ-donor example, California switched to "active choice" (you must check yes or no) in 2011. The other states' donor rate rose 1.4 points while California's fell 0.8, so the DiD is about −2.2 percentage points: active choice lowered sign-ups in California. A two-way fixed effects regression gives the same −0.022, with a standard error of 0.006.

7. Optional: event studies and staggered timing

This section previews material the chapter covers in more depth. It's good to know the ideas, but you won't need the details in this course.

Two-way fixed effects (TWFE)

With many groups and many periods, the regression becomes Y = group fixed effects + time fixed effects + δ·Treated, where Treated = 1 for a treated group in a treated period. With one treatment date, this gives the same answer as the 2×2. Standard errors are usually clustered by group, because the same group's errors are related over time.

Event studies (dynamic effects)

Instead of one "after" effect, estimate a separate effect for each period, relative to the last period before treatment. That's the lower plot in the simulator. It shows whether the effect builds up or fades, and the pre-treatment estimates act as placebo tests. The trade-off: each period's estimate uses less data, so it's noisier.

Staggered timing (rollout designs)

Sometimes groups are treated at different times. The chapter's example: three counties getting broadband in 2001, 2002, and 2003. Plain TWFE then quietly uses already-treated groups as controls for newly treated ones. If effects change over time, that can badly distort the estimate, in rare cases even flipping its sign. The chapter calls this "the secret shame of econometrics." Lecture 8 shows the Goodman-Bacon decomposition (R package bacondecomp) for checking how serious the problem is. Newer estimators, such as Callaway and Sant'Anna (R package did), are built to avoid it.

8. Doing it in R (with AI help)

In this class, AI writes most of the code. Your job is the design and the checking. These are the steps a DiD analysis should include and what to verify at each one:

Step	Typical R	What you check
1. Make the indicators	`mutate(treated = state == "NJ", after = date >= policy_date, treat_post = treated & after)`	Is the right group marked treated, and is the cutoff date right? Is `treated` 1 for the group in every period, and `treat_post` 1 only after?
2. Plot it first	`group_by(treated, year) \|> summarize(y = mean(y))` then `ggplot()` with a line per group	Do the pre-treatment lines move together? Does anything odd happen to the control group at the treatment date?
3. The 2×2 by hand	`group_by(treated, after) \|> summarize(mean(y))`	Compute (D − C) − (B − A) yourself. You'll compare it with step 4.
4. Regression	`lm(y ~ treated * after, data = df)`	The `treatedTRUE:afterTRUE` coefficient should match step 3 exactly. Is its sign sensible?
5. Many groups/periods	`feols(y ~ treat_post \| unit + year, data = df, cluster = ~unit)` (fixest)	Are standard errors clustered by group? Does everyone get treated at the same time? If not, see section 7.
6. Event study	`feols(y ~ i(rel_time, treated, ref = 0) \| unit + year, data = df)`, then `iplot()`	Are the pre-treatment estimates close to zero? Is the reference period really the last one before treatment?
7. Placebo test	Keep only pre-treatment years, set a fake treatment date, re-run step 4 or 5	There should be no "effect" at the fake date.

Watch out with control variablesIn DiD, controls are only there to make parallel trends more believable. The chapter warns that tossing them into a TWFE regression often doesn't do what you'd expect, and that controls measured after treatment can absorb part of the effect. If an AI adds controls, ask why, and report results with and without them.

Common mistakes

Using only before/after for the treated group, or only treated/control after treatment (one difference, not two).
Treating "the groups have similar levels" as the assumption. DiD needs similar changes; different levels are fine.
Saying that a clean pre-trend plot or placebo test proves parallel trends.
Mixing up the two "treated" variables: in the treated group (1 for New Jersey in every period) versus treated right now (1 only for New Jersey after the increase). lm(y ~ treated * after) needs the first; feols(y ~ treat_post | unit + year) needs the second.
Switching between y and log(y) without asking which one parallel trends holds for.
Running plain TWFE when groups are treated at different times.
Forgetting to cluster standard errors by group.
Claiming the effect applies to everyone (it's the ATT, for the treated group).

9. How to read the chapter

The chapter is long (roughly 15,000 words with code and footnotes), and its second half is written for graduate students. Here's where to spend your time:

Section	What it covers
18.1.1 Across Within Variation	Why adding a control group closes both back doors. The heart of the method.	Read closely
18.1.2 DiD and Dirty Water	John Snow and the Lambeth water company.	Read
18.1.3 How DID Does?	The organ-donor example, step by step through the four averages.	Read
18.1.4 Untreated Groups and Parallel Trends	The assumption, the Chicago roads examples, prior trends and placebo tests, and the log warning.	Read closely
18.2.1 Two-Way Fixed Effects	The regression versions and clustering. Read the text; you can skim the code.	Skim
18.2.2 Treatment Effects in DiD	Why DiD gives the ATT. Short.	Read
18.2.3 Supporting the Parallel Trends Assumption	How to run the prior-trends and placebo tests. Directly useful for your assignment.	Read
18.2.4 Long-Term Effects	Event-study (dynamic) DiD and how to read its graph.	Skim
18.2.5 Control Variables	Why controls in DiD are tricky. The first few paragraphs are enough.	Skim
18.2.6 Rollout Designs	The staggered-timing problem with TWFE.	Skim
18.3 How the Pros Do It	New staggered estimators, matching, and triple differences. Beyond this course.	Optional