Using Lagged Outcomes to Evaluate Bias in Value-Added Models

Raj Chetty, John Friedman and Jonah Rockoff

No 21961, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Value-added (VA) models measure the productivity of agents such as teachers or doctors based on the outcomes they produce. The utility of VA models for performance evaluation depends on the extent to which VA estimates are biased by selection, for instance by differences in the abilities of students assigned to teachers. One widely used approach for evaluating bias in VA is to test for balance in lagged values of the outcome, based on the intuition that today’s inputs cannot influence yesterday’s outcomes. We use Monte Carlo simulations to show that, unlike in conventional treatment effect analyses, tests for balance using lagged outcomes do not provide robust information about the degree of bias in value-added models for two reasons. First, the treatment itself (value-added) is estimated, rather than exogenously observed. As a result, correlated shocks to outcomes can induce correlations between current VA estimates and lagged outcomes that are sensitive to model specification. Second, in most VA applications, estimation error does not vanish asymptotically because sample sizes per teacher (or principal, manager, etc.) remain small, making balance tests sensitive to the specification of the error structure even in large datasets. We conclude that bias in VA models is better evaluated using techniques that are less sensitive to model specification, such as randomized experiments, rather than using lagged outcomes.

JEL-codes: C18 H75 I21 J01 J08 J45 M50 M54 (search for similar items in EconPapers)
Date: 2016-02
New Economics Papers: this item is included in nep-ecm and nep-ure
Note: CH ED LS PE
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (12)

Published as Raj Chetty & John N. Friedman & Jonah Rockoff, 2016. "Using Lagged Outcomes to Evaluate Bias in Value-Added Models," American Economic Review, vol 106(5), pages 393-399.

Downloads: (external link)
http://www.nber.org/papers/w21961.pdf (application/pdf)

Related works:
Journal Article: Using Lagged Outcomes to Evaluate Bias in Value-Added Models (2016)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:21961

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w21961

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().