Working Papers Series

Papers below are in pdf.

Cory Koedel

WP 12-01
Test Measurement Error and Inference from Value-Added Models
Cory Koedel, Rebecca Leatherman & Eric Parsons

It is widely known that standardized tests are noisy measures of student learning, but value added models (VAMs) rarely take direct account of measurement error in student test scores. We examine the extent to which modifying VAMs to include information about test measurement error (TME) can improve inference. Our analysis is divided into two parts – one based on simulated data and the other based on administrative micro data from Missouri. In the simulations we control the data generating process, which ensures that we obtain accurate TME metrics with which to modify our value-added models. In the real-data portion of our analysis we use estimates of TME provided by a major test publisher. We find that inference from VAMs is improved by making simple TME adjustments to the models. This is a notable result because the improvement can be had at zero cost.

JEL Codes: I20
Keywords: value added models, value added, teacher value added, test measurement error, teacher evaluation

WP 11-24
Systematic Differences in How Mothers Assess Their Children and Implications for Developmental Research
Cory Koedel & Teerachat Techapaisarnjaroenkit

Researchers often depend on maternal assessments to measure children's non-cognitive development. But the use of maternal assessments can be problematic if mothers differ in how they assess their children in systematic ways. We examine the consistency of maternal assessments across different subgroups of mothers for a large sample of age-4 children. We identify non-negligible differences in maternal assessments by race, and introduce the concept of endogenous maternal assessment bias. Maternal assessments have the potential to be endogenous whenever the intervention of interest is determined, at least in part, by maternal behavior. We illustrate the bias using an application taken from the family-structure literature, and show that developmental differences between children raised in different family structures are understated by maternal assessments.

JEL Codes: I20, J24
Keywords: Maternal Assessment, Maternal Assessment Bias, Maternal Assessment Validity, Endogenous Assessment, Family Structure

WP 11-22
Large-Scale Evaluations of Curricular Effectiveness: The Case of Elementary
Mathematics in Indiana
Cory Koedel & Rachana Bhatt

We use data from one of the few states where information on curriculum adoptions is available – Indiana – to empirically evaluate differences in performance across three elementary-mathematics curricula. The three curricula that we evaluate were popular nationally during the time of our study, and two of the three remain popular today. We find large differences in effectiveness between the curricula, most notably between the two that held the largest market shares in Indiana. Both are best-characterized as traditional in pedagogy. We also show that the publisher of the least-effective curriculum did not lose market share in Indiana in the following adoption cycle; one explanation is that educational decision makers lack information about differences in curricular effectiveness.

JEL Codes: I21, I28 and H75
Keywords: curricular effectiveness, math curricula, non-experimental methods, matching methods, education policy

WP 11-15
Pension-Induced Rigidities in the Labor Market for School Leaders
Cory Koedel, Jason A. Grissom, Shawn Ni & Michael Podgursky

Educators in public schools in the United States are typically enrolled in defined-benefit pension plans, which penalize across-plan mobility. We use administrative data from Missouri to examine how the mobility penalties affect the labor market for school leaders. We show that pension borders greatly affect leadership flows across schools – for two groups of schools separated by a pension border, our estimates indicate that removing the border will increase leadership mobility between them by 97 to 163 percent. We consider the implications of the pension-induced rigidities in the leadership labor market for schools near pension borders in Missouri. Our findings are of general interest given that thousands of public schools operate near pension boundaries nationwide.

JEL Codes: H5, I2, J3
Keywords: Educator pensions, backloaded compensation, principal quality, leadership quality, compensation in education

WP 11-09
Teacher Pension Systems, the Composition of the Teaching Workforce,
and Teacher Quality
Cory Koedel & Michael Podgursky

Teacher pension systems impose large penalties on individuals who separate too soon or remain employed too long. The penalties result in the retention of some teachers who would otherwise choose to leave, and the premature exit of some teachers who would otherwise choose to stay. We examine how these compositional effects of teacher pension systems influence the quality of the teaching workforce, conditional on individuals who initially select into teaching. We find no evidence that the pull and push incentives raise teacher quality, and if anything, we find modest negative effects. Our results support future experimentation with compensation schemes for educators that are not so heavily backloaded.

JEL Codes: I20, J30, J45
Keywords: Educator Pensions, Teacher Pensions, Backloaded Compensation, Teacher Pensions and Teacher Quality, Teacher Compensation, Selection into Teaching

WP 10-13
Math Skills and Labor-Market Outcomes: Evidence from a Resume-Based Field Experiment
Cory Koedel & Eric Tyhurst

We examine the link between math skills and labor-market outcomes using a resume-based field experiment. Specifically, we send fictitious resumes in response to online job postings, randomly assigning some resumes to indicate stronger math skills, and measure employer responses. The resumes that are randomly assigned to indicate stronger math skills receive more interest from employers than the comparison resumes. Our findings add to the body of evidence showing that stronger math skills positively affect labor-market outcomes.

JEL Codes: J23, J24, I20
Keywords: Math skills, math skills and employment, math skills field experiment, math skills resume experiment.

WP 10-09
The Relative Performance of Head Start
Cory Koedel & Teerachat Techapaisarnjaroenkit

In early 2010, the U.S. Department of Health and Human Services released the findings from a large, experimental evaluation of the Head Start program. A common interpretation of the findings is that they show "small" effects, which has lead to, among other things, calls to improve the efficacy of Head Start. However, it is not clear that Head Start is performing worse than should be reasonably expected. To provide a frame of reference for evaluating the program, we compare the performance of Head Start childcare centers to the performance of non-Head Start childcare centers, the latter being the preferred childcare option of wealthier families. We find that, on average, Head Start centers perform similarly to non-Head Start centers, despite serving primarily disadvantaged children. Our results suggest that expectations for the Head Start program may be too high.

JEL Codes: I20, I28
Keywords: Head Start, Preschool, childcare, center-based childcare, standardized difference

WP 10-02
Grading Standards in Education Departments at Universities
Cory Koedel

This paper documents a startling difference in the grading standards between education departments and other academic departments at universities – undergraduate students in education classes receive significantly higher grades than students in all other classes. This phenomenon cannot be explained by differences in student quality or structural differences across departments (i.e., differences in class sizes). Drawing on evidence from the economics literature, the differences in grading standards between education and non-education departments imply that undergraduate education majors, the majority of whom become teachers, supply substantially less effort in college than non-education majors. If the grading standards in education departments were brought in line with those of other major academic departments, student effort would be expected to increase by at least 10-16 percent.

JEL Codes: I20, I23
Keywords: Grading Standards, Grade Inflation, Grading Standards in Departments of Education, Teacher Training

WP 09-13
A Non-Experimental Evaluation of Curricular Effectiveness in Math
Cory Koedel & Rachana Bhatt

We use non-experimental data from a large panel of schools and districts in Indiana to evaluate the impacts of math curricula on student achievement. Using matching methods, we obtain causal estimates of curriculum effects at just a fraction of what it would cost to produce experimental estimates. Furthermore, external validity concerns that are particularly cogent in experimental curricular evaluations suggest that our non-experimental estimates may be preferred. In the short term, we find large differences in effectiveness across some math curricula. However, as with many other educational inputs, the effects of math curricula do not persist over time. Across curriculum adoption cycles, publishers that produce less effective curricula in one cycle do not lose market share in the next cycle. One explanation for this result is the dearth of information available to administrators about curricular effectiveness.

JEL Codes: I21, I28 and H75
Keywords: curricular effectiveness, math curricula, non-experimental methods, matching methods, education policy

Updated substantially as WP 11-22

WP 09-10
The Social Cost of Open Enrollment as a School Choice Policy
Cory Koedel, Julian R. Betts, Lorien A. Rice & Andrew C. Zau

We evaluate the integrating and segregating effects of school choice in a large, urban school district. Our findings, based on applications for fall 2001, suggest that open enrollment, a school-choice program that does not have explicit integrative objectives and does not provide busing, segregates students along three socioeconomic dimensions – race/ethnicity, student achievement and parental-education status. Using information on expenditures to promote integration at the district, we back out estimates of the social cost of open enrollment realized in terms of student segregation. Our social-cost estimates range widely depending on the weights that we place on the different dimensions of integration. However, even using conservative valuations of the different integrative measures suggests a social cost at this single district of over 3.4 million dollars (in year-2000 dollars). When considered in the context of the nation as a whole, where open-enrollment programs are commonplace, this estimate from a single district is substantial. However, we also note that there may be benefits not related to integration that counterbalance some or all of these costs.

JEL Codes: I20, J15, R23
Keywords: school choice, open enrollment, integration, segregation, segregation costs

WP 09-06
Postsecondary Education Structure
Originally listed as:Postsecondary Education Structure and Human Capital Production
Cory Koedel

States differ substantially in the structures of their public four-year university systems. This paper uses micro-level data to evaluate the effects of postsecondary education structure on individuals’ educational and labor-market outcomes. Postsecondary education structure affects whether individuals attend universities at all, whether they attend public or private universities, and whether they attend large or small universities. Individuals who are exposed to more-fractionalized structures are adversely affected in the labor market. In conjunction with evidence that it is more expensive to educate students at smaller universities, this latter result suggests that states with more-fractionalized postsecondary education structures should look to consolidate their resources into fewer, larger universities.

JEL Codes: I20, I23, J24
Keywords: postsecondary education structure, higher education structure,
small university, large university, postsecondary education costs

WP 09-02
Does Student Sorting Invalidate Value-Added Models of Teacher Effectiveness? An Extended Analysis of the Rothstein Critique
Cory Koedel & Julian R. Betts

Value-added modeling continues to gain traction as a tool for measuring teacher performance. However, recent research (Rothstein, 2009, forthcoming) questions the validity of the value-added approach by showing that it does not mitigate student-teacher sorting bias (its presumed primary benefit). Our study explores this critique in more detail. Although we find that estimated teacher effects from some value-added models are severely biased, we also show that a sufficiently complex value-added model that evaluates teachers over multiple years reduces the sorting-bias problem to statistical insignificance. One implication of our findings is that data from the first year or two of classroom teaching for novice teachers may be insufficient to make reliable judgments about quality. Overall, our results suggest that in some cases value-added modeling will continue to provide useful information about the effectiveness of educational inputs.

JEL Codes: I20, I28 and I21
Keywords: value added, measurement of teacher quality, outcome-based teacher
quality

WP 08-08
An Empirical Analysis of Teacher Spillover Effects in Secondary School
Cory Koedel

This paper examines whether educational production in secondary school involves joint production among teachers across subjects. In doing so, it also provides insights into the reliability of value-added modeling. Teacher value- added to reading test scores is estimated for four different teacher types: English, math, science and social studies. While the initial results indicate that reading output is jointly produced by math and English teachers, post-estimation falsification tests debunk the math-teacher “effects” - that is, there is in fact no evidence of joint production in secondary school. The results offer a mixed review of the value-added methodology, suggesting that it may be useful in some contexts but not others.

JEL Codes: I20, I22
Keywords: value-added, teacher quality, secondary school teachers, educational production

WP 08-07
Value-Added to What? How a Ceiling in the Testing Instrument Influences Value-Added Estimation
Cory Koedel and Julian Betts

Value-added measures of teacher quality may be sensitive to the quantitative properties of the testing instruments upon which they are based. This paper focuses on the sensitivity of value-added to a particularly relevant testing-instrument property – test-score-ceiling effects. Test-score ceilings are likely to be increasingly common in testing instruments across the country as education policy continues to emphasize proficiency-based reform. Encouragingly, we show that over a wide range of test-score- ceiling severity, teachers’ value-added estimates are only negligibly influenced by ceiling effects. However, as ceiling conditions approach those found in minimum-competency testing environments, value-added results are significantly altered.

JEL Codes: I20, I21 and J24
Keywords: Value Added, Test Score Ceiling, Ceiling Effects, Teacher Quality, Teacher Value Added

WP 07-13
Teacher Quality and Dropout Outcomes in a Large, Urban School District
Cory Koedel

Recent research shows that variation in teacher quality has large effects on student performance. However, this research is based entirely on student test scores. This paper evaluates teacher quality in terms of another educational outcome of great interest – graduation. Using a unique instrumental variables approach to identify teacher effects, I find that differences in teacher quality have large effects on graduation outcomes. Because teacher effects on graduation outcomes will be more pronounced for students who are on the graduation margin, the results imply an avenue through which high-quality teachers are more productive with disadvantaged students.

JEL Codes: I20, I21 and J24
Keywords: Teacher Quality, Teacher Evaluation, Dropouts, Graduation Outcomes, Multiple Endogenous Treatments

WP 07-08
Re-Examining the Role of Teacher Quality In the Educational Production Function
Cory Koedel & Julian R. Betts

This study uses administrative data linking students and teachers at the classroom level to estimate teacher value-added to student test scores. We find that variation in teacher quality is an important contributor to student achievement – more important than has been implied by previous work. This result is attributable, at least in part, to the lack of a ceiling effect in the testing instrument used to measure teacher quality. We also show that teacher qualifications are almost entirely unable to predict value-added. Motivated by this result, we consider whether it is feasible to incorporate value-added into evaluation or merit pay programs.

JEL Codes: I20, I21, J24
Keywords: teacher quality, educational production, teacher value-added, value-added, test-score ceiling effects, teacher evaluation, teacher accountability, elementary school

WP 07-07
Teacher Quality and Educational Production in Secondary School
Cory Koedel

This study uses administrative data linking students and teachers at the classroom level to evaluate teacher quality and joint production in secondary school. Teacher quality is measured by value-added to student test scores in math and reading. Although empirical research has struggled to link observable teacher qualifications to student achievement, teacher quality measured by student performance varies significantly and has important effects on educational outcomes. I identify which teacher inputs affect which test-score outputs in secondary school and find strong evidence of joint production. The results from this study are applicable to incentive design and teacher accountability.

JEL Codes: I20, I21, J24
Keywords: teacher quality, educational production, joint production, teacher value-added, value-added, teacher evaluation, secondary school

Updated substantially as WP-0808