When U.S. News & World Report’s 2008 list of “America’s Best High Schools” was released in December, Bethesda-Chevy Chase High School Principal Karen Lockard received inquiries from concerned parents. B-CC had been ranked as the nation’s 64th best high school on the 2008 Newsweek list, but it was missing from U.S. News’ top 100. One parent e-mailed: “Should I be worried?” On the school’s Listserv a parent posted the following message about the rankings: “I wonder why B-CC HS is not mentioned at all. I thought B-CC was one of the better schools in Maryland.”
Each year, Newsweek and U.S. News publish lists of what the magazines say are America’s best public high schools. And each year the rankings are the subject of much conversation—and some consternation—among parents, students, teachers and administrators.
In the 2009 Newsweek rankings, released in June, four county schools (Richard Montgomery, B-CC, Thomas S. Wootton and Winston Churchill) were ranked among the nation’s top 100, with two others (Walt Whitman and Walter Johnson) narrowly missing. Three schools (Whitman, Wootton and Churchill) had turned up on the 2008 U.S. News top 100 list.
There’s no disputing the popularity of rankings. We love learning what’s considered “best”—whether it be the top-10 college football teams, the best places to live, the highest rated wines or the best films of the year. Rankings capture our interest as a reflection of our natural competitiveness as well as our curiosity. We want our team, town, company or favorite musician to be included among the most highly regarded. Bethesda-area residents—no slouches when it comes to competition—probably pay even more attention than others to who’s on top. Combine that with being preoccupied with the schools we attended—and those our children attend or will some day—and it shouldn’t be surprising the annual high school rankings generate intense local interest.
Most years, Montgomery County Public Schools (MCPS) has more schools in the top 100 of both lists than any other school system in the country. And MCPS is quick to get the word out about the rankings, often issuing a press release the same day the rankings are published. The MCPS release issued after this year’s Newsweek rankings were published quoted school board President Shirley Brandman as saying the rankings validate that the school system is “seeing the benefits of providing the academic support that allows our students to aim high and achieve at the highest levels."
Many MCPS high school principals say that Superintendent Jerry Weast frequently mentions the rankings as a point of pride at principals’ meetings and other forums. Whitman Principal Alan Goodwin says Weast makes sure he extends individual congratulations to principals whose schools are in the top 100. Such attention from the pinnacle of the county’s education hierarchy can’t help but establish performance standards and set expectations for schools to at least maintain their strong rankings—whether or not the criteria underlying the rankings make good sense as policy.
But do the rankings really mean much? Do they really reflect which schools are the best? As someone who regularly works with statistics (and years ago helped compile survey data for U.S. News’ popular college and graduate school rankings), I feel obliged to question the validity of the methods used to rank high schools.
Newsweek ranks schools based on the Challenge Index, which was developed by Washington Post education reporter (and Bethesda resident) Jay Mathews. A school’s Challenge Index score is the number of Advanced Placement (AP), International Baccalaureate (IB) and Cambridge tests taken by all students in a school year divided by the number of graduating seniors. (AP courses are well-known; IB and Cambridge also consist of rigorous courses for which students can receive college credit. Like AP, their standardized exams are graded by outside examiners.)
By this measure, all 23 Montgomery County public high schools old enough to be included rank among the top 4 percent in the nation on the 2009 Challenge Index.
Mathews is a man on a mission. Inspired by Jaime Escalante’s success with math students at Garfield High School in East Los Angeles (a story told in Mathews’ 1988 book, Escalante: The Best Teacher in America, and made famous in the movie Stand and Deliver), Mathews’ goal is to improve students’ academic preparation, especially in lower- and middle-income neighborhood schools. His solution is to expose more students to challenging course work, and he unapologetically describes his purpose as “advocacy as well as evaluation.”
He says the Challenge Index’s key attributes make it the singular best measure of a school’s quality: It can be easily understood; it points directly to implementing positive change through rigorous course offerings; and it can be applied meaningfully to all schools—unlike quality evaluations based on traditional measures such as test scores, which, he says, are inherently biased toward schools in wealthier, upper-middle-class neighborhoods.
Mathews’ work has won many supporters locally. Walter Johnson Principal Christopher Garran says that Mathews has done a “tremendous job in helping schools open doors to AP for all students” by promoting the need to expose more students to rigorous course work. Principals Joan Benz of Churchill and Michael Doran of Wootton praise Mathews’ accessibility and his willingness to improve the Challenge Index methodology, which he has incrementally revised over the years.
Critics, including Andrew J. Rotherham of Education Sector (a Washington, D.C., think tank that promotes change in education policy), have attacked the Challenge Index for not effectively capturing what it purports to measure (school quality), in part because it doesn’t gauge student achievement, only the number of rigorous course exams taken. Mathews’ response is that exposure to rigorous material alone has proven salutary effects, and that most schools in average-income and lower-income areas would never have a chance to be ranked and recognized if test scores were the main standard. The index has also been criticized by Rotherham, and by Bob Morse, U.S. News’ director of data research, for ignoring graduation rates and achievement gaps among socioeconomic and ethnic segments.
Mathews admits feeling ambivalent about Newsweek headlining the Challenge Index rankings with “America’s Top High Schools.” He recognizes that some readers can be misled by a superficial glance at the list without taking the time to decipher what it’s based on. But he defends the practice on the grounds that “…dramatizing the index with this title is the only sure way to grab readers’ attention, without which its positive impact would be lost.”
U.S. News’ approach to ranking high schools nationally is different from Mathews’ method and far more computationally complex. To be ranked on U.S. News’ “Gold Medal List,” a school must do significantly better on standardized state English and math tests than statistically expected given its economic makeup; be in the top half of its state (approximately) in the performance of its minority students; and, among the schools remaining after these first two hurdles, achieve one of the 100 highest scores on the “College Readiness Index.” The “College Readiness” formula combines two components: the percentage of 12th-graders who had taken an AP or IB exam during or before their senior year; and the percentage who passed at least one exam—equivalent to an AP test score of 3 and an IB score of 4. (The pass rate component is weighted more heavily in the College Readiness formula.) Morse of U.S. News explains: “The first two steps are designed to ensure that the schools serve all their students well; the third assesses the degree to which schools prepare students for college-level work.” (An 18-page explanation of their methodology is available here. )
