Apples to processed and refined applesauce

How Virginia Plays the Accreditation Numbers Games

How effective is Virginia’s SOL test-driven accountability program? State policy makers, other proponents of high stakes testing, and the media offer as evidence of success (and thus celebration) a rise in SOL pass rates. The state proclaims that, despite the continued concerns, the testing program is working. Working, they say, to make sure kids are learning and schools are teaching. Working to sort the bad schools from the good schools and make the bad ones better.

School accreditation in Virginia’s schools is based on pass rates on SOL tests, not on all individual students' test scores. Simply put, schools must have a pass rate of 70% in each of the four core areas in order to be fully accredited. (On a side note, do you ever wonder about the 30% that the state permits to fail?) If you think, however, that a school pass rate of 70% means that 70% of the kids in those particular grades passed the related SOL tests, you would be wrong.

Take, for example, one fully accredited elementary school. (To avoid embarrassing individual schools, we do not identify schools by name.) For accreditation purposes, this school’s reported pass rates for 2001 are as follows:

                                   Figure 1: pass rates adjusted for accreditation purposes (2001)

                           Figure 2: four-year actual pass rates (unadjusted) for school accredited in 2001

To arrive at pass rates for accreditation purposes (see figure 1), the state allows use of various accreditation-inflating strategies, which have been added to state regulations and guidelines for accrediting schools.

Please note that we support strategies designed to make uses of test scores more reasonable and less punitive for schools, e.g., excluding certain English Language Learners’ and transfer students’ scores (though we believe all kids should get corresponding breaks for diploma purposes), and allowing use of three-year rolling averages. (See below). What we object to is the state’s yearly reporting of pass rates and accreditation statistics without noting the changes made in methods for calculating these rates since the program and the reporting began. This gives the impression that we’re comparing apples to apples from year to year, when what we're really comparing is apples to processed and refined apple sauce.  

Here’s how actual pass rates get morphed for accreditation purposes:

"Bonus Points" for Remdiation Recovery Students: Students who fail math and/or English SOL tests in elementary and middle school and are promoted to the next grade can participate in remediation and then re-take the tests. Same goes for those who fail high school math SOL tests. If these students pass the re-tests, their scores are included in the numerator (total test passers) but not the denominator (total test takers) when figuring pass rates for accreditation purposes (for the schools where they re-tested). This can tremendously inflate actual pass rates, depending on how many students re-test after remediation. Indeed, the Board of Education’s oft-stated purpose in approving this strategy in 2000 was to give schools this accreditation booster as an incentive to remediate kids who fail the specified SOL tests, even when they pass the courses. And look how it helped the school in our example! (An explanation of how approximate percentage of pass-rate increase attributable to bonus points is calculated is included in our more technical version of this paper).

Excluding Certain Scores When Figuring Accreditation Pass Rates: The state allows schools to exclude test scores of certain English Language Learners (depending on how long they’ve been in a VA public school), unless it would help schools’ bottom lines to include them. If these students pass, their scores will be included; if they fail, the scores can be excluded. The state also allows schools to exclude scores of kids who transfer in after a certain point in the school year.

Averaging Across Grades When Figuring Accreditation Pass Rates: Schools' pass rates in each subject for accreditation purposes aren't all presented by grade level or course.  Pass rates in certain subjects in elementary schoolare averaged across grade levels , while pass rates in all high school subjects are averaged across courses, to produce a single pass rate for each subject area in each school. In middle schools, pass rates for high school tests are averaged into 8th grade tests in the same subjects. Some grades or, in the case of  middle and high schools, some courses may have very low pass rates but, when combined with higher ones (from other grades or courses), the low ones are masked and the public doesn’t get the true picture.

Rolling Averages: Schools can report their pass rates for purposes of accreditation by averaging across the last three years of testing - if it helps. If it doesn’t, as for our example school, they can simply use the current year’s scores. Given that yearly pass rates tend to fluctuate for various reasons, and given the pass-rate history of this school and others like it, it is quite possible that next year’s rates will go back down.

 Today one would hope that everyone would insist on full disclosure of all formulas and strategies used in calculating and reporting pass rates and accreditation status for all schools, at the time they are reported. To claim that higher pass rates and accreditation percentages show the SOLs are working to improve public education for all students all across the state seems careless and irresponsible, if not downright disingenuous and misleading.