2002: Considering the achievement effects of the SOLs after five years and $100 million state dollars: 

How are Virginia students faring on other measures of student achievement? 

NAEP (National Assessment of Educational Progress) tests  http://nces.ed.gov/nationsreportcard/states/  

NAEP tests are given in subject areas of math, reading, science and writing.  Individual subject areas are not assessed annually, rather they are sampled every couple of years.  They are given at 4th and 8th grade levels in January or February.  When they are given to students, groups of students take a sampling of the questions rather than the entire test and, given the scores on those samples, inferences are made about achievement of the student population as a whole.  Scores are reported at the state level. 

NAEP Reading:  We include a description of NAEP reading because the Virginia Board of Education often uses NAEP Reading to demonstrate the effectiveness of the SOL testing policies.  NAEP reading tests were given in early 1992, 1994, and 1998.  SOL tests were not given until mid-1998 so it is unlikely that the SOL test results (which were not to be announced until late 1998) played a role in the 1998 NAEP scores.  NAEP scores on the 1998 reading tests increased over the results on the 1994 NAEP reading tests.  They have not yet risen to the levels achieved in 1992 (before SOLs were a twinkle in the politicians' eyes).  Additionally, it is important to note that in 1994, NAEP reading scores dropped across the nation, not just in Virginia and it is widely believed that the drop had more to do with technical problems with the tests rather than a drop in student achievement.

Effects of the SOL tests on NAEP reading achievement:  NO BASIS TO COMPARE

NAEP Math: NAEP math tests were given in 1990, 1992, 1996, and 2000.  While NAEP scores rose between 1996 and 2000 for both 4th grade and 8th grade, the percentage of students exclued from NAEP testing increased as well.  Virginia's rate of exclusion of students increased at one of the fastest rates in the nation.  The more students that are excluded (and these are generally the lowest performers), the more likely it is that mean scores will increase. 

Effects of the SOL tests on NAEP math achievement:  UNCLEAR DUE TO INCREASES IN EXCLUSIONS

NAEP Science:  NAEP science tests were given in 1996 and 2000 to 8th graders.  Despite SOL pass rates that rose from 71% in 1998 to 82% in 2000, NAEP Science scores did not differ significantly from 1996 to 2000. 

Effects of the SOL tests on NAEP science achievement:  NO CHANGE

Are rising SOL scores reflected in achievement on NAEP?  We don't think so.

Contrary to the carefully chosen wording of the Virginia Department of Education press release, Virginia students did not "outperform the nation," nor did they uniformly make gains on the 2002 National Assessment of Educational Progress (NAEP) 8^th grade writing tests. The average test scores of African American 8th graders have remained flat since the 1998 administration. In contrast, African American students' average scores nationwide have risen four points since 1998. Virginia's Hispanic 8th graders saw their average test scores fall from 151 in 1998, to 146 in 2002, while Hispanic students nationally saw a five-point gain in average scores. Eighth grade students with disabilities in Virginia had a one-point drop in their average test scores.

The percentage of Virginia's 4^th and 8th graders scoring at or above the Proficient level did not differ significantly from the rest of the nation’s. According to NAEP, "at grade 8, there were 10 jurisdictions with higher percentages of students performing at or above the Proficient level than the nation, 15 with percentages that were not found to differ significantly from the nation, and 22 with percentages that were lower than the nation." (http://nces.ed.gov/nationsreportcard/writing/results2002/stateachieve-g8.asp) (http://nces.ed.gov/nationsreportcard/writing/results2002/stateachieve-g4.asp)

"We continue to be concerned about the quality of public education for all of Virginia's children as all emphasis remains on raising SOL test pass rates. The results of the NAEP writing tests for minority students and students with disabilities unfortunately reinforce our concerns. The SOL pass rates for these students rose, but their NAEP performance did not. We are adding the results of the NAEP writing tests to the list of other achievement measures, such as SATs, diploma rates, NAEP science tests, and Stanford 9 tests, that do not match up to the purported achievement gains shown by SOL pass rates. We are disappointed but not really surprised that state education officials continue to use data selectively to try to make a case that the SOL testing program is improving student achievement," states Mickey VanDerwerker, spokesperson for Parents Across Virginia United to Reform SOLs (PAVURSOL).
 

  Stanford 9 tests:  http://www.pen.k12.va.us/VDOE/Assessment/home.shtml#VirginiaStateAssessmentProgram

Stanford 9 tests are given early in a student's fourth, sixth, and ninth grade years (this fall testing follows the spring SOL testing in 3rd, 5th, and 8th grade).  (Note that some school divisions do additional Stanford 9 tests at other grade levels.)  Both Stanford 9 and SOL tests are developed by the same company.  While SOL tests report scores on the 40-60 question tests on a 600 point scale, Stanford 9 tests give a percentile ranking.  The score on the Stanford 9 compares your child to a normed sample.  For example, the 50th percentile is right in the middle.  Stanford 9 tests students on reading, math and language.   

4th grade:  Stanford 9 scores are up 4 percentile points since 1998 (from 50th percentile to 54th percentile).  SOL reading pass rates have risen from 55% to 72% of children tested passing.  Math has gone up from 53rd percentile to 61st percentile on the Stanford 9.  Compare that to the rise in third grade math SOL pass rates:  from 63% to 80%. 

6th grade:  Standford 9 scores in reading rose 1 percentile point from 1998 to 1999 and then stayed flat since then at 59th percentile.  Fifth grade pass rates on the SOLs, however, rose from 68% to 78% on Reading.  In math, Stanford 9 results showed a jump from 58th to 66nd percentile between 1998 and 2001.  Fifth grade math SOL results showed an increase in students passing from 47% to 71%. 

9th grade:  On Stanford 9 math and reading tests, the average percentile ranks have remained essentially flat at 60th percentile in Reading and 55th percentile in math.  Note, however, that pass rates on SOL reading between 1998 and 2002, rose from 65% to 69% and SOL math from 53% to 71%. 

Effects of SOL testing on Stanford 9 scores:  While there have been increases in fourth grade, the upper grades do not reflect rising achievement.  Scores are STAGNANT at those levels.

Diploma rates: 

While the state has added several new diploma options for students with IEPs, we took a look at the percentage of students in 9th grade getting diplomas three years later.  We particularly looked at the percentage of students receiving either Standard or Advanced Diplomas.  The percentage of Virginia high school students earning a standard or advanced diploma has been falling since 1997, the year before the first SOL tests were administered for real.  In 1997, 75% of the students graduated with a standard or advanced diploma compared to 71.8% in 2002.  The percentage of students receiving a diploma that is tied to an IEP (Modified Standard Diploma, IEP diplomas for example) rose from 1.4% in 1997 to 4.6% in 2002. 

Effects of SOL testing on diploma rates:  STANDARD AND ADVANCED DIPLOMA RATES ARE FALLING

SAT I: 

Average verbal scores have risen a three points since 1998, the first year that SOL results were published and began affecting educational practices.  In the four year period prior to SOL implementation, verbal scores on SAT I rose six points.  Average math scores have risen 7 points in the same period.  However, as you look at the gains in scores, it is important to note that the percentage of students participating in SAT I testing has remained flat at 68%.  Given that these tests are generally taken by students planning for college, if the SOL tests were really increasing more students' achievement, surely we would see higher participation levels in the SAT I testing.

Effects of SOL on SAT I testing:  NO INCREASE IN PARTICIPATION   

SAT II: 

SAT II tests are being required by more and more colleges in Virginia and other states.  Participation in SAT II tests have been falling in Virginia while nationally, participation rates are rising. 

Effects of SOL on SAT II testing:  FALLING RATES OF PARTICIPATION

AP tests: 

http://www.collegeboard.com/ap/library/summary_rtps.html

The percentage of Virginia students taking AP tests rose slightly in 2002 but at a lower rate than nationally.  For Virginia public school students, the percentage of students scoring 3 or better on all AP tests has declined from 65% in 1997 to 62% in 2002.  In particular on AP Biology, English Composition, English Literature, and U.S. History, the percentage of public school students scoring 3 or better has declined since 1997.  Check out the graphic comparison of AP scores of 3 or better and SOL pass rates. 

Effects of SOL on AP testing:  DECLINING NUMBERS OF TESTS WITH SCORES OF 3 OR MORE BUT HIGHER PARTICIPATION MAKES THE EFFECTS UNCLEAR

And how about the SOLs?

	Pass rates on the SOL tests have risen steadily since testing started in 1998.  This is no surprise.  Pass rates start low and rise the first few years of testing (and if they don't, as in the case of Virginia's history tests, lower the cut scores).
	In 2002, statewide pass rates reached 70% or higher on End of Course (high school level) tests. On four of the high school tests, including English and writing, pass rates exceeded 80%.

Students answered, on average, about 1.5 more questions correctly in 2001 than in 1999, on the End of Course tests.

Participation on Chemistry End of Course tests dropped from 61% in 1998 to 58% in 2001.  That could account for rising pass rates!