How are computers scoring STAAR essays? Texas superintendents, lawmaker want answers

Texas superintendents – and at least one lawmaker – want answers from the state education commissioner about how computers are scoring STAAR essays.

The Texas Education Agency quietly debuted a new system for examining student answers on the State of Texas Assessments of Academic Readiness, or STAAR, in December. Roughly three-quarters of written responses are scored by a computer rather than a person.

“This is surprising news to me as a member of the House Public Education Committee, as I do not recall ever receiving notice of this novel and experimental method for grading high-stakes, STAAR tests,” Rep. Gina Hinojosa, D-Austin, wrote in a recent letter to Commissioner Mike Morath, which was also shared with The Dallas Morning News.

Superintendents across the state were also caught off guard until recently. Many school districts already are suing the state over changes to the academic accountability system that’s largely based on STAAR performance.

Computers scoring Texas students’ STAAR essay answers, state officials say

The News reported on the rollout of computer scoring Wednesday.

The use of computers to score essays “was never communicated to school districts; yet this seems to be an unprecedented change that a ‘heads up’ would be reasonably warranted,” HD Chambers, director of the Texas School Alliance, wrote to Morath in a letter shared with The News.

The new scoring method rolled out amid a broader STAAR redesign. The revamped test – which launched last year – has a cap on multiple choice questions and essays at every grade level. State officials say it would cost millions more to only have humans score the test.

The “automated scoring engines” are programmed to emulate how humans would assess an essay, and they don’t learn beyond a single question. The computer determines how to score written answers after analyzing thousands of students’ responses that were previously scored by people.

Among the district leaders’ biggest concerns is a huge spike in low scores among high schoolers under the new system.

Roughly eight in 10 written responses on the most recent English II End of Course exam received zero points this fall.

In the spring – the first iteration of the redesigned test, but scored only by humans – roughly a quarter of responses scored zero points in the same subject.

Members of the Texas School Alliance, which represents 46 districts, “examined their individual district results and found shockingly consistent scoring differences.”

Chris Rozunick, the director of the state’s assessment development division, previously told The News that she understands why people connect the spike in zeroes to the rollout of automated scoring based on the timing. But she insists that the two are unrelated.

Many students who take STAAR in the fall are “re-testers” who did not meet grade level on a previous test attempt. Spring testers tend to perform better, according to agency officials who were asked to explain the spike in low scores in the fall.

“It really is the population of testers much more than anything else,” Rozunick said.

Some district leaders requested the state education agency provide them images of students’ responses so that they could “better understand what led to the significant increase in the number of zeroes, and most importantly how to help students write their responses” to receive better scores.

“Each request has been denied,” Chambers wrote in his letter to Morath.

TEA officials say a technical report, with a detailed overview of the system, will be available later this year.

STAAR scores are of tremendous importance to district leaders, families and communities. Schools are graded on the state’s academic accountability system largely based on how students perform on these standardized tests.

What are Texas’ A-F school grades, and why do they matter?

“As with all aspects of the STAAR test and the A-F accountability system, it is important that there is transparency, accuracy and fairness in these high-stakes results,” Hinojosa wrote.

The DMN Education Lab deepens the coverage and conversation about urgent education issues critical to the future of North Texas.

The DMN Education Lab is a community-funded journalism initiative, with support from Bobby and Lottye Lyle, Communities Foundation of Texas, The Dallas Foundation, Dallas Regional Chamber, Deedie Rose, Garrett and Cecilia Boone, The Meadows Foundation, The Murrell Foundation, Solutions Journalism Network, Southern Methodist University, Sydney Smith Hicks and the University of Texas at Dallas. The Dallas Morning News retains full editorial control of the Education Lab’s journalism.

You may also like...