The regulator responsible for moderating A-level exam results ignored offers of expert help after statisticians refused to sign a highly-restrictive non-disclosure agreement, Sky News has learned.
Ofqual is facing an angry backlash from pupils and teachers after almost 40% of A-level grades were downgraded from teachers’ predictions by its computer algorithm.
Sky News has learned that the Royal Statistical Society (RSS) offered to help the regulator with the algorithm in April, writing to Ofqual to suggest that it take advice from external experts.
The RSS says it proposed two fellows to give assistance: Guy Nason, professor of statistics at Imperial College London, and Paula Williamson, professor of medical statistics at the University of Liverpool.
Ofqual agreed to consider the fellows, but only if the two academics signed a non-disclosure agreement (NDA) which prevented them from commenting in any way on the final choice of the model for five years after the results were released.
“We get the point of non-disclosure agreements: you don’t want someone offering a running commentary while decisions are being made,” said Sharon Witherspoon, vice president of the Royal Statistical Society for Education and Statistical Literacy.
“But constraining independent academic experts from saying, ‘Well, looking at the data, I saw it was clear this would have this effect,’ didn’t fit our principles of transparency.”
The RSS wrote to Ofqual in what Ms Witherspoon said was “the most constructive way” raising its concerns about the NDA, but said it received no reply to its message, so the offer of expert help was never taken.
Ms Witherspoon said she believed independent expertise could have improved the A-level algorithm, which has been criticised for unfairly lowering students’ grades based on schools’ past performance.
“I do feel sorrow about it because I think this could have been averted,” she told Sky News.
Stian Westlake, chief executive of the RSS, said that if Ofqual had been more open, the issues with the algorithm “would have been identified sooner”.
“We could have helped Ofqual reassess the criterial,” he said, citing the decision not to adjust the results for small numbers of students taking a particular subject at a school, a move is believed to have favoured private and selective schools.
“Even if Ofqual decided they didn’t want to change anything, greater transparency would have helped the Department for Education prepare for the response to their decisions.”
The news comes as the RSS has sent a letter to the Office for Statistics Regulation urging it to investigate the algorithm used by Ofqual.
The letter, seen by Sky News, criticises the regulator’s approach, saying it has “not been sufficiently transparent to meet the aim of being trustworthy”.
Ofqual did not respond to a request for comment, but Boris Johnson has shrugged off claims that the A-levels process has been chaotic, describing the results as “robust” and “dependable”.
:: Listen to the Daily podcast on Apple Podcasts, Google Podcasts, Spotify, Spreaker
Ofqual published the methodology used to create its algorithm after results were released.
Ms Witherspoon said that the key decision for the designers of the algorithm was not statistical, but political, because it involved a trade-off between the overall distribution of grades and justice for individual pupils.
Ofqual took the decision to prevent any kind of grade inflation, she said, at the expense of “rough justice” for some students, whose teachers might have given them higher predicted grades.
“That’s the big trade-off,” said Ms Witherspoon. “It’s no good saying there are just a few technical issues. Once you took the decision that you weren’t going to tolerate more than a couple of percentage points of grade inflation… that drove everything else.”