Campus AI Development Use case scenarios

Use case scenarios Discussion case 04 of 06

Discussion case 04

The research code nobody can explain

A lab’s key result rests on an analysis pipeline written almost entirely by a coding agent. The paper is under review. A reviewer asks the team to explain one transformation step, and nobody in the lab can say why it works, only that the outputs looked right.

A hypothetical composite for discussion, not an account of a real institution.

The tension

One viewThe result standsThe pipeline ran, outputs matched expectations, and researchers have always used code they didn’t write, from statistical packages to libraries.
Another viewThe result is not yet trustworthyCode that no one can explain is a method no one can defend. The lab should verify before publishing, even if it delays the paper.

Questions for discussion

  1. 01Is using unexplained AI-written code different from using an unexplained statistical package?
  2. 02What would count as enough verification: tests, a hand-checked subset, an independent rebuild?
  3. 03What should the methods section disclose?
  4. 04Who in the lab is accountable for the code’s correctness?
  5. 05Should the department set a standard, or leave it to each PI?
What the framework suggests
  • Verification and accountability are the lenses at stake.
  • Research integrity is one of the guide’s stakes, and provenance is part of the method.
  • The research software engineer scenario shows the rebuild path.
Where reasonable people disagree
  • How much understanding a researcher owes their own code
  • Whether disclosure is enough without verification
  • Who sets the standard: journals, departments or labs