NiNiProblem statement
Generative AI moved into undergraduate coursework faster than any measurement of its learning effects. Campus debate has settled into two confident and opposite claims: that AI is a personal tutor that lifts weaker students, and that it is a shortcut that hollows out the effortful practice learning depends on. Both claims are plausible, and both are usually argued from anecdote.
The gap this proposal addresses is not whether students use AI — surveys already answer that — but what kind of use produces what kind of change in unaided ability. A study that treats AI use as a single variable will find a weak, noisy correlation and explain nothing. The distinction that matters is between delegation, where the model produces the artefact, and interrogation, where the model is questioned while the student produces the artefact.
Research questions
- RQ1. How do delegation-type and interrogation-type AI use distribute across a first- and second-year undergraduate cohort, and how stable is each pattern across a year?
- RQ2. Controlling for prior attainment, does each pattern predict change in unaided problem-solving performance from term one to term three?
- RQ3. What do students believe they are learning when they use AI, and how does that belief diverge from measured performance?
Design and participants
A three-wave panel of roughly 300 undergraduates in one faculty, sampled across two disciplines with different assessment cultures (a writing-heavy and a problem-set-heavy programme) so that assessment format can be treated as a contrast rather than a confound.
At each wave, participants complete a short unaided performance task under invigilated conditions and a use-log instrument covering the previous four weeks. A subsample of 30 students, stratified by use pattern, completes think-aloud sessions at waves one and three.
Measures
Unaided ability is measured with parallel forms of a discipline-appropriate transfer task — problems that resemble taught material in structure but not in surface content — scored with a rubric that separates correct answers from correct reasoning.
AI use is captured two ways: a self-report log, and, for consenting students, exported chat transcripts coded for delegation versus interrogation. Comparing the two gives an estimate of self-report bias rather than assuming it away.
Analysis plan
Latent class analysis on wave-one use data to recover use patterns empirically instead of imposing them. Then a latent change score model regressing wave-three unaided performance on class membership, prior attainment and assessment culture.
The think-aloud transcripts are analysed separately and only then read against the quantitative result, so that the qualitative account can disagree with the model rather than decorate it.
Limitations
A single faculty limits generalisation, and one academic year is short relative to the development of expertise. Chat transcripts capture only one tool among several. The design cannot randomise AI use, so causal language is avoided throughout; the contribution is a well-identified association with a mechanism attached.
Timeline
- Months 1–2: ethics approval, instrument piloting, parallel-form equating.
- Months 3–4: wave one collection and think-aloud round one.
- Months 7–8: wave two collection.
- Months 11–12: wave three collection and think-aloud round two.
- Months 13–15: analysis and write-up.