Does A.I. stop kids from learning? New data from thousands of students suggest it depends on how A.I. is used. Below are the details... and the implications for how all of us should be using A.I.!
A.I. USE CAN BE BAD FOR LEARNING
• Researchers tracked 27,000 high-school students in China over 2.5 years, comparing each students results before and after they adopted A.I.
• As shown in the attached chart from The Economist: After adopting A.I., homework scores rose 18% and homework time fell ~30% (from 64 minutes to 45).
• But monthly exam scores fell 20% within six months, and high-stakes entrance-exam scores fell 18-24% after about two years.
• Before LLMs, homework used to predict exam performance... Now the top homework scorers are more likely to do *worse* on exams.
BUT A.I. IS NOT ALWAYS BAD
• Losses were concentrated among the ~80% of A.I. users who blatantly outsourced homework (i.e., very short completion times paired with high scores).
• The small proportion of A.I. users who spent about as long on homework as non-users, however, matched non-users on exams.
• In Turkey, ~1,000 high-school math students using standard GPT-4 scored 17% worse on a later unaided exam than students with no A.I., while a GPT Tutor with learning safeguards erased that penalty.
AND SOMETIMES A.I. CAN BOOST LEARNING!
• A Middlebury College study showed A.I. access raised test scores by 0.27 standard deviations, with gains persisting a week later. Gains were largest among students who had A.I. explain concepts rather than write text.
• A Harvard University study showed a custom A.I. physics tutor (fed lecturers solutions, limited to short responses) more than doubled learning vs an active-learning classroom, in less time.
• A six-week World Bank study in Nigerian after-school programs showed using GPT-4 delivered as part of a comprehensive educational solution showed gains equal to 1.5-2 years of typical schooling.
BOTTOM LINE
Learning requires effortful thinking ("desirable difficulty"). A.I. that hands over answers inflates homework grades and erodes skills; on the other hand, A.I. that explains, hints and gives feedback can be an outstanding tutor.
SO WHAT CAN YOU DO?
1. Watch your kids' time on task: homework quality jumping while time collapses signals outsourcing to A.I.
2. Configure A.I. to tutor, not answer: Turn on the study or learning modes many platforms now offer.
3. Lean on assessments done without A.I. since homework grades no longer show what students know.
4. Apply this to *yourself* at work: Try the problem first, ask the model to explain its reasoning and use A.I. to augment your thinking rather than replace it.
The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.
