“Eighty-three percent of LLM users could not quote a single line from essays they had written minutes earlier.”
That’s one of the headline findings from this opinion piece based on a recent MIT Media Lab preprint that has already begun making the rounds in education circles.
Participants were divided into groups and asked to write SAT-style essays. Those using ChatGPT reportedly struggled to recall exact lines from what they had written.
OK…
Now let’s ask the question every researcher should ask before panicking:
Compared to what?
How many SAT writers in 2005 could quote a line from an essay they had just finished?
How many college freshmen in 1995?
How many graduate students?
How many professors?
Because if we’re going to treat “couldn’t quote a sentence from their essay” as evidence of cognitive collapse, I’d like to see the baseline before we start sounding the alarms.
This article treats AI like the arsonist. Looking at the timeline, AI seems a lot more like the fire alarm.
I have written thousands of pages over the course of my career. Blog posts. Research papers. Grant applications. Professional development materials. Dissertation drafts. Articles.
Y’all, if you handed me something I wrote ten minutes ago and asked me to quote a sentence verbatim, there is a non-zero chance I’d stare at you like C-3PO trying to calculate the odds of surviving an asteroid field.
Human beings generally remember ideas. We remember arguments. We remember stories.
We do not typically store exact wording like we’re running a local backup server.
But the more interesting question isn’t whether ChatGPT users remembered sentences. The more interesting question is why in the hell we think that matters?
What strikes me about the growing conversation around AI is that many of the arguments reveal assumptions we’ve been carrying around for decades without examining them.
The underlying assumption is this:
“If a student produces an essay, learning has occurred.”
Has it?
Really?
Because AI didn’t create that problem. AI exposed it.
For years, we’ve designed assignments where the product stood in for the thinking.
- Write the essay.
- Complete the worksheet.
- Build the slideshow.
- Answer the questions.
- Turn it in.
And because producing those things required effort, we assumed the effort itself was evidence of learning.
Now we have technology that can produce many of those artifacts in seconds, and suddenly we’re discovering that the artifact was never the point; it was the thinking.
This shouldn’t be surprising; we’ve seen this film before, and we didn’t like the ending.
- Calculators didn’t destroy mathematics.
- Google didn’t destroy knowledge.
- Spellcheck didn’t destroy writing.
- Wikipedia didn’t destroy research.
Every few years, society education finds a new villain. Every technological shift produces a wave of satanic panic, followed by cooler heads prevailing and an objective realization that what mattered was never the tool. It was the thinking behind the tool.
The irony is that the Chronicle article cites evidence of declining literacy that predates ChatGPT by years.
- NAEP scores were falling before large language models existed.
- Concerns about reading stamina existed before ChatGPT.
- Complaints about student attention spans existed before ChatGPT.
- College professors were writing “students can’t read anymore” essays before ChatGPT was even a research project.
The timeline matters.
If the disease existed before the patient was exposed to the virus, we should probably be careful about diagnosing the cause.
And here’s where I may part company with some of my colleagues: AI isn’t the biggest threat revealed by this study. The biggest threat is discovering how many of our assignments can be completed without demonstrating actual understanding. They are, for all practical purposes, “bullshit” assignments.
The questions are all made up, and the points don’t matter. (Thanks, Whose Line!)
That’s uncomfortable because it gets down to the heart of the problem: the education system itself. It’s the tasks, the assessments, the systems we’ve built around compliance, completion, and artifact production.
I’ll take a moment here and step on all my project-based/deeper learning/authentic learning/transformational learning friends here as well: Just because students produced something doesn’t mean they learned anything. Be careful whom you look down upon, o great creators of “we did this really fun thing, and my kids were so engaged and had a blast, but they didn’t really learn anything and didn’t produce anything new.”
The real question isn’t:
“How do we stop students from using AI?”
The real question is:
“What evidence would convince us that learning actually happened?”
Can students explain their thinking? Defend their reasoning? Apply ideas in a new context? Revise their understanding when presented with new information? Transfer knowledge across disciplines? Create something that wasn’t possible before?
Those are much harder questions than “Did you write the essay?”
But they’re also the questions that matter.
AI is not causing the literacy crisis. It’s just bringing it to the forefront because we’ve been ignoring it for years and just kept doing what we always did – god, I hate that phrase – because that’s what was easy for teachers and schools. Keep cranking kids through the machinery of the education system and ignore any type of quality control that’s SCREAMING at you that something is terribly wrong.
All we’ve ever been concerned about is the end product. We’ve never worried about the process. And that, my friends, is the core issue. To quote Brandon Sanderson, “Journey Before Destination.”
Because the journey is far more important.
