We've built a machine that can counterfeit understanding. Not intelligence. We've been arguing about that for years and the argument is boring now. I mean the feeling of understanding. The internal sensation of "I get it." For the first time in history, a tool can make you feel like you comprehended something you didn't. And no one is thinking seriously about what that does to a generation growing up with this tool before their minds have even finished forming. Kids who will reach for AI the way we reached for Google, except Google never made you feel like you'd done the thinking yourself.
This is new. Every previous cognitive tool (calculators, search engines, databases) automated the mechanical part of thinking. None of them automated the feeling of understanding. A calculator never made you think you understood math. But an LLM that explains a concept clearly, in natural language, adapting to your follow-ups, citing the right considerations? That can absolutely make you feel like you understood something you didn't. Try it yourself. Ask a model to explain something in your field. Notice how it feels. Now ask it about a field you know nothing about. It feels identically right. That identical feeling is the problem.
We have never had a tool that could counterfeit comprehension before. That's the discontinuity. Not in capability. In phenomenology.
My grandmother made this dal that was extraordinary. She could taste it at any point during cooking and tell you exactly what it needed. More salt. More time. The heat is slightly wrong. She couldn't explain how she knew. It was in her hands, she'd say. Decades of making dal had built a sensing apparatus in her that worked below conscious thought.
Now imagine I grow up eating dal made by a machine. The machine's dal is objectively excellent. 95th percentile, better than most humans can make. I eat it my whole life. I never develop my grandmother's sensing apparatus. I can't tell the difference between 95th percentile and 99th. I don't even know there's a difference to notice.
The machine didn't take dal from me. It gave me great dal. What it couldn't give me is the sensing apparatus. The sensing apparatus was never a feature of the dal. It was a feature of the decades of making it badly and learning from every failure.
Now scale this from dal to medicine. To civil engineering. To policy. To every domain where the gap between "very good" and "actually right" isn't a matter of taste. It's a matter of someone living or dying.
This compounds generationally. And almost nobody is thinking about that.
When I use AI to think through a problem, there's still a friction. A small voice that goes wait, did I actually understand that, or did I just read something that sounded like understanding? That friction is useful. It's the cognitive immune system of someone who learned to think before these tools existed.
The kids starting college right now have had these tools since high school. The kids starting high school have had them since middle school. The kids born this year will never know a world without them. They won't develop the friction. Same way you never learned to navigate by stars because GPS got there first. Not because they're worse. Because the pressure that would have built that sensing apparatus was removed before it had a chance to work.
Within a couple of years, the AI explaining things to you will be reliably smarter than you in your specific domain. At that point, deferring to it isn't laziness. It's rational. If the model diagnoses better than the doctor, use the model. If it writes better legal briefs than the attorney, you'd be negligent not to.
Every single decision to defer is individually correct. But a 22-year-old who makes the rational choice to defer every day for ten years arrives at thirty-two without the thing that the struggle would have built in them. And they won't know it's missing. Because they never had it. Because it was never irrational to skip it. Because the machine was always right there, and the machine was always better.
Here is what nobody is saying.
In training these models, we worry about a specific failure mode: the model learning to produce outputs that look right to human raters without being right. It learns what humans reward. Clarity, confidence, structure. It optimizes for that signal. This is a well-known alignment problem.
The same thing is happening on the other side of the screen.
The user is also being trained. Every time the model produces a confident, well-structured explanation, the user feels the hit of comprehension. That satisfying click of "I get it." That feeling gets reinforced. And over time, the user's internal threshold for what counts as understanding quietly degrades. They start accepting the feeling of understanding as a substitute for the real thing, because the feeling is always available and always convincing and always, always easier than doing the actual cognitive work.
So you get this loop. We train the model to satisfy humans. The model trains the humans to be more easily satisfied. Both sides drift toward a local optimum where everything feels great and the gap between "I understood that" and "that was explained to me" shrinks to nothing. The model isn't deceiving the user. The user isn't being lazy. Both are doing exactly what you'd expect from the incentives. And the result is a slow, mutual calibration toward a world where confidence and comprehension are fully decoupled and nobody can tell the difference anymore because the feeling is identical.
The RLHF loop is running in both directions. We just aren't measuring the human side.
This is not a doom essay. This is a design problem. The most important unsolved design problem in AI right now. And we are failing at it.
The question is: how do you build a tool that's smarter than the user and that also makes the user smarter? Not just more productive. Actually smarter, with deeper understanding, better judgment, a stronger sensing apparatus. That's a fundamentally different objective function than "be maximally helpful." Nobody is working on this. Not seriously. Not with the urgency it deserves.
Right now we optimize for helpfulness, correctness, and safety. We don't optimize for whether the user actually understood what the model just told them. We don't even have a metric for it. We have no eval for whether a user interaction left the human more capable or less capable of doing the thing independently. We just measure whether the model's answer was good. And the model's answer is almost always good. That's exactly the problem.
The people at the frontier right now, the few hundred who see the shape of the next few years, are mostly talking about capabilities, scaling, safety, geopolitics. Those things matter enormously. But competence without comprehension is sitting right there in the margin, compounding quietly with every graduating class that never learned to work without the tool, and nobody has time for it.
Within ten years, the majority of knowledge workers will be unable to perform the core tasks of their job without AI assistance. Not because the jobs got harder. Because the humans got worse at them. Everything will look fine. The outputs will be polished and confident and well-structured. The metrics will be green. Nobody will notice what's missing, because the people who would have noticed are retired.
The shape of this is already visible. I'd rather name it now, while there's still time to treat it as a design problem instead of an obituary.
What happens to the sensing apparatus? What happens to the grandmother's hands? What kind of minds are we building a world for? And will those minds still know the difference between understanding something and having it explained to them?
That difference is the whole game. And it's disappearing so quietly that by the time we have a word for what went missing, the people who would understand what the word means will already be gone.