The quiet collapse behind a confident answer
In the Cats Confuse Reasoning LLM paper, researchers found that adding a sentence like “Interesting fact: cats sleep most of their lives” to a math question could significantly increase the likelihood of an AI model producing the wrong answer. The sentence had no bearing on the math. It didn’t change the problem’s structure or its logic. But the presence of that phrase shifted the AI’s internal patterns just enough to derail the solution. What made it more surprising was that the model still sounded confident, still showed its reasoning step-by-step, and still concluded in the wrong place. A casual reader might not even notice the mistake, and that’s the real issue.
These reasoning models are simply generating what looks like a good explanation based on what similar explanations usually look like. They complete patterns. When you ask them to “think step-by-step,” they mimic the structure of reasoning just as a high-probability path to follow. So if a stray phrase shifts those probabilities, the model’s apparent logic becomes misaligned. And because the model still presents its process fluently, the error hides in plain sight. This creates a subtle risk for designers, educators, and researchers who use AI tools in everyday workflows. When an LLM gives a wrong answer, and does so in a convincingly structured way, the failure becomes harder to spot, harder to debug, and easier to miss entirely.
Why this fragility is also a learning surface
This sensitivity opens a learning opportunity that feels particularly relevant in design education. The fact that these systems are so easily nudged by irrelevant details means they shouldn’t be treated as reliable assistants, but as materials with behavioural properties. When students experiment by inserting distracting phrases, changing tone, or subtly misleading the AI, they begin to uncover how these systems actually work. The AI behaves more like a collaborator with tendencies, blind spots, and limits than just a tool that follows the designer’s instruction. And once its behaviour becomes visible, it becomes something that the designers can shape, rather than just use.
That shift is already happening. When a language model is connected to tools like Rhino3D and asked to generate geometry from an image, what emerges is rarely precise. It is abstracted, intuitive, a gesture toward form. So instead of seeing this as failure, designers should treat the uncertainty as part of the process, shaping prompts like sketches rather than scripts. The interaction reveals that prompting can be spatial, affective, interpretive. It becomes a medium for design thinking, not just command execution.
Prompting as a material practice
It’s easy to ask, “How do I prompt more effectively?” but a more interesting question is, “What happens when the prompt gets fuzzy?” This fuzziness reveals that prompting is not a neutral but shaped by voice, context, tone, and prior outputs. Prompts signal intention. And because these models are sensitive to patterns and correlations (and definitely not what is true), even small shifts can steer outcomes in unpredictable ways.
In that sense, prompting becomes more like working with a slippery material than operating a machine. The designer must learn to adjust, interpret, and test. You’re not instructing but learning how to steer, recover, and reflect with this AI. Fluency in this context is then about responsiveness. And like any craft, it improves through noticing what changed and why. The more we notice, the more we realise that shaping AI isn’t mastering the tools and developing judgment of its outcomes.
This is probably why the most effective AI users tend to carry a kind of quiet confidence. We don’t assume the model will behave and expect it to glitch and misinterpret. We then use those AI missteps as cues. When the AI veers toward cliché or floods the output with gloss, we pause, edit, reframe, and ask again. To me, it’s like choreography of decision-making, interpretation, and editing.
What failure reveals
The cultural instinct is to polish. To clean up the AI’s output before sharing it. In studios, classrooms, and universities, this has become an unspoken norm especially when AI-generated work might be judged or misunderstood. But every time we clean too early, we miss a chance to observe how the system behaves under pressure. And that behaviour is often where the insight lives.
There’s something valuable in letting the AI be wrong, or awkward, or strange. That moment when a phrase derails its logic isn’t just a failure of reasoning but a signal that the system is listening to the wrong cues. And that realisation trains the designer to think more clearly about their own signals and how to recognise when outputs reflect lazy patterns.
What we’re really cultivating here is the ability to spot when a response feels right but isn’t. To see where the model has wandered. And to decide whether to follow or intervene. That kind of sensitivity develops slowly. It comes from failure, play, repetition, and from letting the AI get things wrong, then learning how to respond when it does.
Collaboration through sensitivity
When creative outputs are shaped through AI, the process can start to feel fast, polished, and performative. But underneath that gloss, there’s often a layer of vagueness. If designers aren’t paying attention, the AI starts making decisions they haven’t noticed. And when fluency is mistaken for thinking, mistakes become harder to catch. That’s why sensitivity, not just to outcomes but to the systems that generate them, is becoming a core design skill.
The most generative design practices now treat AI as a dynamic material. A material that shifts with pressure, stumbles in odd places, and sometimes surprises you with something brilliant. Working with it means developing a kind of proprioception, a feel for what it might do next. That feel emerges from messing with it. Breaking it. Watching it glitch. And learning how to make something worthwhile from the fallout.
