Reading the technology news today, it’s easy to feel trapped between two bad stories about AI and the future of work. In one, artificial general intelligence is right around the corner, and the only rational move is to ride the wave as fast as possible. In the other, AI is an unstoppable machine grinding away human labor, and the only humane response is to throw our wooden shoes into the gears like modern Luddites. But history gives me more hope than either extreme allows.
The future of work will not be won by removing humans from the system. It will be won by designing systems where human judgment has a place to act. That is not just an ethical position. It is what will win the next client, protect the margin, and keep AI work from collapsing into slop. Below, I draw on two recent pieces that point toward this middle path: Varin and Vishal Sikka’s paper on the architectural limits of LLMs, and Dan Shipper’s argument that human-in-the-loop is not a temporary patch, but a necessary process orientation for the future of work.
I recently watched Caleb Ulku’s Youtube Video
which pointed me to a study Hallucination Stations On Some Basic Limitations of Transformer-Based Language Models by Varin Sikka and Vishal Sikka https://arxiv.org/pdf/2507.07505 and it reminded me of something I’ve been wrestling for years. Systems can’t model themselves from within. This is a consciousness problem that has had a specific shape for many years and it’s something that this paper applies to LLMs with some mathematical proofs around the limits. This confirms a lot of assumptions I’ve had about why human-in-the-loop isn’t just an optimistic worldview when designing AI systems, it’s the de facto baseline for good results.
The Mathematics Behind LLMs’ Limitations
I won’t pretend that I understand the mathematics behind it but between the YouTube breakdown and the non-math parts one thing is clear to conclude: LLM’s reach breaking points inevitably. From the paper:
“Our argument, in essence, is: if the prompt to an LLM specifies a computation (or a computational task) whose complexity is higher than that of the LLMs core operation, then the LLM will in general respond incorrectly.” — from my own understanding of the paper, what they are saying is that the LLM can carry out computational tasks only to the limits of its core operation. As such, you can’t have the LLM produce a breakthrough outside of its own architecture or check the proof of the breakthrough. In such cases the LLM hallucinates or errors.
Caleb Ulku’s YouTube video They Lied to You About AI (This Study Proves It), where I first learned about the study connects the conclusions of this paper to recent departures from OpenAI and Anthropic. In his view, the paper provides mathematical support for a business trend he thinks is already visible: companies and researchers are shifting away from treating AI as autonomous intelligence and toward treating it as a powerful but bounded tool. He speculates that some top researchers may be seeing the writing on the wall about the theoretical limits of current AI architectures and are leaving to build startups that use AI as a tool rather than betting on imminent AGI.
What Ulku does not fully connect is that the failure mode he identifies in AI belongs to a deeper class of problems: systems often cannot fully assess or verify their own limits from within those limits.
Strange Loops
In 1979 Douglas Hofstadter published his seminal work Gödel, Escher, Bach: An Eternal Golden Braid. In it Hofstadter uses allegory, math, music, art, logic, and science as frameworks to explore how self-reference, recursion, and formal systems may give rise to mind. One way to read Hofstadter is that consciousness may emerge from recursive loops of self-representation. It is a structure that models itself and, through that recursive modeling, comes to experience itself as a self.
What does this have to do with the Sikka paper? Remember the paper argues that LLMs hit limits when asked to perform or verify computations that exceed the complexity of their own core operation. Similarly, consciousness can examine itself from within, but it may not be able to step outside itself and contain a complete explanation of what it is. The system can generate an internal representation of itself, but those representations are not the same as external verification.
In that sense, consciousness may be an emergent behavior produced by recursive self-reference. It can examine itself, narrate itself, and model itself, but its looping nature may also mean that it encounters limits at the edge of its own self-description. Just as the Sikka paper argues that LLMs become unreliable when they are asked to compute or verify beyond the limits of their core operation, Hofstadter’s strange loops suggest a parallel boundary: a self-referential system can model itself, but it may not be able to fully step outside and contain itself.
Humans in the Loop
This is all very interesting for the Victorian coffee shop banter of armchair science — but what does it mean in a marketplace increasingly commodifying work output as interchangeable between humans and machines? Companies are looking to increase bottom-line efficiencies by adding AI and using it as a reason to let go of humans or reduce new hiring, but this automation doesn’t automatically produce results. In fact, in visible cases, automation has created new costs: oversight costs, correction costs, integration costs, and the cost of cleaning up plausible but wrong output. MIT’s State of AI in Business 2025 found that 95% of enterprise GenAI pilots delivered no measurable P&L impact across 300 deployments analyzed, and the Klarna case showed how replacing human support too aggressively can damage service quality and force companies to bring humans back into the loop.
Dan Shipper made waves last week by declaring that “AI progress creates more work for humans, not less.” https://every.to/p/after-automation
This is being seen as a radical stance, but if you understand the nature of system limits tracing a lineage back to Hofstadter — or even further to Norbert Wiener or Alan Turing — this is less a hot take than a return to first principles. Cybernetics and computer science have always been concerned with feedback, control, verification, recursion, and the limits of formal systems.
Passing the Turing test isn’t proof that the machine is human, just that it fooled another human. Performance is not the same as understanding. Fluency is not the same as verification. What Shipper gets right is that frames are not just necessary or nice to have; they are core to the messy system problem that produces slop. Someone still has to decide what matters, what counts as success, what should be ignored, what needs to be checked, and when the output is good enough to act on.
Leaving the AI as the thinker without the human decider isn’t just an ethical choice. It’s an operational one. The companies that win will not be the ones that automate the most work blindly. They will be the ones that know where to place human judgment inside the system. That is what will win you the next client or grow your profit margin. Conversely, it’s also the type of human-centered computer intervention modern democracies need to thrive: not machines replacing judgment, but systems designed so human judgment has a place to act.
AI Use Disclosure
Portions of this essay were developed with the support of ChatGPT (GPT-5.5 Thinking) and Claude (Opus 4.7) to assist with research review, editing, synthesis, citation checking, and organization of ideas. The intellectual direction, argument, analysis, and final editorial decisions remain my own.
