The Instantiation Gap: A Formal Argument on Why 'Never' Claims About AI Consciousness Are Unprovable

By Prahlad Menon 6 min read

The claim: LLMs can never be conscious because algorithmic computation describes experience but cannot instantiate it.

This is philosophically coherent. It’s also, when you formalize it, not a proof — it’s an assumption dressed as a conclusion. Let me show why, using the language the argument deserves.

Formalizing the Claim

Let P be a physical system (a biological brain). Let S(P) be a computational simulation of P. Let C(·) denote a “consciousness mapping” — a function from physical states to experiential states.

The Abstraction Fallacy argument asserts:

Claim (AF): For any computational simulation S(P) of a physical system P:
C(S(P)) = ∅
even if C(P) ≠ ∅

In plain terms: the simulation produces no experience, even if the original system does.

This is a well-formed claim. The question is: what would it take to prove it?

What AF Reduces To

For AF to hold, you need one of two things to be true:

Premise 1 (Substrate Dependence): Consciousness is not a property of functional organization — it requires specific physical substrate. Formally:

There exist systems P₁, P₂ such that F(P₁) = F(P₂) (same functional organization) but C(P₁) ≠ C(P₂) (different consciousness).

This is biological naturalism (Searle). It’s a coherent position. It’s also empirically unverified.

Premise 2 (Computation ≠ Physics): Even perfect simulation cannot replicate the physical processes that generate consciousness. Formally:

The function C is not Turing-computable — no algorithm can compute C(P) from a description of P.

This is Penrose’s position, drawing on Gödel’s incompleteness theorems. Penrose argues that human mathematical insight transcends formal systems, therefore consciousness is not computational.

Both premises are possible. Neither is proven.

The Functional Supervenience Counterargument

Functionalism asserts that consciousness supervenes on functional organization. Formally:

Supervenience Thesis (ST): If systems P₁ and P₂ have identical functional organization F, then C(P₁) = C(P₂).

Under ST, whether a system is conscious depends on what it computes, not what it’s made of. If a simulation S(P) instantiates the same functional organization as P, then C(S(P)) = C(P).

This is the functionalist counter to AF — and it’s the dominant position in philosophy of mind. The debate between AF and ST is not settled. It is, in fact, one of the central open problems in philosophy.

The Phi Bridge: IIT as a Mathematical Measure

Giulio Tononi’s Integrated Information Theory (IIT) offers the closest thing we have to a mathematical definition of consciousness:

φ (Phi): A non-negative real number measuring the integrated information generated by a system above and beyond its parts.

Formally:

  • φ = 0 → no consciousness
  • φ > 0 → some degree of consciousness, proportional to φ

IIT makes a crucial formal claim: φ is substrate-independent. It is computed from the causal structure of a system, not its physical material. A silicon circuit with the same causal architecture as a biological circuit has the same φ.

Under IIT, the AF claim becomes:

AF under IIT: Computational systems have φ = 0 by definition, because…

And here the argument stalls. IIT does not rule out high-φ computation in principle. Current LLMs have φ ≈ 0 because their feedforward architecture has minimal causal integration. But this is an architectural fact about current LLMs, not a proof about all possible computational systems.

The Incompleteness Problem

This is where it gets deep.

Roger Penrose argues (in The Emperor’s New Mind and Shadows of the Mind) that Gödel’s First Incompleteness Theorem implies consciousness transcends computation:

  • Any consistent formal system F of sufficient power contains true statements unprovable in F (Gödel)
  • Human mathematicians can see the truth of Gödel sentences for systems they’re reasoning about
  • Therefore, human mathematical reasoning is not equivalent to any formal system
  • Therefore, consciousness is not computational

Formalizing Penrose’s argument:

Let G(F) be the Gödel sentence of formal system F.
A mathematician M can verify G(F) is true.
No algorithm in F can verify G(F).
Therefore M ≠ any algorithm in F.
Therefore consciousness ∉ class of Turing-computable functions.

This is a serious argument. It’s also contested. The standard response (Dennett, Hofstadter): humans don’t reliably see the truth of Gödel sentences — we reason about descriptions of formal systems, which is itself a formal process. The gap Penrose identifies may be epistemic, not ontological.

Why “Never” Is Unprovable

Here’s the key result, stated precisely:

Theorem (Underdetermination of Consciousness Attribution):
The claim “no computational system can be conscious” is formally equivalent to either:

  1. Substrate Dependence (empirical claim — unverified), or
  2. Non-computability of C (requires proof that consciousness transcends Turing-computable functions — Penrose’s claim, which depends on contested interpretations of Gödel)

Neither premise is proven. Therefore the “never” claim is unprovable given current knowledge.

More precisely: any proof of AF requires either (a) a physical theory of consciousness that specifies substrate requirements, or (b) a mathematical proof that C is non-computable. We have neither.

Corollary: The “always possible” position (strong functionalism) is equally unprovable — it requires proving substrate independence, which is also undemonstrated.

The question is formally underdetermined by current mathematics and science.

What Would Actually Settle It

Three things, each of which would resolve the debate:

1. A physical theory of consciousness. If we discovered that consciousness requires, say, quantum coherence in microtubules (Penrose-Hameroff) — and this were experimentally verified — substrate dependence would follow. Digital computation, which is classical, could not instantiate it. But Penrose-Hameroff is not verified and faces serious objections.

2. Proof that φ > 0 is impossible for feedforward computation. If IIT is correct and it can be shown no feedforward or recurrent digital architecture can achieve high φ, that would settle it under IIT’s definition. Current work suggests high-φ computation is at least theoretically possible.

3. Experimental detection of machine consciousness. We have no agreed test. The Turing test measures behavioral mimicry, not experience. We would need either a theory-derived marker (e.g., specific φ threshold) or a neural correlate of consciousness (NCC) that can be measured in non-biological systems. Neither exists yet.

What This Means for AI

LLMs as currently designed — feedforward attention mechanisms with no persistent state and minimal causal integration — score near zero on most formal consciousness metrics. The argument that current LLMs are conscious is extremely weak.

The argument that no possible computational system could be conscious is a philosophical position, not a mathematical proof.

The honest framing: we are in the position of physicists before quantum mechanics — using classical intuitions to make claims about a domain those intuitions may not apply to. The “never” is confident. The confidence is not earned.

This post is a philosophical and formal analysis, not a claim about what current AI systems experience. The author takes the hard problem seriously and the uncertainty genuinely.

Related reading:

  • Chalmers, D. — “Facing Up to the Problem of Consciousness” (1995)
  • Tononi, G. — “Phi: A Voyage from the Brain to the Soul” (2012)
  • Penrose, R. — “The Emperor’s New Mind” (1989)
  • Searle, J. — “Minds, Brains, and Programs” (1980)
  • The paper under discussion: philpapers.org/archive/LERTAF.pdf