There is a fundamental flaw in the idea that consciousness can be defined "functionally" (i.e. "it is what it does").
Consciousness is a purely internal state - we infer it in others through communication, but it is fundamentally something someone experiences about themself, not something defined by how other people perceive them.
Because it is an internally defined property, it *cannot* be defined on the basis of "what it does". It must be defined, at least in part, on the basis of "what it knows".
To take a simple example from robotics:
I have a robot with one gripper and two sensors that moves through its environment. One sensor (A) tells the robot whether its gripper is full or empty. The other sensor (B) tells the robot whether or not it is in the presence of something its gripper can pick up (an object, P).
The robot has three rules.
1. If no P, move in a random direction.
2. If gripper is empty and P sensed, pick up object and then move in a random direction.
3. If gripper is full and P sensed, drop object and then move in a random direction.
This robot has no concept of anything in its world except P/not P, gripper full/empty, and "move in a random direction".
Yet, externally, the effect of these rules is that the robot builds piles of objects [1].
Attributing consciousness to an LLM is the equivalent of saying our robot *intends* to build piles of objects.
1/3
Triggering article: https://www.slowboring.com/p/nobody-knows-what-theyre-talking





