The issue is that the values of the parameters don’t correspond to traditional variables. Concepts in AI are not represented with discrete variables and quantities. A concept may be represented in a distributed way across thousands or millions of neurons. You can look at each individual neuron and say, oh, this neuron’s weight is 0.7142, and this neuron’s weight is 0.2193, etc., across all the billions of neurons in your model, but you’re not going to be able to connect a concept from the output back to the behavior of those individual parameters because they only work in aggregate.
You can only know that an AI system knows a concept based on its behavior and output, not from individual neurons. And AI systems are quite like humans in that regard. If your professor wants to know if you understand calculus, or if the DMV wants to know if you can safely drive a car, they give you a test: can you perform the desired output behavior (a correct answer, a safe drive) when prompted? Understanding how an idea is represented across billions of parameters in an AI system is no more feasible than your professor trying to confirm you understand calculus by scanning your brain to find the exact neuronal connections that represent that knowledge.
The issue is that the values of the parameters don’t correspond to traditional variables. Concepts in AI are not represented with discrete variables and quantities. A concept may be represented in a distributed way across thousands or millions of neurons. You can look at each individual neuron and say, oh, this neuron’s weight is 0.7142, and this neuron’s weight is 0.2193, etc., across all the billions of neurons in your model, but you’re not going to be able to connect a concept from the output back to the behavior of those individual parameters because they only work in aggregate.
You can only know that an AI system knows a concept based on its behavior and output, not from individual neurons. And AI systems are quite like humans in that regard. If your professor wants to know if you understand calculus, or if the DMV wants to know if you can safely drive a car, they give you a test: can you perform the desired output behavior (a correct answer, a safe drive) when prompted? Understanding how an idea is represented across billions of parameters in an AI system is no more feasible than your professor trying to confirm you understand calculus by scanning your brain to find the exact neuronal connections that represent that knowledge.