Isn't this basically just what my comment about the edge of the knowable was and you snarkily replied with the Turing Test?
Like go watch one of the videos I linked if you haven't. I think they'd be really interesting to you, especially the first one.
I agree with you tho. What are we looking for is the question to ask. By that same notion, I can say with certainty for myself that what we have doesn't reason, but I can't elaborate on what it might take to make up something that does. Just as with obscenity in that famous SC case.
To elaborate on the Othello point:
They tested the LLM with a probe and changed a board piece. They used this change and probed the resultant probability distribution to determine whether or not the AI would change its probability distribution to 'prove' that it was creating world representations of the board. The problem is, and this is what makes it kinda fallacious thinking by the study authors, that if you change the input data of course the output data is going to change. That's just a result of training the AI on different legal boardstates, as the way that moves that are made will have a direct result on the placement of the pieces.
Furthermore, they showed that it outperformed random chance at predicting legal moves, but that's just the way that training AI works. An LLM is better at predicting the next word than random chance as a result of its training.