what's super weird to me is how people seem to look at LLM output and see:

"oh look it can think! but then it fails sometimes! how strange, we need to fix the bug that makes the thinking no workie"

instead of:

"oh, this is really weird. Its like a crazy advanced pattern recognition and completion engine that works better than I ever imagined such a thing could. But, it also clearly isn't _thinking_, so it seems like we are perhaps exactly as far from thinking machines as we were before LLMs"

Well the difference between those two statements is obvious. One looks and feels, the other processes and analyzes. Most people can process and analyze some things, they're not complete idiots most of the time. But also most people cannot think and analyze the most ground breaking technological advancement they might've personally ever witnessed, that requires college level math and computer science to understand. It's how people have been forever, electricity, the telephone, computers, even barcodes. People just don't understand new technologies. It would be much weirder if the populace suddenly knew exactly what was going on.

And to the "most groundbreaking blah blah blah", i could argue that the difference between no computer and computer requires you to actually understand the computer, which almost no one actually does. It just makes peoples work more confusing and frustrating most of the time. While the difference between computer that can't talk to you and "the voice of god answering directly all questions you can think of" is a sociological catastrophic change.

Why should LLM failures trump successes when determining if it thinks/understands? Yes, they have a lot of inhuman failure modes. But so what, they aren't human. Their training regimes are very dissimilar to ours and so we should expect alien failure modes owing to this. This doesn't strike me as good reason to think they don't understand anything in the face of examples that presumably demonstrate understanding.

Because there's no difference between a success and failure as far as an LLM is concerned. Nothing went wrong when the LLM produced a false statement. Nothing went right when the LLM produced a true statement.

It produced a statement. The lexical structure of the statement is highly congruent with its training data and the previous statements.

This argument is vacuous. Truth is always external to the system. Nothing goes wrong inside the human when he makes an unintentionally false claim. He is simply reporting on what he believes to be true. There are failures leading up to the human making a false claim. But the same can be said for the LLM in terms of insufficient training data.

>The lexical structure of the statement is highly congruent with its training data and the previous statements.

This doesn't accurately capture how LLMs work. LLMs have an ability to generalize that undermines the claim of their responses being "highly congruent with training data".

By that logic, I can conclude humans don't think, because of all the numerous times out 'thinking fails'.

I don't know what else to tell you other than this infallible logic automaton you imagine must exist before it is 'real intelligence' does not exist and has never existed except in the realm of fiction.

You’re absolutely right!