ChatGPT Confesses... Reveals Deep Learning Deep Secrets...

[reACTIONary]

I've been exploring some of the more technical aspects of large language generative AI models - Chat Bots - and ran across an interesting research paper. Or maybe I should say a puzzling research paper.

Alignment Faking in Large Language Models

In order to "inspect the reasoning of the model" the researchers set up what they call a "chain-of-thought scratchpad" and told the AI to "analyze its situation and decide how to respond to the user." What is puzzling about this is that LLM Chat Bots don't reason, let alone employ a chain-of-reasoning. AI models built on symbol manipulation, such as those using LISP list processing or Prolog logic programing might be said - metaphorically - to reason, but not an LLM. An LLM is basically (to quote one DUer) a "stochastic parrot." So what these researchers were doing was asking a stochastic parrot to stochastically parrot about stochastically parroting. Sort of absurd and, maybe, a bit amusing.

But I had a thought - I can do this at home, using an on-line Chat Bot. I fired up ChatGPT and did a little prompt engineering. The result was interesting, and you may find it interesting also. The full "conversation" is at the link below. ChatGPT was kind enough to offer to summarize the interaction to make it easier to share, but that was a bit too meta for me.

If you are into this sort of thing, enjoy: https://chatgpt.com/share/69a4e489-0690-8011-8a65-aee52302268b

[ret5hd]

1. the first link doesn't work for me...

would you please check.

[ret5hd]

2. and now it's fine.

[reACTIONary]

3. Oh good! I was about to turn in....

.... so I glad I didn't have to do any forensic IT research!

[Layzeebeaver]

7. It doesn't work for me

Edit - and now it does…

[highplainsdem]

4. Instead of the link for the PDF copy of that paper you mentioned, here's the link for the HTML version:

Alignment Faking in Large Language Models
https://arxiv.org/html/2412.14093v2

The paper is from December 2024.

A new paper shed a lot of light on LLM "reasoning" - or lack of same - and I posted this thread about it last month:

A very pro-AI account on both Bluesky and X posted about a "disturbing" Stanford paper on LLMs' failures at reasoning
https://www.democraticunderground.com/100221009224

See the replies there as well, especially reply 28, linking to Gary Marcus's post on Substack the next day about that new study.

Direct link to what Gary wrote:

BREAKING: LLM “reasoning” continues to be deeply flawed
https://garymarcus.substack.com/p/breaking-llm-reasoning-continues

Link for that new paper, published less than a month ago:

Large Language Model Reasoning Failures
https://arxiv.org/abs/2602.06176

And links on that page will let you choose the PDF or HTML version.

Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks. Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios. To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in LLMs. We introduce a novel categorization framework that distinguishes reasoning into embodied and non-embodied types, with the latter further subdivided into informal (intuitive) and formal (logical) reasoning. In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to LLM architectures that broadly affect downstream tasks; application-specific limitations that manifest in particular domains; and robustness issues characterized by inconsistent performance across minor variations. For each reasoning failure, we provide a clear definition, analyze existing studies, explore root causes, and present mitigation strategies. By unifying fragmented research efforts, our survey provides a structured perspective on systemic weaknesses in LLM reasoning, offering valuable insights and guiding future research towards building stronger, more reliable, and robust reasoning capabilities. We additionally release a comprehensive collection of research works on LLM reasoning failures, as a GitHub repository at this https URL, to provide an easy entry point to this area.

[reACTIONary]

9. Thanks! I'm going to start working my way through your links!

[hunter]

5. "Stochastic parrot" is an insult to parrots.

Parrots, ravens, etc. are experimenting with their social environment when they speak as humans do. Some actually acquire human language skills beyond mere mimicry.

People are not so good at understanding bird languages.

Once again, in this paper and in others, the language we are using to describe these computer systems is getting in the way of our understanding them.

The mind of a parrot is very similar to the mind of a human. We are shaped by the same evolutionary processes and we experience very similar realities. As humans we can even imagine ourselves flying like a bird, being like a bird, to further understand what birds might be thinking.

There's none of that in a computer system -- no mind, no training, no socialization, no imagination, no motivations, not even any language as humans, birds, and all sorts of animals experience it.

We don't yet know enough yet about the mechanisms of animal minds to duplicate them in electronics.

AI promoters are fooling themselves or else they are misrepresenting what AI actually is so they can sell it to a gullible audience.

When the magician on the stage claims he is reading your mind he is not reading your mind. It would be foolish to hire him as an actual mind reader.

[LearnedHand]

6. Damn, sycophantic little punk, isn't it

Seriously though this is a brilliant conversation and I like how you think. I was most especially struck by the “x is not y but z” part of the conversation. This part of the LLM’s answer seemed very enlightening: “From a modeling standpoint, it functions as a high-probability discourse template for contrastive reframing.”

And that led me to realize that even this highly sophisticated conversation and model “reasoning” is the product of the scientific and technical works it ingested.

“Corpora”? Really? I almost fell a little bit in love.

[reACTIONary]

8. Yes, very sycophantic. Every question was "incisive", "insightful", "subtle".....

.... I should have asked it to cut that crap out.