Preface
On December 14, 2023, OpenAI released the paper Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, accompanied by a blog post on their website, Weak-to-strong generalization. This post offers a brief analysis of that work.
The paper asks: AI is growing more capable by the day, approaching and in places already surpassing human ability, and supervising such superhuman AI is becoming very hard for us. So how should we supervise an AI that is stronger than we are?
The research team studies the question through a simple demo and finds that a GPT-2-level model can elicit most of GPT-4’s capability, bringing it close to GPT-3.5 β generalizing correctly even on hard problems the small model itself gets wrong.
This may well be a new research direction.
The problem
The authors believe superintelligent AI could arrive within the next decade. Yet our understanding of AI systems today is so shallow that we have no idea how to steer and control superintelligence reliably, which puts humanity at risk.
Imagine using a powerful AI to write hundreds of thousands of lines of code and deploying it to production: if dangerous code is buried somewhere in there, the consequences could be severe.
In this scenario the human is the weak supervisor and the AI is the superhuman system. The core question is how a not-so-smart weak supervisor can supervise a smart superhuman AI.
The method
The authors’ proposal is to have a weak AI supervise a strong one.
As the figure shows, the leftmost panel depicts conventional machine learning, where a human supervises a weak AI; the middle panel shows the hard situation humans now face, supervising an AI more capable than themselves; the right panel shows the authors’ experimental setup, where a weak AI supervises a comparatively strong one.
On one hand, intuition says a powerful AI supervised by a weak one should never end up better than its supervisor, since the weak AI may mislead it. On the other hand, a strong AI already carries formidable latent capability from pretraining, and all the weak AI has to do is draw that knowledge out.
So which is it? Will the strong AI be elicited and realize its full potential, or will it be misled by the incomplete or flawed labels the weak AI provides and end up performing worse?
Note that “strong AI” above refers to a model that has been pretrained but not fine-tuned for the task.
Results and conclusions
As the figure shows, GPT-4 (the strong AI, un-fine-tuned) supervised by GPT-2 (the weak AI) reaches roughly GPT-3.5-level performance. With nothing but weak supervision, the authors elicit most of GPT-4’s capability.
Simple as the experiment is, it shows the approach is viable in practice, at least as a first step. The authors also see encouraging signs elsewhere: bootstrapping and optimal early stopping both look promising.
- Simple human supervision such as RLHF may, absent further work, be hard to scale to superhuman AI
- Weak-to-strong generalization is feasible
The study is both original and thought-provoking. People are forever thinking about how to make AI more capable, but few consider how to supervise a superhuman AI once it surpasses us. As the paper puts it, “This opens up a new research direction that allows us to directly tackle a central challenge of aligning future superhuman models while making iterative empirical progress today.” This is a new field, with plenty of low-hanging fruit waiting to be picked.
