Title: ReAct: Synergizing Reasoning and Acting in Language Models
Institute: Department of Computer Science, Princeton University
Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao
Link: https://arxiv.org/pdf/2210.03629.pdf
Shared Link: https://mp.weixin.qq.com/s/5DFnutIpNIPRyJ7EUlfiEw
Motivation
CoT is black-box reasoning: both the reasoning process and the knowledge it draws on stay inside the model
Earlier work has explored using pretrained language models to plan and act in interactive environments
Inspiration
Between two actions, humans track their progress in language — “all the ingredients are chopped now, so I should boil the water,” for instance. Drawing on this habit, the authors set out to make an LLM’s reasoning process explicit.
ReAct
ReAct is a framework that lets an LLM combine thought and action.
At time $t$, let the observed information be $o_t\in\mathcal{O}$ and the action be $a_t\in\mathcal{A}$, where $a_t$ follows the policy $\pi(a_t|c_t)$ with $c_t=\left(o_1,a_1,\cdots,o_{t-1},a_{t-1},o_t\right)$. Mapping $c_t\mapsto a_t$ directly, however, is implicit and computationally expensive.
ReAct’s approach is simple: augment the set $\mathcal{A}$ into $\hat{\mathcal{A}}=\mathcal{A}\cup\mathcal{L}$, where $\mathcal{L}$ is the space of thoughts, or reasoning steps. Now $\hat a_t\in\mathcal{L}$ is the action obtained from the context $c_t$, and in the next round $c_{t+1}=\left(c_{t},\hat{a}_{t}\right)$ serves as the context for generating the next action.
Every ReAct cycle consists of acting, thinking, and observing the environment.
The approach does come with a drawback: model capacity has to be large enough, and the paper uses a 540B model.
The authors split tasks into two categories, reasoning-dominant and decision-dominant.
Reasoning-dominant: for tasks where reasoning carries most of the weight, such as multi-step reasoning, the authors use a thought-action-observation loop.
Decision-dominant: for decision tasks involving many operations, by contrast, thoughts only need to surface occasionally — you are in a kitchen, the environment is such-and-such, and you need to find such-and-such.
