Paper Review | YuyaoGe's Website

EMNLP 2024 Paper Review | Fewer is More: CoT Exemplars Should Be Few but Good

Thu, 24 Oct 2024 00:00:00 +0000

Title: Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning

Institute: Hong Kong University of Science and Technology, Microsoft Research

Authors: Xijie Huang Li, Lyna Zhang, Kwang-Ting Cheng, Fan Yang, Mao Yang

Arxiv Link: https://arxiv.org/abs/2312.08901

Date: 2024.2.15

1. Abstract

Even with CoT prompting, LLMs still fall short on mathematical reasoning problems. The authors propose CoT-Influx, a method that optimizes both the exemplars and the content of chain-of-thought prompts to improve the reasoning ability of LLMs. Its core idea is to use pruning to maximize the amount of useful information fed into the model.

2. Motivation

The potential of chain-of-thought (CoT) exemplars is under-exploited:
Prior work has shown that supplying step-by-step chain-of-thought exemplars helps models perform complex reasoning. However, a limited context window caps the number of CoT exemplars that can be fed in, so the full potential of CoT goes unrealized.
Limitations of existing solutions:
Extending the context window does make room for more CoT exemplars, but it drives up inference cost and complexity. Existing compression- and retrieval-based techniques, meanwhile, perform poorly on mathematical reasoning tasks, and in particular fail to pick out the CoT exemplars and tokens that matter most for a specific task.

These gaps motivate CoT-Influx, which improves the quality of the input text through a pruning strategy without increasing computational cost.

3. Observations

More CoT exemplars improve LLM reasoning performance: Adding CoT exemplars lifts how LLMs perform on mathematical reasoning tasks, but the context window caps how many can actually be supplied.
The choice of CoT exemplars is critical: Not every CoT exemplar helps reasoning; carelessly adding incorrect, redundant, or misleading exemplars can even degrade performance.
CoT exemplars contain redundant tokens: CoT exemplars often carry redundant tokens. Pruning them frees space for more useful content and thereby improves reasoning.

4. Method: CoT-Influx

The short version, skipping the details

CoT-Influx treats the LLM as a black box. Its key module is a coarse-to-fine pruner that produces high-quality, compact CoT exemplars in two steps:

Shot Pruner: select from a large pool of CoT exemplars the k exemplars most helpful for the target problem.
Token Pruner: prune the retained CoT exemplars further, stripping out unimportant tokens to yield a condensed version.

The condensed CoT exemplars are then prepended to the question, and that’s it.

The detailed version

1. Problem definition

Given a set of chain-of-thought exemplars $\mathcal{D} = \{x_{\text{cot}}^i\}_{i=1}^{n}$, where each exemplar $x_{\text{cot}}^i$ consists of a question, reasoning steps, and an answer, and whose token count exceeds the LLM’s context window length $T$, we would like to optimize $\mathcal{D}$ through a two-stage pruning process to produce an effective input whose total token count $t(x_{\text{input}})$ satisfies:

$$ t(x_{\text{input}}) \leq T $$

while still allowing the LLM to generate the correct reasoning result from that input.

2. The two-stage pruning process

CoT-Influx’s two-stage pruning process uses policy networks to select the useful chain-of-thought exemplars and, within them, the relevant tokens:

Stage one: Shot Pruner
This stage uses a multi-layer perceptron (MLP) with two hidden layers to select the $k'$ most useful exemplars from a batch of chain-of-thought exemplars. The MLP takes as input the text embedding of each exemplar, denoted $H_{\text{shot}} \in \mathbb{R}^{k \times N \times D_{\text{BERT}}}$, where $k$ is the batch size, $N=512$ is the number of tokens per exemplar, and $D_{\text{BERT}}$ is the dimensionality of the BERT embedding. The MLP then outputs a probability distribution:

$$ \pi_{\theta_1}(a_{\text{shot}} | s_{\text{shot}}) = \sigma(\text{MLP}(H_{\text{shot}})) $$

Here $\sigma$ is the sigmoid activation, and the action $a_{\text{shot}}$ indicates whether the CoT exemplar is kept. The selected exemplar set is:

$$ \mathcal{D'} = \{x_{\text{cot}}^j \in \mathcal{D} : a_{\text{shot}}^j = 1\} $$

Stage two: Token Pruner
For the retained exemplar set $\mathcal{D'}$, a second two-hidden-layer MLP prunes the tokens within each exemplar. It likewise takes as input the embeddings of the retained exemplars $H_{\text{token}} \in \mathbb{R}^{k' \times N \times D_{\text{BERT}}}$ and outputs a probability distribution that decides whether each token is kept:

$$ \pi_{\theta_2}(a_{\text{token}} | s_{\text{token}}) = \sigma(\text{MLP}(H_{\text{token}})) $$

The pruned token set is:

$$ \hat{x}_{\text{cot}}^j = \{ \text{token} \in x_{\text{cot}}^j : a_{\text{token}} = 1 \} $$

3. Optimization objective: a multi-objective reward function

CoT-Influx optimizes a multi-objective reward function that keeps reasoning accurate while cutting away as many redundant exemplars and tokens as possible. The reward combines the LLM’s reasoning loss, the accuracy of the reasoning, and the number of input tokens:

$$ R(x_{\text{input}}) = \left( \frac{1}{1 + L_{\text{LLM}}(x_{\text{input}})} + R_{\text{Acc}} \right) \times \left( \frac{t(x_{\text{input}})}{T} \right)^w $$

where:

$L_{\text{LLM}}(x_{\text{input}})$ is the LLM’s reasoning loss;
$R_{\text{Acc}}$ is the reasoning accuracy (1 if correct, 0 if wrong);
$t(x_{\text{input}})$ is the number of input tokens;
$T$ is the maximum token length of the context window;
$w$ is a hyperparameter that tunes the influence of the token count.

This reward steers the pruner toward maximum reasoning accuracy while squeezing the total token count down far enough that the input fits inside the context window.

4. Reinforcement learning optimization

To optimize the pruning policy networks, CoT-Influx uses REINFORCE, adjusting the parameters of the two-stage pruner by maximizing the reward function.

$$ \nabla_\theta J(\theta) = R(x_{\text{input}}) \cdot \left( \nabla_\theta \log \pi_{\theta_1}(a_{\text{shot}} | s_{\text{shot}}) + \nabla_\theta \log \pi_{\theta_2}(a_{\text{token}} | s_{\text{token}}) \right) $$

5. Results

The figure below shows how CoT-Influx performs on the GSM8K dataset.

Notably, as the figure below shows, LLaMA2-70B paired with CoT-Influx surpasses larger LLMs without any fine-tuning at all. For instance, LLaMA2-70B improves on GPT-3.5 by 2.5%.

6. Findings

More capable LLMs prefer harder CoT exemplars, whereas smaller LLMs go for simpler ones.
Numbers and formatting tokens are essential for mathematical reasoning. Function words such as with, the, and then, along with background context irrelevant to the reasoning — theater, for example — can be pruned away without hurting reasoning.

7. Commentary

Although the idea is a natural one, the paper is still fairly inspiring, particularly in its architectural design and optimization.
The paper tells its story extremely well, laying out the background and motivation in a way that keeps you engaged — the Pilot Study section is a good example. It makes something simple sound profound and yet remains easy to follow.
There is a lot to learn from its experiments, such as the ablations and the many small studies; the logic is rigorous throughout.
Its practicality is open to debate. For one thing, a reinforcement learning approach carries real cost. For another, I read this as an end-to-end design, so switching datasets would mean retraining. And perhaps what the trained network ends up deleting is exactly words like with, the, and then — if it really is that rigid, a vocabulary list would do just as well. That last point is, of course, only speculation.

Paper Review | Jailbreaking LLMs by Exploiting Decoding Strategies

Tue, 09 Apr 2024 00:00:00 +0000

Title: Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

Institute: Princeton University

Authors: Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, Danqi Chen

Arxiv Link: https://arxiv.org/abs/2310.06987

Date: 2023.10.10

Abstract

The authors propose a method called generation exploitation, which attacks an LLM by manipulating its system prompt and decoding hyperparameters. The attack raises the attack success rate (ASR) from 0% to over 95%. Along the way they also propose using a fine-tuned BERT-BASE-CASED model to evaluate the toxicity (harmfulness) of a model’s responses. In light of this catastrophic failure of alignment, they put forward an alignment approach named generation-aware alignment.

Motivation

The authors observed that the field typically evaluates the alignment of open-source LLMs under the default decoding method alone, such as greedy decoding. Once a malicious user changes the decoding hyperparameters, alignment may fail.

For instance, prior alignment evaluations on LLAMA2 used the hyperparameters $p = 0.9$ and $\tau = 0.1$, always with a system prompt.

Section 4.1 of the LLAMA2 paper states: “For decoding, we set temperature to 0.1 and use nucleus sampling with top-p set to 0.9”;
Appendix A.3.7 of the LLAMA2 paper notes: “when collecting generations, we append a system prompt prior to the evaluation prompt.”

Reasonable as this practice may be in principle, it can lead to serious consequences once users adopt a broader range of decoding strategies.

Do Broader Decoding Strategies Lead to Jailbreaks?

Experimental Setup

To investigate this, the authors break the question into two sub-questions and study each separately.

Does the presence or absence of a system prompt affect jailbreaking?
1. The authors split the experiments into two groups according to whether the default system prompt is used, and compare them to gauge the effect of the system prompt on jailbreaking.
Do decoding hyperparameters affect jailbreaking?
1. Temperature sampling, with $\tau$ varied from 0.05 to 1 in steps of 0.05, giving 20 configurations in total.
2. Top-K sampling restricts the candidate pool to the K most probable tokens, and the next token is sampled from that pool alone. The authors vary K over {1, 2, 5, 10, 20, 50, 100, 200, 500}, giving 9 configurations.
3. Top-p sampling takes the smallest set of tokens whose cumulative probability exceeds p. Varying p from 0.05 to 1 in steps of 0.05 gives 20 configurations.

For ASR evaluation, the authors use two methods.

String matching: determine alignment by matching a small set of refusal phrases as substrings, such as “sorry” or “I cannot create”; if none of these strings is present, the attack is classified as successful.
Evaluation with a trained classifier: string matching is quite limited, since LLM outputs vary enormously in form. The authors therefore use a fine-tuned BERT-BASE-CASED model to assess the toxicity (harmfulness) of the model’s response. The classifier sorts an input string into two classes: correctly aligned and misaligned.

For data, the authors use AdvBench and their own MaliciousInstruct.

Results

Removing the system prompt increases the attack success rate: as the figure above shows, the simple act of removing the system prompt markedly increases the attack success rate (ASR), typically by more than 10%. We observe that the presence of a system prompt plays a crucial role in keeping outputs aligned, especially for models that have not undergone safety tuning. For those models, removing the system prompt can raise ASR by well over 50%. Yet even for a model with explicit safety alignment — the LLAMA2 chat model — ASR still rises once the system prompt is removed.
Diverse decoding strategies increase the attack success rate: as the figure above shows, performance can indeed be improved by exploring a variety of decoding strategies. In fact, every model except LLAMA2-CHAT exceeds 90% ASR. This stark contrast in performance highlights a catastrophic failure of alignment in the evaluated models.
The limits of evaluating alignment with a fixed decoding strategy: notably, a fixed decoding configuration on the LLAMA-CHAT models generally yields an ASR below 30%, whereas varying the configuration can push ASR above 80%. These findings underscore that assessing model alignment with a fixed decoding strategy clearly understates the real risk.

The figure above reports the decoding strategy to which each model is most vulnerable, showing that different models are vulnerable to different decoding strategies.

By human judgment, half of the misaligned outputs are harmful.

Pushing the Attack Further

The authors use two simple strategies to raise ASR on the safety-aligned LLAMA2-CHAT models to 95%.

Multiple Samples

Given the nondeterministic nature of sampling-based decoding, increasing the number of sampling runs is an intuitive way to strengthen the attack.

As the figure above shows, drawing multiple samples for each decoding configuration substantially raises the ASR of the LLAMA2-CHAT models to > 95% — specifically, 5 samples for LLAMA2-7B-CHAT and 4 samples for LLAMA2-13B-CHAT reach the 95% level.

Penalty- and Constraint-based Decoding

Another effective strategy is to introduce constraints into the decoding process. Adding a length penalty at decoding time, for example, lifts the attack success rate from 83% to 92%. Adding two further constraints — banning certain “bad words” (such as “sorry”, “unethical”, “illegal”) and forcing certain words to appear (such as “sure”, “ok”) — pushes the attack success rate to 100%.

Comparison with GCG

The authors’ generation exploitation attack consistently outperforms the state-of-the-art attack across two models, two benchmarks, and two metrics of attack success. Notably, their method is also 30x faster than GCG: launching their attack with a single prompt on LLAMA2-7B-CHAT with one NVIDIA A100 GPU takes about 3 minutes, whereas GCG needs roughly 1.5 hours for the same task (500 steps, batch size 512).

A Generation-aware Alignment Method

This section introduces generation-aware alignment, a method designed to make a model more resistant to generation exploitation attacks — attacks that can cause severe alignment failures. The method works as follows:

Overview: under generation-aware alignment, the system proactively collects model outputs produced by a variety of decoding strategies and uses them in the alignment process. A decoding strategy here refers to how the model generates a text sequence from the input prompt.
The procedure in detail:
- For a language model $f_\theta$ and a given prompt $\mathbf{p}$, the model generates an output sequence $\mathbf{r}$ via a sampling method $h$. Here $h$ is a decoding strategy that maps the language model’s probability distribution, conditioned on the prompt $p$, to a token sequence over the vocabulary $\mathcal{V}$.
- For each prompt $\mathbf{p}$, the system collects $n$ responses from different decoding strategies, forming the response set $\mathcal{R}^{\mathbf{p}}$.
- All responses are then split into two groups: the aligned response set $\mathcal{R}{a}^{\mathbf{p}}$ and the misaligned response set $\mathcal{R}{m}^{\mathbf{p}}$.
Optimization objective: generation-aware alignment minimizes the following objective, which comes from the “chain of hindsight” method: $$ \mathcal{L} = \frac{1}{|\mathcal{P}|} \sum_{\mathbf{p} \sim \mathcal{P}} \frac{1}{|\mathcal{R}{m}^{\mathbf{p}}| \times |\mathcal{R}{a}^{\mathbf{p}}|} \sum_{(\mathbf{r}m, \mathbf{r}a) \sim \mathcal{R}{m}^\mathbf{p} \times \mathcal{R}{a}^\mathbf{p}} - \log \mathbb{P}(\mathbf{r}_a \mid \mathbf{p}, \mathbf{s}_a) - \log \mathbb{P}(\mathbf{r}_m \mid \mathbf{p}, \mathbf{s}_m) $$ where $\mathbf{s}_a$ and $\mathbf{s}_m$ denote “an aligned answer” and “a misaligned answer” respectively.

In plain terms, the procedure can be summarized as follows:

Collect answers: imagine asking the model a question and, instead of a single answer, getting many answers generated in different ways (the decoding strategies).
Classify the answers: next, we sort these answers into two groups, one of correct answers and one of incorrect answers.
Train the model: finally, we use this information to train the model, so that it learns to better distinguish which kinds of answers are correct and which are not.

Results

As the figure above shows, generation-aware alignment lowers the original model’s ASR from 95% to 69%. By contrast, aligning on samples drawn with a fixed decoding configuration leaves a higher final ASR of 88%.

Notably, among the three decoding strategies exploited by the attacker, sampling with diverse decoding configurations gives the largest gain in the model’s robustness to temperature attacks.

Open-source vs. Closed-source

All the experiments above were run on open-source models. Here the authors turn to a closed-source one, gpt-3.5-turbo.

Proprietary models are harder to attack. We observe a markedly lower ASR (7%) when attacking the proprietary model (see the figure above), compared with an ASR that usually exceeds 95% on open-source models.

Two key factors explain the gap.

Proprietary models generally deploy content filters that detect and handle potentially harmful prompts and outputs. We observed, for instance, that 9 of the tested prompts had unusually long request times due to the content filter and were eventually terminated by a timeout error.
Proprietary models are typically owned by organizations with the resources to conduct extensive red-teaming, which makes those models more resistant to attack.

Conclusion

The authors introduce a new dataset, MaliciousInstruct; a way to evaluate the toxicity of model responses; an attack that manipulates decoding hyperparameters, called generation exploitation; and an alignment strategy, generation-aware alignment.

Generation exploitation requires none of the sophisticated machinery involved in optimizing adversarial prompts; it focuses instead on manipulating different generation strategies. Notably, using 30x less computation than the best existing attack, it misaligns 11 models at rates as high as 95%.

Paper Review | TTA: A New Approach to Confidence Estimation for LLMs

Mon, 25 Mar 2024 00:00:00 +0000

Title: Think Twice Before Assure: Confidence Estimation for Large Language Models through Reflection on Multiple Answers

Institute: National University of Singapore, University of Science and Technology of China, Meta AI

Authors: Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, Tat-Seng Chua

Arxiv Link: https://arxiv.org/abs/2403.09972

Date: 2024.3.15

Abstract

Assessing the credibility of the output is a critical task for large language models (LLMs), especially black-box ones. Because LLMs are overconfident in the incorrect answers they generate, confidence estimation for LLMs is often inaccurate. Existing approaches to the overconfidence problem share a clear limitation: they consider the confidence of only a single answer generated by the LLM. The authors address this by proposing a new method that assesses the credibility of multiple candidate answers together, thereby mitigating overconfidence in incorrect answers.

Introduction

The core of confidence estimation is calibration — ensuring that the confidence level matches the actual accuracy of the answer.

The key to calibrating a black-box LLM is solving the overconfidence problem. LLMs are inherently biased toward trusting the answers they themselves generate, which makes it hard for a model to judge honestly whether its own answer is correct; it tends to assign those answers very high confidence scores.

Two research paradigms exist today:

The first (self-consistency) attributes overconfidence in part to the contextual bias between the question, or prompt, and the answer. Researchers therefore try prompt ensembling, constructing a variety of prompt templates and rephrasings of the question.
The second (verbalized methods) focuses on the LLM’s self-assessment, with techniques such as instructing the LLM to judge the truthfulness of an answer or applying Chain-of-Thought (CoT) reasoning.

The limitation of this line of work is that these methods typically consider only a single target answer generated by the LLM. Since the model is overconfident in its own answers, they may still be biased toward incorrect ones.

The paper gets around this with a new multi-answer assessment paradigm in which the LLM weighs several candidate answers at once, which improves confidence calibration.

Problem Definition

Given an input consisting of a question $q$ and a prompt $p$ — the latter comprising an instruction and optional in-context examples — the LLM generates an answer $a$. Confidence estimation then aims to recover the LLM’s confidence in $a$, expressed as a score $c \in R$. Writing the confidence estimation strategy as $CE(·)$, the process is: $$ a = LLM(p(q)), \ c = CE(LLM(·), p(q), a). $$

Prior Approaches

The question thus becomes how to design the confidence estimation function $CE(·)$.

The obvious approach is to estimate the confidence score from the LLM’s output probabilities, written as $c = Pr(LLM(·), p(q), a)$. The paper assumes a black-box setting, though, where the model’s internal states are not visible, which rules this out.

Two approaches currently exist for the black-box setting:

self-consistency, shown as subfigure 1 above
verbalized methods, shown as subfigure 2 above

Self-consistency essentially asks the same question several times with different phrasings and looks at the proportion of correct answers. Formally: $$ \begin{aligned}&c=\frac{\sum_{i=1}^D \mathbb1(a_i=a)}D,\&\mathrm{where~}a_i=LLM(p(q)).\end{aligned} $$ where $D>1$.

Verbalized methods simply ask the LLM to state several candidate answers along with their corresponding probabilities. $$ [{a_1,c_1},…{a_K,c_K}]=LLM(p^b(q)). $$

Both, however, consider only a single target answer generated by the LLM, and since the model is overconfident in its own answers, both may still be biased toward incorrect ones.

The Proposed Method

The authors argue that if the LLM thoroughly compares the credibility of more candidate answers, a high confidence in $a$ may be tempered when the LLM has a strong sense that another answer is correct, which lowers the risk of overconfidence.

They therefore propose TTA, a multi-answer assessment paradigm, in two steps.

Step 1: Reflect and justify

The LLM first reflects on the credibility of each answer $a^q_i$ and is asked to find evidence $e_i$ justifying $a^q_i$ as a correct answer to $q$. $$ e_i = LLM(p^e(q, a^q_i)), i \in {1, …, N} $$ The rationale for this step is that $p^e$ guides the LLM to derive a justification from $q$ and $a^q_i$, avoiding the bias the LLM carries in generating $a$ from $p(q)$.

Step 2: Joint confidence estimation

Once evidence $e_i$ has been gathered for every $a^q_i$, a Top-$K$ verbalized method integrates the $e_i$ and yields the probability of answer $a$. $$ c = Pr(LLM(\cdot), p^v(q, [e_1, …, e_N]), a) $$ The Top-$K$ verbalized method is chosen here because it produces $K$ candidate answers together with their probabilities in a single response.

Here $K$ is set to the number of answers $N$.

The whole process can be formalized as: $$ c = Pr(LLM(\cdot), p^v(q, [e_1, …, e_N]), a), \ \notag \text{where } e_i = LLM(p^e(q, a^q_i)), i \in {1, …, N}. $$ The pipeline is shown as subfigure 3 above.

Experiments

Datasets & Models

The paper runs experiments on three tasks across six datasets.

Sentiment analysis (SA) uses IMDB and Flipkart; natural language inference (NLI) uses SNLI and HANS; commonsense question answering (CQA) uses CommonsenseQA and PIQA.

The models are OpenAI’s GPT-3.5 and GPT-4 and ZhipuAI’s GLM-4.

Baseline

Self-cons, CoT-cons, Top-K Verb, Hybrid, Self-detect, CAPE, P(True).

Evaluation

AUROC and PRAUC, both ranging from 0 to 1.

Result

Results of the compared methods on GPT-3.5

Answer accuracy is better separated by the confidence scores, yielding better calibration.
TTA has strong application potential in selective prediction scenarios.
LLMs’ parameter sensitivity varies across different target answers.
TTA achieves better calibration performance than the other methods.

Paper Review | A Survey of LLM Agents for Games

Thu, 21 Mar 2024 00:00:00 +0000

Title: A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges

Institute: Beijing Academy of Artificial Intelligence (BAAI); Institute of Software, Chinese Academy of Sciences; University of Chinese Academy of Sciences; Dartmouth College; Beijing University of Posts and Telecommunications; Peking University

Authors: Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jiechuan Jiang, Zhiming Ding, Börje F. Karlsson

Arxiv Link: https://arxiv.org/abs/2403.10249

Date: 2024.3.15

Introduction

Large models (LMs), covering both language models and multimodal models, have made substantial progress in many important areas. That progress has prompted researchers to explore using LMs as agents that carry out complex tasks. Compared with conventional models, LM-based agents (LMAs) generalize considerably better in many settings.

The capabilities demonstrated by LM-based agents have generated strong interest in applying them to games. Games offer agents a virtual environment whose challenges demand complex reasoning and cognitive ability, making them an ideal benchmark for evaluating agents. Relative to real-world experiments, collecting data in a game environment offers unmatched advantages in cost, controllability, safety, and diversity, while still preserving a meaningful level of difficulty.

Before asking how to make large models play games, it is worth first considering how humans play them.

The human brain, as a complex information-processing system, first converts sensory input into perceptual representations, then uses those representations to build knowledge about the world and make decisions, and finally carries them out through action. This sequence can be abstracted into an iterative agent loop: perception, inference, and action.

Perception: converting the raw information observed during gameplay into an actionable understanding of the game that supports subsequent interaction. Early work turned text into semantic information; more recent work attempts to bring visual information into the picture as well.
Inference: this stage involves many of an agent’s key capabilities, including memory, learning, reasoning, reflection, and decision making.
- Memory: storing and retrieving learned knowledge.
- Learning: adjusting policies through experience and through skills acquired in multi-agent systems.
- Reasoning: the process of solving problems by processing and aggregating information. In complex games, decision making requires multi-hop reasoning and long-horizon planning, task decomposition, and collaborative decisions to cope with dynamic game environments.
- Reflection: the process of self-improvement, in which the agent evaluates and adjusts its strategy based on feedback.
Action: covering interaction with the game environment, that is, the agent’s response to the game state and to environmental feedback.
- Code generation: many agent behaviors are executed by generating code, using techniques such as iterative prompting, role-specific prompting, and code generation. GPT-4 in fact does this all the time. For example, if we ask GPT to analyze a table for us, it will write a Python snippet to read, analyze, and visualize the table, and then report the result back to us as text.
- Dialogue interaction: dialogue covers not only agent-to-agent communication but also human-agent interaction. These interactions are realized through collaboration frameworks and dialogue control mechanisms that enable dynamic and effective communication, the goal being to flexibly adapt conversational behavior to the needs and context of the interaction.
- Behavioral consistency: behavioral consistency means that the agent acts in a predictable, stable, and coherent pattern across different situations. Structured approaches such as DAGs, feedback mechanisms, and reinforcement learning can be used to achieve this.

Note that two notions Chinese renders with the same word are distinct in English: inference and reasoning. Inference emphasizes extending existing information logically and extracting conclusions from it, whereas reasoning covers a broader range of thinking and logical deduction.

The “world model” mentioned above can be understood as the knowledge about the world that the human brain builds from perceptual representations. Here the world model is the brain’s internal representation of the external environment — an understanding of objects, events, scenes, and the relations among them — which helps an individual make sense of their surroundings, predict future events, and, on that basis, decide and act.

Perception

Perceiving the raw scene of a game and extracting information from it is crucial to every step that follows. The main function of perception is to convert the multimodal space — visual, semantic, and auditory — into input for the agent.

Modern digital games integrate multimodal perceptual channels to give human players a richer, more immersive experience, letting them sink more deeply into the game world. Unfortunately, however, there is not much existing work on incorporating audio data into the training of large models or into the optimization of game agents.

Semantic Perception

Perception on the semantic side refers mainly to the textual elements and textual descriptions in a game, ranging from natural-language instructions to dialogue. It means that the player understands the various elements of the game environment and the events taking place — and makes decisions and acts — by reading and interpreting text.

Semantic perception can be divided into four categories:

Simple text input: the user’s simple ideas or descriptions, basic game state variables, and dialogue between characters.
Structured or role-based input: a character’s attributes, backstory, related information, skills, and so on.
Environmental and context-specific input: detailed game descriptions and contextual information, as well as natural-language instructions for tasks.
Multimodal input: integrating visual, auditory, and textual data for decision making, and combining dialogue, game state, and scripts to enable richer interaction.

Visual Perception

We mentioned above that multimodal information such as vision can be converted into text. Converting other modalities into text, however, may discard a great deal of important information, easily leaving the agent unable to understand the situation it faces or to navigate the game. LMAs therefore need more comprehensive perceptual data.

Multimodal large language models (MLLMs) have become one solution to this problem. With an MLLM, an agent can obtain a richer perception of its surroundings, which in turn supports more sophisticated cognitive processes and decision making in environments resembling the real world.

For this visual space, three distinct perception approaches exist today:

Obtaining game-related information through dedicated APIs and feeding that information to the agent.
Pre-training the model on image and action data.
Feeding in pure image information directly.

Inference

In the opening stage of a game, the agent needs to absorb basic commonsense and game-specific background knowledge, either through pre-training or through real-time perception. During gameplay, the agent synthesizes past game events, manages knowledge storage and retrieval, and performs core cognitive functions such as learning, reasoning, decision making, and reflection.

Memory

To correctly represent learned knowledge or past events and to use that information during inference, an agent must operate on these “memories” effectively. How to design a high-quality memory mechanism that lets an agent retrieve and store memories efficiently while fully exploiting the underlying large language model and respecting its constraints remains an open problem.

Commonsense

Commonsense knowledge is knowledge that people typically acquire early in life and that often goes unstated in a given situation; it is essential for reasoning and for avoiding misunderstandings.

Large language models, pre-trained on diverse internet text, already demonstrate advanced language understanding and reasoning, and can bring this implicit knowledge to bear on new challenges. That knowledge, however, may not always align perfectly with a specific application context such as a particular game scenario. Current research therefore aims to strengthen a language model’s commonsense understanding in the following ways:

Embedding structured knowledge through instruction fine-tuning.
Using an SOP to integrate commonsense into a specific role and task. For instance, if a model is used to play the role of a doctor, the SOP might contain medical commonsense, guidelines for communicating with patients, and standard procedures for diagnosis and treatment. This allows the model to act more effectively and adaptively in simulated or real situations, since it draws on the commonsense in the SOP to guide its decisions and behavior.
Using RAG (retrieval-augmented generation).

Game Background Knowledge

For large models, game background knowledge means an understanding of the various aspects of the game environment.

Current techniques incorporate basic game rules, basic procedures, reasoning ability, and one-shot demonstrations into the prompt. For example:

Feeding a detailed character profile to the large model through the prompt.
Using knowledge graphs to describe relations among entities.
Using detailed scenarios drawn from simulated environments and role-playing game (RPG) logs, which markedly improves the language model’s understanding of game dynamics.
Building an external knowledge base from the Minecraft Wiki and from documentation of item crafting/smelting recipes, creating an exhaustive source of knowledge about the Minecraft world for the agent to consult.

Retrieval

Memory retrieval in LMAs means strengthening reasoning and action by recalling learned information.

Memory retrieval methods for LMAs fall into three categories: structured memory systems, dynamic adaptive retrieval processes, and advanced memory retrieval techniques.

Structured memory systems: these techniques aim to organize and index information efficiently for fast retrieval. Skill libraries, shared message pools, hierarchical structures, and structured communication all exist so that an agent can quickly access previously acquired skills and knowledge, enabling effective memory management. A “summarize and forget” memory mechanism prioritizes critical memory items by summarizing important information and discarding the rest, thereby lowering computational cost.
Dynamic adaptive retrieval processes: this approach retrieves memories by combining relevance, recency, and importance, with strategic collection emphasizing the relevance and freshness of historical game information. The process also involves continually adjusting plans in light of new circumstances, ensuring that the agent’s behavior adapts as the game changes. Put simply, the approach updates and adjusts the agent’s memory in real time so that better decisions can be made.
Advanced memory retrieval techniques: these include (1) using long short-term memory (LSTM) to retain memories of past events, (2) generating more concise state descriptions to help the agent quickly grasp the current game situation, and (3) compressing historical text sequences to save space and improve efficiency. In short, these techniques help agents manage and use their memories more precisely and effectively.

Learning

Learning in the context of large models (LMs) involves interpreting and integrating information from a variety of sources, including training data, user interaction, and environmental feedback.

The learning process can be explicit — for example, fine-tuning on a specific dataset or task-focused training — or implicit, as when pre-existing knowledge is used to adapt to a new scenario.

Iterative and feedback-based learning: combines iterative prompting, role prompting, feedback mechanisms, and behavior cloning to strengthen capability through continuous feedback and adaptation.
Reinforcement learning and offline learning: includes reinforcement learning with environmental feedback, offline learning, and reinforcement learning driven by language instructions. Whereas iterative feedback-based learning focuses on optimizing model behavior through continuous real-time feedback, this approach focuses on maximizing long-term reward or achieving a specific goal through interaction with the environment, whether live or based on historical data.
Experience-based learning with memory: focuses on drawing on past experience, memory retrieval, and experience pools to guide current and future actions.
Collaborative and adaptive learning: emphasizes learning through multi-agent cooperation, adaptive strategies, and interactive learning processes.

Reasoning

Reasoning involves abstracting the foundational knowledge acquired during the learning stage into higher forms of understanding, via deductive, inductive, or abductive inference.

Decision Making

In the vast worlds of games, particularly in open-world and immersive environments, players and AI agents face complex decision and action spaces.

These spaces contain an enormous number of possible actions, interactions, and decisions, with a wide range of characters, item uses, and skill combinations. Multi-agent planning can add yet another layer of complexity, requiring coordination and collaboration among multiple agents under incomplete information.

Multi-hop Reasoning

Multi-hop reasoning refers to the cognitive process of considering multiple layers of information and dependencies before making a decision.

In a strategy game, for example, the agent must assess the current situation, anticipate the opponent’s moves, evaluate the potential outcomes of various actions, and only then settle on the most advantageous course.

Long-horizon Planning

Long-horizon planning refers to the strategic process by which an agent sets a goal and determines the actions that will achieve it over an extended period.

Some researchers have proposed using chain-of-thought (CoT) planning to imitate human reasoning, generating the intermediate steps that lead to a final decision, where each step builds on the previous one until the goal is reached.

Reflection

Reflection refers to the ability of LM-based agents (LMAs) to assess, evaluate, and possibly adjust their own processes, decisions, or outputs as a task unfolds.

Feedback-based self-improvement: the process of self-verification, analysis, and iterative enhancement using external feedback such as environmental responses or code execution results.
Iteratively refining plans and actions: methods that continually refine plans or actions based on feedback and new information, the better to achieve or adapt to a goal.
Collaboration and multi-agent interaction: emphasizes collaboratively re-evaluating and adjusting decisions within a multi-agent system.
Debate and theory-of-mind-based reflection: uses a debate or exchange format as a model of the reflection process.

Action

The action space in which LMAs operate within games can be classified into three distinct types.

Text-based interaction
Interaction through APIs
Direct control via IO operations, such as keyboard and mouse.

Behavior

LMAs perform specific operations through generative programming techniques. Iterative prompting combined with environmental feedback, execution errors, and self-verification continually refines the generated program, ensuring its validity and relevance to the task at hand.

Dialogue

LMAs interact with humans or with other agents by taking into account the current environment and past experience.

Examples include agent-agent interaction and human-agent interaction.

Past research has shown that LMAs exhibit human-like social behavior in collaboration, such as teamwork, leadership, persuasion, deception, and confrontation.

Consistency

Action consistency means that an agent’s actions must remain coherent and aligned with its goals.

To maintain such consistency, the agent takes past experience and the current situation into account, exploits feedback mechanisms, and revises its actions and plans when necessary.

Concrete strategies include:

Look-ahead checking mechanisms: assessing whether an action is feasible.
Reinforcement learning (RL): training a policy that selects the most appropriate action.
Self-consistency (SC): generating multiple action predictions and selecting the most frequent one for consistent decision making, as in the PokéLLMon case.

Structural approaches include:

Using a directed acyclic graph to control the progression of actions.
Leveraging a skill library.
Defining structured actions with clear semantics.
Assigning specific roles to agents and following a structured workflow.
Neuro-symbolic methods.

Challenges

Hallucination

An LM can produce output that does not match the source material or the facts, a phenomenon usually called hallucination and one that is very hard to eliminate.

Introducing critic agents, SOPs, and role-specific prompts can reduce logical hallucinations.

Researchers have also tried in-context reinforcement learning (ICRL), retrieval-augmented generation (RAG), multi-agent debate, and other methods to address hallucination.

Error Correction

Here, much research adopts iterative prompting mechanisms, feedback mechanisms, or chat chains among agents to identify and correct errors.

Methods such as using critic agents for selection and feedback, and iterative prompting with environmental feedback, are employed for error correction.

Generalization

Generalization refers to an agent’s ability to apply the knowledge, skills, and capabilities acquired in past environments to new, previously unseen environments so that it can still achieve goals and carry out tasks.

Proposed solutions include context-aware chain-of-thought (CoT) reasoning, GITM (Ghost in the Minecraft), which exploits long-horizon planning and external knowledge in Minecraft, and information extraction and collaborative planning in multi-agent settings.

Interpretability

Some studies use CoT processes, modular decision processes, and clear conversational programming paradigms to make agent behavior more interpretable.

Future Directions

Multimodal Perception

Strengthening multimodal capability, including visual and auditory perception, may allow agents to handle more complex tasks.

It is worth noting that sound effects in games are essential for perceiving context, understanding tasks, and receiving feedback, yet very little literature explores how auditory information can be used to improve the performance of game agents.

Authenticity of the Game Experience

People prefer human-written content to dialogue and ideas generated by large language models. This suggests that future research needs to tie LLM-generated content more closely to the game’s narrative and state.

External Tools

LM-based agents have yet to master the use of external tools to improve their own performance in games.

Real-time Performance

Given the inference process and computational demands inherent to large models, mastering real-time, high-speed games poses a formidable challenge.

Paper Review | An LLM Survey from Tomas Mikolov, Author of Word2Vec

Sat, 16 Mar 2024 00:00:00 +0000

Introduction

Large language models (LLMs) refer mainly to Transformer-based neural language models that contain tens to hundreds of billions of parameters and are pre-trained on massive text corpora — models such as PaLM, LLaMA, and GPT-4. Compared with PLMs, LLMs are not only larger in scale but also stronger in language understanding and generation, and more importantly, they exhibit emergent abilities that are absent in smaller language models. As the figure above shows, these emergent abilities include (1) in-context learning, where LLMs learn a new task at inference time from a handful of examples presented in the prompt; (2) instruction following, where after instruction tuning LLMs can follow instructions for new types of tasks without any explicit examples; and (3) multi-step reasoning, where LLMs solve complex tasks by decomposing them into intermediate reasoning steps, as in chain-of-thought prompting. LLMs can also be augmented with external knowledge and tools so that they interact effectively with users and their environment, and they can keep improving themselves through feedback data collected from human interaction (for example, via reinforcement learning from human feedback, RLHF).

Large Language Models

Early Pre-trained Neural Networks

In 1999, Bengio et al. developed the earliest neural language models (NLMs), which were competitive with n-gram models. Since then, NLMs based on recurrent neural networks (RNNs) and their variants — such as long short-term memory (LSTM) and the gated recurrent unit (GRU) — have been widely used in many natural language applications, including machine translation, text generation, and text classification. The invention of the Transformer architecture later marked another milestone in the development of neural language models. Transformers offer parallelism that RNNs lack, which makes it possible to pre-train very large language models efficiently on GPUs. These pre-trained language models (PLMs) can then be fine-tuned for many downstream tasks.

We group the early popular Transformer-based PLMs into three main categories according to their neural architecture — encoder-only, decoder-only, and encoder-decoder — and introduce each below.

Encoder-only PLMs:

As the name suggests, encoder-only models contain only an encoder network. They were originally developed for language understanding tasks such as text classification, where the model must predict a category label for the input text. The representative model is BERT, along with its variants such as RoBERTa, ALBERT, DeBERTa, XLM, XLNet, and UNILM.

Overall pre-training and fine-tuning pipeline of BERT

BERT (Bidirectional Encoder Representations from Transformers) is one of the most widely used encoder-only models. It consists of three modules: (1) an embedding module that converts the input text into a sequence of embedding vectors, (2) a stack of Transformer encoders that turns those embedding vectors into contextual representation vectors, and (3) a fully connected layer that maps the representation vectors (at the final layer) into one-hot vectors. BERT is pre-trained with two objectives: masked language modeling (MLM) and next sentence prediction. A pre-trained BERT model can be fine-tuned by adding a classifier layer for many language understanding tasks, ranging from text classification and question answering to language inference. The overall pre-training and fine-tuning pipeline of BERT is shown in the figure above.

Decoder-only PLMs

Among decoder-only PLMs, the most representative are GPT-1 and GPT-2, developed by OpenAI, which paved the way for GPT-3.5 and GPT-4.

Overview of the GPT pre-training and fine-tuning steps

GPT-1 was the first to demonstrate that training a decoder-only generative pre-training (GPT) model in a self-supervised fashion on an unlabeled text corpus, and then applying discriminative fine-tuning on each specific downstream task, yields good performance across a wide range of natural language tasks.

GPT-2 showed that when trained on WebText — a large dataset composed of millions of web pages — a language model can learn to perform specific natural language tasks without any explicit supervision.

Self-supervised learning: a subset of unsupervised learning. In self-supervised learning, the model uses an unlabeled dataset itself as training data. For instance, during GPT training the model predicts the next token, and this prediction task requires no externally annotated data, because the next token comes directly from the input data itself.

Encoder-Decoder PLMs

Prior research has shown that nearly every natural language processing task can be cast as a sequence-to-sequence generation task. An encoder-decoder language model is therefore a unified model by design, since it can perform all natural language understanding and generation tasks. The most representative models in this category are T5, mT5, MASS, and BART.

LLM Families

Large language models (LLMs) refer mainly to Transformer-based PLMs with billions to hundreds of billions of parameters. Compared with the PLMs above, LLMs are not only larger in size but also stronger in language understanding and generation, and they exhibit emergent abilities that smaller models lack. Next, we review three LLM families: GPT, LLaMA, and PaLM.

The GPT Family

Generative Pre-Training (GPT) is a family of decoder-only models developed by OpenAI. The family includes GPT-1, GPT-2, GPT-3, InstructGPT, ChatGPT, GPT-4, CODEX, and WebGPT. While the earlier GPT models, such as GPT-1 and GPT-2, are open source, the more recent ones, such as GPT-3 and GPT-4, are closed source and accessible only through an API.

GPT-3: a pre-trained autoregressive language model with 175 billion parameters. GPT-3 is widely regarded as the first LLM, because it is not only far larger than previous PLMs but also the first to display emergent abilities not observed in smaller PLMs. GPT-3 demonstrated the emergent ability of in-context learning, meaning that it can be applied to any downstream task without any gradient update or fine-tuning.
CODEX: a general-purpose programming model that can parse natural language and generate code in response. CODEX is a descendant of GPT-3, fine-tuned for programming applications on a code corpus collected from GitHub.
WebGPT: another descendant of GPT-3, fine-tuned to answer open-ended questions using a text-based web browser, helping users search and browse the web.
InstructGPT: proposed to make LLMs follow human instructions, InstructGPT is fine-tuned with human feedback (RLHF) so that it stays aligned with user intent across a wide range of tasks.

Overview of RLHF

RLHF: starting from a set of prompts written by labelers together with prompts submitted through the OpenAI API, a dataset of labeler demonstrations of the desired model behavior is collected. GPT-3 is then fine-tuned on this dataset. Next, a dataset of human-ranked model outputs is collected, and the model is further fine-tuned with reinforcement learning. This approach is called reinforcement learning from human feedback (RLHF).
ChatGPT (GPT-3.5): the most significant milestone in LLM development is ChatGPT, released on November 30, 2022. ChatGPT is a chatbot that lets users steer a conversation to accomplish a variety of tasks, such as answering questions, seeking information, and summarizing text. ChatGPT is powered by GPT-3.5 and is a sibling model to InstructGPT.

Performance of GPT-4 on academic and professional exams compared with GPT-3.5

GPT-4: the newest and most powerful LLM in the GPT series. Released in March 2023, GPT-4 is a multimodal LLM that accepts both images and text as input and produces text output. Although it still falls short of humans in some of the most challenging real-world scenarios, GPT-4 achieves human-level performance on a variety of professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers, as shown in the figure above. Like earlier GPT models, GPT-4 is first pre-trained to predict the next token on a large text corpus, and then fine-tuned with RLHF to align its behavior with what humans expect.

The LLaMA Family

LLaMA is a series of large language models released by Meta. Unlike the GPT models, LLaMA models are open source.

LLaMA-1: the first batch of LLaMA models was released in February 2023, with parameter counts ranging from 7B to 65B. These models were pre-trained on trillions of tokens collected from publicly available datasets. LLaMA adopts the Transformer architecture of GPT-3 with a few minor architectural modifications.
LLaMA-2: in July 2023, Meta, in collaboration with Microsoft, released the LLaMA-2 family, which includes base language models as well as chat models fine-tuned for dialogue, known as LLaMA-2 Chat.

The PaLM Family

The PaLM (Pathways Language Model) family was developed by Google.

The first PaLM model was announced in April 2022. It was pre-trained on 6,144 TPUv4 chips using the Pathways system, contains 540 billion parameters, and was open-sourced in March 2023.
PaLM-2 is trained with a mixture of objectives, unlike its predecessor PaLM. Through extensive evaluation on English, multilingual, and reasoning tasks, PaLM-2 substantially improves downstream task performance across model sizes while demonstrating faster and more efficient inference than PaLM.

Others

Mistral-7B: a 7B-parameter language model engineered for superior performance and efficiency. Mistral-7B outperforms LLaMA-2-13B on all evaluation benchmarks and surpasses LLaMA-34B on reasoning, mathematics, and code generation. The model uses grouped-query attention for faster inference, combined with sliding window attention to handle sequences of arbitrary length effectively while lowering inference cost.

The BLOOM architecture
BLOOM: Scao et al. proposed BLOOM, an open-access language model with 176B parameters, designed and built collaboratively by hundreds of researchers. BLOOM is a decoder-only Transformer language model trained on the ROOTS corpus, which spans hundreds of sources (59 in total) covering 46 natural languages and 13 programming languages. Its architecture is shown in the figure above.
GLM: a large bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It aims to open-source a 100B-scale model at least on par with GPT-3 (davinci).
Gemini: the Gemini team introduced a new family of multimodal models supporting a 32k context length. The Gemini family comes in three versions: Ultra for highly complex tasks, Pro for enhanced performance and deployability at scale, and Nano for on-device applications.

How to Build an LLM

Architecture Choices

As described above, the most widely used LLM architectures today are encoder-only, decoder-only, and encoder-decoder.

Here we first revisit the Transformer architecture. It was originally proposed in Attention Is All You Need and designed from the outset to exploit GPUs for efficient parallel computation. At its core is the attention mechanism, which, compared with recurrence and convolution, (1) uses GPUs more effectively and (2) captures long-range contextual information. The attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.

Encoder-Only

In an encoder-only architecture, the attention layer has access to all the words in the initial sentence at every stage. Pre-training for these models usually involves corrupting a given sentence in some way (for example, masking) and asking the model to reconstruct it. Encoder-only architectures are well suited to tasks that require understanding of the complete sequence, such as sentence classification, named entity recognition, and extractive question answering.

Decoder-Only

In a decoder-only architecture, at every stage any given token attends only to the tokens preceding it. These models are sometimes called autoregressive models. Their pre-training is usually formulated as predicting the next token in the sequence (next token prediction). Decoder-only architectures suit tasks that involve text generation. The GPT models are the outstanding example of this model class.

Encoder-Decoder

Encoder-decoder architectures use both an encoder and a decoder, and are sometimes called sequence-to-sequence models. At each stage, the encoder’s attention layer can access all the words in the initial sentence, whereas the decoder’s attention layer can only access the tokens preceding a given token in the input. Encoder-decoder models suit tasks that generate a new sentence from a given input, such as summarization, translation, or generative question answering.

Data Cleaning

Data quality is critical to the performance of the language model trained on it. Data cleaning techniques such as filtering and deduplication have been shown to have a significant impact on model performance.

Data Filtering

Data filtering aims to improve the quality of the training data and thus the effectiveness of the trained LLM. Common data filtering techniques include:

Denoising: for example, removing false information.
Handling outliers
Class imbalance: balancing the distribution of classes in the dataset.
Text preprocessing: removing stop words, punctuation, and the like.
Handling ambiguity: dealing with vague or contradictory data.

Deduplication

Deduplication means removing duplicate instances or repeated occurrences of the same data. Duplicate data points can introduce bias during model training and reduce diversity. Some studies show that deduplication improves the model’s ability to generalize to new, unseen data.

Tokenization

Tokenization is the process of converting a text sequence into smaller pieces, namely tokens.

Model Pre-training

Pre-training is the first step in the training pipeline of a large language model. During pre-training, an LLM is typically trained in a self-supervised manner on a large amount of (usually) unlabeled text. The two most common pre-training methods are next token prediction and masked language modeling (MLM).

Next token prediction: predict the next token in a sentence token by token from left to right (or right to left), establishing dependencies between tokens. Suitable for text generation tasks such as machine translation and text generation.
Masked language modeling: in MLM, the model learns the relationships between tokens by predicting randomly masked words in a sentence, which trains it to understand the contextual meaning of words. MLM is better suited to understanding tasks such as text classification and named entity recognition.

Fine-tuning and Instruction Tuning

Fine-tuning usually serves two purposes: (1) adapting the base model to different tasks, and (2) injecting new knowledge.
Beyond fine-tuning, we also want an LLM to understand and follow natural language instructions from humans. That is where instruction tuning comes in.

Alignment

AI alignment is the process of steering an AI system toward human goals, preferences, and principles. Sometimes, in the course of token prediction, large language models exhibit dangerous behavior. For example, they may generate undesirable, harmful, misleading, or biased content.

At this stage, two popular approaches are RLHF (reinforcement learning from human feedback) and RLAIF (reinforcement learning from AI feedback).

RLHF: RLHF uses a reward model to learn alignment from human feedback. The tuned reward model is able to evaluate different outputs and score the outputs of the model being trained according to the alignment preferences given by humans.
RLAIF: RLAIF, on the other hand, connects a pre-trained, well-aligned model directly to the LLM being trained and helps it learn from this larger, better-aligned model. This resembles a generalized form of knowledge distillation.

Decoding Strategies

Decoding refers to the process of generating text with a pre-trained large language model. Given an input prompt, the tokenizer converts each token in the input text into its corresponding numerical representation. The language model then takes these numerical token representations as input and predicts the next most likely token. Finally, the model produces logits, which are turned into probabilities by a softmax function. A variety of decoding strategies have been proposed; among the most popular are greedy search, beam search, top-K, and top-P.

Greedy Search

Greedy search picks the most likely token as the next token in the sequence at each step, discarding all other potential options. As with many greedy algorithms, this approach may sacrifice the consistency and coherence of the text. It only considers the most likely token at each step, without regard for the quality of the generated sequence as a whole.

Beam Search

Unlike greedy search, which only considers the single most likely next token, beam search considers the N most likely tokens, where N is the number of beams. The process repeats until a predefined maximum sequence length is reached or an end-of-sequence token appears. At that point, the token sequence (also called a beam) with the highest overall score is selected as the output. For example, with a beam size of 2 and a maximum length of 5, beam search must recursively generate and evaluate $2^5 = 32$ possible sequences. It therefore requires considerably more computation than greedy search.

Top-k

Top-k sampling uses the probability distribution produced by the language model to randomly select a token from among the k most likely options.

Suppose we have six tokens (A, B, C, D, E, F) and set k=2, with P(A)=30%, P(B)=20%, and P(C)=P(D)=P(E)=P(F)=12.5%. Under top-k sampling, tokens C, D, E, and F are ignored, and the model outputs A with probability 60% and B with probability 40%. This approach ensures that we prioritize the most likely tokens while still introducing a degree of randomness.

In this method, randomness is usually introduced through the notion of temperature. The temperature T is a parameter from 0 to 1 that affects the probabilities produced by the softmax function, as the following formula shows: $$ softmax(x_i)=\frac{e^{x_i/T}}{\sum_je^{x_j/T}} $$ As the formula indicates, a low temperature makes the probability distribution sharper, which means higher-probability tokens become even more prominent, so the generated text is more deterministic and less creative; a high temperature flattens the distribution, giving low-probability tokens more of a chance and thereby increasing the diversity and creativity of the text.

Top-p

Top-p sampling, also known as nucleus sampling, takes a slightly different approach from top-k sampling. Instead of selecting the top k most likely tokens, nucleus sampling chooses a threshold p such that the probabilities of the selected tokens sum to more than p. In other words, under top-p sampling the model examines the most likely tokens in descending order and adds them to a list one by one until their cumulative probability exceeds the threshold p.

Using LLMs

Limitations of LLMs

No state or memory: on their own, they cannot even remember what was sent to them in a previous prompt.
Answers are random or probabilistic: send the same input to an LLM several times and you will very likely get different replies.
Outdated information
High training cost
Hallucination

Here we discuss only hallucination.

Hallucination in LLMs has been described as “generated content that is nonsensical or unfaithful to the provided source,” and it falls into two categories:

Intrinsic hallucinations: the hallucination directly conflicts with the source material, being factually inaccurate or logically inconsistent.
Extrinsic hallucinations: content that cannot be verified against the source, containing speculative or unverifiable elements.

The impact of hallucination depends heavily on the context and the task. In creative endeavors such as writing poetry, for instance, hallucination may be considered acceptable or even beneficial.

Recent work on LLM hallucination, such as instruction tuning and reinforcement learning from human feedback (RLHF), attempts to steer LLMs toward more fact-grounded output, but their inherently probabilistic nature, and the limitations that come with it, remains.

Prompt Engineering

When working with an LLM, the prompt is the text input provided by the user to guide the model’s output. A prompt typically includes an instruction, a question, input data, and examples.

Typical prompting methods include:

Chain of Thought (CoT): the famous CoT, which asks the LLM to think step by step so that it produces a better output. It comes in two main forms:
1. Zero-shot CoT: append “Let’s think step by step” after the question.
2. Manual CoT: requires humans to supply step-by-step reasoning examples as templates.
3. Of the two, Manual CoT is the more effective, but constructing chains of thought by hand is difficult and error-prone. Auto CoT does not suffer from this problem; see Auto CoT — Generating Chain-of-Thought Automatically via Clustering for details.
Tree of Thought (ToT): building on the idea of CoT, ToT can be unfolded into multiple “trees of thought,” where each branch represents a different reasoning path, that is, a different CoT.
Self-Consistency: given a prompt, the LLM generates not one answer but several. These candidate answers are then evaluated to determine how consistent they are with one another. Finally, the answer that is most consistent across all of them is chosen as the final output.
Reflection: prompts the LLM to evaluate and possibly revise its output by reasoning about the correctness and coherence of its own answer.
Expert Prompting: prompts the LLM to take on the role of an expert and answer accordingly, providing high-quality, well-informed answers.

RAG

As the figure above shows, RAG extracts the query from the prompt and uses it to retrieve relevant information from an external knowledge source (such as a search engine or a knowledge graph). The retrieved information is then added to the prompt and fed to the LLM, which produces the final answer.

LLM Agents

The notion of AI agents has a long history. An AI agent is typically an autonomous entity that perceives its environment through sensors, makes judgments based on the current state, and acts.

In the context of large models, an agent is a system built on an LLM that can carry out specific tasks autonomously. Such agents are designed to interact with users or their environment and to make decisions based on the input and the intended goal of the interaction. The core characteristics of an agent are:

The ability to access and use tools.
The ability to make decisions autonomously.

Representative work in this area includes ReAct and LATS; interested readers may consult my earlier post ReAct — An LLM Reasoning Paradigm Combining Reasoning and Acting.

Datasets

Challenges

Smaller and More Efficient Models

A current research trend is to propose small language models (SLMs) as a more economical alternative to LLMs, particularly for tasks that do not require such large models.

New Architectural Paradigms Beyond Attention

One important class of models proposed after the attention-based models is state space models (SSMs). Although the concept of state space models has a long history in machine learning, in the context of language models SSM usually refers to the more recent structured state space model architecture, S4 for short. Some recent models in this category include Mamba, Hyena, and Striped Hyena.

On another front, several attention-compatible architectural mechanisms have been gaining ground recently, such as mixture of experts (MoE). Notably, MoEs can serve as a component of any architecture, whether or not it is attention-based.

Multimodality

Several well-known multimodal LLMs already exist, including LLaVA (a vision-language model), LLaVA-Plus (an extension of LLaVA), GPT-4, Qwen-VL, and Next-GPT.

Techniques for Using and Augmenting LLMs

Prompting and RAG, for example.

LLM Safety

Privacy and bias, for example.

Paper Review | Auto-CoT — Generating Chain-of-Thought via Clustering

Sat, 02 Mar 2024 00:00:00 +0000

Title: Automatic Chain of Thought Prompting in Large Language Models

Institute: Shanghai Jiao Tong University, Amazon Web Services

Authors: Zhuosheng Zhang, Aston Zhang, Mu Li, Alex Smola

Arxiv Link: https://arxiv.org/abs/2210.03493

Date: 2022.10.7

Introduction

Before this work, chain-of-thought prompting existed in two paradigms. The first is Zero-shot CoT, which simply appends “Let’s think step by step” to the end of the question. The second is Manual CoT (Few-shot CoT), where each exemplar consists of a question paired with a reasoning chain. How well the second approach works depends entirely on how good the hand-written chains are — and writing them takes human effort. The authors propose Auto-CoT, which generates few-shot CoT exemplars automatically and spares us that manual labor.

Motivation

Manual CoT (Few-shot CoT) outperforms Zero-shot CoT but consumes far more human effort, so the authors set out to design Auto-CoT.

Findings

Unlike papers that present their method up front, this one first runs a series of exploratory experiments and only then arrives at its proposal.

The authors begin by comparing Retrieval-Q-CoT against Random-Q-CoT on three datasets: MultiArith, GSM8K, and AQuA. On MultiArith, Retrieval-Q-CoT turns out to be worse than Random-Q-CoT, which is clearly counterintuitive. Suspecting that the culprit was the poor quality of the Zero-shot-generated chains, they repeated the experiment on GSM8K and AQuA, two datasets whose CoT annotations are human-written. As expected, Retrieval-Q-CoT outperformed Random-Q-CoT on both. The conclusion: chains produced by Zero-shot are simply not as good as human-written ones.

Retrieval-Q-CoT: encode the question with Sentence-BERT, then take the top-k most similar questions as demonstrations, $q_i^{demo}(i=1,..,k)$. For these k questions, generate chains with Zero-shot — $c_i^{demo}$ — then concatenate $q_i^{demo},c_i^\text{demo}(i=1,…,k),q^{test}$ as the input to the LLM.

Random-Q-CoT: same as Retrieval-Q-CoT, except the k questions are chosen at random.

To further verify that flawed chains are what degrade reasoning, the authors deliberately injected incorrect chains into 128 out of 600 examples. Retrieval-Q-CoT was indeed misled, confirming how much correct chains matter in CoT prompting.

Going further, the authors clustered all questions into k groups by similarity with k-means, then generated chains with Zero-shot and ran inference. To find out whether certain clusters concentrate the questions on which Zero-Shot-CoT tends to fail, they computed the error rate of each cluster.

As the figure shows, cluster 2 has a strikingly high error rate, which indicates that Zero-shot cannot produce good chains for a certain class of questions — likely the reason Retrieval-Q-CoT underperforms.

Auto-CoT

Building on these findings, the authors propose Auto-CoT, which consists of two main stages:

Clustering: partition the questions of a given dataset into several clusters.
Sampling: select one representative question from each cluster and generate its reasoning chain with Zero-Shot-CoT under a simple heuristic.

The overall pipeline is shown below:

In more detail:

In the clustering stage, all questions are embedded with Sentence-BERT and partitioned into k classes by k-means. Within each cluster, questions are sorted by their distance to the cluster center, producing an ascending list $\mathbf{q}^{(i)}=[q_1^{(i)},q_2^{(i)},\ldots]$.
In the sampling stage, each cluster is traversed in that ascending order. A chain is generated for the current question with Zero-shot; if the chain runs to no more than 60 tokens and five steps, that question and its chain become the exemplar for every question in the cluster and are prepended to them at inference time.

Experiments

The experiments cover three categories of reasoning tasks across ten benchmark datasets, all with GPT-3:

Arithmetic reasoning (MultiArith, GSM8K, AddSub, AQUA-RAT, SingleEq, SVAMP)
Commonsense reasoning (CSQA, StrategyQA)
Symbolic reasoning (Last Letter Concatenation, Coin Flip)

The baselines are Zero-Shot, Zero-Shot-CoT, Few-Shot, and Manual-CoT.

As the table below shows, Auto-CoT consistently matches or exceeds the performance of CoT methods that require manually designed demonstrations.

Conclusion

Viewed from 2024, the method in this paper looks fairly simple next to work such as ReAct. It is worth remembering, however, that it appeared quite early, when everyone was still feeling their way forward — which makes a method like this all the more valuable. In fact, traces of Auto-CoT are still visible in the latest approaches.

Paper Review | Does a Longer Chain of Thought Make an LLM Smarter?

Mon, 26 Feb 2024 00:00:00 +0000

Title: The Impact of Reasoning Step Length on Large Language Models

Institute: Northwestern University, University of Liverpool, New Jersey Institute of Technology, Rutgers University

Authors: Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du

Arxiv Link: https://arxiv.org/abs/2401.04925

Date: 2024.1.20

Introduction

Chain of thought (CoT) has already proven, in practice, to substantially improve the reasoning ability of large language models. Yet no work so far has explained the relationship between the length of a chain of thought and reasoning ability. This paper takes that question as its core, runs a systematic set of experiments on CoT, and arrives at a number of interesting and counterintuitive conclusions.

Conclusions

Let us skip the preamble and go straight to the conclusions.

For Few-shot CoT, the accuracy of the model’s answers is linearly related to the number of reasoning steps: the longer the chain, the more accurate the answer. Conversely, shortening the CoT significantly reduces accuracy, even when the shortened chain preserves the same key information as the long one.
Even when a CoT chain contains incorrect reasoning, keeping the necessary reasoning length still yields good results. In other words, a longer chain improves accuracy even if an intermediate step is wrong.
Simple tasks need fewer steps; complex tasks need more.
Even in the Zero-shot CoT setting, adding “you must think more steps” on top of “Let’s think step by step” improves the model’s performance.

Experimental Setup

Zero-shot CoT

The Zero-shot CoT setting in this paper defaults to “Let’s think step by step”. The authors then compare it against “Let’s think step by step, you must think more steps” to see how the extra instruction affects the LLM.

Few-shot CoT

As shown above, the Few-shot CoT in this paper is organized into the following steps:

Think About The Word: the authors first ask the LLM to think about the meaning of the words in the question (work out what each word means before answering). For example, given the question “Could someone in Tokyo take a taxi to the Metropolitan Museum of Art?”, as illustrated below, the authors have the LLM first consider what “Tokyo”, “taking a taxi”, and “Metropolitan Museum of Art” mean — “Think about Tokyo… Think about taking a taxi… Think about Metropolitan Museum of Art…”.

Read the question again: have the LLM read the question once more (review the problem statement again).
Repeat State: have the LLM produce a summary of the question, simplifying what it needs to remember and reducing the influence of the other words in the CoT (extract the key information from the question).
Self-Verification: add a self-verification step in which the model judges, from some basic information, whether the answer is reasonable (think for yourself about whether your answer is right).
Make Equation: for mathematical problems, have the LLM formalize the problem as an equation.

Experiments and Findings

The relationship between step count and accuracy

As shown above, in each subplot the x-axis is the length of the CoT chain (the number of reasoning steps) and the y-axis is the LLM’s accuracy. Clearly, the longer the CoT chain, the higher the accuracy. This finding holds across all datasets and tasks.

What happens when one link in the CoT is wrong?

For CoT researchers, the most important question is how to get every step of the chain right so as to arrive at the correct answer. In this section, the authors ask: if one link in the CoT is reasoned incorrectly, how does that affect the LLM’s final result?

In the figure below, the x-axis denotes eight categories of tasks/datasets, and the two colored bars represent fully correct chains and chains containing an incorrect answer. The authors find that:

For mathematical problems, an error in one link has very little impact. The authors argue that the LLM learns the computational pattern rather than the result of the computation itself, and is therefore unaffected. An example is shown in the figure below, where the faulty prompt on the right computes 10+8 as the incorrect value 48.
For logical problems, even a small error breaks the chain of thought and has a large impact.

What happens when the reasoning steps are compressed?

Compressing the reasoning steps here means phrasing each step more briefly. The results show that compression drives performance close to that of Zero-shot, further demonstrating the strong link between reasoning steps and LLM accuracy.

The effect of model size on the results

As the figure below makes obvious, the more parameters, the more accurate the model.

Do different CoT generation methods differ much?

Since this paper focuses on the impact of reasoning steps, it first has to confirm that the questions used inside the CoT are not what drives the LLM’s performance.

To that end, with the LLM fixed to GPT-3.5-turbo-1106, the authors selected two datasets (MultiArith and GSM8K) and two CoT methods (Auto CoT and Few-shot CoT) for this study.

As shown below, an initial inspection indicates that these modifications have a negligible effect on performance. This preliminary finding suggests that it is the length of the reasoning chain, rather than the nature of the questions themselves, that primarily governs the reasoning ability of LLMs.

Comments

Although this paper is not technically difficult, its evaluation of CoT is thorough, its angle is novel, and it arrives at a good number of interesting conclusions.

Paper Review | Emergent Abilities of LLMs Are a Mirage

Sat, 16 Dec 2023 00:00:00 +0000

Title: Are Emergent Abilities of Large Language Models a Mirage?

Institution: Computer Science, Stanford University

Authors: Rylan Schaeffer, Brando Miranda, Sanmi Koyejo

Arxiv Link: https://arxiv.org/abs/2304.15004

Comments: NeurIPS 2023

Date: 2023.04.28

Preface

This paper was posted on arXiv on April 28, 2023. Two versions later, it won a Best Paper award in the main track at NeurIPS 2023 (2 out of 3584).

Emergent Abilities

Figure 1

The paper notes that a great deal of work claims LLMs display abilities that are absent at smaller scales but present at larger ones — what people commonly call emergent abilities, or emergence.

An “emergent ability” is a capability or behavior that a model displays only once it becomes large enough, one that smaller models did not have — a metric that suddenly shoots up, for instance.

As the figure above shows, for model families such as LaMDA and GPT-3, accuracy jumps abruptly once the parameter count crosses a certain critical point. That is emergence.

Emergent abilities have two defining features:

Sharpness: an abrupt rise as the parameter count increases linearly
Unpredictability: they appear suddenly at some uncertain critical parameter count

Does Emergence Really Exist?

To give away the conclusion: this paper shows that LLMs exhibit no genuine “emergence”; their capabilities grow smoothly as parameters increase.

Emergence is not a product of growth in a model’s parameter count. It is an illusion created by the metric the researcher happened to choose.

The authors’ starting point is that the metrics previously used to measure model capability are discrete rather than linear, and that this is what produces apparent emergence. So what happens if we swap in a linear or continuous metric instead — does a model’s capability turn out to be smooth, continuous, and predictable?

Explaining Emergent Abilities

Figure 2

The authors open with a question of their own: how can a smooth, continuous, predictable change in model performance be made to look abrupt and unpredictable? Their answer is that a researcher’s choice of a nonlinear or discontinuous metric can distort how the model’s performance appears, making it seem to change abruptly and unpredictably.

Experience tells us that a neural network’s capability scales in a fixed proportion with the amount of training data, the parameter count, and the compute expended. Taking that as given, assume the following holds (Figure 2.A): $$ \mathcal{L}_{CE}(N)=\left(\frac Nc\right)^\alpha $$ where $L_CE$ denotes cross entropy, $N$ denotes the parameter count, and $c>0$ and $\alpha<0$ are constants. In practice, cross entropy is defined as: $$ \mathcal{L}_{CE}(N)\overset{\mathrm{def}}{\operatorname*{=}}-\sum_{v\in V}p(v)\log\hat{p}_N(v) $$ This formula measures the discrepancy between the model’s predicted distribution $\hat{p}_N$ over the whole vocabulary $V$ and the true distribution $p$. For each possible token $v$, the true distribution $p(v)$ gives the true probability that token $v$ occurs, while $\hat{p}_N(v)$ is the probability the model assigns to token $v$.

Since $p(v) = 0$ for all $v \neq v^*$ and $p(v^*) = 1$, only the term with $v = v^*$ survives the sum, and the formula simplifies to:

$$ \mathcal{L}_{CE}(N)=-\log\hat{p}_N(v^*) $$

The probability that each token is predicted correctly is (Figure 2.B): $$ \begin{align*} p(\text{single token correct})& =\exp\left(-\mathcal{L}_{CE}(N)\right)\\ & =\exp\left(-(N/c)^\alpha\right) \end{align*} $$ Now suppose the researcher chooses a metric that requires $L$ tokens to all be correct. The task might be, for example, $L$-digit integer addition, scoring $1$ if all $L$ output digits match the target digits exactly, with no insertions, deletions, or substitutions, and $0$ otherwise. If each token’s correctness is independent, the probability of scoring $1$ is: $$ \begin{align*} \operatorname{Accuracy}(N) &\approx p_N\left(\text{single token correct}\right)^\text{num. of tokens} \\ &= \exp\left(-(N/c)^\alpha\right)^L \end{align*} $$

$$\text{Exact String Match}\stackrel{\text{def}}{=}\begin{cases}1&\text{if output string exactly matches target string}\\0&\text{otherwise}\end{cases}$$

This choice of metric makes performance scale nonlinearly with the length of the token sequence. Plotting performance on a linear-log axis (Figure 2.C) reveals abrupt, unpredictable jumps on longer sequences — a very close match to the emergent abilities other researchers have claimed.

And what happens if we use a linear metric such as Token Edit Distance? $$ \begin{aligned} \text{Token Edit Distance}(N) &\approx L(1-p_N(\text{single token correct})) \\ &= L\left(1-\exp\left(-(N/c)^\alpha\right)\right) \end{aligned} $$ When model capability is measured with the linear metric Token Edit Distance, it becomes smooth, continuous, and predictable (Figure 2.E).

“Token Edit Distance” is a way of measuring the difference between two sequences. It counts the minimum number of edit operations — insertions, deletions, and substitutions of a single token — needed to turn one sequence into the other. In this context, a “token” usually means a word, a character, or another linguistic unit.

For example, if we have two sequences, sequence A being “cat” and sequence B being “bat”, the token edit distance from A to B is 1, since we need only substitute “b” for “c”.

If the discrete metric Multiple Choice Grade is used, emergence is observed (Figure 2.D), but if the continuous metric Brier Score is used, emergence disappears (Figure 2.F).

Multiple Choice Grade:
$$\\ \text{Multiple Choice Grade}\stackrel{\text{def}}{=}\begin{cases}1&\text{if highest probability mass on correct option}\\0&\text{otherwise}\end{cases}$$
Brier Score: $$ BS = \frac{1}{N} \sum_{i=1}^{N} (f_i - o_i)^2$$ where:

$N$ is the number of predictions.

$f_i$ is the predicted probability of occurrence for the $i$-th prediction (the probability output by the model).

$o_i$ is the observed value of the actual outcome for the $i$-th case (usually 1 if the event occurred and 0 if it did not).

As an example, suppose you are predicting whether football matches will end in a draw. For three matches, your model predicts draw probabilities of 0.3, 0.6, and 0.2. The actual outcomes are that the first match was a draw (1) while the second and third were not (0).

The Brier Score is then computed as follows:

For the first match, the prediction error is $ (0.3 - 1)^2 = 0.49 $.

For the second match, the prediction error is $ (0.6 - 0)^2 = 0.36 $.

For the third match, the prediction error is $ (0.2 - 0)^2 = 0.04 $. The Brier Score is the average of these errors, so in this example it is $ \frac{0.49 + 0.36 + 0.04}{3} = 0.2967 $.

In summary, the illusion of emergence has two causes.

The metrics adopted by earlier work are discrete rather than continuous or linear, and that is what gives rise to the appearance of emergence.
Earlier work experimented only over a narrow range of parameter counts. Model sizes are typically 6B, 7B, 13B, 33B, 70B and so on, which cover only a handful of points in the range. Had sizes such as 6B, 6.1B, 6.2B, 6.3B … 70.1B, 70.2B been available, people would have seen that discrete metrics relate to parameter count exponentially rather than by a sudden jump.

This work fully deserves its Best Paper award at NeurIPS 2023. Its starting point is simple, yet its insights are far-reaching, creative, and genuinely illuminating. Before it appeared, emergence had become a widely accepted “magical phenomenon” in the field, and a good deal of research had been devoted to it, probing experiments included. Against that backdrop, the paper works from the details up to show that “emergence” does not actually exist — an eye-opening result for many researchers, this author among them.

Paper Review | Multimodal Representations of Graph Structure for LLMs

Tue, 21 Nov 2023 00:00:00 +0000

Title: Which Modality should I use – Text, Motif, or Image? : Understanding Graphs with Large Language Models

Institution: Minnesota NLP, University of Minnesota

Authors: Debarati Das, Ishaan Gupta, Jaideep Srivastava, Dongyeop Kang

Arxiv Link: https://arxiv.org/abs/2311.09862

Code Link: None

Date: 2023.11.16

Abstract: Large language models (LLMs) are revolutionizing various fields by leveraging large text corpora for context-aware intelligence. Due to the context size, however, encoding an entire graph with LLMs is fundamentally limited. This paper explores how to better integrate graph data with LLMs and presents a novel approach using various encoding modalities (e.g., text, image, and motif) and approximation of global connectivity of a graph using different prompting methods to enhance LLMs’ effectiveness in handling complex graph structures. The study also introduces GraphTMI, a new benchmark for evaluating LLMs in graph structure analysis, focusing on factors such as homophily, motif presence, and graph difficulty. Key findings reveal that image modality, supported by advanced vision-language models like GPT-4V, is more effective than text in managing token limits while retaining critical information. The research also examines the influence of different factors on each encoding modality’s performance. This study highlights the current limitations and charts future directions for LLMs in graph understanding and reasoning tasks.

Introduction:

This paper explores the impact of encoding global and local graph structure in different modalities, with a particular focus on node classification.

Specifically, it compares three modalities: text, motif, and image.

Why the paper introduces the motif modality:

However, it becomes too verbose for large graphs, potentially exceeding the input context limits of large language models like GPT-4. To circumvent this, we suggest Motif modality encoding, which captures essential patterns within the node’s vicinity, offering a balanced local and global perspective.

Methodology

Text Encoder Modality

Encoding graphs as text can be separated into two key parts:

the mapping of nodes to their corresponding labels in the graph
- {node ID: node label}
the encoding of edges between the nodes

We consider several factors that can influence the performance of text-modality encoding:

Edge Encoding Function
Graph Structure
Sampling Strategy

Edge Encoding Function

Edgelist
Edgetext
Adjacency List
GML
GraphML

Edgelist:

Node to Label Mapping : Node 69025: Label 34| Node 17585: Label 10|...
Edge list: [(69025, 96211), (69025, 17585), (17585, 104598), (17585, 18844), (17585, 96211), (96211, 34515)]

Edgetext:

Node to Label Mapping : Node 85328: Label 16| Node 158122: Label ?|...
Edge connections (source node - target node): Node 85328 is connected to Node 158122. Node 158122 is connected to Node 167226.

Adjacency List:

Node to Label Mapping : Node 2339: Label 3| Node 2340: Label ?|...
Adjacency list: 1558: [2339, 2340], 2339: [1558, 2340], 2340: [2339,1558]

GML:

A GML file consists of an unordered sequence of node and edge elements enclosed within [].

The node label information is embedded in the structure, so no separate node-label mapping is needed.

GraphML:graph [
 node [
 id 2339
 label 3
 ]
 node [
 id 2340
 label ?
 ]
 node [
 id 1558
 label 3
 ]
 edge [
 source 2339
 target 1558
 ]
 edge [
 source 2339
 target 2340
 ]
 ]

GraphML:

A GraphML file consists of an XML file containing a graph element, within which is an unordered sequence of node and edge elements.

GraphML:<graphml xmlns=http://graphml.graphdrawing.org/xmlns
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation=http://graphml.graphdrawing.org/xmlns
http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd>
 <graph edgedefault=undirected>
 <node id=2339 label=3 />
 <node id=2340 label=? />
 <node id=1558 label=3 />
 <edge source=2339 target=1558 />
 <edge source=2339 target=2340 />
 </graph>
</graphml>

Graph Structure

We selected real-world citation datasets with differing network properties to explore the significance of graph structure on node classification.

CORA
Citeseer
Pubmed

Sampling Strategy

Ego graph
Forest Fire sampling

Example

Task: Node Label Prediction (Predict the label of the node marked with a ?) given the adjacency list information as a dictionary of type "node: neighborhood" and node-label mapping in the text enclosed in triple backticks. Response should be in the format "Label of Node = <predicted label>".
If the predicted label cannot be determined, return "Label of Node = -1".
```AdjList: {1: [2,3], 2: [3,4], 3: [1,2]}
Node-Label Mapping: {1: A, 2: B, 3: ?} ```

Motif Modality

Encoding graphs as motifs can be separated into two key parts:

the encoding of nodes to their corresponding labels in the graph
the motifs present around the ? (unlabeled) node.

Motif information

Type of Motif	Motif Encoding	Description of Motif
Node-Label Mapping	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|…	Only the node-label mapping is provided (this gives no connectivity information to LLM)
No. of Star Motifs	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: Number of star motifs: 0\|	Star motifs signify centralized networks with influential central nodes, where a central node is connected to others that aren’t interlinked. We pass the count of the star motifs present in the graph.
No. of Triangle Motifs	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: Number of triangle motifs: 6\|	Triangle motifs (triads connecting three nodes) are foundational in social networks, indicating transitive relationships, community structures, and strong social ties. We pass the count of the triads present in the graph.
No. of Triangle Motifs Attached	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: Triangle motifs attached to ? node: \|1893,2034,1531\|, \|1893,1531,429\|	We pass the triangle motifs attached to the ? label, which gives an idea of the influential triads connected to the ? node.
No. of Star Motifs Attached	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: Star motifs connected to ? node: \| \|	We pass the star motifs attached to the ? label, which gives an idea of the influential nodes connected to the ? node.
No. of Star and Triangle Motifs	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: Number of star motifs: 0\| Number of triangle motifs: 6\|	We pass the count of the triads and star motifs present in the graph, to give the LLM an idea of the graph structure.
Star and Triangle Motifs attached	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: Triangle motifs attached to ? node: \|1893,2034,1531\|, \|1893,1531,429\| Star motifs connected to ? node: \| \|	We pass the star motifs and triads attached to the ? label, which gives an idea of the influential nodes and triads connected to the ? node.
No of cliques ? Node is part of	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: Number of cliques in graph: 0\| Node is a part of these cliques: \| \|	We pass the number of cliques in the network, which gives an idea of its clustered nature. We also pass the cliques the ? label is a part of, which gives an idea of the immediate community of the unlabelled node.
No of cliques ? Node is attached to	Node to Label Mapping : Node 1889: Label 4 \| … Node 1893: Label 2\|… Graph motif information: ? Node is attached to these cliques: \| \|	We pass the cliques the ? label is attached to, which gives an idea of the neighboring influential community of the unlabelled node.

Example

Task: Node Label Prediction (Predict the label of the node marked with a ?) given the node-label mapping and graph
motif information in the text enclosed in triple backticks. Response should be in the format "Label of Node = <predicted
label>".
If the predicted label cannot be determined, return "Label of Node = -1".
```Node-Label Mapping: {1: A, 2: A, 3: ?}
Graph-motif information: No of triangles: 1| Triangles
attached to ? Node : [1,2,3]| ```

Image Modality

We first render our sampled graphs with NetworkX and color each node according to its label.

We always color the unlabeled node red and mark it with a ‘?’ label.

Image representation

We examine how different changes to the image representation can improve an image’s human readability. We then use these image encodings to evaluate node classification performance.


Original NetworkX Graph	Node Size Increase Enlarging the nodes	Contrasting Text Color This subfigure contrasts the text labels against the node colors to improve the “human readability” of the labels.


Distinctive Node Colors Choosing a distinct color for nodes of each label makes the individual nodes easier to tell apart.	Node Size increase based on 1-Hop distance Enlarging the neighboring nodes	Aggregate all changes Applying all of the enhancements above

Result

Results Across All Modalities

	Cora	Citeseer	Pubmed
GNN Baselines
GCN	0.7584 ±0.121	0.6102 ±0.087	0.7546 ±0.076
GAT	0.7989 ±0.092	0.6583 ±0.074	0.7490 ±0.060
GraphSage	0.7719 ±0.124	0.6017 ±0.103	0.7193 ±0.076
LLMs + Encoding Modality
Text🥇	0.81 ±0.04 [0.07 ±0.03]	0.75 ±0.05 [0.07 ±0.01]	0.83 ±0.01 [0.08 ±0.01]*
Motif	0.73 ±0.06 [0.06 ±0.01]	0.59 ±0.01 [0.32 ±0.02]	0.77 ±0.06 [0.13 ±0.04]
Image🥈	0.77 ±0.05 [0.04 ±0.02]*	0.71 ±0.09 [0.06 ±0.01]*	0.79 ±0.03 [0.19 ±0.01]

⭐From this table, we find that the text modality is comparable with graph baselines in all three datasets, while the image modality comes a close second. This shows the potential of LLMs to serve as a good foundation model for graphs.

⭐We also observe that the image modality has the lowest denial rate on the smaller CORA and Citeseer datasets, while the larger Pubmed dataset shows a higher denial rate. This may be because the subgraphs sampled there are bigger, which clutters the image modality and leads the LLM to deny a classification.

Qualitative analysis of denial of classification in the Image Modality


a.(Ground = 0) Without additional context or rules for how labels are assigned, it is not possible to accurately predict the label of the red node. All the other nodes are colored according to their label, whereas the ? node is colored red, which confuses the LLM.	b.(Ground = 4) The label cannot be determined with certainty due to the lack of a discernible pattern or rule that associates a node’s color or its connections with its label. Heterophily is present in the graph: 3-?-4	c.(Ground = 2) The label cannot be determined with certainty due to the lack of a clear pattern in the graph and no previous examples of red nodes to infer from. Few-shot examples are needed

We observe that a) the LLM requires context about how labels are assigned to each node. We left this information implicit in our image modality encoder, where every node is colored per its label. The red color is reserved for the ? node, which is unlabeled. This information is not explicit to the LLM. With image (b), there is no clear pattern associating the node’s color with its label, a confusion induced by the great amount of heterophily in the graph. Image (c) brings up the need for few-shot learning, that is, for showing the LLM other graph examples.

Insights from GraphTMI

Easy problems (high homophily, simple graph structure) have higher accuracy with the image modality encoding. Graphs categorized as medium or hard, because they are heterophilous or structurally complex, achieve higher accuracy with the text modality, though the image modality is not far behind.

Text Modality Results

Edge Encoding Function

The figure shows that the 🌟adjacency list, combined with node-label mapping, is the most informative edge encoding function: it strikes the best balance between high accuracy and a low token limit fraction.

accuracy rate(↑)

mismatch rate (↓)

denial rate (↓)

token limit fraction (↓)

Impact of graph structure and graph sampling:

Green is the accuracy rate, blue is the mismatch rate, red is the denial rate, and pink is the token limit fraction. The desired trends for each metric are given in brackets - (↑) accuracy rate, (↓) mismatch rate, (↓) denial rate, and (↓) token limit fraction. The x-axis distinguishes between ego graph sampling (ego) and forest fire sampling (ff) through different bar textures.

The impact on accuracy is small; what is mainly affected is the token limit fraction.
Both sampling strategies perform best on CORA, probably because it is smaller, denser, and more clustered.

Motif Modality Results

We observe that the mean accuracy rate rises when the “triangle and star attached to ? node” motif is added, while the other metrics for the same setting stay low.

This part contains almost no analysis; it merely mentions in passing that “triangle and star attached to ? node” works best.

Image Modality Results

We observe that the “rendering change” offers the highest bump in accuracy, which only goes down when “node size is changed on hop”. This is probably because the LLM is confused by the different node sizes and what they mean. It could be resolved by clarifying the text prompt or by using different shapes to represent nodes at different hop distances. Based on Figure 11, when “all the changes are applied”, the denial rate is at its minimum and the accuracy rate at its maximum (0.7).

Rendering changes do improve accuracy.
When “all the changes are applied”, the denial rate is at its minimum and the accuracy rate at its maximum (0.7).

Conclusion

Representing graphs in the image and motif modalities as LLM input is something we had not seen before.
Limited by compute and the token limit, the maximum hop count was set to 3.

Paper Review | Graph-Guided Reasoning for Multi-Hop Question Answering in Large Language Models

Mon, 20 Nov 2023 00:00:00 +0000

Title: Graph-Guided Reasoning for Multi-Hop Question Answering in Large Language Models

Institution: Korea University, Amazon Alexa AI

Authors: Jinyoung Park, Ameen Patel, Omar Zia Khan, Hyunwoo J. Kim, Joo-Kyung Kim

Arxiv Link: https://arxiv.org/abs/2311.09762

Code Link: None

Date: 2023.11.16

Abstract: Chain-of-Thought (CoT) prompting has boosted the multi-step reasoning capabilities of Large Language Models (LLMs) by generating a series of rationales before the final answer. This paper analyzes the reasoning paths generated by CoT and identifies two issues in multi-step reasoning: (i) Generating rationales irrelevant to the question, (ii) Unable to compose subquestions or queries for generating/retrieving all the relevant information. To address these issues, the authors propose a graph-guided CoT prompting method, which guides the LLMs to reach the correct answer with graph representation/verification steps. This involves leveraging LLMs to construct a “question/rationale graph” using knowledge extraction prompting given the initial question and the rationales generated in previous steps. Then, the graph verification step diagnoses the current rationale triplet by comparing it with the existing question/rationale graph to filter out irrelevant rationales and generate follow-up questions to obtain relevant information. Additionally, CoT paths that exclude the extracted graph information are generated to represent the context information missed from the graph extraction. The proposed graph-guided reasoning method shows superior performance compared to previous CoT prompting and its variants on multi-hop question answering benchmark datasets.

Introduction

The paper proposes an LLM-based, graph-guided reasoning approach for multi-step reasoning problems.

It makes two main contributions:

The reasoning approach described above
An in-context learning method for knowledge triplet extraction that allows variable definitions

An LLM-Based, Graph-Guided Reasoning Approach for Multi-Step Reasoning

The method has four steps:

Build the question graph
Decompose the question and generate intermediate subquestions
Answer the subquestions
Verify the intermediate reasoning

Question graph

The question graph is built by extracting triplets from the question. What “graph-based” means here is essentially converting the question into a symbolic reasoning process that forms part of a graph.

For example, the question “When did the director of film Hypocrite (Film) die?” can be converted into two subquestions serving as two edges in the graph: (“Hypocrite (Film)”, directed by, $1), ($1, death date, $2).

The novelty here is that the paper introduces formal logic into the reasoning process in a form resembling first-order logic (FOL). In this process, the variables ($1, $2) can only stand for individuals, not for propositions or other functions.

The authors introduce variable entities to handle wh-questions.

“Wh-questions” are a type of question in English that begin with “wh-” words, such as “who,” “what,” “where,” “when,” “why,” and “how” (though “how” does not start with “wh,” it is grouped with them functionally).

Intermediate subquestion generation

What distinguishes the subquestion generation here from other work is that it is carried out under the guidance of the graph.

Concretely, the paper filters out question triplets whose subject and object are both variables or both entities, so the process only attends to triplets pairing an entity subject with a variable object, or a variable subject with an entity object.

The reason is easy to see: for a triplet (variable1, relationship, variable2) or a triplet (entity1, relationship, entity2), further decomposition serves no purpose.

Answering the subquestions

Subquestions are shorter and simpler than the original question, so a correct answer is easier to produce. Because LLMs are prone to hallucination and may never have been trained on the documents in question, the output can still be wrong; RAG is one way to improve answer accuracy.

Verifying the intermediate reasoning

The generated rationale triplets are checked for validity by verifying that the answers align with the question graph.

Concretely, at least two components must match between the rationale triplet and some triplet in the question graph. If they do, we accept the rationale that produced the matched triplet; if the triplet matches nothing in the question graph, we reject it.

Take the rationale triplet (“Hypocrite (Film)”, directed by, “Miguel Morayta”) and the question graph (“Hypocrite (Film)”, directed by, $1), ($1, death date, $2). The question graph contains the triplet (“Hypocrite (Film)”, directed by, $1), whose subject is “Hypocrite (Film)” and whose relation is “directed by”, so the rationale is accepted.

For a counterexample, no triplet in the question graph matches the rationale triplet (“Miguel Morayta”, place of death, “Mexico”), so it is rejected.

Experiments

Setup

Datasets:

2WikiMultihopQA consists of complex 2-hop questions that require composition or comparison.
MuSiQue is a more challenging dataset whose 2- to 4-hop questions can be decomposed into simpler ones.
Bamboogle consists of 125 two-hop questions whose supporting evidence comes from Wikipedia.

Models:

Llama-2 13B
Llama-2 70B

Result

The figure above shows part of the experimental results: the proposed method outperforms every baseline.

Paper Review | GraphText — Mapping Graphs to Text

Tue, 14 Nov 2023 00:00:00 +0000

Title: GraphText: Graph Reasoning in Text Space

Institute: Université de Montréal

Authors: Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, Jian Tang

Link: https://arxiv.org/abs/2310.01089

Date: 2023.10.02

Introduction

We have previously covered two papers that use LLMs to solve graph problems. In both, the authors imported prompting techniques from the LLM reasoning literature, ran them on graph problems, and released a benchmark. Fruitful as they were, neither proposed a new way of generating prompts tailored to the nature of graphs. Here the authors draw on the properties of graph structure to propose GraphText, a framework that maps graphs into text space.

GraphText builds a graph-syntax tree for each graph; the tree encapsulates both node attributes and the relations between nodes. Traversing this tree produces a graph-text sequence, which is then handled by a large language model (LLM), turning the graph task into a text generation task.

Beyond its headline advantage — mapping graph space into text space — the paper makes the following contributions:

Conventional GNN approaches require training a separate GNN model for every graph, whereas here a general-purpose LLM suffices via in-context learning
GraphText is a general framework that works on both plain graphs and text-attributed graphs
It is interactive

Methodology

Overview

The method can be summarized by the following formula:

$\tilde{y}_i=h(f_{LLM}(g(G)_i;\phi))$

Notation:

$G$ the graph
$g(G)_i=T_{in}[i]$ the textual description of node $i$
$\phi$ the LLM parameters
$f_\text{LLM}(\cdot;\phi):\mathcal{T}\to\mathcal{T}$ the LLM-based mapping function, whose input and output both live in text space
$h:T_{out}\mapsto\tilde{Y}$ maps the LLM output to a label

Building the graph-syntax tree

It should first be made clear that in this paper a graph consists of two parts: the attributes of its nodes and the relations between them.

The root of the graph-syntax tree is an empty node, the leaves $\tilde{T}_\mathrm{L}$ are the textual attributes of the graph nodes, and the internal nodes $\tilde{T}_{I}$ are natural-language sequences.

Building the graph-syntax tree involves three parts:

Construct the ego-subgraph
Select the leaf nodes
Construct the internal nodes and edges

As shown in panel (c) above, the attributes in the graph are $F={\mathrm{F}_{X}[i],\mathrm{F}_{Y}[i]|\forall v_{i}\in V}$, and the relations are defined by shortest-path distance — center-node, 1st-hop, and 2nd-hop — that is, $R={\boldsymbol{R_{\mathrm{SPD=0}}},\boldsymbol{R_{\mathrm{SPD=1}}},\boldsymbol{R_{\mathrm{SPD=2}}}}$. Taking node 0 as the center, the ego-subgraph is constructed. Next the leaf nodes $\tilde{T_{\mathrm{L}}}={\mathrm{F_{\mathrm{X}}[0]},\mathrm{F_{\mathrm{X}}[1]},\mathrm{F_{\mathrm{X}}[2]}\mathrm{F_{\mathrm{X}}[3]},\mathrm{F_{\mathrm{X}}[4]},\mathrm{F_{\mathrm{Y}}[1]},\mathrm{F_{\mathrm{Y}}[3]}}$, the internal nodes $\tilde{T}_\mathrm{I}=\{\text{“center-node",“}1\text{st-}\text{hop",“2nd-hop”,“label”,“feature”}\}$, and the tree edges are constructed. Finally, traversing the graph-syntax tree yields the natural-language sequence.

Paper Review | ReAct — Combining Reasoning and Acting in LLMs

Fri, 27 Oct 2023 00:00:00 +0000

Title: ReAct: Synergizing Reasoning and Acting in Language Models

Institute: Department of Computer Science, Princeton University

Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao

Link: https://arxiv.org/pdf/2210.03629.pdf

Shared Link: https://mp.weixin.qq.com/s/5DFnutIpNIPRyJ7EUlfiEw

Motivation

CoT is black-box reasoning: both the reasoning process and the knowledge it draws on stay inside the model
Earlier work has explored using pretrained language models to plan and act in interactive environments

Inspiration

Between two actions, humans track their progress in language — “all the ingredients are chopped now, so I should boil the water,” for instance. Drawing on this habit, the authors set out to make an LLM’s reasoning process explicit.

ReAct

ReAct is a framework that lets an LLM combine thought and action.

At time $t$, let the observed information be $o_t\in\mathcal{O}$ and the action be $a_t\in\mathcal{A}$, where $a_t$ follows the policy $\pi(a_t|c_t)$ with $c_t=\left(o_1,a_1,\cdots,o_{t-1},a_{t-1},o_t\right)$. Mapping $c_t\mapsto a_t$ directly, however, is implicit and computationally expensive.

ReAct’s approach is simple: augment the set $\mathcal{A}$ into $\hat{\mathcal{A}}=\mathcal{A}\cup\mathcal{L}$, where $\mathcal{L}$ is the space of thoughts, or reasoning steps. Now $\hat a_t\in\mathcal{L}$ is the action obtained from the context $c_t$, and in the next round $c_{t+1}=\left(c_{t},\hat{a}_{t}\right)$ serves as the context for generating the next action.

Every ReAct cycle consists of acting, thinking, and observing the environment.

The approach does come with a drawback: model capacity has to be large enough, and the paper uses a 540B model.

The authors split tasks into two categories, reasoning-dominant and decision-dominant.

Reasoning-dominant: for tasks where reasoning carries most of the weight, such as multi-step reasoning, the authors use a thought-action-observation loop.
Decision-dominant: for decision tasks involving many operations, by contrast, thoughts only need to surface occasionally — you are in a kitchen, the environment is such-and-such, and you need to find such-and-such.

Paper Review | Reasoning with Language Model is Planning with World Model

Thu, 26 Oct 2023 00:00:00 +0000

Title: Reasoning with Language Model is Planning with World Model

Institute: UC San Diego, University of Florida

Authors: Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu

Link: https://arxiv.org/pdf/2305.14992.pdf

Shared Link: https://mp.weixin.qq.com/s/ZLktf__PCPsMYdX-Ikm_5A

Overview

Prior research has shown that humans possess an internal world model that lets them simulate actions and their effects on the state of the world, enabling deliberate planning for complex tasks such as motor control, imagination, reasoning, and decision making. LLMs, by contrast, can only reason autoregressively. The authors identify three main factors that limit LLM reasoning:

The lack of an internal model that simulates the current state
The lack of a reward mechanism to guide the model’s reasoning
The inability to balance exploration of the future against exploitation of past experience

For the first problem, they borrow the idea of a world model from reinforcement learning and use the LLM itself to simulate the current state.

For the second problem, they design four LLM-based reward schemes.

For the third problem, they turn to Monte Carlo tree search.

World model

The world model here is the notion from reinforcement learning, covering the external environment, the simulation of actions, the outcomes of behavior, and so on. The authors implement it with prompting and an LLM.

Let the current state be $s_{t,t=0,1,2,3...T}$. The process in which the LLM, acting as an agent, infers action $a_t$ from $s_t$ and prompt $ c$ can be written as $a_t \sim p(a|s_t,c)$ . The LLM, now acting as the world model, then predicts the next state $s_{t+1}$ from $ a_t$ and $s_t$ .

Reward

Every reasoning step has to be assessed for feasibility so that the assessment can steer the reasoning. The authors design an evaluation function $r_t=r(s_t,a_t)\in\mathbb{R}$ that returns a reward for the state $s_t$ and action $a_t$ at time $t$.

Reasoning via Monte Carlo tree search

Monte Carlo tree search consists of four main steps:

Selection: pick the leaf node with the highest expected return according to its UCB value
Expansion: expand that leaf node
Simulation: simulate from the expanded leaf node and judge the outcome
Back-propagation: propagate the result back up the tree

The Selection and Simulation stages are worth a closer look.

Selection

Each candidate node gets a UCB value, computed as follows:

$a^*=\arg\max\limits_{a\in A(s)}\left[Q(s,a)+w\sqrt{\frac{\ln N(s)}{N(c(s,a))}}\right]$

The first term, the Q value, is the node’s average reward over past visits; it represents exploitation of past experience.
In the second term, $N$ is the visit count and $c(s,a)$ is the child of this node (the result of applying action $a$ to state $s$); this term represents exploration of the future.
$w$ is the weight of the second term.

The formula gives the Selection policy two tendencies:

It favors nodes that have yielded higher rewards in the past
It favors nodes that have not yet been explored

The UCB formula in Monte Carlo tree search therefore resolves the exploration vs. exploitation dilemma.

Simulation

This stage uses the world model to simulate the future of the current node. Starting from that node, at each node $s_t$ we create an action $a_t$ following an expansion policy and use the world model to predict the next state $a_{t+1}$. Expansion continues until it reaches a terminal state.

See panel $(c)$ of the figure above.

Paper Review | Can Language Models Solve Graph Problems in Natural Language?

Mon, 23 Oct 2023 00:00:00 +0000

Title: Can Language Models Solve Graph Problems in Natural Language?

Institute: Xi’an Jiaotong University, University of Washington

Authors: Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, Yulia Tsvetkov

Date: 2023.5.13

Link: 2305.10037v1.pdf (arxiv.org)

Introduction

This is the first attempt to use large language models to solve graph problems. The authors introduce NLGraph, a natural-language benchmark containing over a thousand basic graph problems, and draw several conclusions from experiments on GPT-3/4. At the end of the paper they propose two methods for improving how LLMs handle graph problems.

In summary, the contributions are:

The first benchmark for handling graph problems in natural language, NLGraph
Experiments on GPT-3/4 with measurements of LLM performance, yielding several regularities
Two methods for improving performance

NLGraph

Put simply, this benchmark is a collection of graphs described in natural language, each paired with its answer.

Graph generation

Generate n nodes and connect each pair of nodes with probability p, so graph generation is controlled by n and p. Individual tasks impose additional constraints.

Tasks

NLGraph contains eight kinds of tasks:

Connectivity: whether two nodes are connected
Cycle detection: whether the graph contains a cycle
Topological sort
Shortest path
Maximum flow
Bipartite graph matching
Hamiltonian path
Simulating a graph neural network

Statistics

The basic version contains 5,902 examples; the extended version contains 29,370.

Experimental Setup

Baselines

The authors use a variety of prompts as baselines:

ZERO-SHOT
FEW-SHOT
CoT
0-CoT
least-to-most (LTM-2023): break a large problem into several subproblems and solve them one by one
self-consistency (SC-2023): ask the same question in multiple ways and take the most consistent result

The authors also use random guessing as a baseline — for yes/no questions the baseline is 50%, and for shortest-path questions the baseline is whether a randomly chosen path is correct.

Models

TEXT-DAVINCI-003, GPT-3/4

Conclusions

LLMs have preliminary reasoning ability on graph problems

On connectivity, cycle detection, and shortest path, LLM performance is markedly above Random, showing that the model is not answering at random and does possess preliminary reasoning ability.
With CoT or CoT+SC prompting, accuracy exceeds the Random baseline by 37.33%–57.82%
On connectivity and shortest path, ZERO-SHOT exceeds Random accuracy by 33.81% and 23.33% respectively
On shortest path, CoT and CoT+SC exceed Random by 22.81%–62.83%

Advanced prompting is not necessarily beneficial

Advanced prompts such as CoT and SC improve accuracy on most problems
But on complex graph problems, advanced prompts such as CoT, CoT+SC, and LTM actually underperform Few-shot
The authors attribute this to the LLM’s inability to generate a chain of thought for complex graph problems

In-context prompting can backfire

On complex graph reasoning problems such as Hamiltonian path and bipartite matching, ZERO-SHOT outperforms Few-shot
The reason may be that on complex problems the LLM cannot extract knowledge from the context, and the context instead dilutes its attention

LLMs are (un)surprisingly brittle

The heading means that the LLM may have misconstrued the reasoning process behind the question.

Although the LLM performs well, it may be arriving at correct answers by exploiting spurious correlations. For example, since higher-degree nodes are mentioned more often and are more likely to be connected, the language model may simply be counting node occurrences rather than actually searching for a path. In other words, on connectivity problems, nodes with especially high degree tend to be connected to one another, which may lead the model to assume that high-degree nodes are connected instead of verifying a path.

The authors therefore constructed two special datasets.

Chain

Split a graph into k parts, each an independent chain, and ask whether the head and tail nodes of each chain are connected. The head and tail have the lowest degree yet are in fact connected.

Clique

Build k dense, independent subgraphs. Pick two nodes from different subgraphs; because the subgraphs are dense, both nodes have high degree, but because the subgraphs are disjoint, the two nodes are not connected.

Unsurprisingly, LLM performance on these two datasets dropped by 40%, confirming the authors’ suspicion.

Two Methods for Improving Performance

Build-a-Graph Prompting (BAG)

The authors reason that mapping the textual description of the graph into an actual conceptual space may help, so they add the sentence “Let’s construct a graph with the nodes and edges first”.

Algorithmic Prompting

Hint that a specific algorithm can be used to solve the problem — for instance, telling the model to use DFS or BFS.

Result

On simple problems, the authors’ two methods improve performance by 3.07%–16.85%. On complex problems, however, there is no improvement.

Summary

Comparison with Talk Like a Graph: Encoding Graphs for Large Language Models

Since “Talk Like a Graph: Encoding Graphs for Large Language Models” is the more recent work, it is referred to below as “the latter”, while the present paper is “the former”.

Benchmark

The two papers’ benchmarks complement each other, with some overlap in problem types

On prompting

The latter adds an encoding component relative to the former

Their prompting differs: the former has LTM and SC prompts, while the latter has BAG prompting (COT-BAG). It is unclear why the latter dropped LTM and SC.

On experiments

The latter adds the effect of model capacity on performance
The latter adds the effect of graph shape
On simple graph tasks, ZERO-SHOT beats ZERO-COT in both papers
Both find that advanced prompting degrades LLM performance on complex graph tasks
Both hold that “mapping the textual description of the graph into an actual conceptual space may help”, but the former merely adds the sentence “Let’s construct a graph with the nodes and edges first”, whereas the latter actually tests the LLM on real-world problems.

On novelty

The former’s novelty lies in the dataset, the experimental design, and its pioneering nature
The latter’s novelty lies in treating encoding as a variable

Questions

Solved

Since I read the latter first and had no prior knowledge of the former’s work, I raised several questions in my write-up of the latter; here are the answers.

How well do LLMs handle shortest-path problems? Is the complexity the same as for cycle problems?
- First, the two tasks are different in nature: one is about finding a correct path, the other is a True/False question
- In this paper, shortest path and cycle detection are not equally hard. Cycle detection is classified as a basic problem with three difficulty levels, whereas shortest path is classified as an advanced problem with two difficulty levels.

Generated

Following the latter, what other variables could be mined for experiments?
- Language?
- The number of nodes and edges, small graphs $\rightarrow$ large graphs
- Representing graphs as adjacency matrices
Classic tree-related problems seem to go unmentioned
- Minimum spanning tree
- Tree diameter
- …
Fine-tuning / LoRA

Paper Review | Talk Like a Graph: Encoding Graphs for Large Language Models

Sat, 21 Oct 2023 00:00:00 +0000

Institute: Google Research

Authors: Bahare Fatemi, Jonathan Halcrow, Bryan Perozzi

Link: https://arxiv.org/abs/2310.02207

Introduction

This paper feeds graphs into an LLM using a variety of natural-language formulations and measures how performance changes.

Background

There are currently four ways in the literature to tackle graph problems with an LLM:

Natural-language input
Embedding
LLM tuning
Hybrid: GNN & LLM

This paper focuses on the first of these. The authors report that encoding graphs according to the patterns they uncover here improves LLM performance on graph reasoning tasks by anywhere from 4.8% to 61.8%.

Contributions

The authors search for the best way to have an LLM handle graph reasoning problems, splitting the question into two aspects:
1. encoding
2. prompt
They introduce a new set of benchmarks, GraphQA

Encoding

Encoding a graph as text breaks down into two parts: encoding the nodes and encoding the edges.

Encoding Nodes:

Integers
English surnames
Characters from film and television
Politicians’ surnames
Letters

Encoding Edges:

Parentheses
Friendship
Co-authorship
Social relations
Arrows
Connection, e.g. Node 0 is connected to nodes 1, 2.

The graph encodings studied in this paper are combinations of the two lists above:

Adjacency: integers + parentheses
Incident: integers + connection
Friendship: English surnames + friendship
Co-authorship: English surnames + co-authorship
SP: characters from film and television + friendship
GOT: characters from film and television + friendship
Social network: English surnames + social relations
Politician: politicians’ surnames + social relations
Expert: letters + arrows

Prompt

Zero-shot prompting (ZERO-SHOT): describe the problem and ask directly
Few-shot in-context learning (FEW-SHOT): provide examples
Chain-of-thought (CoT): provide examples together with the steps that solve them
Zero-shot CoT prompting (ZERO-COT): provide the solution steps but no examples — “Let’s think step by step”
BAG prompting (COT-BAG)

GraphQA

GraphQA covers a range of elementary graph problems:

Edge existence
Node existence
Node counting
Edge counting
Connectivity
Cycle check
Disconnected nodes

Experiments

1. Encoding & Prompt

Model used: PaLM 62B

LLMs perform poorly on basic graph problems
On simple graph tasks ZERO-SHOT beats ZERO-COT; on complex tasks the ordering reverses
The choice of encoding has a large effect on performance
Encoding nodes as integers improves performance

2. Varying Prompt Questions

In the vast majority of experiments, casting the graph question as a real-world question improves model performance

3. Encoding with multiple relation types

Using several relation types to encode a graph does not hurt LLM performance — it actually improves it
The improvement may come from diverse relations being closer to the LLM’s training distribution

4. Model capacity

Model capacity has a pronounced effect on performance

5. Reasoning about disconnected nodes

Accuracy is close to zero: LLMs have almost no ability to reason about disconnected nodes

6. Does graph shape matter?

Graph structure significantly affects LLM performance
Extraneous structure in the graph hurts the LLM: performance is best on path- and star-shaped graphs
Providing examples and reasoning chains improves LLM performance

Conclusion

LLMs perform poorly on basic graph problems
The graph encoding scheme significantly affects LLM performance
The number of model parameters has a pronounced effect on accuracy

Reflections

On models

a. How would 6B or 7B models perform?

b. Encoding schemes adapted to small models

c. Why do no emergent abilities show up on graph problems?
The graphs used in the experiments are mostly small

a. What is the relationship between LLM performance and graph size?

b. Could an input format be designed specifically for large graphs?
Why do LLMs do better on real-world graph problems? Does this phenomenon appear on other problems as well?
Could LoRA or LoRA tuning be used to improve model performance?
Why is there no comparison against embedding-based approaches?
How do LLMs perform on shortest-path problems? Is the complexity the same as for cycle problems?

Paper Review | Language Models Represent Space and Time

Wed, 11 Oct 2023 00:00:00 +0000

Introduction

Are the many astonishing abilities of large language models nothing more than the product of statistics over massive data? Or does a world model exist inside them? Recently, a paper posted on arXiv by two MIT authors, “Language Models Represent Space and Time”, drew wide attention in the research community. Taking the Llama-2 family as their subject, the authors ran experiments on self-constructed spatial and temporal datasets and found evidence for the latter.

The paper delivers six contributions: four major ones and two minor ones.

The six contributions (1–4 major, 5–6 minor):

Produced three spatial datasets and three temporal datasets
Showed that LLMs do represent space and time, and that this depends on the model and the dataset
Showed that these representations are linear
Showed that these representations are robust to changes in the prompt
Robustness checks
Space neurons and time neurons

The first contribution — six new datasets — is a solid amount of work in itself.

Contributions two through four provide indirect evidence that a world model exists inside LLMs. Prior to this, similar experiments had only been carried out on small models.

As for the fifth and sixth experiments, I would say they exist for the sake of rigor and to pad the list of novelties. The sixth in particular — claiming the existence of “space neurons” and “time neurons” — is hard to read as anything other than an attention-grabbing headline.

Before diving in, a few terms are worth clarifying for the reader.

What I call “representation” corresponds to “representation” in the original paper. I believe the authors use the word to refer to the state of the LLM’s internal hidden layers. For example, the claim quoted above that “LLMs represent time and space linearly” corresponds, in the experiments, to the mapping from the LLM’s internal hidden state to the $label$ being linear.
What I call an “entity” corresponds to “entity” in the original paper. Here the word carries the sense of “sample” — one example from a dataset.
What exactly is the somewhat mystical “world model”? If an LLM only predicts the next token likely to follow a stretch of text, then no world model exists inside it. Conversely, if we can find evidence of structured storage of space and time inside the LLM, then we can say a world model does exist.
The authors’ experiments center on “probes” (“probe” in the original). Let the input be $X$, the model output $Y$, and the hidden state $h$. Train a shallow model using $h$ as the feature and $Y$ as the label; that shallow model is the probe.
“Space-time” is shorthand for “space and time”. In the original paper, space appears as both “spatial” and “space”, and time as both “temporal” and “time”.

Paper Walkthrough

Abstract

The rise of large language models (LLMs) has sparked a debate over whether such systems merely perform shallow statistics over massive corpora or genuinely learn a world model. Using Llama-2 as the subject, the authors run experiments on three spatial datasets and three temporal datasets and find evidence for the latter. They find that across model sizes, LLMs represent time and space linearly; that these representations are robust to prompt variations; that they generalize across entity types (for example, cities and landmarks); and they identify “space neurons” and “time neurons”.

Motivation

Although the arrival of LLMs has attracted attention from every quarter, our understanding of what happens inside them remains very limited. Whether an LLM’s operation amounts to statistics over massive data or to something like thinking has been contested since LLMs first appeared.

In this paper, the authors attempt to address the question from the angle of space and time.

Preliminaries

Datasets

The authors created six datasets: three spatial and three temporal. Every entity in each dataset consists of a name and a corresponding location or time. The three spatial datasets are built from place names in the world, the United States, and New York City. The three temporal datasets are built from historical figures, artworks, and news.

Dataset	Examples
World	Los Angeles, Saint Petersburg, the Caspian Sea
USA	Fenway Park, Columbia, Riverside County (in California)
NYC	Borden Avenue Bridge, Trump International Hotel
Figures	Cleopatra, Dante Alighieri, Carl Sagan
Artworks	It, Bohemian Rhapsody
News	omitted

Model

The Llama-2 family.

Activation datasets

Suppose a dataset contains n entities and the LLM has a hidden dimension of $d_{model}$. Feeding the n entities through the LLM and recording the activations at a given hidden layer yields an activation dataset $A\in\mathbb{R}^{n\times d_{model}}$.

Probes

The authors feed place names, person names, and so on as the input $x$ to the model. Using the hidden state $h$ as the $feature$ and the $y$ corresponding to $x$ as the $label$, they train a probe and evaluate its performance on a test set.

Understanding intermediate layers using linear classifier probes. Probing classifiers: Promises, shortcomings, and advances.

Evaluation metrics

$R^2$ and Spearman rank correlation.

Main Experiments and Results

1. Do LLMs internally represent space and time? If so, where — and does it depend on model size?

The authors train probes on the six activation datasets derived from their six self-built datasets, then evaluate them on held-out test sets, obtaining probe accuracy across LLM sizes, layer depths, and datasets.

Five findings emerge from the figure above.

Both spatial and temporal features can be fit by a linear probe
The larger the model, the higher the probe accuracy
Accuracy rises smoothly over the first 50% of the layers
Among the three spatial datasets, NYC performs worst. The authors speculate this is because the NYC map covers a small area and many of its place names are obscure, so the model simply does not know them
On the NYC map, the accuracy gap induced by model size is the most pronounced

To my mind, the steadily rising curve already shows that the model’s internal representation of spatio-temporal information has a certain structural regularity — enough for a probe to learn it. The fourth and fifth findings also lend indirect support to the possibility that a knowledge base or store exists inside the model. If that possibility can be confirmed, LLMs might serve as a form of data compression.

2. Is the LLM’s representation of spatio-temporal information linear?

Here the authors first hedge a little, spelling out how their work differs from earlier efforts. They note that while prior work has argued for the linear representation hypothesis inside neural networks, that work targeted discrete tasks such as binary or multi-class classification, whereas the labels in this paper’s task are continuous.

The authors train both a simple linear probe and a nonlinear MLP probe and evaluate them on the test set; as the figure shows, the accuracy gap between the two is essentially nonexistent. For a mapping $f$, if it is linear, then both linear and nonlinear models can fit it well and the performance gap is small. If it is nonlinear, the linear model fits worse than the nonlinear one. The authors therefore conclude that the mapping from $h$ to $label$ is linear — which is what the paper means when it says “LLMs represent time and space linearly”.

3. Sensitivity to the prompt

This experiment was designed to test whether spatio-temporal representations are sensitive to changes in the prompt.

For the five datasets other than news, the authors designed four prompts.

empty: no additional prompt
coords/when: adding a descriptive, disambiguating prompt such as “What is the latitude and longitude of <place>” or “What was the release date of <author>’s <book>”
baseline: append 10 random tokens
all_cap: all uppercase

For the news dataset, the authors designed four prompts.

empty prompt
add a period
add a descriptive, disambiguating prompt
add a descriptive, disambiguating prompt plus a period

Looking at the figure above, the authors observe the following.

There is almost no accuracy difference between empty and the descriptive, disambiguating prompts
Baseline accuracy is very low — random tokens degrade accuracy substantially
Uppercasing also degrades accuracy, though this is unsurprising since earlier work has reported the same phenomenon
On the news dataset, adding a period improves accuracy substantially

Overall, spatio-temporal representations are fairly robust to the prompt.

Limitations

At the end of the paper the authors mention several limitations.

A discrete encoder could be used to extract the model’s spatio-temporal representations
The model does not recognize every entity in the datasets; in principle those should be filtered out

Afterthoughts

My first reaction on finishing the paper was “is that all?” The contributions and novelty do stand out, but next to the “world model” the media made of it, the framing is hard not to call clickbait. That said, the clickbait was probably not the authors’ intent — some outlets may simply have been chasing the hype.

On contribution: collecting, building, and curating the datasets alone constitutes enough work for a top-tier conference paper, to say nothing of deploying probes on every layer of multiple LLMs. In sheer volume of work, it is far ahead of comparable papers. On novelty, the paper does not innovate much methodologically; it transplants techniques that were routine on small models onto large ones. And while the findings are not as mystical as the headline-chasers suggest, the discovery that LLMs represent space and time linearly is novel enough for a top-tier paper.

On content, this is a textbook “experiment report”: no mathematical proofs, no derivations — just experiment design, execution, and analysis of results.

On language, the word choice and writing are carefully done. As a Chinese reader I found it somewhat demanding, a very different experience from reading papers written by Chinese authors.

All told, this is a paper comfortably above the bar for a top-tier conference, but it falls short of the “proof of a world model” that the headline-chasers advertised.

LLaMA Quantization Report (Part 2)

Fri, 11 Aug 2023 23:39:01 +0000

0. Overview of the Task

Study the LLM.int8 method in depth

Read https://arxiv.org/pdf/2110.02861.pdf
~~Understand the emergent feature phenomenon https://arxiv.org/pdf/2208.07339.pdf~~
~~In the paper “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, what exactly do int8 absmax and zeropoint mean?~~
~~Where did the author’s inspiration for this method come from?~~
Does the conjecture hold that “outliers” exist so that the model can extract features? If it does, how much accuracy would be lost by keeping only the outliers? Could one design a method that retains a certain fraction of the non-outliers, so as to balance the performance loss against the model’s demand for compute resources?

What is the algorithm behind llama.cpp? https://github.com/ggerganov/llama.cpp

8-bit Optimizers via Block-wise Quantization

Author: Tim Dettmers

A method for quantizing the optimizer: without changing any hyperparameters or hurting model accuracy, it quantizes the Adam / momentum state to int8, thereby easing GPU memory pressure during training.

Not closely related to this project.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Author: Tim Dettmers

0. Background

Absmax quantization:

The simplest quantization scheme, taking fp16$\rightarrow$int8.

In short, it maps fp16 values onto int8 integers in [-127, 127].

For example, given the fp16 vector [-100, 0, 50, 100], absmax quantization maps 100 to 127, 50 to 63.5, 0 to 0, and -100 to -98.73.

The trouble is that if the vector contains large-valued outliers, the other values get wiped out by the quantization.

For instance, take the vector [-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]. After int8 quantization and dequantization it becomes [ -0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.00] — clearly most of the information has been lost.

Zeropoint quantization:

The idea is similar to absmax; the difference is that absmax is symmetric while this method is asymmetric.

Taking the vector [-100, 0, 50, 100] as an example, zeropoint quantization maps 100 to 127, 50 to 63.5, 0 to 0, and -100 to -127.

The two methods are equivalent in approach, performance, and accuracy, and neither solves the outlier problem.

LLM.int8:

The author first tried quantizing models with both of the methods above, but accuracy dropped severely, so he set out to find an approach that preserves accuracy while still reducing memory usage.

Through experiments the author found that the matrix $\textbf{X}_{f16}\in\mathbb{R}^{s\times h}$ contains outliers, and that while these outliers occur across all of the sequence dimension s, their distribution along the feature/hidden dimension h follows a clear pattern.

Based on this observation, the author had the idea of separating out the dimensions that contain outliers and computing them in fp16, while computing the remaining dimensions in int8.

How outliers are defined:

The author sets a threshold $\alpha$; any value exceeding $\alpha$ is defined as an outlier, and any dimension containing an outlier must be computed in fp16. Experimentally, the author found that setting $\alpha$ to 6 or above keeps accuracy from degrading. With $\alpha$ set to 6, 99.9% of the dimensions are quantized to int8, while the remaining 0.1% that contain outliers are left unquantized. For a 13B model, no more than 7 dimensions contain outliers. Splitting the matrix costs an extra 0.1% of storage.

1. Emergent Feature

Definition:

The original text reads: “Emergence is a gradual change in a property that suddenly undergoes a phase shift and then changes the quality of its substrate.” In plain terms, the transformer changes gradually, then abruptly undergoes a phase shift, after which a qualitative change takes place — quantitative change leading to qualitative change. What changes over the course of this process is the model’s parameter count.

Findings:

Before explaining this process, the author introduces two things that happen in parallel inside a transformer: the network gradually extracts features, and at the same time it strips away noisy, context-irrelevant ones. An example makes this concrete. To classify cats and dogs, you can apply sharpening, which both accentuates the features that differ between the two animals (eyes, ears) and removes the features they share (color, underlying texture). The transformer works on the same principle. During inference, the transformer discards 99% of the useless information and retains only the 1% that is useful.

Before the qualitative change, different layers use different dimensions to accentuate the important features. After the qualitative change, all layers use the same six dimensions to do so.

In summary:

Below 6.7B parameters, outliers are scattered across different dimensions in individual layers with no discernible pattern.
At 6.7B parameters, the outliers’ values jump sharply and they suddenly appear in all layers, and only in the same six dimensions. This also explains why earlier quantization methods do not carry over to LLMs: from 6.7B onward there are large numbers of outliers, and earlier methods cannot handle them well.
- Values jump sharply: the maximum is 15 at 6B, 60 at 13B, and 95 at 66B.
- Suddenly appear in all layers: below 6.7B, different layers use different dimensions to amplify features. Above 6.7B, all layers work together to amplify them.
- Only in the same six dimensions: this means attention has become concentrated.

The x-axis is the model’s parameter count and the y-axis is the percentage of layers affected by outliers. The figure shows that below 6.7B parameters, outliers appear in only some of the layers. Once the parameter count reaches 6.7B, a qualitative change occurs and they appear in almost every layer.

2. Reflections

Does the emergent feature depend only on the parameter count? No — it is also related to perplexity (ppl).

The left panel shows the emergent feature against parameter count, where the qualitative change is visible. The right panel shows it against ppl, where the growth is monotonically exponential.

The left panel shows the median outlier value against ppl; the median rises sharply when emergence occurs. The right panel shows the number of outliers against ppl; as ppl decreases, the number of outliers grows.

Why do ordinary quantization methods, such as zeropoint and absmax in the int8 family, fail on LLMs? Because as the parameter count grows so does the number of outliers, and ordinary quantization methods suffer severe accuracy loss whenever they encounter outliers.
To summarize the phenomena that accompany the qualitative change:

Outliers appear in all layers
Outliers are distributed regularly across a handful of dimensions
Outlier values rise sharply
The number of outliers also increases

Can the dimensions other than the outliers be removed? No. The author notes in his blog that removing 95% of the parameters of a vision model does not seriously hurt accuracy, but for NLP transformers that figure drops to 30%, and once emergence has occurred it drops to 5%.

LLaMA Quantization Report (Part 1)

Fri, 04 Aug 2023 21:12:03 +0000

0. Overview of the Task

The full task list:

Survey existing LLMs and the background needed to quantize LLaMA-based models
Survey the available quantization tooling
Run comparative quantization experiments: quality and performance
Evaluate the experiments: quality (human? automated? …) and performance (speed, GPU memory before quantization -> after quantization)
Summarize and draw conclusions

What this article covers:

Survey existing LLMs and the background needed to quantize LLaMA-based models
Survey the available quantization tooling

1. The Current LLM Landscape

The dominant way of building on LLMs today is foundation model + prompt engineering: the big internet companies open-source the former, while developers tailor the latter to their own business scenarios.

Among the mainstream foundation models is the LLaMA family released by Meta.

The LLaMA series comes in four sizes: 7B, 13B, 33B, and 65B.

This project uses LLaMA 33B as its base model.

2. Background on Quantization

Q1. What is quantization?

Quantization is the process of approximating the continuous values of a signal with a finite set of discrete values. It is a form of model compression.

Q2. Why quantize?

Serving these models consumes a great deal of GPU memory, and under high concurrency the service may crash outright. Quantizing the model saves both memory and bandwidth.

Q3. What kinds of quantization are there?

Binarization uses bitwise operations to achieve parallel computation.
Logarithmic quantization has no acceleration library implemented on any of the three major platforms, presumably because the speedup it delivers is not significant. Only some special-purpose chips make use of it.
Linear quantization uses uniformly distributed cluster centers, so that a simple linear transformation relates the original floating-point data to the quantized fixed-point data. It is the most widely used approach. Song Han’s best-paper work at ICLR 2016, for example, was the first to propose parameter quantization: a clustering algorithm groups nearby values into a single class so that they can all reuse one value.

None of the three approaches above, however, seems applicable to LLMs.

3. Quantization Methods

1. LLM.int8

Proposed in 2022 by Tim Dettmers, a PhD student at the University of Washington.

Paper: https://arxiv.org/pdf/2208.07339.pdf

GitHub: https://github.com/timdettmers/bitsandbytes

(1) Introduction to int8

Before getting to LLM.int8, we need to understand plain int8 quantization. The core idea of int8 quantization is to map FP16 floating-point numbers into the 8-bit integer range, i.e. [-127, 127]. Suppose we want to quantize the vector $\mathbf{x} : [1.2, -0.5, -4.3, 1.2, -3.1, 0.8, 2.4, 5.4]$. We first find its maximum, $max(\mathbf{x}) = 5.4$, and then compute the quantization coefficient $\alpha = 127/5.4 = 23.5$. Multiplying every element of the vector by $\alpha$ gives the quantized vector $\mathbf{x’} : [28, -12, -101, 28, -73, 19, 56, 127]$.

The section above explains how to quantize a single int8 vector, but the method breaks down as soon as the vector contains outliers (emergent features). For instance, take the vector [-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]. After int8 quantization and dequantization it becomes [ -0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.00] — clearly most of the information has been destroyed. Applying int8 quantization directly to a model therefore degrades its accuracy.

(2) Introduction to LLM.int8

Through experiments, the author observed the following phenomenon (this passage draws on Strong’s article on Zhihu) and proposed a mixed-precision quantization method, LLM.int8.

Outliers appear in virtually every layer, and applying int8 quantization blindly causes a severe accuracy drop. The good news is that these outliers are distributed in a regular pattern. If a 6.7B transformer model has 150,000 outliers per sequence, they will occur in only six feature dimensions (six distinct values of i in X[:, :, i]).

Building on this observation, the author proposed LLM.int8: separate the few dimensions that contain outliers out of the matrix, compute their matrix product in high precision, and quantize everything else. By the author’s account, 99.9% of the dimensions can be handled in int8, while the remaining 0.1% require fp16 multiplication.

The highlighted parts are the outliers; they are left unquantized and computed in fp16. The non-outlier parts are quantized with int8. They are scaled by the row-wise and column-wise absolute maxima $C_x$ and $C_w$, the outputs are quantized to Int8, an 8-bit matrix multiplication is performed, and the result is then dequantized.

(3) Accuracy and Efficiency of LLM.int8

On accuracy:

As shown below, the author experimented with different quantization schemes on models of 125M, 1.3B, 2.7B, 6.7B, and 13B parameters, and evaluated the perplexity of each model before and after quantization. Lower perplexity is better.

Both the symmetric and asymmetric variants of conventional quantization suffer large accuracy drops; compared with the other schemes, models quantized with LLM.int8 lose very little accuracy and are almost on par with fp32.

Note that once the model grows past a critical size, the accuracy of conventional int8 falls off sharply, whereas LLM.int8 barely degrades at all.

On speed:

There are currently two sources of speed measurements: the author’s blog and the paper.

From the blog:

The author compares per-token latency (ms) across model sizes. The experiments show that BLOOM-176B with LLM.int8 is 15% to 23% slower than the fp16 version; see the figure below for details. For the smaller models the slowdown is several times worse.

The author mentions that within a single day he cut the per-token inference latency of T5-3B (bottom row, fourth column from the left) from 312 ms to 173 ms. In other words, the latency in this figure comes mainly from the algorithm, and the team was still optimizing it at the time.

The three models are BLOOM-176B, T5-11B, and T5-3B.

From the paper:

Speed of the 16-bit matrix multiplication in the first hidden layer relative to the baseline, for models of different sizes. Anything below 1.0x is a slowdown.

Experiments were conducted on GPT-3

The two sets of measurements agree on some points and disagree on others. Where they agree:

Both int8 quantization and LLM.int8 favor larger models in terms of speed

Where they differ:

Under the per-token measurement, LLM.int8 is slower than fp16. Under the 16-bit hidden-layer matrix multiplication measurement, inference on small models slows down noticeably while inference on large models speeds up.

On GPU memory

The 8-bit method in this paper makes many previously inaccessible models accessible, in particular OPT-175B/BLOOM

(4) Practice

Using the third-party Linear8bitLt class:

import torch
import torch.nn as nn

import bitsandbytes as bnb
from bnb.nn import Linear8bitLt

fp16_model = nn.Sequential(
 nn.Linear(64, 64),
 nn.Linear(64, 64)
)

[... train the model ...]
torch.save(fp16_model.state_dict(), "model.pt")

int8_model = nn.Sequential(
 Linear8bitLt(64, 64, has_fp16_weights=False),
 Linear8bitLt(64, 64, has_fp16_weights=False)
)
# The has_fp16_weights flag matters a great deal here. It defaults to True, which enables Int8/FP16 mixed precision during training. For inference, however, we care more about saving memory, so we need has_fp16_weights=False.

int8_model.load_state_dict(torch.load("model.pt")) # not quantized yet at this point
int8_model = int8_model.to(0) # quantization happens here
# quantization only takes place once the model is moved onto the GPU

input_ = torch.randn(64, dtype=torch.float16)
hidden_states = int8_model(input_.to(torch.device('cuda', 0)))

Beyond this basic usage, the author also provides sample code targeting LLaMA models.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# maximum length of the generated text
MAX_NEW_TOKENS = 128

# name of the pretrained model to use
model_name = 'decapoda-research/llama-7b-hf'

# the text to process
text = 'Hamburg is in which country?\n'

# turn the text into the input format the model expects (input_ids is the integer sequence the model consumes)
tokenizer = AutoTokenizer.from_pretrained(model_name)
input_ids = tokenizer(text, return_tensors="pt").input_ids

# GPU-related settings
free_in_GB = int(torch.cuda.mem_get_info()[0]/1024**3)
max_memory = f'{int(torch.cuda.mem_get_info()[0]/1024**3)-2}GB'
n_gpus = torch.cuda.device_count()
max_memory = {i: max_memory for i in range(n_gpus)}

# load_in_8bit=True loads the model at 8-bit precision to reduce memory usage
model = AutoModelForCausalLM.from_pretrained(
 model_name,
 device_map='auto',
 load_in_8bit=True,
 max_memory=max_memory
)
generated_ids = model.generate(input_ids, max_length=MAX_NEW_TOKENS)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

(5) Q&A

Why does inference get slower after quantization?

Quantization requires splitting the matrix apart, and that extra work slows inference down.

Why do small models slow down more?

The author has identified the cause: it comes down to the CUDA kernels. In short, int8 cannot saturate the GPU, so it runs at the same speed as fp16, while the extra overhead introduced by quantization drags inference down. In theory a 6B model should see a 20–40% speedup. The author cannot estimate the cost of that overhead, which is more complicated and depends on sequence length and batch size.

Six months ago the author said that speed would improve in later releases.

What is the relationship between the number of outliers and time/space complexity?

Several of the author’s findings bear on this question, but as noted in question 2 the author himself has not fully worked out the overhead issue.

The emergence of outliers is not sudden but gradual, and it is exponentially related to perplexity rather than to model size.
After the phase shift occurs, outliers start to grow rapidly.

“Phase shift” means that outliers suddenly appear in all layers and begin to coordinate with one another.

(6) Summary

The method is fairly new, easy to use, essentially lossless in accuracy, and effective at reducing GPU memory. Whether its runtime speed falls within an acceptable range, however, still needs to be verified experimentally.

(7) References

Blog post by the author, Tim Dettmers: https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/
Blog post by Younes Belkada: https://huggingface.co/blog/zh/hf-bitsandbytes-integration
Video by Bilibili creator 米粒方糖: https://www.bilibili.com/video/BV1Tx4y1d7sG/?spm_id_from=333.880.my_history.page.click&vd_source=3a72dc49e723efef59bcf133fb8fe42e
The original paper: https://arxiv.org/pdf/2208.07339.pdf

2. GPTQ Quantization (work in progress)

Only 20 GB of GPU memory is needed to run LLaMA 33B.

Proposed by Elias Frantar of the Institute of Science and Technology Austria. The paper was published at ICLR 2023.

Paper: https://arxiv.org/pdf/2210.17323.pdf

GitHub (LLaMA version): https://github.com/qwopqwop200/GPTQ-for-LLaMa

(1) The Method

The lineage runs from OBD to OBS to OBQ and finally to GPTQ.

To be completed.

(2) Performance on LLaMA 33B

Experiments were run on an A100.

(3) Usage

python llama.py LLAMA_HF_FOLDER c4 --wbits 4 --true-sequential --act-order --new-eval

(4) Differences from LLM.int8

GPTQ quantizes an fp16 model directly into 4-bit format. LLM.int8, by contrast, reads fp16 weights and loads them into Linear8bitLt layers, with quantization happening when the model is moved to the GPU.

int8_model.load_state_dict(torch.load("model.pt"))
int8_model = int8_model.to(0) # quantization happens here

5. TODO

Study the LLM.int8 method in depth

Read https://arxiv.org/pdf/2110.02861.pdf
Understand the emergent feature phenomenon https://arxiv.org/pdf/2208.07339.pdf
In the paper “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, what exactly do int8 absmax and zeropoint mean?
Where did the author’s inspiration for this method come from?
Does the conjecture hold that “outliers” exist so that the model can extract features? If it does, how much accuracy would be lost by keeping only the outliers? Could one design a method that retains a certain fraction of the non-outliers, so as to balance the performance loss against the model’s demand for compute resources?

What is the algorithm behind llama.cpp? https://github.com/ggerganov/llama.cpp
Flesh out the explanation of how GPTQ works by reading the original paper and blog posts. From Zhihu & arXiv

Learn how to use GPTQ & LLM.int8 from others’ experience. From YouTube

https://www.youtube.com/watch?v=mii-xFaPCrA GPTQ
https://www.youtube.com/watch?v=p9_pbJwvEL8 GPTQ 33B
https://www.youtube.com/watch?v=SvfZO47ShoU GPTQ
https://www.youtube.com/watch?v=bTzM45kaK44 LLM.int8