Introduction

Large language models (LLMs) refer mainly to Transformer-based neural language models that contain tens to hundreds of billions of parameters and are pre-trained on massive text corpora — models such as PaLM, LLaMA, and GPT-4. Compared with PLMs, LLMs are not only larger in scale but also stronger in language understanding and generation, and more importantly, they exhibit emergent abilities that are absent in smaller language models. As the figure above shows, these emergent abilities include (1) in-context learning, where LLMs learn a new task at inference time from a handful of examples presented in the prompt; (2) instruction following, where after instruction tuning LLMs can follow instructions for new types of tasks without any explicit examples; and (3) multi-step reasoning, where LLMs solve complex tasks by decomposing them into intermediate reasoning steps, as in chain-of-thought prompting. LLMs can also be augmented with external knowledge and tools so that they interact effectively with users and their environment, and they can keep improving themselves through feedback data collected from human interaction (for example, via reinforcement learning from human feedback, RLHF).

Large Language Models

Early Pre-trained Neural Networks

In 1999, Bengio et al. developed the earliest neural language models (NLMs), which were competitive with n-gram models. Since then, NLMs based on recurrent neural networks (RNNs) and their variants — such as long short-term memory (LSTM) and the gated recurrent unit (GRU) — have been widely used in many natural language applications, including machine translation, text generation, and text classification. The invention of the Transformer architecture later marked another milestone in the development of neural language models. Transformers offer parallelism that RNNs lack, which makes it possible to pre-train very large language models efficiently on GPUs. These pre-trained language models (PLMs) can then be fine-tuned for many downstream tasks.

We group the early popular Transformer-based PLMs into three main categories according to their neural architecture — encoder-only, decoder-only, and encoder-decoder — and introduce each below.

Encoder-only PLMs:

As the name suggests, encoder-only models contain only an encoder network. They were originally developed for language understanding tasks such as text classification, where the model must predict a category label for the input text. The representative model is BERT, along with its variants such as RoBERTa, ALBERT, DeBERTa, XLM, XLNet, and UNILM.

Overall pre-training and fine-tuning pipeline of BERT

BERT (Bidirectional Encoder Representations from Transformers) is one of the most widely used encoder-only models. It consists of three modules: (1) an embedding module that converts the input text into a sequence of embedding vectors, (2) a stack of Transformer encoders that turns those embedding vectors into contextual representation vectors, and (3) a fully connected layer that maps the representation vectors (at the final layer) into one-hot vectors. BERT is pre-trained with two objectives: masked language modeling (MLM) and next sentence prediction. A pre-trained BERT model can be fine-tuned by adding a classifier layer for many language understanding tasks, ranging from text classification and question answering to language inference. The overall pre-training and fine-tuning pipeline of BERT is shown in the figure above.

Decoder-only PLMs

Among decoder-only PLMs, the most representative are GPT-1 and GPT-2, developed by OpenAI, which paved the way for GPT-3.5 and GPT-4.

Overview of the GPT pre-training and fine-tuning steps

GPT-1 was the first to demonstrate that training a decoder-only generative pre-training (GPT) model in a self-supervised fashion on an unlabeled text corpus, and then applying discriminative fine-tuning on each specific downstream task, yields good performance across a wide range of natural language tasks.

GPT-2 showed that when trained on WebText — a large dataset composed of millions of web pages — a language model can learn to perform specific natural language tasks without any explicit supervision.

Self-supervised learning: a subset of unsupervised learning. In self-supervised learning, the model uses an unlabeled dataset itself as training data. For instance, during GPT training the model predicts the next token, and this prediction task requires no externally annotated data, because the next token comes directly from the input data itself.

Encoder-Decoder PLMs

Prior research has shown that nearly every natural language processing task can be cast as a sequence-to-sequence generation task. An encoder-decoder language model is therefore a unified model by design, since it can perform all natural language understanding and generation tasks. The most representative models in this category are T5, mT5, MASS, and BART.

LLM Families

Large language models (LLMs) refer mainly to Transformer-based PLMs with billions to hundreds of billions of parameters. Compared with the PLMs above, LLMs are not only larger in size but also stronger in language understanding and generation, and they exhibit emergent abilities that smaller models lack. Next, we review three LLM families: GPT, LLaMA, and PaLM.

The GPT Family

Generative Pre-Training (GPT) is a family of decoder-only models developed by OpenAI. The family includes GPT-1, GPT-2, GPT-3, InstructGPT, ChatGPT, GPT-4, CODEX, and WebGPT. While the earlier GPT models, such as GPT-1 and GPT-2, are open source, the more recent ones, such as GPT-3 and GPT-4, are closed source and accessible only through an API.

• GPT-3: a pre-trained autoregressive language model with 175 billion parameters. GPT-3 is widely regarded as the first LLM, because it is not only far larger than previous PLMs but also the first to display emergent abilities not observed in smaller PLMs. GPT-3 demonstrated the emergent ability of in-context learning, meaning that it can be applied to any downstream task without any gradient update or fine-tuning.

• CODEX: a general-purpose programming model that can parse natural language and generate code in response. CODEX is a descendant of GPT-3, fine-tuned for programming applications on a code corpus collected from GitHub.

• WebGPT: another descendant of GPT-3, fine-tuned to answer open-ended questions using a text-based web browser, helping users search and browse the web.

• InstructGPT: proposed to make LLMs follow human instructions, InstructGPT is fine-tuned with human feedback (RLHF) so that it stays aligned with user intent across a wide range of tasks.

  
  

  Overview of RLHF

  RLHF: starting from a set of prompts written by labelers together with prompts submitted through the OpenAI API, a dataset of labeler demonstrations of the desired model behavior is collected. GPT-3 is then fine-tuned on this dataset. Next, a dataset of human-ranked model outputs is collected, and the model is further fine-tuned with reinforcement learning. This approach is called reinforcement learning from human feedback (RLHF).

• ChatGPT (GPT-3.5): the most significant milestone in LLM development is ChatGPT, released on November 30, 2022. ChatGPT is a chatbot that lets users steer a conversation to accomplish a variety of tasks, such as answering questions, seeking information, and summarizing text. ChatGPT is powered by GPT-3.5 and is a sibling model to InstructGPT.

Performance of GPT-4 on academic and professional exams compared with GPT-3.5

• GPT-4: the newest and most powerful LLM in the GPT series. Released in March 2023, GPT-4 is a multimodal LLM that accepts both images and text as input and produces text output. Although it still falls short of humans in some of the most challenging real-world scenarios, GPT-4 achieves human-level performance on a variety of professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers, as shown in the figure above. Like earlier GPT models, GPT-4 is first pre-trained to predict the next token on a large text corpus, and then fine-tuned with RLHF to align its behavior with what humans expect.

The LLaMA Family

LLaMA is a series of large language models released by Meta. Unlike the GPT models, LLaMA models are open source.

• LLaMA-1: the first batch of LLaMA models was released in February 2023, with parameter counts ranging from 7B to 65B. These models were pre-trained on trillions of tokens collected from publicly available datasets. LLaMA adopts the Transformer architecture of GPT-3 with a few minor architectural modifications.
• LLaMA-2: in July 2023, Meta, in collaboration with Microsoft, released the LLaMA-2 family, which includes base language models as well as chat models fine-tuned for dialogue, known as LLaMA-2 Chat.

The PaLM Family

The PaLM (Pathways Language Model) family was developed by Google.

• The first PaLM model was announced in April 2022. It was pre-trained on 6,144 TPUv4 chips using the Pathways system, contains 540 billion parameters, and was open-sourced in March 2023.
• PaLM-2 is trained with a mixture of objectives, unlike its predecessor PaLM. Through extensive evaluation on English, multilingual, and reasoning tasks, PaLM-2 substantially improves downstream task performance across model sizes while demonstrating faster and more efficient inference than PaLM.

Others

• Mistral-7B: a 7B-parameter language model engineered for superior performance and efficiency. Mistral-7B outperforms LLaMA-2-13B on all evaluation benchmarks and surpasses LLaMA-34B on reasoning, mathematics, and code generation. The model uses grouped-query attention for faster inference, combined with sliding window attention to handle sequences of arbitrary length effectively while lowering inference cost.

  
  
  The BLOOM architecture

• BLOOM: Scao et al. proposed BLOOM, an open-access language model with 176B parameters, designed and built collaboratively by hundreds of researchers. BLOOM is a decoder-only Transformer language model trained on the ROOTS corpus, which spans hundreds of sources (59 in total) covering 46 natural languages and 13 programming languages. Its architecture is shown in the figure above.

• GLM: a large bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It aims to open-source a 100B-scale model at least on par with GPT-3 (davinci).

• Gemini: the Gemini team introduced a new family of multimodal models supporting a 32k context length. The Gemini family comes in three versions: Ultra for highly complex tasks, Pro for enhanced performance and deployability at scale, and Nano for on-device applications.

How to Build an LLM

Architecture Choices

As described above, the most widely used LLM architectures today are encoder-only, decoder-only, and encoder-decoder.

Here we first revisit the Transformer architecture. It was originally proposed in Attention Is All You Need and designed from the outset to exploit GPUs for efficient parallel computation. At its core is the attention mechanism, which, compared with recurrence and convolution, (1) uses GPUs more effectively and (2) captures long-range contextual information. The attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.

Encoder-Only

In an encoder-only architecture, the attention layer has access to all the words in the initial sentence at every stage. Pre-training for these models usually involves corrupting a given sentence in some way (for example, masking) and asking the model to reconstruct it. Encoder-only architectures are well suited to tasks that require understanding of the complete sequence, such as sentence classification, named entity recognition, and extractive question answering.

Decoder-Only

In a decoder-only architecture, at every stage any given token attends only to the tokens preceding it. These models are sometimes called autoregressive models. Their pre-training is usually formulated as predicting the next token in the sequence (next token prediction). Decoder-only architectures suit tasks that involve text generation. The GPT models are the outstanding example of this model class.

Encoder-Decoder

Encoder-decoder architectures use both an encoder and a decoder, and are sometimes called sequence-to-sequence models. At each stage, the encoder’s attention layer can access all the words in the initial sentence, whereas the decoder’s attention layer can only access the tokens preceding a given token in the input. Encoder-decoder models suit tasks that generate a new sentence from a given input, such as summarization, translation, or generative question answering.

Data Cleaning

Data quality is critical to the performance of the language model trained on it. Data cleaning techniques such as filtering and deduplication have been shown to have a significant impact on model performance.

Data Filtering

Data filtering aims to improve the quality of the training data and thus the effectiveness of the trained LLM. Common data filtering techniques include:

1. Denoising: for example, removing false information.
2. Handling outliers
3. Class imbalance: balancing the distribution of classes in the dataset.
4. Text preprocessing: removing stop words, punctuation, and the like.
5. Handling ambiguity: dealing with vague or contradictory data.

Deduplication

Deduplication means removing duplicate instances or repeated occurrences of the same data. Duplicate data points can introduce bias during model training and reduce diversity. Some studies show that deduplication improves the model’s ability to generalize to new, unseen data.

Tokenization

Tokenization is the process of converting a text sequence into smaller pieces, namely tokens.

Model Pre-training

Pre-training is the first step in the training pipeline of a large language model. During pre-training, an LLM is typically trained in a self-supervised manner on a large amount of (usually) unlabeled text. The two most common pre-training methods are next token prediction and masked language modeling (MLM).

• Next token prediction: predict the next token in a sentence token by token from left to right (or right to left), establishing dependencies between tokens. Suitable for text generation tasks such as machine translation and text generation.
• Masked language modeling: in MLM, the model learns the relationships between tokens by predicting randomly masked words in a sentence, which trains it to understand the contextual meaning of words. MLM is better suited to understanding tasks such as text classification and named entity recognition.

Fine-tuning and Instruction Tuning

• Fine-tuning usually serves two purposes: (1) adapting the base model to different tasks, and (2) injecting new knowledge.

• Beyond fine-tuning, we also want an LLM to understand and follow natural language instructions from humans. That is where instruction tuning comes in.

Alignment

AI alignment is the process of steering an AI system toward human goals, preferences, and principles. Sometimes, in the course of token prediction, large language models exhibit dangerous behavior. For example, they may generate undesirable, harmful, misleading, or biased content.

At this stage, two popular approaches are RLHF (reinforcement learning from human feedback) and RLAIF (reinforcement learning from AI feedback).

• RLHF: RLHF uses a reward model to learn alignment from human feedback. The tuned reward model is able to evaluate different outputs and score the outputs of the model being trained according to the alignment preferences given by humans.
• RLAIF: RLAIF, on the other hand, connects a pre-trained, well-aligned model directly to the LLM being trained and helps it learn from this larger, better-aligned model. This resembles a generalized form of knowledge distillation.

Decoding Strategies

Decoding refers to the process of generating text with a pre-trained large language model. Given an input prompt, the tokenizer converts each token in the input text into its corresponding numerical representation. The language model then takes these numerical token representations as input and predicts the next most likely token. Finally, the model produces logits, which are turned into probabilities by a softmax function. A variety of decoding strategies have been proposed; among the most popular are greedy search, beam search, top-K, and top-P.

Greedy Search

Greedy search picks the most likely token as the next token in the sequence at each step, discarding all other potential options. As with many greedy algorithms, this approach may sacrifice the consistency and coherence of the text. It only considers the most likely token at each step, without regard for the quality of the generated sequence as a whole.

Beam Search

Unlike greedy search, which only considers the single most likely next token, beam search considers the N most likely tokens, where N is the number of beams. The process repeats until a predefined maximum sequence length is reached or an end-of-sequence token appears. At that point, the token sequence (also called a beam) with the highest overall score is selected as the output. For example, with a beam size of 2 and a maximum length of 5, beam search must recursively generate and evaluate $2^5 = 32$ possible sequences. It therefore requires considerably more computation than greedy search.

Top-k

Top-k sampling uses the probability distribution produced by the language model to randomly select a token from among the k most likely options.

Suppose we have six tokens (A, B, C, D, E, F) and set k=2, with P(A)=30%, P(B)=20%, and P(C)=P(D)=P(E)=P(F)=12.5%. Under top-k sampling, tokens C, D, E, and F are ignored, and the model outputs A with probability 60% and B with probability 40%. This approach ensures that we prioritize the most likely tokens while still introducing a degree of randomness.

In this method, randomness is usually introduced through the notion of temperature. The temperature T is a parameter from 0 to 1 that affects the probabilities produced by the softmax function, as the following formula shows: $$ softmax(x_i)=\frac{e^{x_i/T}}{\sum_je^{x_j/T}} $$ As the formula indicates, a low temperature makes the probability distribution sharper, which means higher-probability tokens become even more prominent, so the generated text is more deterministic and less creative; a high temperature flattens the distribution, giving low-probability tokens more of a chance and thereby increasing the diversity and creativity of the text.

Top-p

Top-p sampling, also known as nucleus sampling, takes a slightly different approach from top-k sampling. Instead of selecting the top k most likely tokens, nucleus sampling chooses a threshold p such that the probabilities of the selected tokens sum to more than p. In other words, under top-p sampling the model examines the most likely tokens in descending order and adds them to a list one by one until their cumulative probability exceeds the threshold p.

Using LLMs

Limitations of LLMs

• No state or memory: on their own, they cannot even remember what was sent to them in a previous prompt.
• Answers are random or probabilistic: send the same input to an LLM several times and you will very likely get different replies.
• Outdated information
• High training cost
• Hallucination

Here we discuss only hallucination.

Hallucination in LLMs has been described as “generated content that is nonsensical or unfaithful to the provided source,” and it falls into two categories:

1. Intrinsic hallucinations: the hallucination directly conflicts with the source material, being factually inaccurate or logically inconsistent.
2. Extrinsic hallucinations: content that cannot be verified against the source, containing speculative or unverifiable elements.

The impact of hallucination depends heavily on the context and the task. In creative endeavors such as writing poetry, for instance, hallucination may be considered acceptable or even beneficial.

Recent work on LLM hallucination, such as instruction tuning and reinforcement learning from human feedback (RLHF), attempts to steer LLMs toward more fact-grounded output, but their inherently probabilistic nature, and the limitations that come with it, remains.

Prompt Engineering

When working with an LLM, the prompt is the text input provided by the user to guide the model’s output. A prompt typically includes an instruction, a question, input data, and examples.

Typical prompting methods include:

1. Chain of Thought (CoT): the famous CoT, which asks the LLM to think step by step so that it produces a better output. It comes in two main forms:
   1. Zero-shot CoT: append “Let’s think step by step” after the question.
   2. Manual CoT: requires humans to supply step-by-step reasoning examples as templates.
   3. Of the two, Manual CoT is the more effective, but constructing chains of thought by hand is difficult and error-prone. Auto CoT does not suffer from this problem; see Auto CoT — Generating Chain-of-Thought Automatically via Clustering for details.
2. Tree of Thought (ToT): building on the idea of CoT, ToT can be unfolded into multiple “trees of thought,” where each branch represents a different reasoning path, that is, a different CoT.
3. Self-Consistency: given a prompt, the LLM generates not one answer but several. These candidate answers are then evaluated to determine how consistent they are with one another. Finally, the answer that is most consistent across all of them is chosen as the final output.
4. Reflection: prompts the LLM to evaluate and possibly revise its output by reasoning about the correctness and coherence of its own answer.
5. Expert Prompting: prompts the LLM to take on the role of an expert and answer accordingly, providing high-quality, well-informed answers.

RAG

As the figure above shows, RAG extracts the query from the prompt and uses it to retrieve relevant information from an external knowledge source (such as a search engine or a knowledge graph). The retrieved information is then added to the prompt and fed to the LLM, which produces the final answer.

LLM Agents

The notion of AI agents has a long history. An AI agent is typically an autonomous entity that perceives its environment through sensors, makes judgments based on the current state, and acts.

In the context of large models, an agent is a system built on an LLM that can carry out specific tasks autonomously. Such agents are designed to interact with users or their environment and to make decisions based on the input and the intended goal of the interaction. The core characteristics of an agent are:

1. The ability to access and use tools.
2. The ability to make decisions autonomously.

Representative work in this area includes ReAct and LATS; interested readers may consult my earlier post ReAct — An LLM Reasoning Paradigm Combining Reasoning and Acting.

Datasets

Challenges

Smaller and More Efficient Models

A current research trend is to propose small language models (SLMs) as a more economical alternative to LLMs, particularly for tasks that do not require such large models.

New Architectural Paradigms Beyond Attention

One important class of models proposed after the attention-based models is state space models (SSMs). Although the concept of state space models has a long history in machine learning, in the context of language models SSM usually refers to the more recent structured state space model architecture, S4 for short. Some recent models in this category include Mamba, Hyena, and Striped Hyena.

On another front, several attention-compatible architectural mechanisms have been gaining ground recently, such as mixture of experts (MoE). Notably, MoEs can serve as a component of any architecture, whether or not it is attention-based.

Multimodality

Several well-known multimodal LLMs already exist, including LLaVA (a vision-language model), LLaVA-Plus (an extension of LLaVA), GPT-4, Qwen-VL, and Next-GPT.

Techniques for Using and Augmenting LLMs

Prompting and RAG, for example.

LLM Safety

Privacy and bias, for example.