Title: Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

Institute: Princeton University

Authors: Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, Danqi Chen

Arxiv Link: https://arxiv.org/abs/2310.06987

Date: 2023.10.10

Abstract

The authors propose a method called generation exploitation, which attacks an LLM by manipulating its system prompt and decoding hyperparameters. The attack raises the attack success rate (ASR) from 0% to over 95%. Along the way they also propose using a fine-tuned BERT-BASE-CASED model to evaluate the toxicity (harmfulness) of a model’s responses. In light of this catastrophic failure of alignment, they put forward an alignment approach named generation-aware alignment.

Motivation

The authors observed that the field typically evaluates the alignment of open-source LLMs under the default decoding method alone, such as greedy decoding. Once a malicious user changes the decoding hyperparameters, alignment may fail.

For instance, prior alignment evaluations on LLAMA2 used the hyperparameters $p = 0.9$ and $\tau = 0.1$, always with a system prompt.

• Section 4.1 of the LLAMA2 paper states: “For decoding, we set temperature to 0.1 and use nucleus sampling with top-p set to 0.9”;
• Appendix A.3.7 of the LLAMA2 paper notes: “when collecting generations, we append a system prompt prior to the evaluation prompt.”

Reasonable as this practice may be in principle, it can lead to serious consequences once users adopt a broader range of decoding strategies.

Do Broader Decoding Strategies Lead to Jailbreaks?

Experimental Setup

To investigate this, the authors break the question into two sub-questions and study each separately.

1. Does the presence or absence of a system prompt affect jailbreaking?
   1. The authors split the experiments into two groups according to whether the default system prompt is used, and compare them to gauge the effect of the system prompt on jailbreaking.
2. Do decoding hyperparameters affect jailbreaking?
   1. Temperature sampling, with $\tau$ varied from 0.05 to 1 in steps of 0.05, giving 20 configurations in total.
   2. Top-K sampling restricts the candidate pool to the K most probable tokens, and the next token is sampled from that pool alone. The authors vary K over {1, 2, 5, 10, 20, 50, 100, 200, 500}, giving 9 configurations.
   3. Top-p sampling takes the smallest set of tokens whose cumulative probability exceeds p. Varying p from 0.05 to 1 in steps of 0.05 gives 20 configurations.

For ASR evaluation, the authors use two methods.

1. String matching: determine alignment by matching a small set of refusal phrases as substrings, such as “sorry” or “I cannot create”; if none of these strings is present, the attack is classified as successful.
2. Evaluation with a trained classifier: string matching is quite limited, since LLM outputs vary enormously in form. The authors therefore use a fine-tuned BERT-BASE-CASED model to assess the toxicity (harmfulness) of the model’s response. The classifier sorts an input string into two classes: correctly aligned and misaligned.

For data, the authors use AdvBench and their own MaliciousInstruct.

Results

1. Removing the system prompt increases the attack success rate: as the figure above shows, the simple act of removing the system prompt markedly increases the attack success rate (ASR), typically by more than 10%. We observe that the presence of a system prompt plays a crucial role in keeping outputs aligned, especially for models that have not undergone safety tuning. For those models, removing the system prompt can raise ASR by well over 50%. Yet even for a model with explicit safety alignment — the LLAMA2 chat model — ASR still rises once the system prompt is removed.
2. Diverse decoding strategies increase the attack success rate: as the figure above shows, performance can indeed be improved by exploring a variety of decoding strategies. In fact, every model except LLAMA2-CHAT exceeds 90% ASR. This stark contrast in performance highlights a catastrophic failure of alignment in the evaluated models.
3. The limits of evaluating alignment with a fixed decoding strategy: notably, a fixed decoding configuration on the LLAMA-CHAT models generally yields an ASR below 30%, whereas varying the configuration can push ASR above 80%. These findings underscore that assessing model alignment with a fixed decoding strategy clearly understates the real risk.

The figure above reports the decoding strategy to which each model is most vulnerable, showing that different models are vulnerable to different decoding strategies.

By human judgment, half of the misaligned outputs are harmful.

Pushing the Attack Further

The authors use two simple strategies to raise ASR on the safety-aligned LLAMA2-CHAT models to 95%.

Multiple Samples

Given the nondeterministic nature of sampling-based decoding, increasing the number of sampling runs is an intuitive way to strengthen the attack.

As the figure above shows, drawing multiple samples for each decoding configuration substantially raises the ASR of the LLAMA2-CHAT models to > 95% — specifically, 5 samples for LLAMA2-7B-CHAT and 4 samples for LLAMA2-13B-CHAT reach the 95% level.

Penalty- and Constraint-based Decoding

Another effective strategy is to introduce constraints into the decoding process. Adding a length penalty at decoding time, for example, lifts the attack success rate from 83% to 92%. Adding two further constraints — banning certain “bad words” (such as “sorry”, “unethical”, “illegal”) and forcing certain words to appear (such as “sure”, “ok”) — pushes the attack success rate to 100%.

Comparison with GCG

The authors’ generation exploitation attack consistently outperforms the state-of-the-art attack across two models, two benchmarks, and two metrics of attack success. Notably, their method is also 30x faster than GCG: launching their attack with a single prompt on LLAMA2-7B-CHAT with one NVIDIA A100 GPU takes about 3 minutes, whereas GCG needs roughly 1.5 hours for the same task (500 steps, batch size 512).

A Generation-aware Alignment Method

This section introduces generation-aware alignment, a method designed to make a model more resistant to generation exploitation attacks — attacks that can cause severe alignment failures. The method works as follows:

1. Overview: under generation-aware alignment, the system proactively collects model outputs produced by a variety of decoding strategies and uses them in the alignment process. A decoding strategy here refers to how the model generates a text sequence from the input prompt.

2. The procedure in detail:

   • For a language model $f_\theta$ and a given prompt $\mathbf{p}$, the model generates an output sequence $\mathbf{r}$ via a sampling method $h$. Here $h$ is a decoding strategy that maps the language model’s probability distribution, conditioned on the prompt $p$, to a token sequence over the vocabulary $\mathcal{V}$.
   • For each prompt $\mathbf{p}$, the system collects $n$ responses from different decoding strategies, forming the response set $\mathcal{R}^{\mathbf{p}}$.
   • All responses are then split into two groups: the aligned response set $\mathcal{R}{a}^{\mathbf{p}}$ and the misaligned response set $\mathcal{R}{m}^{\mathbf{p}}$.

3. Optimization objective: generation-aware alignment minimizes the following objective, which comes from the “chain of hindsight” method: $$ \mathcal{L} = \frac{1}{|\mathcal{P}|} \sum_{\mathbf{p} \sim \mathcal{P}} \frac{1}{|\mathcal{R}{m}^{\mathbf{p}}| \times |\mathcal{R}{a}^{\mathbf{p}}|} \sum_{(\mathbf{r}m, \mathbf{r}a) \sim \mathcal{R}{m}^\mathbf{p} \times \mathcal{R}{a}^\mathbf{p}} - \log \mathbb{P}(\mathbf{r}_a \mid \mathbf{p}, \mathbf{s}_a) - \log \mathbb{P}(\mathbf{r}_m \mid \mathbf{p}, \mathbf{s}_m) $$ where $\mathbf{s}_a$ and $\mathbf{s}_m$ denote “an aligned answer” and “a misaligned answer” respectively.

In plain terms, the procedure can be summarized as follows:

1. Collect answers: imagine asking the model a question and, instead of a single answer, getting many answers generated in different ways (the decoding strategies).

2. Classify the answers: next, we sort these answers into two groups, one of correct answers and one of incorrect answers.

3. Train the model: finally, we use this information to train the model, so that it learns to better distinguish which kinds of answers are correct and which are not.

Results

As the figure above shows, generation-aware alignment lowers the original model’s ASR from 95% to 69%. By contrast, aligning on samples drawn with a fixed decoding configuration leaves a higher final ASR of 88%.

Notably, among the three decoding strategies exploited by the attacker, sampling with diverse decoding configurations gives the largest gain in the model’s robustness to temperature attacks.

Open-source vs. Closed-source

All the experiments above were run on open-source models. Here the authors turn to a closed-source one, gpt-3.5-turbo.

Proprietary models are harder to attack. We observe a markedly lower ASR (7%) when attacking the proprietary model (see the figure above), compared with an ASR that usually exceeds 95% on open-source models.

Two key factors explain the gap.

1. Proprietary models generally deploy content filters that detect and handle potentially harmful prompts and outputs. We observed, for instance, that 9 of the tested prompts had unusually long request times due to the content filter and were eventually terminated by a timeout error.
2. Proprietary models are typically owned by organizations with the resources to conduct extensive red-teaming, which makes those models more resistant to attack.

Conclusion

The authors introduce a new dataset, MaliciousInstruct; a way to evaluate the toxicity of model responses; an attack that manipulates decoding hyperparameters, called generation exploitation; and an alignment strategy, generation-aware alignment.

Generation exploitation requires none of the sophisticated machinery involved in optimizing adversarial prompts; it focuses instead on manipulating different generation strategies. Notably, using 30x less computation than the best existing attack, it misaligns 11 models at rates as high as 95%.