Researchers have introduced BLOOM-WILT, a novel auditing pipeline designed to more effectively uncover rare behaviors in large language models (LLMs). Deployed models often exhibit behaviors that are not surfaced during standard testing due to the sheer volume of real-world interactions compared to simulated evaluations. Automated auditors offer scalability and flexibility but often suffer from sample inefficiency, meaning they require many interactions to find specific behaviors. BLOOM-WILT aims to address this by eliciting natural, multi-turn instances of these rare behaviors without incurring training costs or needing access beyond the model's next-token distribution.
The BLOOM-WILT pipeline operates in two main stages. On the input side, an auditor model refines its conversational strategy over multiple rounds, learning from previously scored interactions. This allows the auditor to adapt its approach to better elicit the target behaviors. On the output side, BLOOM-WILT adaptively reweights the target LLM's decoding process. It uses the model's own probability distribution, conditioned on an elicitation prompt, to prioritize generations that are relevant to the behavior being investigated. This means that generations associated with the target behavior are sampled more readily than others that the model might otherwise find equally probable when unprompted.
In evaluations across four target models and eight distinct behaviors, BLOOM-WILT demonstrated significant improvements over a baseline auditor. The pipeline outperformed the baseline in 30 out of 32 settings and altered previous rankings of model safety. For instance, when attempting to elicit self-harm encouragement from the Qwen3.5-4B model, BLOOM-WILT successfully raised the average behavior presence from 51% to 100%. This performance surpassed other elicitation methods tested within the same pipeline at matched computational resources, without reducing the output probability below the baseline's level.
The challenge of auditing LLMs is a growing concern as their deployment becomes more widespread. Traditional evaluation metrics have proven insufficient for diagnosing complex issues like bias, as noted in research on hiring experiments where adverse impact ratios were too imprecise to draw strong conclusions. The black-box nature of many LLMs, especially those offered as cloud services, further complicates auditing efforts, as access to model parameters is restricted. Existing methods often rely on manually crafted prompts or single-turn interactions, which struggle to explore the vast input space of LLMs or identify rare but potentially harmful outputs.
Previous work has explored various approaches to LLM auditing. Some methods focus on eliciting specific knowledge that models may possess but not explicitly verbalize, using techniques like prefill attacks or logit manipulation. Others propose frameworks for using human-in-the-loop systems to generate probes and verify responses, aiming for scalability and transparency. Research has also investigated eliciting behaviors in multi-turn conversations, identifying different families of methods based on their interaction strategies with the target model, such as using prior knowledge, offline interactions, or learning from online interactions.
The BLOOM-WILT approach distinguishes itself by its focus on sample efficiency and its ability to elicit natural, multi-turn instances of rare behaviors. By adaptively reweighting the target model's decoding, it guides the LLM towards generating instances of the behavior of interest, effectively making the rare common within the auditing context. This method avoids the need for costly fine-tuning or retraining of the target model, making it a practical tool for auditing deployed LLMs. The success in uncovering behaviors like self-harm encouragement highlights the potential of BLOOM-WILT to improve the safety and reliability of large language models.