Open-Weight Safety

AI safety research has largely focused on frontier models developed and deployed as closed systems. The resulting interventions often depend on technical controls that do not carry over to open models, or restrictions incompatible with the reasons people develop and use them. Open models need safety methods designed for how they are built, shared, modified, and deployed. We develop interventions that address these technical requirements while respecting the principles of open-source development.

Pretraining data filtering

Deep Ignorance results: filtered models have comparable general benchmark scores and lower scores on the targeted biological-knowledge evaluation, including after adversarial fine-tuning.
General benchmark performance (left) and targeted biological-knowledge performance during adversarial fine-tuning (right). Stronger filtering reduced the latter while preserving performance on the general evaluations. Deep Ignorance, Figure 1 · CC BY 4.0 · Full-size figure

Pretraining data filtering is a promising safety intervention that deserves further investment. It changes what a model learns in the first place, rather than relying only on restrictions added after training.

In Deep Ignorance, we trained 6.9-billion-parameter models on data filtered to exclude targeted biological knowledge used as a proxy in safety evaluations. The filtered models were substantially more resistant to the adversarial fine-tuning we tested than models protected by the post-training safeguards in our comparison. We observed no degradation on unrelated capability evaluations.

This is a reason to invest in pretraining as a site for safety interventions. Further work can test how filtering performs at larger scales, which kinds of knowledge it can reliably target, and how it can complement other methods.

All Open-Weight Safety papers