The Sequence Research #471: One of the New Techniques Powering in OpenAI GPT-o3
Deliberate Aligment is a method to improve the safety and trustworthiness of LLMs
A few weeks ago, OpenAI dazzled the AI world once again by unveiling its newest reasoning model GPT-o3. There is very little that we know about this model at the moment but, together with the release OpenAI published some research about one of the techniques used to train reasoning LLMs in a way that follow safety spec.
Under the catching name of Deliberative Alignment, this method is a pioneering approach to improve the safety and trustworthiness of LLMs. It diverges from conventional safety training methods by directly instructing the model on safety specifications and training it to explicitly recall and reason over these specifications before generating a response. This approach tackles the limitations of implicit, pattern-based learning, resulting in improved data efficiency and generalization capabilities, particularly when encountering unfamiliar scenarios or adversarial attacks.