Edge 445: A New Series About Knowledge Distillation

Created Using Midjourney

In this issue:

1. An introduction to knowlege distillation.

2. A review of one of the first papers about knowledge distillation.

3. Google’s Data Commons framework to ground LLMs on factual knowledge.

💡 ML Concept of the Day: An Intro to Knowledge Distillation

Making foundation models smaller and more cost effective is one of the key challenges of generative AI. While large frontier models have literally changed the world, they result cost prohibited for most applications. Distillation is one of the new emerging techniques focused on reducing the size while maintaining the accuracy of large generative AI models. These days, we are constaintly seeing distilled versions of large models being able to run on smaller compute environments such as mobile devices.

How does distillation work exactly?