Edge 343: Understanding Llama-Adapter Fine-Tuning

One of the most intriguing fine-tuning methods that combines prefix-tuning and PEFT.

Created Using DALL-E

💡 ML Concept of the Day: What is Llama-Adapter?

In this series about fine-tuning, we have explored concepts such as prefix-tuning or parameter-efficient fine-tuning(PEFT). Today, we would like to dive into a concept that combines ideas from those two methods: Llama-Adapter.

The ideas behind Llama-Adapter come from adapter fine-tuning techniques. The original adapter method is somewhat related to the aforementioned prefix tuning as they also add additional parameters to each transformer block. However, instead of prepending prefixes to the input embeddings, the adapter method adds adapter layers in two places, as illustrated in the figure below.