The Sequence Chat: Rohan Taori on Stanford's Alpaca, Alpaca Farm and the Future of LLMs

Alpaca was one of the first open LLMs to incorporate instruction following capabilities. Now one of the project's main researchers shares his insights about modern LLMs.

👤 Quick bio

Tell us a bit about yourself. Your background, current role and how did you get started in machine learning and data labeling?  

I’m currently a PhD student at Stanford, where my research has focused around understanding and improving the use of real-world machine learning systems. My interest in ML started during my undergrad at Berkeley where I met a fantastic group of people (who later became good friends) who introduced me to DeepDream, a method for style transfer with image classification models. Though these models are nothing compared to the generative text-to-image models we have today, I was enthralled by the art pieces created by DeepDream, and that set me on a path delving into deep learning.

🛠 ML Work  

Alpaca has rapidly become one of the most influential open-source releases in the foundation model space. Can you tell us a bit about the inspiration and vision for the project? 

Alpaca was really sort of a happy surprise along the way of our bigger project, AlpacaFarm. Our goal for AlpacaFarm was to study methods for learning from human feedback (e.g. RLHF) in depth, as this process has been used to train models like ChatGPT. The first step for any RLHF method is actually to perform supervised fine-tuning on a small set of initial questions and answers, which is the baseline that any method builds off of. Alpaca is precisely this baseline! We were stunned at how just a little bit of fine-tuning made the base LLaMA 7B model so much easier to interact with, and so we released the model as a way to share this information with everyone.

Alpaca's biggest contribution was the addition of RLHF fine-tuning to the LLaMA model. Could you describe the process of building the instruction training dataset and the process for training the model? 

Alpaca itself did not have any RLHF component and was only supervised fine-tuned on a small question-answer set. To build this dataset, we queried text-davinci-003 to generate questions and answers to open-ended instructions in a format similar to a set of seed questions written by the authors of SelfInstruct (https://arxiv.org/abs/2212.10560). Our subsequent project, AlpacaFarm, studied the RLHF component in more detail. In this stage, we asked the model to generate two outputs for a set of instructions, and then asked humans to select which output they preferred. We then trained a binary classifier (or reward model) with this data, and optimized against it with a variety of methods, including PPO.

Your research group followed the work on Alpaca with AlpacaFarm, a method that drastically improves the performance and efficiency of RLHF processes. How does AlpacaFarm works and what are some of its key components?

AlpacaFarm is a simulator that enables research on learning from feedback quickly and cheaply. The primary component is a set of LLM prompts designed to simulate humans - we ask GPT-4 and ChatGPT for the output preferences to train the reward model, and we designed this automated feedback to align with learning from real human feedback. We also include a standardized evaluation protocol for judging the performance of any instruction-following model, and a set of reference implementations for PPO and other RLHF methods. AlpacaFarm is really cool because the rankings of different methods developed in simulation roughly line up with the rankings of the methods when trained on actual human feedback, which means that the simulator can serve as an effective way to study and develop new RLHF methods.

Instruction following with RLHF has become one of the key capabilities to complement LLs. However, there are other interesting areas of research, such as knowledge augmentation, chain of thought reasoning, and tool learning, that are starting to show potential. What new areas of LLM are you excited about, and what is directly applicable to Alpaca? 

I think that tool learning is a really exciting direction, as it equips LLMs with the ability to execute very complex tasks. At the same time, there is a greater security risk in giving an automated agent control over various external resources. So developing new methods for tool use and mitigating the security risks is a really important research discussion for the near future. While Alpaca doesn’t come with tool use directly, AlpacaFarm shows that simulation is a viable way to easily study real-world LM behaviors, and I’m excited to see what this direction can bring.

Alpaca decided to take a semi open-source approach and has sparked an interesting wave of innovation within the foundation model community. How do you see the balance between open source versus API-based distribution for foundation models? What are the risks of making these powerful models available to everyone? 

Alpaca is not actually open-source - it came with a non-commercial license due to the assets involved in its creation. However, since then, other truly open-source instruction-following models have been released to the community, which is great. Just like any new technology, there are new risks and consequences with LLMs. My view is that having better models more openly available will enable research on these important questions. Many of the issues surrounding their use involve different communities, different norms, and different cultures; open releases are a great way of inviting the broader participation needed in the development of these models.

💥 Miscellaneous – a set of rapid-fire questions 

What is your favorite area of AI research outside of generative AI? 

Having worked in image classification for a while, I have to say that I am quite fond of CLIP (https://arxiv.org/abs/2103.00020) - a contrastive learning paradigm to align text and images found on the internet. Constructing a new classifier for a downstream task can now be much easier, and potentially even accomplished with just a bit of prompting.

If you had to pick one ML architecture to displace transformers in the near future, what would that be? 

I really don’t know too much about the intricacies of different ML architectures. As a broader vision, I am pretty excited to see video understanding capabilities be added into language models, and there are lots of questions around how to best encode video data efficiently that have yet to be answered.

Have we hit a scale limit with LLMs? What are the new techniques needed to enhance LLM knowledge without depending on size? 

This question is hard to answer, in part because we don’t know how big GPT-4 is :). More generally, as a community, we lack a good understanding of how the quality of pre-training data impacts model performance. For example, what sources should be included in the dataset? What types of automated filters should we train, and how aggressively should we use them? How feasible is it to learn from multi-lingual data? All these and many more data engineering questions remain, and we will probably see model capabilities steadily improve as we solve more pieces of this puzzle.

What's next for Alpaca? 

AlpacaFarm is really the completion arc for Alpaca! I’m really excited to see the methods innovation that the simulator can bring in the following months. Apart from that, I have a few loose ideas about what to work on next, but nothing concrete decided for now.