The Sequence Chat: Nathan Benaich, Air Street Capital About Investing in Generative AI

A conversation about the state of the generative AI market, new research breakthroughts, open source , opportunities for startups and many other insights.

Quick Bio

Nathan Benaich is the Founder and General Partner of Air Street Capital, a venture capital firm investing in early-stage AI-first technology and life science companies. His investments include Allcyte (acq. Exscientia), Intenseye, Thought Machine, Profluent, V7, Synthesia, and Valence Discovery (acq. Recursion). Nathan is the co-author of the annual State of AI Report and the newsletter, your guide to AI. Nathan also leads Spinout.fyi, which seeks to improve university spinout creation starting with open data on deal terms, and The RAAIS Foundation, a non-profit that runs the annual RAAIS summit and funds open source AI fellowships. He holds a Ph.D. in cancer biology from the University of Cambridge and a BA from Williams College.

Quick bio

Tell us a bit about yourself. Your background, current role and how did you get started in artificial intelligence and venture capital?   

I took what was then an unorthodox route into VC, but I believe it’s now becoming more and more common. My original plan was to go to medical school to become a physician-scientist working on translational research in cancer and stem cell biology. I studied biology at Williams College and worked on breast cancer metastasis at the Whitehead Institute at MIT during my summers.

I was fascinated by the incredible engine of the Boston biotech ecosystem that systematically translates inventions in the lab into spinouts to create products that make a difference in the world. At the same time, I saw the many challenges associated with building a life science company - ranging from the huge amounts of capital required through to the gauntlet of clinical trials and human biology.

As I pursued my PhD in experimental and computational cancer research at Cambridge in the UK, I became more and more fascinated with technology and startups beyond biotech. Products like Dropbox, iPhones, and Twitter were launching during my undergraduate studies and had properly scaled by the time I got to grad school. These innovations not only engaged me, but made peoples' lives more enjoyable and efficient. I started teaching, immersing myself into the tech and venture capital scene (VC). I kept seeing that the most exciting and ambitious companies were financed with VC and I discovered a passion for discovering inspirational founders trying to change their fields. I found that VC could fulfill my scientific urge to formulate and test a data-driven thesis.

When my research group moved to London, I took full advantage of the nascent startup ecosystem there. I met entrepreneurs and investors, went to meetups and through lots of hard work, found myself with a shot in VC.

Air Street was built on the idea that AI-first companies require AI-first investors. Since day one, I’ve worked hard to deepen our connection and role in the AI community, whether it’s through our international meetups, producing the State of AI Report, and supporting the RAAIS conference and the RAAIS Foundation.

🛠 AI Work  

Air Street recently announced that it has raised $121,212,121 for its second fund. Other than being the first palindrome fund in AI 😉, could you please elaborate on your investment thesis and vision?

At Air Street, we are unashamedly positive about the potential of AI. We believe it will unlock a new era of economic progress and scientific discovery, by acting as a force multiplier on technology.

With that in mind, we look for founders building AI-first companies helping to solve real-world challenges. By AI-first, we mean that AI is central to what they are building and if you removed it, they wouldn’t have a functioning product.

The founders we back combine two main traits. Firstly, deep insight into their customers’ operating context, pressures, pain-points, and how new technology would fit into their way of working. Secondly, technical brilliance combined with pragmatism when it comes to selecting the right tools to use or build. For example, it may mean accepting that even in 2023, not every problem has a GenAI-shaped solution…

One of the most challenging questions when investing in generative AI is determining where the value will accrue in the long term. How do you differentiate potentially disruptive companies from mere features of other products powered by generative AI?

A few years ago, long before the generative AI boom, I wrote an essay making the case for the full-stack machine learning company. This means, instead of building part of the stack and licensing it out to another business to solve a problem, you build a business that creates a fully-integrated ML product that solves the problem end-to-end.

If we take an example from the life sciences, a company that licenses out a model to help big pharma is probably going to capture less economic upside than one that builds an end-to-end drug discovery platform that owns drug assets. I’m much more excited by that kind of ambitious vision than the many LLM-as-a-service businesses that have sprung up in the last year.

Air Street recently published its State of the Art report, which covers some of the most important areas of the generative AI ecosystem. What were some of the most important findings you learned while putting together the report?

The State of AI is our annual report, covering the biggest developments across research, industry, politics, and safety.

Unsurprisingly, there was a heavy generative AI presence in this year’s report - including both an assessment of model performance and promising applications, we try to write with a wider lens. Progress in GenAI doesn’t just have implications for researchers or investors, it powers other trends.

For example, the GenAI boom has clearly had a knock-on effect on the semiconductor wars between the US and China, and the report charts how the US has aggressively mobilized its allies amid a faltering Chinese response. It’s also accelerated the global governance conversation, as lawmakers have scrambled to respond in the face of a bad-tempered debate about risk inside the AI community.

It’s also important to note that it’s the State of AI, not the State of LLMs, and we cover much more besides. This year’s report also contains a range of material, covering everything from the life sciences, weather-forecasting, and autonomy through to the potential impact of AI on sensitive political subjects, including elections and job losses.

One of the most intriguing frictions in the generative AI space is the balance between open-source and closed-source/API-based models. How do you see the evolution of these two distribution models over the next few years? Who will emerge as the winner in the end?

This year’s report finds that there’s a clear capability gap between GPT-4 and its more open counterparts. Meta’s Llama 2, a more open alternative, is competitive with Chat-GPT on most tasks, with the exception of coding, where it lags it significantly. Code Llama, a code-specialized version of Llama 2, is competitive with GPT-4, which demonstrates that task-specific models still have a chance.

We’ve definitely seen a push by incumbents for more closed source AI - stemming from a combination of genuine safety concerns and obvious commercial cynicism. A low light for us in this year’s report was OpenAI’s hollow technical report on GPT-4 and Anthropic’s decision not to publish one at all for Claude 2, despite both being built on the shoulders of open source.

However, I’ve been encouraged by the resilience of the open source ecosystem. Hugging Face, the town hall for open source AI has seen record levels of traffic. August of this year alone saw 600 million model downloads.

Open source models have been continually improving in performance and I see no reason why this trend won’t continue. In smaller, more specialized applications, there’s a particularly clear role for teams that don’t have the resources to build multi-trillion parameter models.

A lot of the large investments in generative AI have been in companies building massive foundation models. How far do you think the scaling laws can go in this area? Would we see LLMs that surpass $10B in pretraining/fine-tuning costs?

It’s theoretically possible we might hit $10 billion models, but I think we’re still some way off; after all, the report predicts the emergence of $1 billion models next year. I think it’s possible, however, that we may have hit the point of diminishing returns before the $10 billion point.

In the report, we point to how researchers at Epoch AI have already warned about us hitting a data ceiling. They argue that we could run out of high-quality language data as early as 2026 and low-quality language in the 2030s.

Similarly, there are interesting case studies of teams building smaller, curated language models designed for specific tasks, with impressive performance. For specialized applications, these may prove a cheaper, more efficient route, rather than striving for ever greater scale.

💥 Miscellaneous – a set of rapid-fire questions 

Do you think there will be trillion-dollar native generative AI-based companies? If so, would you venture to predict that OpenAI will be one of them?

I don’t think it’s impossible, but we’re a long way off from this happening. OpenAI has blasted past its revenue targets, but its losses are continuing to mount steeply. Trillion-dollar businesses, like Alphabet, Apple, and Microsoft, have achieved clear, sustained commercial success with significant moats and cash cow products. While OpenAI has generous financial backers in Microsoft, at some point (particularly in a high-interest rate environment), gravity is likely to reassert itself, triggering tough questions about the business model. 

I don’t doubt that the team at OpenAI will be able to figure out the answer, but until the wider foundation model space moves from producing technical breakthroughs to building scalable businesses, talk of trillion-dollar valuations seems premature.

You can make a case that RLHF enabled the transition from GPT-3 to the mainstream ChatGPT phenomenon. What do you think is the next research breakthrough that can unlock the next wave of innovation in generative AI?

Language alone obviously doesn’t capture the full scope of human reasoning or communication, or how we plan and take action in the world. That’s why we see multimodality as the new frontier in this year’s report.

We see this already in GPT-4 (and GPT-4V), which unlike its predecessors, was trained on both text and images and can generate text from images.

Multimodality is already beginning to underpin a range of exciting potential applications. Google’s Med-PaLM 2 language model exceeded expert physician performance on the US Medical Licensing Examination, but obviously real-world medicine isn’t a purely text-based profession.

With this in mind, Google created a dataset called MultiMedBench that has medical questions along with matching images, allowing them to train multimodal systems that understand both text and images. A version of MedPaLM was trained on this dataset, with a single set of weights to handle multiple data types. This helps it generalize and perform new medical tasks.

We’ve also seen UK self-driving start-up Wayve build a model called LINGO-1, a model that combines videos of journeys with expert commentary on driving behavior and the scene. You can also ask the model questions via natural language. As well as improving reasoning and planning, it potentially marks a big step forward in the explainability of end-to-end driving models.

What are the most significant mistakes that you see entrepreneurs making while building in the generative AI space? How about the most common mistake investors make?

These mistakes usually happen at the conceptual stage. Anyone operating in this space, whether they’re a founder or investor should ask themselves a few fundamental questions.

Firstly, is generative AI actually the most efficient, pragmatic solution to the challenge you’re approaching? Fashion will change and getting side-tracked by whatever’s vogueish will set you up for failure later on.

Secondly, is there a viable business model? Going back to the full-stack machine learning company, there are many generative AI businesses that risk missing out on a lot of the value they’re helping to create.

Finally, are you playing in a space you’re likely to win in? Certain spaces, whether it’s foundational model creation or finetuning-as-a-service are disproportionately likely to be captured by well-funded incumbents. A new entrant needs a clear edge.

Who is your favorite mathematician or computer scientist, and why?

Chris Ré is an exceptional computer scientist at Stanford with a track record of focusing his research on real-world problems for data and AI teams. For example, his group recently produced FlashAttention, which makes the memory footprint of attention smaller, to produce faster and higher quality transformer models. Alongside his students, Chris has also spun out now large-scale companies in AI such as Snorkel, SambaNova Systems and Lattice Data (acq. Apple).

Eric Lander is a Professor of Biology at MIT and Professor of Systems Biology at Harvard Medical School. He made seminal contributions to population genetics and led the formation of the Whitehead Institute/MIT Center for Genome Research, which became a key center for the Whole Genome Project Human Genome Project that started in 1990. He then drove the founding of the Broad Institute in 2004, which is hands-down one of the best research centers for human biology in the world, particularly when it comes to approaches that unite software and biological experimentation at large automated scales.