🌅 The Era of Foundation Models is Here
Weekly news digest curated by the industry insiders
The term ‘foundation models’ is becoming one of the hottest buzzwords in the machine learning (ML) lingo. Researchers from Stanford University originally coined the term to describe models that have been trained in large amounts of unlabeled data and can be fine-tuned to specific domains. Think about fine-tuning GPT-like models for domains such as law or science. Foundation models are shifting the ML development paradigm from creating brand-new models to fine-tuning large pretrained models.
The efforts around foundation models are increasing remarkably fast. Stanford University created the Center for Research on Foundation Models (CRFM), a new initiative focused on studying best practices around foundation models. Just this week, Snorkel AI released Data-centric Foundation Model Development, a new series of addition to the Snorkel Flow platform to fine-tune and distill foundation models. Meta AI also unveiled details about MultiRay, their platform for running foundation models at scale. Finally, the CRFM team unveiled a new benchmark to facilitate the holistic evaluation of foundation models. Foundation models efforts are popping up everywhere, from large AI labs to innovative startups.
Building by fine-tuning the new paradigm. The era of foundation models is definitely upon us!
🔺🔻TheSequence Scope – our Sunday edition with the industry’s development overview – is free. To receive high-quality content about the most relevant developments in the ML world every Tuesday and Thursday, please subscribe to TheSequence Edge 🔺🔻
🗓 Next week in TheSequence Edge:
Edge#245: we start a new series about machine learning interpretability; discuss Manifold, an architecture for debugging ML models; explore Meta’s Captum, a framework for deep learning interpretability.
Edge#246: we discuss OpenAI’s best practices that they used to mitigate risks while training DALL-E2
📌 Our LinkedIn account
In this uncertain times for Twitter, we’d like to introduce to you TheSequence’s LinkedIn account. We are building a unique resource and support system for all ML&AI aficionados. Let’s connect!
Now, let’s review the most important developments in the AI industry this week
🔎 ML Research
Stanford University published HELM, a benchmark for the holistic evaluation of foundation models →read more
Meta AI discusses MultiRay, the architecture used to power large foundation ML models at scale across their different organizations →read more
Data Enrichment Practices
DeepMind published an insightful paper discussing human data collection best practices used in real-world ML scenarios →read more
MoE with Expert Routing
Google Research published a research paper proposing a routing algorithm in mixture of experts (MoE) neural networks →read more
🤖 Cool AI Tech Releases
Data-centric Foundation Model Development
Snorkel AI released Data-centric Foundation Model Development, a new set of capabilities in the Snorkel Flow platform to adapt large foundation models to domain-specific scenarios →read more
Data Cards Playbook
Google Brain released Data Cards Playbook, a toolkit for transparency in ML datasets →read more
🛠 Real World ML
Anomaly Detection in Prime Video
Amazon Science discusses the ML techniques used for anomaly detection in their Prime Video application →read more
Einstein Search Answers
Salesforce Research discusses the ML techniques powering Einstein Search Answers, a new search architecture for customer support →read more
Netflix Video Quality
Netflix discusses the neural network techniques used for video encoding optimizations in the media giant →read more
💸 Money in AI