👾🤖 Simpler, More Efficient Transformers
Weekly news digest curated by the industry insiders
📝 Editorial
It is hard to argue that transformers have become the most relevant architectures in modern machine learning (ML). Since the publication of the now-iconic Attention is All You Need paper, transformers have revolutionized fields like natural language understanding and computer vision. Now they are becoming highly relevant across most ML domains. However, transformer models require incredibly sophisticated ML infrastructures and remain too complex to be applied in many domains. If transformers hope to achieve mainstream adoption, they need to get simpler, more efficient and easier to operate.
The simplification transformers present challenges on both the ML research and engineering fronts. The good news is that the AI research community very well understands these challenges. Just this week, AI labs at Amazon Research and Stanford University published three papers about simplifying the training and efficiency of transformer architectures. Similarly, Apple Research published a reference implementation of transformer models optimized for Apple devices. Finally, OpenAI unveiled a series of best practices used to train large transformer models at scale. The efforts in this research area help achieve simpler and more efficient transformers that have the potential to further revolutionize ML. We will be discussing some of these papers in the future editions of TheSequence.
🔺🔻TheSequence Scope – our Sunday edition with the industry’s development overview – is free. To receive high-quality content about the most relevant developments in the ML world every Tuesday and Thursday, please subscribe to TheSequence Edge 🔺🔻
🗓 Next week in TheSequence Edge:
Edge#199: we discuss building blocks and types of GNN architectures; explain GraphWorld, which provides insights about how to test GNNs; explore Spektral, a library for building GNNs in Keras and TensorFlow.
Edge#200: we discuss PyWhy, Microsoft’s new home for causal inference.
Now, let’s review the most important developments in the AI industry this week
🔎 ML Research
Training Large ML Models
OpenAI published a series of best practices for training large neural networks →read more on OpenAI blog
Improving BERT
Amazon Research published a paper detailing Pyramid-BERT, a new method that improves the efficiency and memory footprint of BERT-based models →read more on Amazon Research blog
Compressing BART Models
Amazon Research published a paper outlining a technique for compressing bidirectional autoregressive transformers( BART) models for resource constrained operations →read more on Amazon Research blog
LinkBERT
Stanford University published a paper discussing LinkBERT, a method that incorporates document linking to improve the training of transformer models →read more on Stanford’s AI Lab blog
📌 Event: June 29th – Arize:Observe Unstructured
Join us for Arize:Observe Unstructured to learn how top teams use unstructured data in their ML initiatives! Speakers at this free, half-day event include OpenAI, Hugging Face 🤗, the creator of UMAP & more!
🤖 Cool AI Tech Releases
Transformers on Apple Devices
Apple Research released a reference implementation incorporating best practices for deploying transformer models in the Apple Neural Engine →read more on Apple Research blog
TFMOT Update
The TensorFlow team released some updates to its model optimization toolkit (TFMOT) to better support computer vision models →read more on TensorFlow blog
🛠 Real World ML
Data Quality at LinkedIn
LinkedIn discusses its data quality management architecture and processes →read more on LinkedIn engineering blog
💸 Money in AI
ML&AI&Data
MLOps startup Gantry raised a combined $28.3 million in Seed and Series A funding led by Amplify and Coatue. Watch their demo. Hiring remote.
AI platform Continual raised a $14.5 million Series A funding round, led by Innovation Endeavors. Hiring in San Francisco, New York, remote/US.
AI-powered
Open banking platform Bud Financial raised an $80 million Series B funding round led by Bellis Phantom Holdco Ltd. Hiring in London/UK.
Drug discovery company Insilico Medicine raised a $60 million Series D financing round from a syndicate of global investors. Hiring worldwide.
Drug discovery company CHARM Therapeutics raised a $50 million Series A financing round co-led by F-Prime Capital and OrbiMed. Hiring in the UK and remote.
Video localization startup Papercup raised $20 million in a Series A financing round led by Octopus Ventures. Hiring in London/UK.
Environmenttech startup Virridy raised $5.5 million in a Series A funding round led by Accord Capital.