👥 Edge#15: Machine Teaching; Uber's Generative Teaching Networks; and Snorkel-Flow~
In this issue:
we discuss the idea of machine teaching;
we review a research paper that debates the concept of generative teaching networks and its usage at Uber;
we deep dive into Snorkel-Flow, one of the most innovative data labeling platforms in the market.
Enjoy the learning!
💡 ML Concept of the Day: What is Machine Teaching?
The teacher-student paradigm is deeply instituted in our learning processes as a human to human interaction. From infancy, we typically grasp the idea of having a “teacher” who guides us through the process of mastering specific tasks and correcting mistakes. Our entire education system is based on that. Machine teaching is a new paradigm that tries to incorporate this process in the area of machine learning.
The idea of machine teaching is conceptually simple and based on the creation of intelligent models that can teach other models. The principles can be summarized in a simple problem statement: given a set of machine learning models, a dataset, and a target task; determine the most efficient teachers to optimize the learning process. A complementary way to compare machine learning and machine teaching is that the former aims to make “learners” more efficient via better algorithms, while the latter aims to make “teachers” more productive by improving the learning models.
Let’s get back to human teacher-student relationships for a second. A teacher typically breaks tasks into smaller subtasks, explains the concepts, and then evaluates students’ knowledge with questions and tests. That constant feedback loop is missing in today’s machine learning models. Let’s take an image classification scenario in which a model is attempting to recognize decorative dining room artifacts such as tables, chairs, cabinets, etc. In the traditional machine learning approach, we will train a model with a large dataset that includes labels for those objects. The machine teaching alternative will identify the concepts that are relevant to identify the target objects (e.g. a table has four legs, etc.), train the model in those concepts and evaluate for potential errors.
The idea of machine teaching has been originally attributed to Microsoft Research, which has built an entire group dedicated to this new discipline. However, machine teaching has already been gaining traction within other large software companies and is steadily becoming part of mission-critical machine learning applications.
🔎 ML Research You Should Know: Machine Teaching at Uber
The paper Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data outlines how Uber uses a machine teaching technique to improve the training of machine learning models.
The objective: Uber proposes a technique known as generative teaching networks (GTNs) to accelerate the training and evaluation of machine learning models.
Why is it so important: Initial experiments at Uber showed that GTNs were able to train machine learning models 9-to-10 times faster than traditional methods.
Diving deeper: By the name itself, you can probably infer that GTNs draw inspiration from another famous generative model: generative adversarial neural networks (GANs). Like GANs, GTNs involve two neural networks competing/collaborating to master a task. Unlike GANs, GTNs do not have the pressure of making training data more realistic. Instead, GTNs can be seen as GANs with a teacher (generator) and learner networks. GTNs regularly train the learner network in generated data and evaluate its knowledge using real data. Functionally, the algorithm of GTNs can be summarized in the following five steps:
1) Noise is fed to the input generator which is used to create new synthetic data.
2) The learner is trained to perform well on the generated data.
3) The trained learner is then evaluated on the real training data in the outer-loop to compute the outer-loop meta-loss.
4) The gradients of the generator parameters are computed to the meta-loss to update the generator.
5) Both a learned curriculum and weight normalization substantially improve GTN performance.
Image credit: the original paper
The magic of GTNs is that they are able to train a neural network using synthetic data to the point that it can outperform a version of the same network trained using real data. This is achieved by constant interactions with the teacher network, which regularly looks to improve the knowledge of the learner networks.
Image credit: the original paper
GTNs are a very nascent area of research, but the applications at Uber certainly validate its value proposition at large scale. One interesting area to explore is the combination of GTNs with neural architecture search or meta-learning techniques to improve the efficiency of neural network architectures. We should expect to see more research in this area in the near future.
🤖 ML Technology to Follow: Snorkel Flow is a Platform to Unlock the Training Data Bottleneck in Machine Learning
Why should I know about this: Incubated at Stanford University in 2016, Snorkel became a very popular open-source project for data labeling. Snorkel Flow is a new end-to-end platform built on the principles of the Snorkel project.
What is it: In 2016, Snorkel quickly became a popular open-source stack to include in the life cycle of machine learning models. Over the years, the demand for more robust tooling, support, and SLAs around for Snorkel kept increasing. As a result, the project team decided to launch Snorkel AI, a company centered around expanding the original vision of the Snorkel project. Snorkel Flow is the first product created by the Snorkel AI team.
Conceptually, Snorkel Flow is presented as a platform for the end-to-end development and deployment of machine learning models. However, the area in which Snorkel Flow excels is the creation and maintenance of training datasets. More specifically, Snorkel Flow enables the programmatic labeling, augmentation, and building of training datasets. Let’s unpack that statement into more specific concepts.
Snorkel Flow enables the creation of programmatic operators for labeling training datasets. Operators are, essentially, labeling functions that can contain complex rules and heuristics to label training datasets. For instance, an operator can classify the subject of a text as “Politics” if it contains the text “White House” in it. The approach of using programmatic operators departs from the traditional way of hand labeling training datasets, making it more scalable and maintainable. Snorkel Flow enables the infrastructure for executing a large number of operators in order to construct and maintain training records.
Data augmentation is another area of innovation of the Snorkel Flow platform. It is very common in the training of machine learning models to use transformations in the training datasets to validate or improve the performance of models. For instance, small variations in images within a dataset can be used to test the robustness of a classifier against adversarial attacks. Snorkel Flow automates the creation of data augmentation routines that can be incorporated into training datasets. In addition to the data labeling and augmentation elements, Snorkel Flow enables the monitoring of machine learning models in a way that expands to the underlying training datasets. This can facilitate the long-term versioning and maintenance of models at scale.
Image credit: Snorkel
Snorkel Flow offers a rich toolset for streamlining the lifecycle of training datasets. Developers can integrate Snorkel Flow into machine learning models using the platform’s Python SDK and the corresponding CLI tools.
How can I use it: Snorkel Flow is available at https://www.snorkel.ai/. The former open-source Snorkel project is available at https://www.snorkel.org/.
🧠 The Quiz
Now, let’s check your knowledge. The questions are the following:
What’s the core concept behind machine teaching?
With what objective Generative Teaching Networks were first introduced by Uber?
Please use an email you are signed up with, so we can track your success.
That was fun! Thank you. See you on Thursday 😉
TheSequence is a summary of groundbreaking ML research papers, engaging explanations of ML concepts, and exploration of new ML frameworks and platforms. TheSequence keeps you up to date with the news, trends, and technology developments in the AI field.
5 minutes of your time, 3 times a week – you will steadily become knowledgeable about everything happening in the AI space. Make it a gift for those who can benefit from it.
