Conceptual diagram of TVKD: After teaching the human preference dataset to the teacher model, learning proceeds by delivering the teacher's information and the dataset to the student model. Credit: The Korea Advanced Institute of Science and Technology (KAIST)
No matter how much data they learn, why do artificial intelligence (AI) models often miss the mark on human intent? Conventional comparison learning, designed to help AI understand human preferences, has frequently led to confusion rather than clarity. A KAIST research team has now presented a new learning solution that allows AI to accurately learn human preferences even with limited data by assigning it a "private tutor."
A research team led by Professor Junmo Kim of KAIST School of Electrical Engineering developed TVKD (Teacher Value-based Knowledge Distillation), a reinforcement learning framework that significantly improves data efficiency and learning stability while effectively reflecting human preferences. Their paper is published on the arXiv preprint server.
Limitations of traditional AI training
Existing AI training methods typically rely on collecting massive amounts of preference comparison data—simple structures like "A is better than B." However, this approach requires vast datasets and often causes the AI to become confused in ambiguous situations where the distinction is unclear.
To solve this problem, the research team proposed a method in which a "teacher model" that has first deeply understood human preferences delivers only the core information to a "student model." This can be compared to a private tutor who organizes and teaches complex content, and the research team named this preference distillation.
How preference distillation works
The biggest feature of this technology is that instead of simply imitating good or bad, it is designed so that the teacher model learns a value function that numerically judges how valuable each situation is, and then delivers this to the student model. Through this, the AI can learn by making comprehensive judgments about "why this choice is better" rather than fragmentary comparisons, even in ambiguous situations.
The core of this technology is twofold. First, by reflecting value judgments that consider the entire context into the student model, learning that understands the overall flow rather than fragmentary answers has become possible. Second, a technique was introduced to adjust learning importance according to the reliability of preference data. Clear data is significantly reflected in learning, while the influence of ambiguous or noisy data is reduced, allowing the AI to learn stably even in realistic environments.
Results and future implications
As a result of the research team applying this technology to various AI models and conducting experiments, it showed more accurate and stable performance than methods previously known to have the best performance. In particular, it recorded achievements that stably outperformed existing top technologies in major evaluation indices such as MT-Bench and AlpacaEval.
Professor Junmo Kim said, "In reality, human preference data is not always sufficient or perfect," and added, "This technology will allow AI to learn consistently even under such constraints, so it will be highly practical in various fields."
