MixKD: Towards Efficient Distillation of Large-scale Language Models

Engineering

• Electric Power Utilization 100%
• Applicability 100%
• Give Rise 100%
• Linear Interpolation 100%
• Neural Network System 100%
• Demonstrates 100%
• Tasks 100%

Computer Science

• Knowledge Distillation 100%
• Language Modeling 100%
• Power Consumption 50%
• Generalization Error 50%
• Training Instance 50%
• Training Example 50%
• Generalization Ability 50%
• Memory Resource 50%
• Data Augmentation 50%
• Data Distribution 50%
• Neural Network 50%
• Linear Interpolation 50%

Keyphrases

• Large Language Models 100%
• Big Model 100%
• Student Model 100%
• Empirical Error 50%
• Teacher Model 50%
• Teacher Behavior 50%