Dispersion Loss Undermines Knowledge Compression in Small Language Models

A new study has found that a phenomenon called dispersion loss counteracts the embedding condensation process in small language models, limiting their effectiveness. This means that compressing knowledge into smaller LLMs is harder than previously assumed. The findings have implications for designing more efficient, lightweight AI models.