Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models

The remarkable success of Large Language Models (LLMs) has ushered naturallanguage processing (NLP) research into a new era. Despite their diversecapabilities, LLMs trained on different corpora exhibit varying strengths andweaknesses, leading to challenges in maximizing their overall efficiency andversatility. To address these challenges, recent studies have exploredcollaborative strategies for LLMs. This paper provides a comprehensive overviewof this emerging research area, highlighting the motivation behind suchcollaborations. Specifically, we categorize collaborative strategies into threeprimary approaches: Merging, Ensemble, and Cooperation. Merging involvesintegrating multiple LLMs in the parameter space. Ensemble combines the outputsof various LLMs. Cooperation leverages different LLMs to allow full play totheir diverse capabilities for specific tasks. We provide in-depthintroductions to these methods from different perspectives and discuss theirpotential applications. Additionally, we outline future research directions,hoping this work will catalyze further studies on LLM collaborations and pavingthe way for advanced NLP applications.

Academy

IntPDF

Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models

Further reading

Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models

Further reading#

Further reading