一张进化树图看懂大模型血缘：GPT、BERT、T5 的分叉与汇合

核心洞察

这篇论文最值得看的就是那张进化树图，一张图把GPT、BERTBidirectional Encoder Representations from Transformers, a pre-trained natural language processing model for deep bidirectional language understanding.、T5几条技术路线的分叉和汇合讲清楚了。我比较意外的是Meta在开源这件事上的投入力度，翻遍主流模型列表，能做到全部开源的商业公司基本就它一家。另外文中提到闭源趋势导致学术界越来越难做预训练实验，这个判断在2023年看是准的，现在回头看更是如此。

---

大语言模型这两年发展太快了，各种新模型层出不穷，感觉跟寒武纪物种大爆发似的。各模型之间的血缘关系也让人看得云里雾里。最近有学者整理了一份ChatGPT等语言模型的进化树图，把LLM之间的家族关系理了一遍，一目了然。

论文地址：https://arxiv.org/abs/2304.13712

Github相关资源：https://github.com/Mooler0410/LLMsPracticalGuide

核心结论

1. 论文通过一张进化树图系统梳理了GPT、BERT、T5等主流大语言模型的技术血缘关系，按架构分为仅解码器（蓝色）、仅编码器（粉红色）和编码器-解码器（绿色）三条分支，实心方块代表开源模型、空心方块代表闭源模型。

2. 2021年GPT-3A large language model developed by OpenAI with 175 billion parameters, known for its advanced text generation capabilities.推出后，仅解码器架构逐渐占据主导地位，而BERT带动的仅编码器架构增长随之消退。

3. Meta是主流商业公司中开源力度最大的一家，其开发的所有语言模型均为开源；相比之下，2020年之前大部分模型开源，但GPT-3之后PaLM、LaMDA、GPT-4等均选择闭源。

4. 闭源趋势导致学术研究者越来越难开展语言模型预训练实验，基于API的研究可能成为学术界的主流方式。

5. 编码器-解码器架构仍有前景且大部分开源，Google在该方向贡献显著，但仅解码器模型的灵活性和多功能性使这一路线的前景不太乐观。

最重要的进化树图

这张现代语言模型进化树追溯了近年来的发展脉络，标出了一些最知名的模型。同一分支上的模型关系更近。基于TransformerA deep learning neural network architecture using self-attention mechanisms for sequence processing.的模型用非灰色标注：仅解码器模型是蓝色分支，仅编码器模型是粉红色分支，编码器-解码器模型是绿色分支。模型在图上的垂直位置对应发布日期。实心方块代表开源模型，空心方块代表闭源模型。右下角的堆叠条形图统计了各公司和机构发布的模型数量。

另外还有一张按年份演化的动图，内容和上图一致。

论文内容简介

论文标题：Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond

论文地址：https://arxiv.org/abs/2304.13712

趋势

a) 仅解码器模型逐渐占据主导。早期仅解码器模型不如仅编码器和编码器-解码器流行。2021年之后，GPT-3的推出带动了仅解码器模型的爆发。与此同时，BERT带来的那波仅编码器增长逐渐消退。

b) OpenAI在这个领域一直保持领先。其他公司和机构在努力追赶GPT-3和GPT-4的水平。这种领先可以归因于OpenAI即使在技术路线不被广泛认可的时候也坚持走下去。

c) Meta对开源语言模型的贡献很大。翻一遍开源模型列表会发现，Meta开发的所有语言模型都是开源的，在商业公司里算是最慷慨的一家。

d) 语言模型有转向闭源的趋势。2020年之前大部分模型是开源的。GPT-3推出之后，PaLM、LaMDA、GPT-4这些模型都选择了闭源。学术研究者做语言模型训练实验越来越难，基于API的研究可能成为学术界的主流方式。

e) 编码器-解码器架构仍有前景，这个方向还在被积极探索，而且大部分是开源的。Google对开源编码器-解码器架构贡献很大。不过仅解码器模型的灵活性和多功能性，让Google坚持这条路的前景不太乐观。

仅解码器模型和开源模型在近年占据主导，OpenAI和Meta在推动创新和开源方面出力最多。编码器-解码器模型和闭源模型也在推动发展。各家公司面临的技术路径前景不太一样。

模型实用指南

这是一份LLM实用指南资源列表，还在持续更新。基于上述调查论文整理，帮从业者了解大型语言模型及其在NLP应用中的用法。

BERT风格的语言模型：编码器-解码器或仅编码器

• BERT：BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding，2018，论文
• RoBERTa：ALBERT: A Lite BERT for Self-supervised Learning of Language Representations，2019，论文
• DistilBERT：DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter，2019，论文
• ALBERT：ALBERT: A Lite BERT for Self-supervised Learning of Language Representations，2019，论文
• UniLM：Unified Language Model Pre-training for Natural Language Understanding and Generation，2019，论文
• ELECTRA：ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS，2020，论文
• T5：Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer，Colin Raffel et al.，JMLR 2019，论文
• GLM：GLM-130B: An Open Bilingual Pre-trained Model，2022，论文
• AlexaTM：AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model，Saleh Soltan et al.，arXiv 2022，论文
• ST-MoE：ST-MoE: Designing Stable and Transferable Sparse Expert Models，2022，论文

GPT风格语言模型：仅解码器

• GPT：Improving Language Understanding by Generative Pre-Training，2018，论文
• GPT-2：Language Models are Unsupervised Multitask Learners，2018，论文
• GPT-3：Language Models are Few-Shot Learners，NeurIPS 2020，论文
• OPT：OPT: Open Pre-trained Transformer Language Models，2022，论文
• PaLM：PaLM: Scaling Language Modeling with Pathways，Aakanksha Chowdhery et al.，arXiv 2022，论文
• BLOOM：BLOOM: A 176B-Parameter Open-Access Multilingual Language Model，2022，论文
• MT-NLG：Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model，2021，论文
• GLaM：GLaM: Efficient Scaling of Language Models with Mixture-of-Experts，ICML 2022，论文
• Gopher：Scaling Language Models: Methods, Analysis & Insights from Training Gopher，2021，论文
• chinchilla：Training Compute-Optimal Large Language Models，2022，论文
• LaMDA：LaMDA: Language Models for Dialog Applications，2021，论文
• LLaMAA collection of large language models released by Meta (Facebook) supporting multiple languages including Latin and Cyrillic scripts.：LLaMA: Open and Efficient Foundation Language Models，2023，论文
• GPT-4：GPT-4 Technical Report，2023，论文
• BloombergGPT：BloombergGPT: A Large Language Model for Finance，2023，论文
• GPT-NeoX-20B：GPT-NeoX-20B: An Open-Source Autoregressive Language Model，2022，论文

数据实用指南

预训练数据

• RedPajama，2023，代码库
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling，Arxiv 2020，论文
• How does the pre-training objective affect what large language models learn about linguistic properties?，ACL 2022，论文
• Scaling laws for neural language models，2020，论文
• Data-centric artificial intelligence: A survey，2023，论文
• How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources，2022，博客

微调数据

• Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach，EMNLP 2019，论文
• Language Models are Few-Shot Learners，NIPS 2020，论文
• Does Synthetic Data Generation of LLMs Help Clinical Text Mining?，Arxiv 2023，论文

测试数据/用户数据

• Shortcut learning of large language models in natural language understanding: A survey，Arxiv 2023，论文
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective，Arxiv 2023，论文
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems，Arxiv 2019，论文

NLP任务实用指南

研究者为NLP应用构建了一个选择LLM还是微调模型的决策流程。这个流程帮用户评估手头的下游NLP应用是否满足特定条件，据此判断LLM或微调模型哪个更合适。

传统的非语言处理任务

• A benchmark for toxic comment classification on civil comments dataset，Arxiv 2023，论文
• Is chatgpt a general-purpose natural language processing task solver?，Arxiv 2023，论文
• Benchmarking large language models for news summarization，Arxiv 2022，论文

生成任务

• News summarization and evaluation in the era of gpt-3，Arxiv 2022，论文
• Is chatgpt a good translator? yes with gpt-4 as the engine，Arxiv 2023，论文
• Multilingual machine translation systems from Microsoft for WMT21 shared task，WMT2021，论文
• Can ChatGPT understand too? a comparative study on chatgpt and fine-tuned bert，Arxiv 2023，论文

知识密集型任务

• Measuring massive multitask language understanding，ICLR 2021，论文
• Beyond the imitation game: Quantifying and extrapolating the capabilities of language models，Arxiv 2022，论文
• Inverse scaling prize，2022，链接
• Atlas: Few-shot Learning with Retrieval Augmented Language Models，Arxiv 2022，论文
• Large Language Models Encode Clinical Knowledge，Arxiv 2022，论文

缩放能力

• Training Compute-Optimal Large Language Models，NeurIPS 2022，论文
• Scaling Laws for Neural Language Models，Arxiv 2020，论文
• Solving math word problems with process- and outcome-based feedback，Arxiv 2022，论文
• Chain of thought prompting elicits reasoning in large language models，NeurIPS 2022，论文
• Emergent abilities of large language models，TMLR 2022，论文
• Inverse scaling can become U-shaped，Arxiv 2022，论文
• Towards Reasoning in Large Language Models: A Survey，Arxiv 2022，论文

具体任务

• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks，Arxiv 2022，论文
• PaLI: A Jointly-Scaled Multilingual Language-Image Model，Arxiv 2022，论文
• AugGPT: Leveraging ChatGPT for Text Data Augmentation，Arxiv 2023，论文
• Is gpt-3 a good data annotator?，Arxiv 2022，论文
• Want To Reduce Labeling Cost? GPT-3 Can Help，EMNLP findings 2021，论文
• GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation，EMNLP findings 2021，论文
• LLM for Patient-Trial Matching: Privacy-Aware Data Augmentation Towards Better Performance and Generalizability，Arxiv 2023，论文
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks，Arxiv 2023，论文
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment，Arxiv 2023，论文
• GPTScore: Evaluate as You Desire，Arxiv 2023，论文
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality，Arxiv 2023，论文
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study，Arxiv 2023，论文

现实世界的任务

• Sparks of Artificial General Intelligence: Early experiments with GPT-4，Arxiv 2023，论文

效率

1. 成本

• Openai's gpt-3 language model: A technical overview，2020，博客
• Measuring the carbon intensity of ai in cloud instances，FaccT 2022，论文
• In AI, is bigger always better?，Nature Article 2023，文章
• Language Models are Few-Shot Learners，NeurIPS 2020，论文
• Pricing，OpenAI，博客

2. 延迟

• HELM：Holistic evaluation of language models，Arxiv 2022，论文

3. 参数高效微调

• LoRA: Low-Rank Adaptation of Large Language Models，Arxiv 2021，论文
• Prefix-Tuning: Optimizing Continuous Prompts for Generation，ACL 2021，论文
• P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks，ACL 2022，论文
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks，Arxiv 2022，论文

4. 预训练系统

• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models，Arxiv 2019，论文
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism，Arxiv 2019，论文
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM，Arxiv 2021，论文
• Reducing Activation Recomputation in Large Transformer Models，Arxiv 2021，论文

信用

1. 稳健性和校准

• Calibrate before use: Improving few-shot performance of language models，ICML 2021，论文
• SPeC: A Soft Prompt-Based Calibration on Mitigating Performance Variability in Clinical Notes Summarization，Arxiv 2023，论文

2. 杂散偏置

• Shortcut learning of large language models in natural language understanding: A survey，2023，论文
• Mitigating gender bias in captioning system，WWW 2020，论文
• Calibrate Before Use: Improving Few-Shot Performance of Language Models，ICML 2021，论文
• Shortcut Learning in Deep Neural Networks，Nature Machine Intelligence 2020，论文
• Do Prompt-Based Models Really Understand the Meaning of Their Prompts?，NAACL 2022，论文

3. 安全问题

• GPT-4 System Card，2023，论文
• The science of detecting llm-generated texts，Arxiv 2023，论文
• How stereotypes are shared through language: a review and introduction of the aocial categories and stereotypes communication (scsc) framework，Review of Communication Research，2019，论文
• Gender shades: Intersectional accuracy disparities in commercial gender classification，FaccT 2018，论文

基准指导调优

• FLAN：Finetuned Language Models Are Zero-Shot Learners，Arxiv 2021，论文
• T0：Multitask Prompted Training Enables Zero-Shot Task Generalization，Arxiv 2021，论文
• Cross-task generalization via natural language crowdsourcing instructions，ACL 2022，论文
• Tk-INSTRUCT：Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks，EMNLP 2022，论文
• FLAN-T5/PaLM：Scaling Instruction-Finetuned Language Models，Arxiv 2022，论文
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning，Arxiv 2023，论文
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization，Arxiv 2023，论文

校准

• Deep Reinforcement Learning from Human Preferences，NIPS 2017，论文
• Learning to summarize from human feedback，Arxiv 2020，论文
• A General Language Assistant as a Laboratory for Alignment，Arxiv 2021，论文
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback，Arxiv 2022，论文
• Teaching language models to support answers with verified quotes，Arxiv 2022，论文
• InstructGPT：Training language models to follow instructions with human feedback，Arxiv 2022，论文
• Improving alignment of dialogue agents via targeted human judgements，Arxiv 2022，论文
• Scaling Laws for Reward Model Overoptimization，Arxiv 2022，论文
• Scalable Oversight：Measuring Progress on Scalable Oversight for Large Language Models，Arxiv 2022，论文

安全校准

• Red Teaming Language Models with Language Models，Arxiv 2022，论文
• Constitutional ai: Harmlessness from ai feedback，Arxiv 2022，论文
• The Capacity for Moral Self-Correction in Large Language Models，Arxiv 2023，论文
• OpenAI: Our approach to AI safety，2023，博客

真实性一致性

• Reinforcement Learning for Language Models，2023，博客

提示实用指南

• OpenAI Cookbook，博客
• Prompt Engineering，博客
• ChatGPT Prompt Engineering for Developers!，课程

开源社区的工作

• Self-Instruct: Aligning Language Model with Self Generated Instructions，Arxiv 2022，论文
• Alpaca，代码库
• Vicuna，代码库
• Dolly，博客
• DeepSpeed-Chat，博客
• GPT4All，代码库
• OpenAssitant，代码库
• ChatGLM，代码库
• MOSS，代码库
• Lamini，代码库/博客

常见问题（FAQ）

LLM进化树图怎么看？不同颜色和方块代表什么？

进化树图中，蓝色分支为仅解码器模型，粉红色为仅编码器，绿色为编码器-解码器。实心方块代表开源模型，空心代表闭源。垂直位置对应发布日期，同一分支上的模型关系更近。

大语言模型发展趋势有哪些？

仅解码器模型逐渐占据主导，OpenAI保持领先，Meta对开源贡献大，语言模型有转向闭源趋势，编码器-解码器架构仍有前景但仅解码器模型更灵活。

有哪些LLM实用指南资源？

论文提供了模型、数据、NLP任务的实用指南资源列表，包括BERT风格和GPT风格模型的论文链接，并持续更新，帮助从业者了解LLM在NLP应用中的用法。