Textbooks Are All You Need
- Suriya Gunasekar ,
- Yi Zhang ,
- Jyoti Aneja ,
- Caio Cesar ,
- Teodoro Mendes ,
- Allie Del Giorno ,
- Sivakanth Gopi ,
- Mojan Javaheripi ,
- Piero Kauffmann ,
- Gustavo de Rosa ,
- Olli Saarikivi ,
- Adil Salim ,
- Shital Shah ,
- Harkirat Singh Behl ,
- Xin Wang ,
- Sébastien Bubeck ,
- Ronen Eldan ,
- Adam Tauman Kalai ,
- Yin Tat Lee ,
- Yuanzhi Li
We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of “textbook quality” data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval.
Publication Downloads
Phi-1
December 11, 2023
The language model phi-1 is a Transformer with 1.3 billion parameters, specialized for basic Python coding. Its training involved a variety of data sources, including subsets of Python codes from The Stack v1.2, Q&A content from StackOverflow, competition code from code_contests, and synthetic Python textbooks and exercises generated by gpt-3.5-turbo-0301. Even though the model and the datasets are relatively small compared to contemporary Large Language Models (LLMs), phi-1 has demonstrated an impressive accuracy rate exceeding 50% on the simple Python coding benchmark, HumanEval.
Research Forum Keynote: Research in the Era of AI
Microsoft Research Forum, January 30, 2024 Peter Lee, Corporate Vice President, Microsoft Research and Incubations, discusses how recent developments in AI have transformed the way Microsoft approaches research. See more at https://aka.ms/ResearchForum-Jan2024