Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation

We introduce Bonito, an open-source model for conditional task generationthat converts unannotated text into task-specific training datasets forinstruction tuning. We aim to enable zero-shot task adaptation of largelanguage models on users’ specialized, private data. We train Bonito byfine-tuning a pretrained large language model on a new large-scale dataset with1.65M examples created by remixing existing instruction tuning datasets intometa-templates. The meta-templates for a dataset produce training exampleswhere the input is the unannotated text and the task attribute and the outputconsists of the instruction and the response. We use Bonito to generatesynthetic tasks for seven datasets from specialized domains with unannotatedtext across three task types – yes-no question answering, extractive questionanswering, and natural language inference – and adapt language models. We showthat Bonito significantly improves the average performance of pretrained andinstruction tuned models over the de facto self supervised baseline. Forexample, adapting Mistral-Instruct-v2 and instruction tuned variants of Mistraland Llama2 with Bonito improves the strong zero-shot performance by 22.1 F1points whereas the next word prediction objective undoes some of the benefitsof instruction tuning and reduces the average performance by 0.8 F1 points. Weconduct additional experiments with Bonito to understand the effects of thedomain, the size of the training set, and the choice of alternative synthetictask generators. Overall, we show that learning with synthetic instructiontuning datasets is an effective way to adapt language models to new domains.The model, dataset, and code are available athttps://github.com/BatsResearch/bonito.

Further reading#

• Access Paper in arXiv.org