Microsoft researchers have found a new way to make computer programs better by improving the instructions they use. They created a method called CodeOcean, which helps generate better and more varied instruction data from open-source code. This makes the programs understand and perform tasks more effectively. CodeOcean addresses challenges like duplicate data and lack of control over data quality during instruction data generation.
The researchers built a dataset called CodeOcean with 20,000 examples of instructions for four types of code-related tasks: Code Summarization, Code Generation, Code Translation, and Code Repair. Their aim is to boost the performance of Code Language Models (LLMs) through a process called instruction tuning. They also introduced a fine-tuned model called WaveCoder, designed to enhance instruction tuning for Code LLMs, showing better generalization across various code-related tasks compared to other similar models.
WaveCoder is based on recent advancements in Large Language Models (LLMs), highlighting the potential of instruction tuning in improving model capabilities for different tasks. Instruction tuning has proven effective in enhancing the generalization abilities of LLMs in various studies. The researchers introduced the concept of alignment, where pre-trained models learn from self-supervised tasks to better understand text inputs. Instruction tuning allows these models to extract more information from instructions and improve their interaction with users.
Compared to existing methods like self-instruct and evol-instruct, which rely on teacher models and may produce duplicate data, the proposed LLM Generator-Discriminator framework in CodeOcean controls data quality by using source code. It generates more realistic instruction data by taking raw code as input and selecting a core dataset while adjusting data diversity through raw code distribution adjustments.
The study classified instruction instances into four code-related tasks, refined the instruction data to create CodeOcean, and introduced WaveCoder models fine-tuned with CodeOcean. These models demonstrated superior generalization abilities and efficiency in code generation tasks compared to other open-source models. WaveCoder consistently outperformed other models on various benchmarks, emphasizing the importance of data quality and diversity in the instruction-tuning process. The research highlights CodeOcean’s superiority in refining instruction data and improving the instruction-following ability of base models compared to the CodeAlpaca dataset.
In conclusion, the research introduces CodeOcean, a multi-task instruction data approach, and WaveCoder models to enhance the generalization ability of Code LLMs. The proposed LLM Generator-Discriminator framework is effective in generating realistic, diverse instruction data, contributing to improved performance across various code-related tasks. Future work may explore the interplay among different tasks and larger datasets to further enhance performance and generalization abilities.