论文标题
培训和评估Jupyter笔记本数据科学助理
Training and Evaluating a Jupyter Notebook Data Science Assistant
论文作者
论文摘要
我们通过训练所有公开可用的jupyter笔记本github存储库并开发新的指标：数据科学问题（DSP），研究了由序列到序列变压器提供支持的数据科学助理的可行性。 DSP是来自306个教学笔记本，具有92个数据集依赖性，自然语言和降价问题描述以及基于断言的单位测试的1119个问题的集合。这些笔记本旨在测试大学生对数学和数据科学实施各种Python实施的掌握，现在我们利用它们来研究Jupyt5理解和通过测试的能力。我们分析了DSP的含量，验证其质量，并且发现Jupyt5的100次抽样尝试可以解决DSP问题的77.5％。我们进一步介绍了各种消融和统计分析，并将DSP与其他最新自然语言与编码基准进行比较。
We study the feasibility of a Data Science assistant powered by a sequence-to-sequence transformer by training a new model JuPyT5 on all publicly available Jupyter Notebook GitHub repositories and developing a new metric: Data Science Problems (DSP). DSP is a collection of 1119 problems curated from 306 pedagogical notebooks with 92 dataset dependencies, natural language and Markdown problem descriptions, and assert-based unit tests. These notebooks were designed to test university students' mastery of various Python implementations of Math and Data Science, and we now leverage them to study the ability of JuPyT5 to understand and pass the tests. We analyze the content of DSP, validate its quality, and we find that given 100 sampling attempts JuPyT5 is able to solve 77.5\% of the DSP problems. We further present various ablation and statistical analyses and compare DSP to other recent natural language to code benchmarks.
