Evaluation and Implementation of Code Search using Transformers to Enhance Developer Productivity

Abstract: With the rapid advancements in the field of Natural Language Processing and Artificial Intelligence, several aspects of its use cases and impact on productivity are largely unexplored. Many of the recent machine learning models are based on an architecture called Transformers that allows for faster computation and for more context to be preserved. At the same time, tech companies face the dilemmas of how to navigate their code bases, spanning over millions of lines of code. The aim of this thesis is to investigate whether the implementation and fine-tuning of a Transformers-based model can be utilised to improve the code search process in a tech company, leading to improvements in developer productivity. Specifically, the thesis will evaluate the effectiveness of such implementation from a productivity perspective in terms of velocity, quality, and satisfaction. The research uses a mixed method design consisting of two distinct methodologies as well as analyses of quantitative and qualitative data. To assess the level of accuracy that can be obtained by optimising a Transformers-based model on internal data, an evaluative experiment with various internal datasets was conducted. The second methodology applied was a usability test, investigating potential impacts on velocity, quality, and satisfaction by testing a contextual code-search prototype with developers. Data from the tests was analysed through a heat map-, trade-off- and template analysis. Results indicate that a Transformers-based modes can be optimised for code search on internal data and has the potential to improve code search from the aspects of velocity, quality, and satisfaction.