LOCK-IN FREE : Lock-In Free Research Enablement Environment

Abstract: Lock-in Free with its Lock-in-Free Research Enablement Environment aims to provide platforms for academic and industrial researchers to conduct research requiring big data, while at thesame time, preventing them from getting locked-in to a specific vendor of a data scientist notebook and a cloud platform. The thesis benefits the particular graduate-level course titled ScaDaMaLe for PhD students in Swedish Universities and industrial data scientists in general. The thesis provides automation tools that users can conveniently use to start up environments for running notebooks in a format of their choice (databricks, Zeppelin, Jupyter) either on local machines (using Docker containers and Docker compose) or on multi-cloud providers (using Terraform by HashiCorp) with the concept of Infrastructure as Code. The thesis integrates a tool called pinot for users to convert notebooks between these three different formats and work on the format of their preference. With the use of pinot and mdbook, instructors can also convert notebooks into static websites and e-books (hosted on GitHub pages) enabling students to access the course contents when there is no kernel at hand to run notebooks. Lastly, the thesis provides an autograder system capable of generating student lecture and assignment notebooks and grading student submission notebooks in different frameworks in a vendor-agnostic manner.