End user interface for collecting and evaluating company data : Real-time data collection through web-scraping

Abstract: The demand of open and reliable data, in the Era of Big Data is constantly increasing as thediversity of research and the need of trustworthy data as high-quality data is increasesconsiderably the quality of the findings . However, it is very hard to get reliable data for free witha small effort. With an immense progress of tools, on one hand for data scraping, data cleansing,data storing, and on the other hand so many platforms with data that can be scrapped, it isabsolutely crucial to make use of them and easily build data sets with real and trustworthy data,for free and in a user-friendly way. Using several available tools, an application with a graphicaluser interface (GUI) was developed. The possibilities of the applications are: collecting financialdata for any given list of companies, updating an existent data set, build a data set out of thewhole data warehouse(DW), based on several filters, make the data sets available to anyone whouses the application, and build simple visualization of the existent data. To make sure that‘garbage data in – garbage data out’ concept is avoided, a constant analysis of the data quality isperformed, and the quality of the data is adjusted so that it is ready for use in a research project.The work provides a viable solution for collecting data and making it borderless while respectingthe standards of data sharing. The application can collect data from 2 sources, with more than250 features per company. The application is updated with more functionalities and more sourcesof data.