Pengembangan Aplikasi Web Scraping untuk Crawling Web Data dari Situs E-Commerce Properti

Abstract

Technological developments drive the need for automated web data collection solutions for market analysis on real-estate e-Commerce sites, so this study will discuss the development of a web scraper application as a solution to overcome the challenges of manual data analysis from the original site. The web scraping feature in this application uses Selenium Webdriver and BeautifulSoup technology, and MongoDB as a flexible storage database. This system is also equipped with features for filtering, sorting, and exporting data to Excel for more efficient comparative analysis of real-estate data. This project was developed based on a website with REST API implementation to integrate the frontend with the backend. The frontend was developed using React and Tailwind, while the backend was developed using Flask. Based on the project that has been developed, the test results show that the application is able to dynamically scrape data from various domains at a fairly fast speed. In addition, functional testing also proves that the features in the application function according to specifications and are able to provide effective solutions for real-estate data analysis needs. With this research and development, it is hoped that application users can analyze real-estate data efficiently and be the first step for further development.