IDX and KSEI Web Scraping

The purpose of this project is to scrape stocks and government / corporate bonds data available in Indonesia from Indonesia Stock Exchange (IDX / BEI) and KSEI websites.

  • Type Jupyter Notebook
VIEW PROJECT

Goals:

Scrape stocks and bonds data and store it in Excel format to support future data analysis. Data will also be available through FastAPI on Render.
 

Features:

  1. Scrape IDX individual stock summary and details :
    • Company Profiles (scheduled to run every first day of the month)
    • Today Trading Info (scheduled to run every Saturday)
    • Financial Reports Links (scheduled to run every first day of the month)
  2. Scrape IDX stock sectoral summary
  3. Scrape IDX stock index summary
  4. Scrape corporate and government bonds summary and details
  5. FastAPI deployment

Code Methodology:

1. Identify what kind of data that are extractable from the websites
2. Run web scraping with BeautifulSoup and Selenium (IDX has Cloudflare protection, KSEI has none) / Requests library
3. Utilize multi-threading library in Python
4. Apply data cleaning and normalization techniques to ensure the data was consistent and accurate