Projects Category: Data Engineering

Home
Data Engineering

Data Analysis | Data Engineering

Synthetic Transaction Data Analysis

Synthetic Transaction Data Analysis

Here is my project about Analyzing Transaction Data. The dataset for this project is synthetically generated using SDV library from a transaction data sample that I have.

Type Website

VIEW PROJECT

Goals:

Analyze transaction data and create a dashboard with Streamlit library, then deploy it to Render.

Data Engineering

IDX and KSEI Web Scraping

IDX and KSEI Web Scraping

The purpose of this project is to scrape stocks and government / corporate bonds data available in Indonesia from Indonesia Stock Exchange (IDX / BEI) and KSEI websites.

Type Jupyter Notebook

VIEW PROJECT

Goals:

Scrape stocks and bonds data and store it in Excel format to support future data analysis. Data will also be available through FastAPI on Render.

Features:

Scrape IDX individual stock summary and details :
- Company Profiles (scheduled to run every first day of the month)
- Today Trading Info (scheduled to run every Saturday)
- Financial Reports Links (scheduled to run every first day of the month)
Scrape IDX stock sectoral summary
Scrape IDX stock index summary
Scrape corporate and government bonds summary and details
FastAPI deployment

Code Methodology:

1. Identify what kind of data that are extractable from the websites

2. Run web scraping with BeautifulSoup and Selenium (IDX has Cloudflare protection, KSEI has none) / Requests library

3. Utilize multi-threading library in Python

4. Apply data cleaning and normalization techniques to ensure the data was consistent and accurate