Indian Market Historical Data
An automated open-source data pipeline and master catalog delivering 35M+ historical records across Indian equities, mutual funds, SGBs, macro indicators, and corporate actions.
Lead Architect & Developer
Capital Markets & Data Rails
Python • Parquet • GitHub Actions
Quant Researchers, Analysts & Developers
Problem Statement & Landscape
The Core Challenge
Reliable historical market data in India is traditionally locked behind expensive proprietary enterprise terminals (Bloomberg, Refinitiv) or fragmented across disparate public sources with breaking schema changes and missing adjustments. The challenge was to architect a zero-cost, automated data engineering pipeline that ingests, cleans, normalizes, and packages over 35 million records across stocks, indices, mutual funds, SGBs, and macro indicators into high-performance Apache Parquet partitions updated daily.
Execution Scope
Product Responsibilities
Architected end-to-end automated ETL data pipelines via GitHub Actions
Designed partitioned Apache Parquet storage hierarchy for high-speed columnar querying
Built resilient scrapers and API consumers with automated retry and fallback logic
Curated master catalogs for 43,000+ corporate actions and all 67 Sovereign Gold Bond tranches
Engineered automated daily delta updates and data validation checks
Published open-source documentation and zero-dependency Python starter scripts
System Breakdown
Key Contributions & Workflows
The core product features, technical workflows, and data pipelines I spearheaded:
High-Performance Parquet Data Lake
Engineered partitioned columnar storage spanning 6,970+ NSE/BSE equities, 37,800+ mutual fund schemes (35.3M+ daily NAV records), and 50+ years of bullion history compressed into just ~1.05 GB.
Automated Daily GitHub Actions ETL
Configured multi-stage scheduled CI/CD workflows that automatically scrape, compute corporate action adjustments, validate checksums, and commit daily market closes with zero manual intervention.
Institutional Flows & Macro Tracking
Built daily aggregators tracking FII/DII cash market investments, derivatives participant open interest, FII Long Ratio %, India 10Y Benchmark G-Sec yield, and monthly CPI inflation from 1957.
Sovereign Gold Bond Master Engine
Compiled a complete catalog of all 67 SGB tranches (2015-2024), computing semi-annual coupon schedules, redemption payouts, secondary market quotes, and real-time discount/premium vs. spot gold.
Infrastructure Rails
Product Ecosystem & Integrations
Third-party APIs, data feeds, and core services mapped to power this product:
Data Pipelines
Data Sources
Storage & Format
Distribution
Outcomes & Results
Product Impact
Assembled 35.3M+ mutual fund records and 6,970+ equities into open Parquet formats
Zero-cost automated pipeline executing daily without external database hosting overhead
Empowered quant researchers, algotraders, and developers with clean, adjusted historical data
Built comprehensive master catalogs for 43,700+ corporate actions and 67 SGB tranches
Retrospective
“Data quality in financial systems is built on deterministic validation. Packaging massive datasets into columnar Parquet files drastically reduces compute and bandwidth costs while unlocking sub-second analytics for the community.”