Data Sources

Overview

Every dataset in this project is free and public. CAISO market data is pulled with the open-source gridstatus library, which reads directly from CAISO’s own publication channels (OASIS and caiso.com) - no API key, no quota, no paid services. Weather comes from the free Open-Meteo historical archive (ERA5 reanalysis), and plant metadata from EIA’s public 860M monthly release. The GridStatus.io hosted API service is not used at this time.

Datasets

Dataset	Source	Granularity	Description
CAISO LMPs (DAM, RT 15-min, RT 5-min)	CAISO (GridStatus)	Hourly / 15-min / 5-min	Hub APNode prices for the spreads workstream
CAISO fuel prices and regions	CAISO (GridStatus)	Hourly / snapshot	Citygate gas legs for MIHR, spark spread, HRCO
CAISO load, fuel mix, renewables	CAISO (GridStatus)	5-min / hourly	Grid operational context (net load, duck curve)
Curtailed generator report + derived outages	CAISO (GridStatus)	Daily / hourly derived	Generator-level outage reasons; quanto conditioning
Station temperatures	Open-Meteo (ERA5)	Daily	HDD/CDD legs at six California stations
EIA-860M generators	EIA (bulk download)	Monthly snapshot	Plant coordinates, technology, capacity (plant map)

Full per-dataset documentation lives on the DataFrames pages of this site.

Data Pipeline

• Pull: one module per dataset under src/pulls/, orchestrated by pull_all_caiso.py (the pull_caiso doit task). Each run tops every dataset up through today, repairs any gaps, and refreshes the snapshot tables (fuel regions, EIA-860M).

• Store: typed Parquet caches under data/opensource/ - naive-UTC timestamps, Float64 numerics, local-time columns dropped at ingest. Merges dedup on each dataset’s natural key, so no date range is ever pulled twice.

• Derive: the hourly outage aggregation is rebuilt from the cached generator report at the end of every pull run; DAM-RT spreads are computed in the analysis layer from the cached LMP legs.