# Data Sources

## Overview

Every dataset in this project is free and public. CAISO market data is pulled
with the open-source [gridstatus](https://opensource.gridstatus.io) library,
which reads directly from CAISO's own publication channels (OASIS and
caiso.com) - no API key, no quota, no paid services. Weather comes from the
free Open-Meteo historical archive (ERA5 reanalysis), and plant metadata from
EIA's public 860M monthly release. The GridStatus.io hosted API service is
not used at this time.

## Datasets

| Dataset | Source | Granularity | Description |
|---------|--------|-------------|-------------|
| CAISO LMPs (DAM, RT 15-min, RT 5-min) | CAISO (GridStatus) | Hourly / 15-min / 5-min | Hub APNode prices for the spreads workstream |
| CAISO fuel prices and regions | CAISO (GridStatus) | Hourly / snapshot | Citygate gas legs for MIHR, spark spread, HRCO |
| CAISO load, fuel mix, renewables | CAISO (GridStatus) | 5-min / hourly | Grid operational context (net load, duck curve) |
| Curtailed generator report + derived outages | CAISO (GridStatus) | Daily / hourly derived | Generator-level outage reasons; quanto conditioning |
| Station temperatures | Open-Meteo (ERA5) | Daily | HDD/CDD legs at six California stations |
| EIA-860M generators | EIA (bulk download) | Monthly snapshot | Plant coordinates, technology, capacity (plant map) |

Full per-dataset documentation lives on the DataFrames pages of this site.

## Data Pipeline

- **Pull:** one module per dataset under `src/pulls/`, orchestrated by
  `pull_all_caiso.py` (the `pull_caiso` doit task). Each run tops every
  dataset up through today, repairs any gaps, and refreshes the snapshot
  tables (fuel regions, EIA-860M).
- **Store:** typed Parquet caches under `data/opensource/` - naive-UTC
  timestamps, Float64 numerics, local-time columns dropped at ingest. Merges
  dedup on each dataset's natural key, so no date range is ever pulled twice.
- **Derive:** the hourly outage aggregation is rebuilt from the cached
  generator report at the end of every pull run; DAM-RT spreads are computed
  in the analysis layer from the cached LMP legs.