Project-Lab-Energy-Weather-Natgas-Quanto-CAISO

Project-Lab-Energy-Weather-Natgas-Quanto-CAISO#

Last updated: Aug 27, 2026, 3:09:02 PM

Project Notes

Table of Contents#

Notebooks 📖

Pipeline Charts 📈

Pipeline Dataframes 📊

Pipeline Specs#

Pipeline Name

Project-Lab-Energy-Weather-Natgas-Quanto-CAISO

Pipeline ID

CAISO

Lead Pipeline Developer

Brian Nguyen

Contributors

Brian Nguyen, Claire Kim

Git Repo URL

Pipeline Web Page

Pipeline Web Page

Date of Last Code Update

2026-08-27 10:08:05

OS Compatibility

Windows, Linux, macOS

Linked Dataframes

CAISO:caiso_lmp_day_ahead_hourly
CAISO:caiso_lmp_real_time_15_min
CAISO:caiso_lmp_real_time_5_min
CAISO:caiso_fuel_prices
CAISO:caiso_fuel_regions
CAISO:caiso_load
CAISO:caiso_load_hourly
CAISO:caiso_fuel_mix
CAISO:caiso_renewables_hourly
CAISO:caiso_curtailed_non_operational_generator_report
CAISO:caiso_outages_aggregated_hourly
CAISO:openmeteo_temperature_daily
CAISO:eia860m_generators
CAISO:us_ba_boundaries

Build Commands:

doit

About this project#

Summer 2026 Project Lab (Fama-Miller). Two workstreams:

  1. Spreads - DAM-RT (DART), location spreads, time-block spreads, TBx

  2. Quantos - HDD/CDD, MIHR, HRCO strips, HDD-conditional HRCOs (Quantos), Margrabe pricing under zero net cost of carry.

Data Note#

GridStatus.io free tier is 250 API requests / 500k rows per user per month while licensing is finalized. This is purely for Academic research use only; written work must acknowledge the Fama-Miller Center for Research in Finance.

Full-history pulls go through the per-dataset src/pulls/pull_caiso_*.py modules, orchestrated by src/pulls/pull_all_caiso.py (free open-source gridstatus channel), are cached to Parquet in data/opensource/, and are ledger-guarded so no date range is ever pulled twice. The data/ folder is gitignored and fully regenerable - a fresh clone rebuilds it with doit.

Quick Start#

The quickest way to run code in this repo is to use the following steps.

You must have TexLive (or another LaTeX distribution) installed on your computer and available in your path. You can do this by downloading and installing it from here (windows and mac installers).

First, you must have the conda package manager installed (e.g., via Anaconda). However, I recommend using mamba, via miniforge as it is faster and more lightweight than conda.

Create and activate the conda environment:

conda env create -f environment.yml
conda activate Project-Lab-Summer-2026-CAISO

Finally, run the project tasks:

doit

And that’s it!

CAISO data pulls run over the free open-source gridstatus channel and are part of doit: the pull_caiso task tops up every dataset through today whenever any cache is behind. The batch can also be run directly:

python src/pulls/pull_all_caiso.py   # top up every dataset through today (ledger-guarded)

A dataset never pulled before is fetched from START_DATE, and each run ends with a gap-repair pass. Single datasets can be pulled directly, e.g. python src/pulls/pull_caiso_load.py; that resumes from the dataset’s own ledger tail rather than re-fetching history the cache already holds.

To audit coverage or deepen history (e.g. after moving START_DATE back), use the gap tools from Python - scan for missing days, repair, and repeat until the scan comes back empty (OASIS occasionally drops chunks under rate limiting, so one pass is not always enough):

# list missing days per dataset over a range you name
python -c "import sys; sys.path.insert(0,'src/pulls'); from pull_all_caiso import gap_scan; print(gap_scan('2023-05-11','2026-08-27'))"

# pull exactly those missing windows (ledger-recorded, safe to re-run)
python -c "import sys; sys.path.insert(0,'src/pulls'); from pull_all_caiso import backfill; backfill()"

backfill() with no arguments scans START_DATE (2018-01-01) through today and pulls only the ranges gap_scan reports missing, so a clean cache costs nothing. Each dataset’s real span is in data/opensource/ledger.json. Pass dates for a scoped repair, e.g. backfill('2026-01-01', '2026-02-01').

Other commands#

Unit Tests and Doc Tests#

You can run the unit test, including doctests, with the following command:

pytest --doctest-modules

You can build the chartbook site (rendered into docs/) with:

doit build_chartbook_site

or directly:

chartbook build -f

Setting Environment Variables#

You can export your environment variables from your .env files like so, if you wish. This can be done easily in a Linux or Mac terminal with the following command:

set -a  # automatically export all variables
source .env
set +a

On Windows (PowerShell):

Get-Content .env | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [Environment]::SetEnvironmentVariable($matches[1], $matches[2], 'Process') } }

Formatting#

This project uses Ruff for linting and formatting Python code.

# Auto-fix linting issues (e.g., unused imports, undefined names)
ruff check . --fix

# Format code (consistent style, spacing, line length)
ruff format .

# Sort imports, then fix linting issues, then format
ruff format . && ruff check --select I --fix . && ruff check --fix .

  • ruff check --fix applies safe auto-fixes for linting violations

  • ruff format formats code similar to Black

  • --select I targets only import sorting rules (isort-compatible)

General Directory Structure#

  • The assets folder is used for things like hand-drawn figures or other pictures that were not generated from code. These things cannot be easily recreated if they are deleted.

  • The _output folder, on the other hand, contains dataframes and figures that are generated from code. The entire folder should be able to be deleted, because the code can be run again, which would again generate all of the contents.

  • The data folder is this project’s Parquet cache of CAISO pulls (data/opensource/, with a JSON ledger). Like the chartbook-standard _data/ folder it is gitignored and regenerable - doit rebuilds it; only the folder name differs (src/settings.py points DATA_DIR at data).

  • The data_manual is for data that cannot be easily recreated. This data should be version controlled.

  • I’m using the doit Python module as a task runner. It works like make and the associated Makefiles. To rerun the code, install doit (https://pydoit.org/) and execute the command doit from the project root directory (where dodo.py lives). Note that doit is very flexible and can be used to run code commands from the command prompt, thus making it suitable for projects that use scripts written in multiple different programming languages.

  • I’m using the .env file as a container for absolute paths that are private to each collaborator in the project. You can also use it for private credentials, if needed. It should not be tracked in Git.

Data and Output Storage#

Output is stored in the “_output” directory. This includes dataframes, charts, and rendered notebooks. When the output is small enough, I’ll keep this under version control. I like this because I can keep track of how dataframes change as my analysis progresses, for example.

The data directory and _output directory can be kept elsewhere on the machine. To make this easy, I always include the ability to customize these locations by defining the path to these directories in environment variables, which I intend to be defined in the .env file, though they can also simply be defined on the command line or elsewhere. The settings.py is responsible for loading these environment variables and doing some preprocessing on them. The settings.py file is the entry point for all other scripts to these definitions. That is, all code that references these variables and others are loaded by importing config.

Naming Conventions#

  • pull_ vs load_: Files or functions that pull data from an external data source are prepended with “pull_”, as in “pull_caiso_load.py”. Functions that load data that has been cached in the data folder are prepended with “load_”. In this project, each dataset has its own src/pulls/pull_caiso_<dataset>.py module containing its pull_ and load_ functions (plus src/derive_caiso_outages_aggregated_hourly.py for the derived outages dataset), orchestrated by src/pulls/pull_all_caiso.py.

Dependencies and Virtual Environments#

Working with conda environments#

This project uses conda for environment management. The dependencies are stored in environment.yml.

To create/update the environment:

conda env create -f environment.yml
# or to update an existing environment:
conda env update -f environment.yml

To activate the environment:

conda activate Project-Lab-Summer-2026-CAISO

To export the current environment:

conda env export > environment.yml

Tip: Consider using mamba instead of conda for faster package resolution. Install via miniforge.