# Project-Lab-Energy-Weather-Natgas-Quanto-CAISO Last updated: {sub-ref}`today` ```{toctree} :maxdepth: 1 :caption: Project Notes project_overview ``` ## Table of Contents ```{toctree} :maxdepth: 1 :caption: Notebooks 📖 Data Exploration Outage Reasons Weather Indices Regional Temperature Mapping Spark Spreads Generator Weather Map Quanto Pricing Quanto in Practice Basis Risk Exploration Negative Binomial Outage Model County Outage Model Outage Specification Search Outage Reason Codes and Derate Depth Outage Granularity Comparison ``` ```{toctree} :maxdepth: 1 :caption: Pipeline Charts 📈 cb/charts.md ``` ```{postlist} :format: "{title}" ``` ```{toctree} :maxdepth: 1 :caption: Pipeline Dataframes 📊 cb/dataframes/CAISO/caiso_curtailed_non_operational_generator_report.md cb/dataframes/CAISO/caiso_fuel_mix.md cb/dataframes/CAISO/caiso_fuel_prices.md cb/dataframes/CAISO/caiso_fuel_regions.md cb/dataframes/CAISO/caiso_lmp_day_ahead_hourly.md cb/dataframes/CAISO/caiso_lmp_real_time_15_min.md cb/dataframes/CAISO/caiso_lmp_real_time_5_min.md cb/dataframes/CAISO/caiso_load.md cb/dataframes/CAISO/caiso_load_hourly.md cb/dataframes/CAISO/caiso_outages_aggregated_hourly.md cb/dataframes/CAISO/caiso_renewables_hourly.md cb/dataframes/CAISO/eia860m_generators.md cb/dataframes/CAISO/openmeteo_temperature_daily.md cb/dataframes/CAISO/us_ba_boundaries.md ``` ```{toctree} :maxdepth: 1 :caption: Appendix 💡 myst_markdown_demos.md apidocs/index ``` ## Pipeline Specs | Pipeline Name | Project-Lab-Energy-Weather-Natgas-Quanto-CAISO | |---------------------------------|--------------------------------------------------------| | Pipeline ID | [CAISO](./index.md) | | Lead Pipeline Developer | Brian Nguyen | | Contributors | Brian Nguyen, Claire Kim | | Git Repo URL | | | Pipeline Web Page | Pipeline Web Page | | Date of Last Code Update | 2026-08-27 10:08:05 | | OS Compatibility | Windows, Linux, macOS | | Linked Dataframes | [CAISO:caiso_lmp_day_ahead_hourly](cb/dataframes/CAISO/caiso_lmp_day_ahead_hourly.md)
[CAISO:caiso_lmp_real_time_15_min](cb/dataframes/CAISO/caiso_lmp_real_time_15_min.md)
[CAISO:caiso_lmp_real_time_5_min](cb/dataframes/CAISO/caiso_lmp_real_time_5_min.md)
[CAISO:caiso_fuel_prices](cb/dataframes/CAISO/caiso_fuel_prices.md)
[CAISO:caiso_fuel_regions](cb/dataframes/CAISO/caiso_fuel_regions.md)
[CAISO:caiso_load](cb/dataframes/CAISO/caiso_load.md)
[CAISO:caiso_load_hourly](cb/dataframes/CAISO/caiso_load_hourly.md)
[CAISO:caiso_fuel_mix](cb/dataframes/CAISO/caiso_fuel_mix.md)
[CAISO:caiso_renewables_hourly](cb/dataframes/CAISO/caiso_renewables_hourly.md)
[CAISO:caiso_curtailed_non_operational_generator_report](cb/dataframes/CAISO/caiso_curtailed_non_operational_generator_report.md)
[CAISO:caiso_outages_aggregated_hourly](cb/dataframes/CAISO/caiso_outages_aggregated_hourly.md)
[CAISO:openmeteo_temperature_daily](cb/dataframes/CAISO/openmeteo_temperature_daily.md)
[CAISO:eia860m_generators](cb/dataframes/CAISO/eia860m_generators.md)
[CAISO:us_ba_boundaries](cb/dataframes/CAISO/us_ba_boundaries.md)
| **Build Commands:** ``` doit ``` ## About this project Summer 2026 Project Lab (Fama-Miller). Two workstreams: 1. **Spreads** - DAM-RT (DART), location spreads, time-block spreads, TBx 2. **Quantos** - HDD/CDD, MIHR, HRCO strips, HDD-conditional HRCOs (Quantos), Margrabe pricing under zero net cost of carry. ### Data Note GridStatus.io free tier is **250 API requests / 500k rows per user per month** while licensing is finalized. This is purely for Academic research use only; written work must acknowledge the Fama-Miller Center for Research in Finance. Full-history pulls go through the per-dataset `src/pulls/pull_caiso_*.py` modules, orchestrated by `src/pulls/pull_all_caiso.py` (free open-source `gridstatus` channel), are cached to Parquet in `data/opensource/`, and are ledger-guarded so no date range is ever pulled twice. The `data/` folder is gitignored and fully regenerable - a fresh clone rebuilds it with `doit`. ## Quick Start The quickest way to run code in this repo is to use the following steps. You must have TexLive (or another LaTeX distribution) installed on your computer and available in your path. You can do this by downloading and installing it from here ([windows](https://tug.org/texlive/windows.html#install) and [mac](https://tug.org/mactex/mactex-download.html) installers). First, you must have the `conda` package manager installed (e.g., via Anaconda). However, I recommend using `mamba`, via [miniforge](https://github.com/conda-forge/miniforge) as it is faster and more lightweight than `conda`. Create and activate the conda environment: ```bash conda env create -f environment.yml conda activate Project-Lab-Summer-2026-CAISO ``` Finally, run the project tasks: ```bash doit ``` And that's it! CAISO data pulls run over the free open-source `gridstatus` channel and are part of `doit`: the `pull_caiso` task tops up every dataset through today whenever any cache is behind. The batch can also be run directly: ```bash python src/pulls/pull_all_caiso.py # top up every dataset through today (ledger-guarded) ``` A dataset never pulled before is fetched from `START_DATE`, and each run ends with a gap-repair pass. Single datasets can be pulled directly, e.g. `python src/pulls/pull_caiso_load.py`; that resumes from the dataset's own ledger tail rather than re-fetching history the cache already holds. To audit coverage or deepen history (e.g. after moving `START_DATE` back), use the gap tools from Python - scan for missing days, repair, and repeat until the scan comes back empty (OASIS occasionally drops chunks under rate limiting, so one pass is not always enough): ```bash # list missing days per dataset over a range you name python -c "import sys; sys.path.insert(0,'src/pulls'); from pull_all_caiso import gap_scan; print(gap_scan('2023-05-11','2026-08-27'))" # pull exactly those missing windows (ledger-recorded, safe to re-run) python -c "import sys; sys.path.insert(0,'src/pulls'); from pull_all_caiso import backfill; backfill()" ``` `backfill()` with no arguments scans `START_DATE` (2018-01-01) through today and pulls only the ranges `gap_scan` reports missing, so a clean cache costs nothing. Each dataset's real span is in `data/opensource/ledger.json`. Pass dates for a scoped repair, e.g. `backfill('2026-01-01', '2026-02-01')`. ### Other commands #### Unit Tests and Doc Tests You can run the unit test, including doctests, with the following command: ``` pytest --doctest-modules ``` You can build the chartbook site (rendered into `docs/`) with: ``` doit build_chartbook_site ``` or directly: ``` chartbook build -f ``` #### Setting Environment Variables You can [export your environment variables](https://stackoverflow.com/questions/43267413/how-to-set-environment-variables-from-env-file) from your `.env` files like so, if you wish. This can be done easily in a Linux or Mac terminal with the following command: ```bash set -a # automatically export all variables source .env set +a ``` On Windows (PowerShell): ```powershell Get-Content .env | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [Environment]::SetEnvironmentVariable($matches[1], $matches[2], 'Process') } } ``` ### Formatting This project uses [Ruff](https://docs.astral.sh/ruff/) for linting and formatting Python code. ```bash # Auto-fix linting issues (e.g., unused imports, undefined names) ruff check . --fix # Format code (consistent style, spacing, line length) ruff format . # Sort imports, then fix linting issues, then format ruff format . && ruff check --select I --fix . && ruff check --fix . ``` - `ruff check --fix` applies safe auto-fixes for linting violations - `ruff format` formats code similar to Black - `--select I` targets only import sorting rules (isort-compatible) ### General Directory Structure - The `assets` folder is used for things like hand-drawn figures or other pictures that were not generated from code. These things cannot be easily recreated if they are deleted. - The `_output` folder, on the other hand, contains dataframes and figures that are generated from code. The entire folder should be able to be deleted, because the code can be run again, which would again generate all of the contents. - The `data` folder is this project's Parquet cache of CAISO pulls (`data/opensource/`, with a JSON ledger). Like the chartbook-standard `_data/` folder it is gitignored and regenerable - `doit` rebuilds it; only the folder name differs (`src/settings.py` points `DATA_DIR` at `data`). - The `data_manual` is for data that cannot be easily recreated. This data should be version controlled. - I'm using the `doit` Python module as a task runner. It works like `make` and the associated `Makefile`s. To rerun the code, install `doit` (https://pydoit.org/) and execute the command `doit` from the project root directory (where `dodo.py` lives). Note that doit is very flexible and can be used to run code commands from the command prompt, thus making it suitable for projects that use scripts written in multiple different programming languages. - I'm using the `.env` file as a container for absolute paths that are private to each collaborator in the project. You can also use it for private credentials, if needed. It should not be tracked in Git. ### Data and Output Storage Output is stored in the "_output" directory. This includes dataframes, charts, and rendered notebooks. When the output is small enough, I'll keep this under version control. I like this because I can keep track of how dataframes change as my analysis progresses, for example. The `data` directory and `_output` directory can be kept elsewhere on the machine. To make this easy, I always include the ability to customize these locations by defining the path to these directories in environment variables, which I intend to be defined in the `.env` file, though they can also simply be defined on the command line or elsewhere. The `settings.py` is responsible for loading these environment variables and doing some preprocessing on them. The `settings.py` file is the entry point for all other scripts to these definitions. That is, all code that references these variables and others are loaded by importing `config`. ### Naming Conventions - **`pull_` vs `load_`**: Files or functions that pull data from an external data source are prepended with "pull_", as in "pull_caiso_load.py". Functions that load data that has been cached in the data folder are prepended with "load_". In this project, each dataset has its own `src/pulls/pull_caiso_.py` module containing its `pull_` and `load_` functions (plus `src/derive_caiso_outages_aggregated_hourly.py` for the derived outages dataset), orchestrated by `src/pulls/pull_all_caiso.py`. ### Dependencies and Virtual Environments #### Working with `conda` environments This project uses conda for environment management. The dependencies are stored in `environment.yml`. To create/update the environment: ```bash conda env create -f environment.yml # or to update an existing environment: conda env update -f environment.yml ``` To activate the environment: ```bash conda activate Project-Lab-Summer-2026-CAISO ``` To export the current environment: ```bash conda env export > environment.yml ``` **Tip:** Consider using `mamba` instead of `conda` for faster package resolution. Install via [miniforge](https://github.com/conda-forge/miniforge).