Residential Electricity Demand Forecasting from Weather
End-to-end pipeline. DSGrid demand profiles plus ERA5 weather via Open-Meteo for NYC.
Summary. Solo end-to-end project. I built a pipeline that integrates DSGrid synthetic residential demand profiles with ERA5 daily weather data via the Open-Meteo API for New York City, producing a multi-year aligned dataset. I benchmarked supervised regression and classification baselines (linear, logistic, gradient boosting) alongside unsupervised methods (PCA, t-SNE, K-means, DBSCAN, hierarchical clustering), and published the full reproducible workflow.
DSAN 5000, Data Science & Analytics, Fall 2025. My first end-to-end project at Georgetown.
Why this project
Utilities plan for peak demand, not average demand, and weather is the biggest lever on where that peak lands. Hong and Fan (2016) and Fumo and Biswas (2015) both show that temperature and calendar features explain broad electricity-demand patterns but fall short of accurate load forecasts on their own. I wanted to test that claim directly on real data: how far can daily weather and seasonal features alone take you, for both predicting and classifying demand, before you need building-level or behavioral data to close the gap?
Data engineering
The unglamorous half of this project was getting two data sources with very different shapes to align cleanly. Synthetic demand profiles at one resolution, ERA5 daily weather at another, all keyed to NYC. I used DSGrid’s synthetic residential demand profiles as the target series and pulled daily ERA5 weather data for New York City through the Open-Meteo API, then joined both to a common daily calendar, carrying forward day-of-week, month, and holiday indicators as calendar features. The result is a multi-year, date-aligned dataset ready for both supervised and unsupervised analysis, with the full join and cleaning logic published in the repo’s technical-details pages.
Models
I ran two parallel tracks. On the supervised side: linear regression to predict daily demand from weather and calendar features, then logistic regression, and gradient boosting to classify days into low, medium, and high demand bins. On the unsupervised side: PCA and t-SNE to look for structure in the weather/demand relationship, then K-means, DBSCAN, and hierarchical clustering to test whether demand days form natural groups or vary continuously. The contrast between the two tracks is the real story, if the supervised models are only picking up broad signal, the clustering should show it too.
Findings
It matched the literature. Regression and classification models captured seasonal and weather-driven variation but performed weakly as predictors, consistent with Hong and Fan’s and Fumo and Biswas’s findings that weather and calendar features aren’t sufficient for strong load-forecast accuracy on their own. The clustering methods reinforced this: rather than separating into clean demand tiers, the data showed continuous variation, no natural “low/medium/high” clusters emerged from the unsupervised methods, which tracks with demand behaving more like a smooth function of temperature than a set of discrete regimes. Pfenninger’s (2017) point about weather-driven time series needing careful handling also proved out in practice, most of the project’s effort went into the join and cleaning pipeline, not the modeling.
What this project taught me
This was my first end-to-end project at Georgetown, and the biggest lesson was that data engineering is the project, not the preamble to it. I went in expecting the modeling comparison to be the interesting part; it turned out the harder and more valuable work was getting two mismatched data sources to line up honestly. If I did this again, I’d bring in a third data source with behavioral or building-level detail (occupancy, building type, appliance mix) from the start, since the literature and my own results both point to weather alone hitting a ceiling.