Forecasting demand before the peak hits, on Databricks, for an electricity utility

The forecast a utility can’t afford to get wrong at the peak

For an electricity utility the whole game is the peak, not the average. Demand sits in a comfortable band for most of the year, then a heatwave lands and the system is pushed to its limit in a few hours on a few afternoons. Those hours decide whether there is enough generation committed, whether the network holds, and what the day costs. Get the average right and the peak wrong, and the forecast has failed where it mattered.

The old method does not cope. Demand forecasting in many utilities still leans on a trend line over historical reads, nudged by an analyst’s judgement and a weather rule of thumb. It reads the easy middle of the range well and the extremes poorly, which is the wrong way round. A trend cannot tell a 38-degree Thursday from a mild one, cannot see a building heatwave the way a model trained on temperature can, and gives its least reliable answer on exactly the days the utility most needs to trust it.

Two forces have made this harder, not easier. Demand is increasingly driven by weather as more cooling load comes online, so the peaks are sharper and more temperature-sensitive than the historical average suggests. And rooftop solar has quietly broken the meter as a measure of demand. Behind-the-meter generation means the grid sees less than the underlying demand on a sunny day, and the peak shifts later into the evening as the sun drops. The obligations sit underneath all of it. Forecasts feed bidding and planning in the National Electricity Market that AEMO operates, under the rules the Australian Energy Regulator oversees, so a forecast is not an academic exercise. It informs real positions and real spend.

Why Databricks, and what the forecasting stack looks like

The aim is a demand forecast the operations and trading teams can plan against, accurate where it counts, and explainable when someone asks why. We headline these builds on Databricks because the work is machine learning on a large, high-frequency time series, and that is what the lakehouse is built for. Years of half-hourly interval and SCADA reads across many connection points run to billions of rows. Databricks handles that volume, lets us engineer features over it in one place, and tracks every model version in MLflow so a forecast can be reproduced rather than trusted on faith.

The stack sits around that core. A governed data layer in Snowflake holds the interval reads, the weather history and forecasts, and the calendar of holidays and seasons, so the features are built from one consistent, governed source. On Databricks we engineer the features that actually drive demand, lagged load, temperature and recent heat, time of day and day of week, holiday and seasonal flags, and the rooftop-solar effect, then train and validate the models. MLflow records each run, its features and its accuracy on held-out peak days, so models are versioned and comparable instead of living in one analyst’s notebook. Power BI then puts the forecasts and their error in front of the planning team, where the people making the calls already work.

A trend or a BI-only approach cannot do this part. A dashboard can show what demand was. It cannot learn how temperature, time and solar interact to set tomorrow’s peak, and it cannot be trained to be most accurate on the days that cost the most. That is the line between reporting the past and forecasting the peak, and it is why the modelling sits on a platform built for it rather than in a spreadsheet.

Building it, and where it got hard

The friction in demand forecasting is not the algorithm. It is that the thing you most need to predict, the peak, is both the rarest case and the one the data hides. One problem stood in for the rest. The model that scored well on average was wrong in the worst possible place.

Early on the models posted a respectable error across the whole year and still missed the summer peaks. Two causes ran together. First, the peaks are rare, so a model trained to minimise average error learns the easy middle and treats the extremes as noise. Second, rooftop solar masks the true demand. On a sunny afternoon metered demand understates the underlying demand and the real peak slides into the early evening as solar output falls, so a model trained on raw metered reads was learning the wrong shape of the day. A forecast that is accurate on a mild Tuesday and wrong on a 40-degree Friday is the opposite of useful here.

The fix was about the data and the training, not a fancier model. We built explicit weather and calendar features so the model could read a building heatwave and a holiday the way the system actually responds. We handled the rooftop-solar effect directly, modelling behind-the-meter generation so the forecast reflected underlying demand rather than what the meter saw after the panels had worked. We weighted the training toward peak periods, so the model paid for its errors where errors are expensive. And we validated on held-out peak days specifically, tracking that in MLflow, so peak accuracy was something we measured and could trust rather than hoped for. One constraint shaped the rest. Weather forecasts carry their own error, so we were careful not to present a demand forecast as more certain than the weather it rests on, and we kept the planning teams sighted on that.

What changed

In a representative build the forecast got noticeably better where it counts. Error on the daily maximum demand fell by roughly a third against the previous trend-based method, because the model was trained and validated to be accurate on the peaks rather than the easy middle. The operations and trading teams could trust forecasts that ran several days further ahead, which gave them a real window to position generation and network capacity before a hot spell rather than reacting on the day. And because every model and its validation lived in MLflow, a forecast could be explained and rerun, not defended from memory.

These figures are illustrative. They describe the pattern we see rather than a published result for a named utility. The shape is the point. The forecast stops failing at the peak, the planning teams get more lead time on the days that matter, and the call on how much generation and network to commit runs on a model that reads weather, calendar and solar the way the system actually does. That supports both reliable supply and the efficient use of generation that the utility is trying to plan for.

Where this fits

Demand forecasting on Databricks is one application of our Artificial Intelligence service, for the realities of an Australian electricity utility. It is a contained, high-value place to start, because the data already exists and the value comes from modelling the peaks properly and getting the forecast in front of the planning team. It sits apart from a consumption-reporting dashboard, which tells you what demand was. This tells you what the peak will be. If your demand forecast is most wrong on the days it most needs to be right, the place to start is to map your interval, weather and calendar data and decide which decisions a better peak forecast would change.