A more accurate forecast can still produce a worse decision

A distributor improves its average forecast accuracy across the whole catalogue. The data science team is pleased. The operations director is not: important products still run out, and the warehouse is still full of things nobody ordered.

The model got better at the metric. The decisions didn’t get any easier.

Often the problem isn’t the model. It’s the question the model was judged against. A forecast that works for a monthly revenue outlook can be useless for replenishing an item with a long supplier lead time. And an average can bury the particular errors that cost the most.

Start with the decision and its deadline

Write down what the forecast is supposed to change. Purchasing needs demand across the replenishment horizon. Staffing needs demand by location and shift. Cash planning needs the timing of receipts and payments. Those uses shouldn’t automatically share the same target, level of detail or measure of success.

Include the time available to act. A forecast that arrives after the order deadline can be statistically excellent and operationally irrelevant. Evaluate it using the data that would really have been available when the decision was made, delays and revisions included.

Hyndman and Athanasopoulos explain the difference between fitted values and genuine forecast evaluation in Forecasting: Principles and Practice. They test on data the model never saw and compare against simple methods. Before you accept a claim of improvement, ask for exactly that comparison.

Ask which mistakes are expensive

Running short of an essential item costs something very different from over-ordering a discretionary one. A business owner should explain those consequences before the team picks what to optimise. The aim is to connect evaluation to the decision, not to invent one universal accuracy score.

Review performance by the segments that drive action: product family, location, horizon, demand pattern. Agree those segments in advance so nobody can pick only the flattering ones. Show the overall picture alongside the failures that matter.

Bring uncertainty into the planning conversation. A single number can hide a wide range of plausible outcomes. Decision-makers need to know which actions stay sensible across that range and which commitments only work if the forecast is exactly right.

Write down every human adjustment

Commercial teams know things the model doesn’t: a promotion, a supplier problem, a product being withdrawn. Keep the original forecast and the adjusted one, with the reason and timing of the change. Later, you can tell the model’s performance apart from management judgement.

The same authors cover structured judgement in their chapter on judgmental forecasts. The leadership takeaway is simple: make adjustments reviewable. An unexplained override means you never learn whether it helped.

And don’t let the budget become the forecast. A target expresses ambition; a forecast is your best honest estimate. Nudging the forecast towards the target removes exactly the early warning you need.

Keep the data, the model and the decision connected

Document changes in product identity, customer scope, source timing and business definitions. A model can appear to drift because what it measures changed underneath it. Investigate the model without checking the data history and you may rebuild something that wasn’t broken.

Keep preparation, definitions and model work joined up, so an analyst can investigate a bad result without reconstructing months of history. When a source changes, decide whether to update the model, adjust how you read it or limit its use for a while.