Modelling For Decisions III. Energy Modelers and Black and Dying Swans

Yesterday’s piece was by economists- today’s is about energy systems modeling, from some energy policy and analysis experts. Of course, the energy sector is key to reducing carbon emissions, so perhaps their experience and perspective is valuable.

My favorite quote is the last line.

“But perhaps a start is for decision-makers to adapt to an increasingly uncertain and dynamic world by creating a more imaginative discourse, one that welcomes nuance and doubt as spaces for opportunity and transformative change, and sees forecasts as the beginning of a policy or investment discussion rather than the end, and forecasters not as Delphic oracles of outcome, but as the people who know best why attempts at prediction must fall short.”

I would argue that we stakeholders and public, and of course, practitioners and scientists outside the modelling community should participate in that imaginative discourse. Nuance and doubt are our friends, IMHO, and yield more robust paths forward.

“We explore two challenges to forecasting complex systems which can lead to large forecast errors. Though not an exhaustive list, these two challenges lead to a significant fraction of large forecast errors and are of central importance to energy system modeling. The first challenge is that in complex systems, there are more variables than can be considered. Often described as epistemic uncertainty, these un-modeled variables—the unknown unknowns—can lead to reality diverging dramatically from forecasts. The second challenge in forecasting complex systems is from the inherently nonlinear nature of many such systems. This results in a compounding of stochastic uncertainties—the known unknowns—which in turn can result in real-world outcomes that deviate significantly from forecasts.”

Remember from yesterday, Idea 1 was “weather-like vs. climate-like” forecasting with weather-like having more chances to check in with the real world. Idea 2 was the concept of Big Surprises which seem unpredictable. Today’s authors’ experience is that “the future is often directed by unlikely events.” They also relate epistemic uncertainty to black swans and stochastic uncertainties to dying swans. Perhaps the longer the projection, the greater the probability that unlikely events will overwhelm likely events? For that reason, some have suggested focusing on the short to medium term for projections.

In many cases, modeling apologetics are insightful and accurate, and meaningfully contribute to improved future forecasts. But there are reasons to question the universality of this narrative. Explaining away modeling errors as due to one-off unlikely events misses the prevalence of errors caused by such events, and may lure us (especially those of us who are non-modelers but rely upon model outputs) toward a heuristic of naturalistic equilibrium: a belief that “now things are normal,” or that they will soon be. The history of the energy system teaches us that the future is often directed by unlikely events, and that there is value in questioning whether naturalistic analogies of equilibrium are appropriate in many cases. Energy systems may experience multiple years, or even decades, of disequilibrium due to complex and shifting market rules, uncertainties of technological or economic feasibility at nonlinear scales of deployment, and extraordinary diversity in market structure, composition, and actors. The enormity of such extraneous uncertainties places any forecaster in very deep water.

The questions raised here, and the types of forecast errors described, should be expanded to other sectors. The rapid pace of advancement and interconnectedness of the world means that epistemic uncertainty is larger than ever [50]. For many newer technologies, the degree of uncertainty regarding future generation has increased in recent years. Cost declines in technologies such as wind, solar, and energy storage place them on competitive terms with conventional generation technologies. These technologies have shown even more stark learning-by-doing effects than shale gas production. Markets for electric cars and demand response, to name a few, similarly pose the possibility for dramatic shifts. Even moderate changes in cost and policies can lead to large changes in the future adoption of these technologies.

Decision-makers and investors would benefit from learning more about why they were caught unawares by the shale revolution, and how they can be better prepared the next time such a surprise occurs. The answer to that second question is not immediately apparent. Tautologically, if we were prepared for them, surprises would cease to be surprises. But perhaps a start is for decision-makers to adapt to an increasingly uncertain and dynamic world by creating a more imaginative discourse, one that welcomes nuance and doubt as spaces for opportunity and transformative change, and sees forecasts as the beginning of a policy or investment discussion rather than the end, and forecasters not as Delphic oracles of outcome, but as the people who know best why attempts at prediction must fall short.