Role of science

In this post, we’ll juxtapose two articles about the science of coronavirus, one from Dan Sarewitz, who is a scientist who studies the interface between science, technology, and policy, and one a journalist at the WaPo. Plus we’ll also link to an essay by a law professor about Federalism and pandemic responses, and finally go to an article in the journal Science about pandemic modeling. Apologies for the length of this post, but I’ve tried to point you to some interesting takes on the same issue and also relate it to science and our standard TWS policy disputes.

Here are excerpts from Dan’s essay (worth reading in its entirety):

The facts, that is, are being made authoritative not through scientists telling us what to believe about an invisible virus, but by occurrences in the real world, visible for all to see. If a researcher claims that a certain chemical in the environment, such as the glyphosate in Roundup, will cause a certain number of increased cancer deaths per year or that a particular economic policy will lead to a certain number of new jobs, in most cases no one will ever be able to confirm that prediction. Even if the mechanism by which the chemical causes some variety of cancer is clear in lab rats, it is likely to have many plausible causes in humans. Even if the new jobs do appear, the cause might be trade decisions made by other countries, or the expansion of new industries. In the years that might be necessary to test such claims (though usually they cannot be tested), other researchers may come up with entirely new explanations. No wonder scientific and political debates about such matters never seem to end. But for COVID-19, the basic scientific inferences quickly play out—through changing incidence of the disease and its consequences—in ways that allow both scientists and the public to assess the current level of scientific understanding and the facts on the ground.

..

For many problems at the intersection of science and policy, scientists use mathematical models to make inferences about the future, for time periods ranging from decades to centuries or more: How can new energy technologies best be deployed to reduce greenhouse gas emissions? How will nuclear waste behave in a geological repository over coming millennia? How much will economic productivity increase if more investments are made in research? But such questions always involve enormous uncertainties, and the models used to try to answer them are laden with assumptions about more basic questions that are themselves unanswerable: How will the price of solar panels change in the coming decades? How many centuries will it take for groundwater to corrode the nuclear waste storage vessels? How efficiently do universities create economically valuable knowledge? Different assumptions about these sorts of questions allow models to fuzz the boundary between science and politics by providing competing views of the future, in support of competing political agendas.

While epidemiological models used for predicting the future of COVID-19 are also assumption-laden and highly uncertain, they can be constantly tested and refined based on data that is emerging on a daily basis, to accomplish what everyone agrees must be done. For the most part models are being used to help put boundaries around the range of plausible futures that we face, and we can see different versions of these futures unfold as different countries implement different policies at different speeds. The models are valuable because they allow us to test our assumptions about both the behavior of the virus and the impacts of different policy approaches, in real time. They are not crystal balls deployed to make the case for one preferred future or another, but navigation charts that help us narrow the plausible pathways to the future that we all hope for.

..

But when it comes to fighting COVID itself, rather than fixing the economy, the combination of shared values and clear chains of causation makes it tough to import second-order political agendas into debates about what actions to take—despite the ongoing and acknowledged uncertainties. Politicians as ideologically distinct as New York’s Mayor Bill de Blasio, a liberal Democrat, and Ohio’s Governor Mike DeWine, a conservative Republican, are implementing essentially equivalent strategies for addressing the pandemic. While President Donald Trump is at the moment threatening to loosen up social distancing rules, his spasmodic approach to pandemic policies isn’t turning out to be significantly different from that of many other national political leaders. For this crisis, the things that unite us are outranking those that divide us; pandering and opportunism, while never absent from politics, are being brought to heel by the pincer combination of shared values and facts on the ground.

Now, let’s take a brief aside to Federalism and the coronavirus response from Dan Farber posted on Legal Planet (also worth reading in its entirety, the Constitution is only one part of the discussion):

These constitutional rules reinforce the statutory and practical reasons why states have been doing so much of the heavy lifting during this viral outbreak. The federal government could do a lot more than it has so far, but its powers are not unbounded. Don’t get me wrong, the role of the federal government in addressing the pandemic is vitally important. The Feds have resources and funding the states can’t match. But the way our system of government is designed, states and cities are inevitably going to be on the front lines.

Finally, let’s check back with the WaPo.. It’w worth reading the whole thing thinking about “what does the author mean by “politicization”? What evidence does the author use to support that claim?

This is why epidemiology exists. Its practitioners use math and scientific principles to understand disease, project its consequences, and figure out ways to survive and overcome it. Their models are not meant to be crystal balls predicting exact numbers or dates. They forecast how diseases will spread under different conditions. And their models allow policymakers to foresee challenges, understand trend lines and make the best decisions for the public good.

But one factor many modelers failed to predict was how politicized their work would become in the era of President Trump, and how that in turn could affect their models.

I don’t find the WaPo’s evidence more convincing that “what do do” has “become politicized” than Sarewitz’s. Some people disagree about models (including modelers, I’m sure) and President Trump issues statements that don’t make sense (same old, same old). I guess having one’s models “politicized” is bad, but models being used in policy is necessary. Which goes back to our old forest discussion about what is the role of elected political leaders (legitimate?) or is “politics” really bad when making decisions? What is the bad part- values of elected officials or only those you happen to disagree with? Or is the bad part of politics only when the decision is solely based on tribal loyalties (party politics) or light or dark money, or ???

If you want to dive into the detail of some of the models without going too far into the weeds, this Science article “Mathematics of life and death: How disease models shape national shutdowns and other pandemic policies” seems to cover it.

Here’s a quote about models from that piece:

Policymakers have relied too heavily on COVID-19 models, says Devi Sridhar, a global health expert at the University of Edinburgh. “I’m not really sure whether the theoretical models will play out in real life.” And it’s dangerous for politicians to trust models that claim to show how a little-studied virus can be kept in check, says Harvard University epidemiologist William Hanage. “It’s like, you’ve decided you’ve got to ride a tiger,” he says, “except you don’t know where the tiger is, how big it is, or how many tigers there actually are.”

Models are at their most useful when they identify something that is not obvious, Kucharski says. One valuable function, he says, was to flag that temperature screening at airports will miss most coronavirus-infected people.

There’s also a lot that models don’t capture. They cannot anticipate, say, the development of a faster, easier test to identify and isolate infected people or an effective antiviral that reduces the need for hospital beds. “That’s the nature of modeling: We put in what we know,” says Ira Longini, a modeler at the University of Florida. Nor do most models factor in the anguish of social distancing, or whether the public obeys orders to stay home. Recent data from Hong Kong and Singapore suggest extreme social distancing is hard to keep up, says Gabriel Leung, a modeler at the University of Hong Kong. Both cities are seeing an uptick in cases that he thinks stem at least in part from “response fatigue.” “We were the poster children because we started early. And we went quite heavy,” Leung says. Now, “It’s 2 months already, and people are really getting very tired.” He thinks both cities may be on the brink of a “major sustained local outbreak”.

Yesterday’s piece was by economists- today’s is about energy systems modeling, from some energy policy and analysis experts. Of course, the energy sector is key to reducing carbon emissions, so perhaps their experience and perspective is valuable.

My favorite quote is the last line.

“But perhaps a start is for decision-makers to adapt to an increasingly uncertain and dynamic world by creating a more imaginative discourse, one that welcomes nuance and doubt as spaces for opportunity and transformative change, and sees forecasts as the beginning of a policy or investment discussion rather than the end, and forecasters not as Delphic oracles of outcome, but as the people who know best why attempts at prediction must fall short.”

I would argue that we stakeholders and public, and of course, practitioners and scientists outside the modelling community should participate in that imaginative discourse. Nuance and doubt are our friends, IMHO, and yield more robust paths forward.

“We explore two challenges to forecasting complex systems which can lead to large forecast errors. Though not an exhaustive list, these two challenges lead to a significant fraction of large forecast errors and are of central importance to energy system modeling. The first challenge is that in complex systems, there are more variables than can be considered. Often described as epistemic uncertainty, these un-modeled variables—the unknown unknowns—can lead to reality diverging dramatically from forecasts. The second challenge in forecasting complex systems is from the inherently nonlinear nature of many such systems. This results in a compounding of stochastic uncertainties—the known unknowns—which in turn can result in real-world outcomes that deviate significantly from forecasts.”

Remember from yesterday, Idea 1 was “weather-like vs. climate-like” forecasting with weather-like having more chances to check in with the real world. Idea 2 was the concept of Big Surprises which seem unpredictable. Today’s authors’ experience is that “the future is often directed by unlikely events.” They also relate epistemic uncertainty to black swans and stochastic uncertainties to dying swans. Perhaps the longer the projection, the greater the probability that unlikely events will overwhelm likely events? For that reason, some have suggested focusing on the short to medium term for projections.

In many cases, modeling apologetics are insightful and accurate, and meaningfully contribute to improved future forecasts. But there are reasons to question the universality of this narrative. Explaining away modeling errors as due to one-off unlikely events misses the prevalence of errors caused by such events, and may lure us (especially those of us who are non-modelers but rely upon model outputs) toward a heuristic of naturalistic equilibrium: a belief that “now things are normal,” or that they will soon be. The history of the energy system teaches us that the future is often directed by unlikely events, and that there is value in questioning whether naturalistic analogies of equilibrium are appropriate in many cases. Energy systems may experience multiple years, or even decades, of disequilibrium due to complex and shifting market rules, uncertainties of technological or economic feasibility at nonlinear scales of deployment, and extraordinary diversity in market structure, composition, and actors. The enormity of such extraneous uncertainties places any forecaster in very deep water.

The questions raised here, and the types of forecast errors described, should be expanded to other sectors. The rapid pace of advancement and interconnectedness of the world means that epistemic uncertainty is larger than ever [50]. For many newer technologies, the degree of uncertainty regarding future generation has increased in recent years. Cost declines in technologies such as wind, solar, and energy storage place them on competitive terms with conventional generation technologies. These technologies have shown even more stark learning-by-doing effects than shale gas production. Markets for electric cars and demand response, to name a few, similarly pose the possibility for dramatic shifts. Even moderate changes in cost and policies can lead to large changes in the future adoption of these technologies.

Decision-makers and investors would benefit from learning more about why they were caught unawares by the shale revolution, and how they can be better prepared the next time such a surprise occurs. The answer to that second question is not immediately apparent. Tautologically, if we were prepared for them, surprises would cease to be surprises. But perhaps a start is for decision-makers to adapt to an increasingly uncertain and dynamic world by creating a more imaginative discourse, one that welcomes nuance and doubt as spaces for opportunity and transformative change, and sees forecasts as the beginning of a policy or investment discussion rather than the end, and forecasters not as Delphic oracles of outcome, but as the people who know best why attempts at prediction must fall short.

Here’s a link to a paper called “Escape From Model-Land”. It’s written by an economist/modeler at the London School of Economics and a mathematician/statistician at Pembroke College, Oxford and LSE.

From the abstract:

The authors present a short guide to some of the temptations and pitfalls of model-land, some directions towards the exit, and two ways to escape. Their aim is to improve decision support by providing relevant, adequate information regarding the real-world target of interest, or making it clear why today’s model models are not up to that task for the particular target of interest.

I like how the authors (Idea 1) distinguish between “weather-like” tasks, and “climate-like” tasks. In our world, a “weather-like task” might be a growth and yield or fire behavior model, for which more real-world data is always available to improve the model (that is, if there is a feedback process and someone whose job it is to care for the model). Model upkeep, however, it not as scientifically cool as model development, so there’s that.

Our image of model land is intended to illustrate Whitehead’s (1925) “Fallacy of Misplaced Concreteness”. Whitehead (1925) reminds us that “it is of the utmost importance to be vigilant in critically revising your modes of abstraction”. Since obviously the “disadvantage of exclusive attention to a group of abstractions, however well-founded, is that, by the nature of the case, you have abstracted from the remainder of things”. Model-land encompasses the group of abstractions that our model is made of, the real-world includes the remainder of things.
Big Surprises, for example, arise when something our simulation models cannot mimic turns out to have important implications for us. Big Surprises invalidate (not update) model-based probability forecasts: the conditions I in any conditional probability P(x|I) changes. In “weather-like” tasks, where there are many opportunities to test the outcome of our model against a real observed outcome, we can see when/how our models become silly (though this does not eliminate every possibility of a Big Surprise). In “climate-like” tasks, where the forecasts are made truly out-of-sample, there is no such opportunity and we rely on judgements about the quality of the model given the degree to which it performs well under different conditions.

In economics, forecasting the closing value of an exchange rate or of Brent Crude is a weather-like task: the same mathematical forecasting system can be used for hundreds or thousands of forecasts, and thus a large forecast-outcome archive can be obtained. Weather forecasts fall into this category; a “weather model” forecast system produces forecasts every 6 hours for, say, 5 years. In climate-like tasks there may be only one forecast: will the explosion of a nuclear bomb ignite and burn off the Earth’s atmosphere (this calculation was actually made)? How will the euro respond if Greece leaves the Eurozone? The pound? Or the system may change so much before we again address the same question that the relevant models are very different, as in year-ahead GDP forecasting, or forecasting inflation, or the hottest (or wettest) day of the year 2099 in the Old Quad of Pembroke College, Oxford.

……
(Idea 2). The unpredictable or Big Surprise. People working with “weather-like” tasks may develop a high level of humility regarding the many unknowns small and Big that can happen in the real world. It’s possible that people working with “climate-like” tasks have fewer opportunities to develop that appreciation, or perhaps that they must trudge forward knowing about known unknowns and unknown unknowns. The authors’ discussion of the difference between how economists use model outputs compared to climate modelers (intervening expert judgement)is also interesting.

And the questions we asked in the last post:

“It is helpful to recognise a few critical distinctions regarding pathways out of model-land and back to reality. Is the model used simply the “best available” at the present time, or is it arguably adequate for the specific purpose of interest? How would adequacy for purpose be assessed, and what would it look like? Are you working with a weather-like task, where adequacy for purpose can more or less be quantified, or a climate-like task, where relevant forecasts cannot be evaluated fully? Mayo (1996) argues that severe testing is required to build confidence in a model (or theory); we agree this is an excellent method for developing confidence in weather-like tasks, but is it possible to construct severe tests for extrapolation (climate-like) tasks? Is the system reflexive; does it respond to the forecasts themselves? How do we evaluate models: against real-world variables, or against a contrived index, or against other models? Or are they primarily evaluated by means of their epistemic or physical foundations? Or, one step further, are they primarily explanatory models for insight and under-standing rather than quantitative forecast machines? Does the model in fact assist with human understanding of the system, or is it so complex that it becomes a prosthesis of understanding in itself?

“A prosthesis of understanding” (for prosthesis, I think “substitute”) reminds me of linear programming models (most notably, in our case, Forplan).

One of the challenges facing policy folks and politicians today is “how seriously should we take the outputs of models”? Countries spend enormous amounts of research funds attempting to predict changes from climate change. How far down the predictive ladder from GCM’s to “the distribution of this plant species in 2050” or “how much carbon will be accumulated by these trees in 2100” can we go, before we say “probably not worth calculating, let’s spend the bucks on CCS research or other methods of actually fixing the climate problem.”.

There have been several recent papers and op-eds looking at modeling with regards to economics and climate. And because the models used by IPCC include economic prediction to predict future climate, there is an obvious linkage. Before we get to those, though, I’d like to share some personal history of modeling efforts back in the relatively dark ages (1970s, that is 50 years ago). Modeling came about because of the availability of computer programs. I actually took a graduate class in FORTRAN programming at Yale for credit towards my degree. My first experience with models was with Dr. Dave Smith, our silviculture professor at Yale, talking about JABOWA (this simulation model is still available here on Dan Botkin’s website). Back in the 70’s, the model was pretty primitive. Dr. Smith was skeptical. He told us students something like “”that model would grow trees to 200 feet, but when they know they only grow to 120 feet, they just but a card in that says “if height is greater than 120, height equals 120.”” The idea at the time was that models would be helpful for scientists to explore what parameters are important, and then to test their hypotheses derived from the models with real world data. I also remember Dr. Franklin commenting on Dr. Cannell’s physiological models (Cannell is Austrialian) “it’s modeling to you, but it sure sounds like muddling to me.” Back in those days, the emphasis was on collecting data with modeling as an approach to help understand real world data.

It is interesting, and perhaps understandable, that as models improved, that the concept of “what models are good for” would change to “if we could predict the future using models, we would make better decisions.” Hence, the use of models in decision-making.

I’m not saying that they’re all unhelpful (say, fire models), but there is a danger in taking them too far in decision-making, not considering the disciplinary and cross-disciplinary controversies with each one, not linking them to real-world tests of their accuracy, or especially having open discussions of accuracy and utility with stakeholders, decision-makers and experts. Utility, as we shall see in the papers in the next posts, is in the eye of the beholder. Even the least scientifically “cool” model (say, compared to GCM’s), an herbicide drift model, or a growth and yield model, usually has a feedback mechanism by which you contact a human being somewhere whose job it is to make tweaks to the model to improve it. The more complex the model, though, the less likely there is a guru (such as a Forplan guru-remember them?) who understands all the moving parts and can make appropriate changes. The more complex, therefore, the more acceptance becomes “belief in its community of builders and their linkages” rather than “outputs we can test.” And for predictive models, empirical tests are conceptually pretty difficult because just because you predicted 2010 well doesn’t mean you will predict 2020 or 2100 well.

And at some level of complexity, it’s all about trust and not so much about data at all. But who decides where the model fits between “better than nothing, I guess” and “oracle”? And on what basis?

Please feel free to share your own experiences, successes and not so much, with models and their use in decision-making.

In this piece, Roger Pielke, Jr. argues for the importance of “facilitating democratic discourse” and suggests that there is room for improvement in academia. My own experiences have been that we had more open discussions within the Forest Service (due to having different experts needing to agree on documents) than my recent experiences with academia, both as a student and as an alumna. As Roger says, it’s not for everyone, but I think it should be encouraged, especially in institutions whose mission involves education of young people who need to enter a diverse work world. Of course, it’s also one of the main missions of this blog.

More than a century ago, American pragmatist John Dewey emphasized the importance of “intellectual hospitality.” By this he meant “An attitude of mind which actively welcomes suggestions and relevant information from all sides.” Today academics and other experts face a crisis of intellectual hospitality, with implications not just for the art of science communication but also for the broader roles of experts in democracy.

….

A counter argument is that in an era where politics might matter more than ever, why should experts engage with their opponents? Maybe the stakes nowadays are just too high for the high-minded luxury of intellectual hospitality. In fact, perhaps we should actively be opposing delegitimized academics, the GMO industry, climate skeptics and even Republicans, lest we help their causes. I hear this argument a lot, as do others who seek to cross political lines in scientific engagement and communication.

Despite my experiences, I persist in believing that not just science but also democracy is well served by intellectual hospitality.

Experts reinforce democracy and work against authoritarianism when we adopt a stance of intellectual hospitality. A half century ago, American political scientist E. E. Schattschneider made a powerful case for the importance to democracy of a willingness to engage different points of view: “Democracy is based on a profound insight into human nature, the realization that all men are sinful, all are imperfect, all are prejudiced, and no one knows the whole truth.”

To paraphrase Walter Lippmann, democracy is not about getting everyone to think alike, but about getting people who think differently to act alike. Intellectual hospitality will not lead to uniform thinking, but it may facilitate collective action.

As experts we face important choices in how we deploy the authority that we have gained in society. We can use that power to delegitimize those we disagree with, to seek to dominate the intellectual arena. Alternatively, we can allow some in our ranks to try to serve as a corrective to the hyper-partisan politics of our era, to seek to facilitate democratic discourse. The choice is profound, not just for the politics of specific issues like climate change and GMOs, but for the practice of democracy itself.

Lackey raises many points worth examining. Many people don’t trust scientists generally as voices of authority. Based on the ideological proclivities of university folks (seen in studies), could we expect them to design and carry out unbiased studies? Can choices of research priorities be carried out in an unbiased way by biased people? What about ways to mediate these impacts like co-production, co-design and extended peer review (as noted by Sir Peter Gluckman here).

Do these biases lead to an imbedded assumption in research studies that “natural is better”? If it is, shouldn’t we be discussing that openly in a forum of all disciplines and people that may be involved or impacted? If we really believe that, what does it say about human beings.. we are pretty much cockroaches in the kitchen of the planet? Ultimately those are not scientific beliefs, those are about what humankind is about- philosophical or metaphysical, depending on your worldview.

I’m not saying “natural” isn’t better, but there are different reasons that it might be better or not and those reasons could be important to policy. Further, with the climate changing, if we impute greater than 50% of the reason for climate change to human forces (at the risk of not, and seeming to be Zinkean or Pruittesque), then nothing will really be “natural” in the original sense. Where does that leave us policy-wise (we can already see this playing out in some ESA discussions)?

Here’s a quote related to that point:
“

Is Our Science Biased Toward Natural?”
A simple question, but I’m working in academia these days, so a straightforward yes or no will not suffice.
To start, put on your science hat, and be honest here, imagine that the public owns a 5,000 acre stand of old growth fir. Is preserving this stand of old growth preferable to removing the trees and building a destination resort and golf course on the same 5,000 acres?
It is not! At least not without assuming, perhaps unwittingly, a policy preference, a value choice. The result? A classic example of normative science.
It may look like a scientific statement. It may sound like a scientific statement. It is often presented by people who we assume to be operating as scientists. But such statements in science are nothing more than policy advocacy masquerading as science.

Anyway, there’s plenty of discussion fodder in this paper, so have at it!