Quantitative information is one of the means used to interface science with policy. As a consequence, a lot of effort is invested in producing quantitative information for policy and a lot of criticism is directed towards the use of numbers in policy. In this post, five approaches are analysed, which have emerged from these criticisms and propose alternative uses of quantitative information for governance: (i) valuation of ecosystem services, (ii) social multi-criteria evaluation, (iii) quantification of uncertainty through the NUSAP approach, (iv) Quantitative Story-Telling and (v) the heuristic use of statistics. Results show the diversity of conceptualisations of numbers that emerge from the practices analysed and the constitutive role of different conceptualisations for the science-policy interface. Alternative conceptualisations of numbers are used to challenge the model of science speaking truth to power. Elements such as uncertainty, complexity, pluralism, malpractice and values are mobilised to redefine the relations between science and policy. Results show that alternative quantification may run the risk of producing alternative facts. As a remedy against this risk, reflexive approaches use numbers to discuss the relevance of equity, positionality and quality in science for policy.
A summary of the main characteristics of each approach is given in table 1. The classification used in this post to distinguish between the different approaches overemphasises the differences. In practice, there are overlaps between these approaches and between the criticisms they pose to the use of quantitative information for policy.
The types of challenges tackled by the alternative approaches to quantification considered in this post are framing, pluralism, uncertainty, complexity and malpractice. Some of these challenges have been amply discussed in academia and in the media. The issue of framing is central to advocacy groups, both within and outside on academia. The replicability crisis in psychology and statistics (Baker, 2015, 2016; OSC, 2015) has brought the issue of malpractice to the forefront. I argue that the approaches analysed question the idea that there is something special about numbers as opposed to other kinds of scientific analysis or argumentation in the interface with policy. For example, quantitative Story-Telling suggests that numbers are stories no different from qualitative argumentations.
The conceptualisation of numbers that emerges from the approaches analysed is varied and nuanced. The valuation of ecosystem services consists of value-based problem framing. Numbers are conceptualised as value-driven facts. The conceptualisation of numbers as bearers of information does not change. What is different is the framing that determines which type of information numbers should bear.
In social multi-criteria evaluation, numbers are used to rank the criteria according to which policy alternatives are measured and to rank preferences of different actors. The approach can be described as ranking of incommensurable values. In this case, numbers are used as a cardinal ranking tool in the definition of social preferences. Numbers are means to deal with pluralism, through the multi-criteria matrix, weighting factors, and the social equity matrix.
According to the NUSAP approach, numbers are something to be studied, they are the object of investigation, and their use changes according to the level of uncertainty. The approach offers a quantification of uncertainty in so-called facts. At low levels of uncertainty, numbers can be used as facts, while in the context of irreducible uncertainty numbers themselves must be assessed as sources of uncertainty, or even ignorance (Ravetz, 1987).
In Quantitative Story-Telling, numbers are used to test the usefulness of simplifications in guiding policy rather than as sources of information. Quantitative Story-Telling is based on the assumption that numbers are not descriptions of a reality that is out-there but play an important role in the organisation of one’s perception of reality. Numbers describe the perception of a particular story-teller. Counting cells in an organism reflects the analytical choice of the observer, but does not exhaust all there is to know about that organism. Quantification in this approach is used to acknowledge complexity.
Lastly, numbers have a secondary role in the heuristic use of statistics. The focus is on the pertinent use of statistical methods, as a way of ensuring that data are analysed according to the information they provide (or fail to provide), rather than according to the assumptions imposed on data. The approach used is prominently quantitative, but numbers do not carry any information per se. The heuristic use of statistics is based on assumption hunting. Numbers become useful only once the constraints are known and the pertinence of the statistical methods can be established.
Collectively, these conceptualisations move away from the positivist use of numbers. Numbers are used as heuristic tools, as means to tell a story. Heuristics are understood as opposed to blue-print procedures for the use of quantitative information. They can be seen as a strategy to deal with uncertainty and complexity. Heuristics should not be confused with improvisation, but rather with the ability to adapt to the irreducible singularities of the object of study (Serrano and Romero, 2014).
These conceptualisations of numbers matter for the interface between science and policy. Different conceptualisations of numbers reflect the role of science advice in opening or closing the policy option space, and make it possible to rethink the type of engagement with power.
Three variations can be observed in the role of science envisaged by the approaches analysed: advocacy, reflexivity, and quality assurance. In the evaluation of ecosystem services, science provides a value-based problem definition upon which policy is expected to act. The separation between experts and non-experts is maintained and used to make authoritative claims over the values that should inform policy. Science thus plays an advocacy role. Social multi-criteria evaluation and the NUSAP notation system pursue reflexivity in the use of numbers, preference rankings and expert advice in policy processes. Both approaches refrain from giving substantive advice, and offer analytical tools that lead social actors and experts to reflect upon their own positionality in the definition and assessment of policy options. Science and policy are understood as intertwined processes, rather than as separate activities. Quantitative Story-Telling and the heuristic use of statistics retain a role of quality assurance. The former approach is used to assess the quality of policy narratives, and the latter is used to ensure the appropriate use of statistical methods.
The production of quantitative information is not just outward looking (with regard to the type of policy that science wants to inform) but also inward looking. Reflexivity is practiced by analysing uncertainty, pluralism and complexity and using numbers as heuristic tools. Reflexivity leads to humble science, which offers decision-support tools rather than authoritative claims. By contrast, value-based problem framing takes the insights of science studies and uses them to advocate for specific policies. The advocacy function of the evaluation of ecosystem services creates a paradox, expressed by Jasanoff as, “how … can a sceptical and reflexive stance in relation to scientific knowledge be reconciled with making authoritative recommendations for social policy?” (1996: 193). With regard to the treatment of policy options, there is a clear difference in the evaluation of ecosystem services and the other approaches, which open the option space.
Reflexivity invites also reflection on science’s blind spots. Quantitative Story-Telling and the heuristic use of statistics both adopt a via negativa use of numbers. Quantitative story-telling uses quantification as a means to identify and test the constraints associated with different narratives. The heuristic use of statistics is used to determine when statistical methods cannot be used. This analysis suggests that instead of describing the world through numbers, numbers can be useful in defining what cannot be known: for example, uncertainties, constraints, probabilities. The relevance of negative information is also discussed by Taleb (2012). Taleb warns that absence of evidence is not the same as evidence of absence. In the case of new pharmaceutical drugs, side effects may not be evident in the beginning, which should not be mistaken for absence of side effects.
This diversity of uses and conceptualisations of numbers has important consequences for the science-policy interface. The model of science speaking truth to power is based on the definition of numbers are bearers of objective information. If the definition of numbers changes, the role of science, the role of policy, and their interactions also changes. Just like the positivist conceptualisation of numbers creates new objects and modes of governance through statistics and indicators, alternative conceptualisations of numbers have to be studied in relation to the processes they may challenge or stabilise.
To highlight the variations introduced by these approaches to the science-policy interface, I play around with the ‘science speaking truth to power’ refrain. The valuation of ecosystem services is an example of what Funtowicz and Strand (2007) have defined as the framing model. The framing model can be expressed as science speaking valuable facts to power, where valuable can be defined in terms of ethics, relevance, urgency, et cetera.
The reference to iterative and participatory processes suggests that science is done with policy, rather than for policy, reflecting the logic of care. The relationship between science and policy in social multi-criteria evaluation can be characterised as science speaking pluralism with power, in the NUSAP approach as science speaking uncertainty with power, and in Quantitative Story-Telling as science speaking complexity with power.
Similarly to the demarcation model (Funtowicz & Strand 2007), which establishes which science should be used for policy, the heuristic use of statistics establish in which cases statistics should not be used for policy, bringing attention to context. While in the case of low uncertainty quantification and statistical analysis are useful tools, in the context of irreducible uncertainty and complexity, the limits of quantitative information become the object of study. This approach aims at creating awareness about constraints, and does not give any substantive advice to policy. The heuristic use of statistics can be thought of as science speaking context to power.
These approaches offer a variety of means towards a more reflexive use of numbers, in which quantification engages with pluralism, complexity and uncertainty. Reflexivity consists of the observation of science’s own assumptions, subjectivities and ignorance. The effects of such alternative approaches on power relations and science-policy processes remains to be seen. What distinguishes the heuristic use of numbers is that uncertainties and contradictions are not swept under the scientific rug, but are part of the interface with policy and with society. These approaches attempt to build trust from a humble, rather than an authoritative, position.
References
Baker, M. (2015) ‘Over half of psychology studies fail reproducibility test: Largest replication study to date casts doubt on many published positive results.’, Nature Online.
Baker, M. (2016) ‘Statisticians issue warning on P values’, Nature, 531, p. 151.
Funtowicz, S. O. and Strand, R. (2007) ‘Models of science and policy’, Biosafety First-Holistic Approaches to Risk and Uncertainty in Genetic Engineering and Genetically Modified Organisms, pp. 263–278.
Jasanoff, S. (1996) ‘Beyond epistemology: Relativism and engagement in the politics of science’, Social, 26, pp. 393–418.
OSC (2015) ‘Estimating the reproducibility of psychological science’, Science, 349, p. 6251.
Ravetz, J. R. (1987) ‘Usable Knowledge, Usable Ignorance Incomplete Science with Policy Implications’, Science Communication, 9(1), pp. 87–116. doi: 10.1177/107554708700900104.
Serrano, E. and Romero, J. M. (2014) El derecho a habitar y cómo hacerlo realidad. Malaga.
Taleb, N. N. (2012) Antifragile. New York: Random House.
***This is an excerpt from my latest publication in Science, Technology & Human Values: Conceptualizing numbers at the science-policy interface. (2018)
0 Comments