Evidence-informed deliberative processes

Value assessment frameworks for HTA agencies: The organization of evidence-informed deliberative processes.



Priority setting in health care has been long recognized as an intrinsically complex and value-laden process. Yet, health technology assessment agencies (HTAs) presently employ value assessment frameworks that are ill fitted to capture the range and diversity of stakeholder values and thereby risk compromising the legitimacy of their recommendations. We propose "evidence-informed deliberative processes" as an alternative framework with the aim to enhance this legitimacy. This framework integrates two increasingly popular and complementary frameworks for priority setting: multicriteria decision analysis and accountability for reasonableness. Evidence-informed deliberative processes are, on one hand, based on early, continued stakeholder deliberation to learn about the importance of relevant social values. On the other hand, they are based on rational decision-making through evidence-informed evaluation of the identified values.


The framework has important implications for how HTA agencies should ideally organize their processes. First, HTA agencies should take the responsibility of organizing stakeholder involvement. Second, agencies are advised to integrate their assessment and appraisal phases, allowing for the timely collection of evidence on values that are considered relevant. Third, HTA agencies should subject their decision-making criteria to public scrutiny. Fourth, agencies are advised to use a checklist of potentially relevant criteria and to provide argumentation for how each criterion affected the recommendation. Fifth, HTA agencies must publish their argumentation and install options for appeal. The framework should not be considered a blueprint for HTA agencies but rather an aspirational goal-agencies can take incremental steps toward achieving this goal.

                                                                                                           Link to full paper  



Priority setting for universal health coverage: We need evidence-informed deliberative process, not just more evidence on cost-effectiveness




Priority setting of health interventions is generally considered as a valuable approach to support low- and middle-income countries (LMICs) in their strive for universal health coverage (UHC). However, present initiatives on priority setting are mainly geared towards the development of more cost-effectiveness information, and this evidence does not sufficiently support countries to make optimal choices. The reason is that priority setting is in reality a value-laden political process in which multiple criteria beyond cost-effectiveness are important, and stakeholders often justifiably disagree about the relative importance of these criteria.


Here, we propose the use of ‘evidence-informed deliberative processes’ as an approach that does explicitly recognise priority setting as a political process and an intrinsically complex task. In these processes, deliberation between stakeholders is crucial to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. Such processes then result in the use of a broader range of explicit criteria that can be seen as the product of both international learning (‘core’ criteria, which include eg, cost-effectiveness, priority to the worse off, and financial protection) and learning among local stakeholders (‘contextual’ criteria). We believe that, with these evidence-informed deliberative processes in place, priority setting can provide a more meaningful contribution to achieving UHC.

                                                                                                                       Link to full paper


Fair processes for priority setting: Putting theory into practice




Embedding health technology assessment (HTA) in a fair process has great potential to capture societal values relevant to public reimbursement decisions on health technologies. However, the development of such processes for priority setting has largely been theoretical. In this paper, we provide further practical lead ways on how these processes can be implemented.


We first present the misconception about the relation between facts and values that is since long misleading the conduct of HTA and underlies the current assessment-appraisal split. We then argue that HTA should instead be explicitly organized as an ongoing evidence-informed deliberative process, that facilitates learning among stakeholders. This has important consequences for whose values to consider, how to deal with vested interests, how to consider all values in the decision-making process, and how to communicate decisions. This is in stark contrast to how HTA processes are implemented now. It is time to set the stage for HTA as learning.


Link to full paper


Related publications

Time to recognize countries’ preferences in HIV control. Lancet 2016. Tromp N, Prawiranegara R, Siregar S, Jansen MPM, Baltussen R.


PrEP: why we are waiting. Lancet HIV. 2015 M Jansen, N Tromp, R Baltussen

Priority setting in HIV/AIDS control in West Java Indonesia: an evaluation based on the accountability for reasonableness framework. 
Health Policy Plan. 2014 Apr 15. Tromp N, Prawiranegara R, Subhan Riparev H, Siregar A, Sunjaya D, Baltussen R


Multiple Criteria Decision Analysis for Health Care Decision Making--An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, Longrenn T, Mussen F, Peacock S, Watkins J, Ijzerman M.


Multiple Criteria Decision Analysis for Health Care Decision Making--Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, Lönngren T, Mussen F, Peacock S, Watkins J, Devlin N; ISPOR Task Force.. Value Health. 2016 Mar-Apr;19(2):125-37. doi: 10.1016/j.jval.2015.12.016. Epub 2016 Mar 7. Review.


Multi-criteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value in Health 2012 Sep-Oct;15(6):961-70. Youngkong S, Baltussen R, Tantivess S, Mohara A, and Teerawattananon Y.


Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis.  Cost Eff Resour Alloc. 2014 Aug 29;12:18. doi: 10.1186/1478-7547-12-18. eCollection 2014. Norheim, O.F., Baltussen, R.M., Johri, M., Chisholm, D., Nord, E., Brock, D., Carlsson, P., Cookson, R., Daniels, N., Danis, M., Fleurbaey, M., Johansson, K.A., Kapiriri, L., Littlejohns, P., Mbeeli, T., Rao, K.D., Edejer, T.T. & Wikler, D.

Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care. 2013 Oct;29(4):456-65. doi: 10.1017/S0266462313000573. Tanios N, Wagner M, Tony M, Baltussen R, van Til J, Rindress D, Kind P, Goetghebeur MM; International Task Force on Decision Criteria.


Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers BMC Health Serv Res. 2012; 12: 454. Tromp N, Baltussen R.


From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012 Jul 18;10(1):9. Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, Goetghebeur MM.


Decision-Making Criteria among National Policymakers in Five Countries: A Discrete Choice Experiment Eliciting Relative Preferences for Equity and Efficiency. Value in Health 2012 May;15(3):534-9. Mirelman A, Mentzakis E, Kinter E, Paolucci F, Fordham R, Ozawa S, Ferraz M, Baltussen R, Niessen LW.


The EVIDEM framework and its usefulness for priority setting across a broad range of health interventions Cost Effectiveness and Resource Allocation. 2011;9(8). Youngkong S, Tromp N, Chitama D


Fairness in service choice: an important yet underdeveloped path to universal coverage Tropical Medicine and International Health. 2011 Jul;16(7):838-9. Baltussen R, Norheim O, Johri M


Criteria for priority setting of HIV/AIDS interventions in Thailand: A discrete choice experiment BMC Health Service Research. 2010;10:197-206. Youngkong S, Baltussen R, Tantivess S, Koolman X, Teerawattananon Y


Multi-criteria decision analysis of breast cancer control in low- and middle- income countries: development of a rating tool for policy makers. Cost Eff Resour Alloc. 2014 May 17;12:13. doi: 10.1186/1478-7547-12-13. eCollection 2014.Venhorst K, Zelle SG, Tromp N, Lauer JA


Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experience. Health Policy. 2010;96(3):262-4. Baltussen R, Youngkong S, Paolucci F, Niessen L


Incorporating equity-efficiency interactions in cost-effectiveness analysis-three approaches applied to breast cancer control  Value Health. 2010;13(5):573-9. Baeten S, Baltussen R, Uyl-de Groot C, Bridges J, Niessen L


Setting priorities for health interventions in developing countries: a review of empirical studies Tropical Medicine and International Health. 2009;14(8):930-9. Youngkong S, Kapiriri L, Baltussen R


Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012 Feb 17;10: Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R.


From papers to practices: district level priority setting processes and criteria for family planning, maternal, newborn and child health interventions in Tanzania BMC Womens Health. 2011 Oct 21;11:46. Chitama D, Baltussen R, Ketting E, Kamazima S, Nswilla A, Mujinja PG.