Narrative Review of Multi-Stakeholder Engagement in Health Technology Assessment

Introduction

With the support of Health Canada, CADTH launched a learning period in November 2021 to better understand how to optimize the use of Real-World Evidence (RWE) to inform decision-making for drugs for rare diseases. As part of this learning period, CADTH is coordinating collaborative “learning by doing” projects that will inform the development of 4 key pillars related to the optimal use of RWE, one of which involves multi-stakeholder engagement and dialogue.

This is a high-level narrative review about how to conduct multi-stakeholder engagement in health technology assessment (HTA). The focus is on processes and approaches used in multi-stakeholder sessions with all stakeholder groups around the table, rather than engagement that involves compiling input collected from various stakeholder groups individually. The purpose of this review is to learn more about existing guidance on multi-stakeholder engagement for HTA. CADTH will also use learnings from this review to develop policies and procedures for multi-stakeholder dialogue for rare diseases and to inform future CADTH initiatives to optimize the use of RWE to support decision-making.

Methods

A rapid review approach was used to summarize the aspects of HTA literature that specifically address how organizations conduct multi-stakeholder engagement. This rapid approach is similar to other CADTH publications.2 The research question was “How is multi-stakeholder engagement approached in HTA?” Refer to the following table for the criteria used to select literature for this review.

A CADTH information specialist conducted a limited literature search in 2 parts. The first part of the search used key resources including MEDLINE, the Cochrane Database of Systematic Reviews, and the International HTA Database. The search strategy was comprised of both controlled vocabulary, such as the National Library of Medicine’s MeSH (Medical Subject Headings), and keywords. The main search concepts were HTA and stakeholder engagement. No filters were applied to limit the retrieval by study type. Where possible, retrieval was limited to the human population. The search was limited to English-language documents published between January 1, 2018, and March 4, 2022. Two hundred and forty abstracts were reviewed, and 28 articles fit the inclusion criteria above. One project member reviewed the abstracts, and 2 other project members validated the selection approach.

The second part of the search was a focused web search for grey literature related to multi-stakeholder engagement by major HTA agencies in Canada and elsewhere. Thirteen grey literature reports fit the inclusion criteria above.

This review focused on information from the articles and reports that explained how multi-stakeholder engagement was approached, not on the topics discussed or the outcomes of the engagement activities that related to the topics.

Overview of the Literature

The literature included discussion summaries from panel sessions,^3,4 cross-jurisdiction comparisons of approaches ⁵, studies to create conceptual frameworks,^6-8 treatment-specific discussions,^9-12 and guides on various aspects of HTA.^13,14

Participation in HTA processes by stakeholders such as the HTA workforce, the academic community, industry partners, payers, health care providers, and the patient community is increasing¹⁵ and is recognized as important.^16-18 Patient participation is very important, as described in CADTH’s framework for patient engagement in HTA.¹⁹ Patients are experts in their pathologies;¹⁷ therefore, their participation is particularly important for rare diseases, considering the limited clinical knowledge⁴ and because of the many uncertainties that may occur about the care pathway, natural history, longer-term clinical outcomes, and so on.¹⁸

Involving multiple stakeholders contributes to legitimacy of decision-making,¹⁷ and may increase stakeholders’ acceptance of the decisions.²⁰ This is partly because stakeholders learn new information through engaging with others and change their opinions accordingly.⁶ Multi-stakeholder involvement builds understanding among stakeholders, allows knowledge exchange, incorporates a range of values, and supports quality decision-making.^6,7,21,22

Some authors refer to involving multiple stakeholders simply as engagement.²³ Others use terms like collaboration^7,22,24-26 or partnership.^5,27 No matter the terminology, successful engagement requires transparent processes^9,28 along with clear roles, responsibilities, and objectives of the collaboration.^26,28

Techniques for engagement vary and may include multiple iterations and combinations of asynchronous activities such as surveys,^8,12,23 and synchronous activities with both small group^10,29,30 and large group31 deliberations. Engagement is sometimes done with combined stakeholder groups10 or with both combined and separate stakeholder groups at various phases.12,29

Planning and delivering multi-stakeholder engagement that supports high levels of participation and high-quality decision-making requires HTA agencies to commit time and resources.^3,23 Competencies are needed to design processes that ensure input is meaningful, captured, and considered.^3,15,23 Expertise in qualitative and quantitative methodologies23 and expertise in facilitation to create a safe space9 and build trust among participants are needed.^27,30 Also important are skills to develop materials that are understandable for a variety of audiences.^4,20,23,32

Engagement Principles and Practices

The literature identified several principles and practices that support multi-stakeholder involvement:

• Identify shared goals or purpose²² along with clear roles, responsibilities, and terms of collaboration.^26,28 Co-create solutions in a way that meets stakeholder needs.²²
• Establish trust among all stakeholders.^27,30 Trust involves each member recognizing other members’ goals as legitimate, even if they differ from their own.³³
• Be transparent with stakeholders, including the public,^9,28 throughout the process, including how stakeholders are recruited,¹³ the methods used to consider and prioritize input,^15,23 the types of stakeholder involvement, and the decisions made.⁶ Transparency contributes to the legitimacy of decision-making.²¹
• Make assumptions explicit so that stakeholders understand each other’s positions.³⁴
• Ensure impartiality so that decisions are free of undue external or internal influences.⁶
• Foster inclusivity by removing barriers to participation so that all values around a decision are considered.⁶ Inclusivity involves patient representatives from groups who experience vulnerabilities.²⁰ Key to supporting inclusivity is using materials and terminology that are understandable for a variety of audiences.^4,20,23,32 Also, CADTH’s Patient and Community Advisory Committee recommended seeking out engagement opportunities with nurses, nurse practitioners, social workers, support workers, outreach workers, and peer facilitators who work in communities identified as marginalized.³⁵

Considerations for Organizations Hosting Multi-Stakeholder Engagement

The literature also identified considerations for organizations bringing multiple stakeholders together:

• Significant time and resources are required^3,23 for engagement that supports participation, builds understanding, and supports quality decision-making.
• Transparent and consistent processes and methodologies help ensure input is meaningful, captured, and considered.^3,15,23
• A strong project team needs people with disease and product-specific knowledge, process knowledge such as understanding of regulatory tools, and methodology knowledge such as clinical trial design, data analysis, and RWE generation.³³
• Project team members with experience in qualitative and quantitative methodologies are needed to elicit and incorporate feedback from diverse stakeholders.²³
• Stakeholders need guidance about how they can participate and the input they are expected to provide. Project teams need guidance on how to use and report that input.^3,16
• All stakeholders (such as the academic community, health care providers, industry partners, payers, public and patient community representatives) should be given opportunities to participate throughout the process in a safe space with iterative communication.^9,17,18
• The role of regulators is to observe and share general information on regulatory requirements. They should avoid providing regulatory advice or making decisions.⁹
• Address conflict of interest barriers with a code of conduct and a transparent conflict of interest policy and breach of trust procedure.³⁶
• Confidentiality and nondisclosure agreements can help manage the risk of sharing confidential commercial information and support transparency among stakeholders.²⁶
• There are opportunities and risks of virtual engagement. NICE found the move to virtual meetings (because of COVID-19) has benefits, including accessibility for people with health or geographic barriers, ease of participation with the “raise hand” function, and functionality for people with hearing difficulties to read lips and adjust volume. Its drawbacks include risk of exclusion for people with low technology literacy (NICE provides communication technology resources and training), and less opportunity to form relationships through informal conversations.³⁷

Approaches to Multi-Stakeholder Engagement

Approaches to collect feedback vary and may include combinations of methods.

• Group interviews and round-by-round brainstorming,⁷ semi-structured discussions,⁸ and focus groups³⁴ are often used. Roundtable discussions require focused questions and well-identified goals.²³ Meeting notes can be analyzed qualitatively.¹²
• While large group discussions are often used,^31,38 small group discussions are also used.^10,23,29,30 Small groups sometimes work better for people with specific expertise.²³
• Asynchronous tactics such as written submissions,³¹ surveys,^8,12,23 and voting^10,13,31 can be interspersed with discussion activities.
• Activities are sometimes done with combined stakeholder groups¹⁰ or with both combined and separate stakeholder groups engaged at various phases of the process.^12,29

Specific techniques mentioned in the literature include:

• consensus processes, such as the modified Delphi technique, which allow individual voting on criteria followed by group discussion to validate, eliminate, and add criteria.^39,40 A variation is to not eliminate outcomes patients consider important until they are discussed in-person.¹¹
• multi-criteria decision analysis, which involves a combination of questionnaires and meetings to select and rank criteria, and to build consensus about the results.¹⁷ Decision-conferencing is used to construct preferences for decision analysis. Effective decision-conferencing requires trust among participants; a group between 7 and 15 participants, which is small enough to reach agreement and large enough to represent a variety of interests; and facilitation from a decision analysis expert.³⁰
• evidence-informed deliberative processes, which allow stakeholders to reflect on values, informed by evidence.²¹ This involves coordinated activities where stakeholders exchange information, examine an issue, and come to a judgment as a group.⁶ Using deliberation to make social and scientific judgments leads to better understanding of preferences among participants.¹³
• priority-setting processes such as the James Lind Alliance (JLA) method, which involves both patients and clinicians in setting priorities for research.⁴¹ Focusing on specific conditions, or areas of health and social care, the JLA facilitates priority-setting partnerships by bringing patient, carer, and clinician groups together on equal footing to jointly prioritize and produce a “Top 10” list of agreed knowledge gaps as research questions.

Challenges Associated With Multi-Stakeholder Engagement

Challenges that arose from the literature include the following:

• It is not always clear how to identify all relevant stakeholders, find ways to include them meaningfully, and determine how many should represent each type and when they should be included.¹⁵
• Frameworks to measure impact and optimal methods of engagement are needed.^22,27
• Although patients with direct connection to the topic are often involved in HTA activities, members of the public rarely are.⁵
• External stakeholders who do not directly interact with HTA processes in general (such as the public, regulatory agencies, political bodies, and the judiciary) may challenge the autonomy of decisions and recommendations.¹⁵
• A common challenge identified by HTA agencies, funders, and payers was that patient input was not part of decision-making. Common challenges identified by the patient community was lack of clear and open communication, and lack of shared vision and goals.²²

Examples of Multi-Stakeholder Engagement

This section highlights examples of approaches that organizations have used for multi-stakeholder engagement. These examples were selected from the search results because of the detailed information provided on how the engagement was conducted.

Approach: Working Group, Survey, and Workshop 
CADTH⁴²

The development of a 2022 health technology trends list was guided by the JLA approach to priority setting partnerships.⁴¹

• A working group of CADTH staff and external experts guided the process and championed the project.
• A survey about technology trends was developed by the working group. Survey respondents included patients and members of advocacy groups, academia, health care services, industry, and professional associations. The working group narrowed down the survey results to a short list of the most important technology trends.
• A workshop brought together 20 stakeholders to prioritize the top items on the short list. Participants included people with lived experiences with health technologies, consumer advocates, health care professionals, government policy analysts, academic researchers, and industry representatives.
• Before the workshop, participants were given a workshop guide, the short list of trends with summary information, and a worksheet to rank the priorities.
• During the workshop, 3 small group sessions and one plenary session were held for participants to discuss the short-listed technologies and rank them in order of importance using a consensus-based approach.

Approach: Round Tables and Voting 
World Federation of Haemophilia Gene Therapy¹⁰

A round table was held to reach consensus on statements to inform policy on gene therapy.

• Four topics were prioritized from an earlier round table as areas around which to seek consensus.
• Each participant self-selected, or was assigned to, 1 of 4 small topic working groups that each had equal multi-stakeholder representation.
• Working groups drafted consensus statements on their topic.
• A spokesperson presented their group’s consensus statement to all participants and discussed the thought process and challenges in developing the statement.
• Voting was conducted after each presentation, from which statements were either accepted as reaching consensus or prioritized for discussion.

Approach: Presentations, Surveys, and Discussions With Mixed and Separate Stakeholder Groups 
The European Biomarker Enterprise to Attack Diabetic Kidney Disease (BEAt-DKD) Consortium¹²

Meetings were held to build consensus on implementing precision medicine in diabetic kidney disease.

• Patient associations and health care provider groups were identified through contacts and web searches.

Day 1:

• Experts from the pharmaceutical industry, European HTAs, and regulators presented at plenaries about their perspectives on precision medicine.
• Participants completed short surveys based on the previously identified topics using a real-time online survey tool.
• A breakout session was held with mixed stakeholder groups so that everyone could understand other stakeholders’ perspectives.
• A second breakout session was held with separate stakeholder groups where participants answered Likert scale items about the topics previously identified and then discussed the answers in a moderated semi-structured discussion.

Day 2:

• A summary of findings of the second breakout session was presented and this formed the basis of a consensus discussion.
• Consultations with selected stakeholder representatives led to identifying issues for implementation of precision medicine in DKD.

Afterward:

• Answers to the open-ended questions in the survey, the discussion results of the breakout sessions, and the consensus discussion were analyzed qualitatively.

Approach: Iterative Workshops Over 2 Years 
National Health Service, UK²⁹

Stakeholders co-developed specifications of a modelling tool to support decision-making for a risk assessment program targeted to adults without pre-existing conditions.

• A key objective was to engage with a diverse group of stakeholders through 4 workshops.
• The project team developed a stakeholder recruitment grid and invited organizations and individuals. If specific stakeholders were unable to attend, the team used snowballing techniques to identify other people from their organizations.
• Workshops were delivered in months 4, 8, 16, and 24.
• Workshop planning started 3 months in advance of each one. Full rehearsals took place a week before the actual workshops.
• Stakeholders were either grouped with others from similar organizations or mixed for a broad range of perspectives, depending on the objectives of the workshop session.
• The workshops were delivered by an independent facilitator.
• Small group exercises with specific objectives, questions, activities, and outputs were held. Exercises were based on validated scripts.
• In workshop 1, stakeholders committed to their involvement and the modelling team embraced the value of the co-production process.
• In workshop 2, stakeholders and the modelling team explored how end users would use the decision support tool.
• In workshop 3, stakeholders adjusted and approved the decision support tool.
• In workshop 4, the iterative co-production process culminated, and the decision support tool was demonstrated.

Approach: Written Feedback, Pre-Selected Speakers, Semi-Structured Discussion 
Institute for Clinical and Economic Review (ICER)^31,43

ICER collects written and verbal feedback from specific stakeholders at various stages of their clinical and economic reviews. When a draft evidence report is almost complete, they hold a roundtable discussion at a public meeting with patients, clinical experts, manufacturers, payers, and policy experts. The meeting includes the following:

• Pre-selected speakers: Each manufacturer involved in the review can request time during the oral comment period. The public is encouraged to submit written comments, as time during the meeting for everyone to comment is not guaranteed. Priority for oral comment slots is given to patients with the condition being discussed and to subject matter experts who are patient advocates, clinicians, or researchers.
• Deliberation and voting on key questions by an appraisal committee.
• An expert roundtable on how to apply the evidence to the real world: This is a semi-structured discussion with manufacturers about the clinical and economic considerations of the intervention. An ICER contact has a one-on-one conversation with roundtable participants before the meeting about topics that may arise.

Approach: Three Dialogues Over the Course of an Intervention Study 
TRUST4RD18

An approach using iterative multi-stakeholder dialogue is proposed to resolve uncertainties and evidence gaps in assessing treatments for rare diseases. Building understanding requires iterative communication between stakeholders: manufacturers, clinicians, patients, regulators, HTA agencies, and payers.

The iterative process includes 3 milestones:

• At least 1 early dialogue is held between industry representatives, regulatory authorities, and payers from various countries, along with clinicians and patient representatives, to help solve design issues of intervention studies at an early stage.
• A second dialogue is held after the completion of the pivotal trial to address uncertainties and data gaps. This should involve the same stakeholders as the first dialogue.
• A third dialogue is held to discuss the results of additional evidence generation after launch of the HTA. This focuses on assessing whether the predictions in the HTA submission are validated in the real-world setting.

Areas for Further Exploration

A widened search strategy could include targeted literature published earlier than 2018, and fields such as health research, participatory action research, public health, international development, and organizational behaviour for detailed guidance on best practices. The search strategy could also include terms such as value assessment, deliberative processes, deliberations, deliberative engagement, evidence-informed deliberative processes, collaboration, and partnerships. The following are potential topics to further explore in the published and grey literature:

• how to match engagement goals such as common understanding or consensus building to engagement tactics
• the types of engagement that are effective in allowing all stakeholders to participate in decision-making, as opposed to simply informing the decisions
• details on optimal stakeholder selection (criteria/skill set requirements) and recruitment
• details on meeting preparation (training, introduction to the evidence, question grid) and facilitation process
• details on how to effectively execute and combine different types of tools, methods, and activities for multi-stakeholder engagement. This includes asynchronous and synchronous deliberations and other types of meetings (such as focus groups) to ensure inclusivity, transparency, and meaningful input
• how to best analyze and report the input
• how to assess effectiveness of multi-stakeholder engagement from a process perspective (such as how feedback is collected, the quality and quantity of feedback, and how to ensure everyone’s voice is included) and from an outcome perspective (how decisions or actions were impacted by the input received)
• partnership development and collaboration models in support of multi-stakeholder engagement
• value framework development, deliberations, and deliberative processes
• ethical considerations in multi-stakeholder engagement
• management of suspected bias from different stakeholders.

References

1. CADTH. Real-world data and real-world evidence at CADTH. 2022; https://www.cadth.ca/sites/default/files/events/5%20-%20Nicole%20Mittmann%20-%20Life%20Cycle%20Management%20Update.pdf. Accessed 2022 Oct 6.
2. CADTH. Narrative review of patient involvement in organizational governance. 2021; https://www.cadth.ca/narrative-review-patient-involvement-organizational-governance. Accessed 2022 Oct 6.
3. Panel session report: multi-stakeholder dialogues addressing needs, HTA requirements and good practices in emerging countries. HTAi Conference, Vancouver, June 2018: Strengthening the Evidence-to-Action Connection: HTAi; 2018: https://htai.org/wp-content/uploads/2018/09/180921-HTAi-2018-Panel-Report-Emerging-Countries-final.pdf. Accessed 2022 Mar 31.
4. Panel session report: multi-stakeholder approaches to improve evidence-based decisions in rare diseases: engagement of patients and patient organizations. HTAi Conference, Tokyo May 2016: Informing Health Care Decisions with Values and Evidence: HTAi; 2016: https://htai.org/wp-content/uploads/2018/02/PCISG-Resource-HTAi-2016-Panel-Report-final-for-web-Sep2016.pdf. Accessed 2022 Mar 31.
5. Poder TG, Safianyk C, Fournier MF, et al. Patients, users, caregivers, and citizens' involvement in local health technology assessment unit in Quebec: a survey. Int J Technol Assess Health Care. 2020;37:e5.
6. Baltussen R, Jansen M, Oortwijn W. Evidence-Informed Deliberative Processes for Legitimate Health Benefit Package Design - Part I: Conceptual Framework. Int J Health Policy Manag. 2021;10:10.
7. Jiu L, Hogervorst MA, Vreman RA, Mantel-Teeuwisse AK, Goettsch WG. Understanding innovation of health technology assessment methods: the IHTAM framework. Int J Technol Assess Health Care. 2022;38(1):e16.
8. Augustovski F, Alfie V, Alcaraz A, Garcia Marti S, Drummond MF, Pichon-Riviere A. A Value Framework for the Assessment of Diagnostic Technologies: A Proposal Based on a Targeted Systematic Review and a Multistakeholder Deliberative Process in Latin America. Value Health. 2021;24(4):486-496.
9. Karres D, Lesa G, Ligas F, et al. Can a Multistakeholder Prioritization Structure Support Regulatory Decision Making? A Review of Pediatric Oncology Strategy Forums Reflecting on Challenges and Opportunities of this Concept. Clin Pharmacol Ther. 2020;108(3):553-556.
10. Pierce GF, Pasi KJ, Coffin D, et al. Towards a global multidisciplinary consensus framework on haemophilia gene therapy: Report of the 2nd World Federation of Haemophilia Gene Therapy Round Table. Haemophilia. 2020;26(3):443-449.
11. Tambor E, Robinson M, Hsu L, Chang HY, Al Naber J, core SCDP. coreSCD: multi-stakeholder consensus on core outcomes for sickle cell disease clinical trials. BMC Med Res Methodol. 2021;21(1):219.
12. Bakker E, Mol PGM, Nabais J, et al. Perspectives on a Way Forward to Implementation of Precision Medicine in Patients With Diabetic Kidney Disease; Results of a Stakeholder Consensus-Building Meeting. Front Pharmacol. 2021;12:662642.
13. Oortwijn W, Jansen M, Baltussen R. Evidence-Informed Deliberative Processes for Health Benefit Package Design - Part II: A Practical Guide. Int J Health Policy Manag. 2021;10:10.
14. Medical technologies evaluation programme methods guide. (NICE programme methods guide PMG33). London (GB): National Institute for Health and Care Excellence; 2017: https://www.nice.org.uk/process/pmg33/resources/medical-technologies-evaluation-programme-methods-guide-pdf-72286774205893. Accessed 2022 Mar 31.
15. Trowman R, Ollendorf DA, Sampietro-Colom L. Burning Issues in Health Technology Assessment and Policy Making: What's Keeping Senior Health Technology Assessment Users and Producers up at Night? Int J Technol Assess Health Care. 2020;36(1):5-7.
16. Garrett Z, Imaz-Iglesia I, Willemsen A. Building a model of health technology assessment cooperation: lessons learned from EUnetHTA joint action 3. Int J Technol Assess Health Care. 2022;38(1):e14.
17. de Andres-Nogales F, Cruz E, Calleja MA, et al. A multi-stakeholder multicriteria decision analysis for the reimbursement of orphan drugs (FinMHU-MCDA study). Orphanet J Rare Dis. 2021;16(1):186.
18. Annemans L, Makady A. TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases. Orphanet J Rare Dis. 2020;15(1):127.
19. Berglas S, Vautour N, Bell D. Creating a patient and community advisory committee at the Canadian Agency for Drugs and Technologies in Health. Int J Technol Assess Health Care. 2021;37:e19.
20. Feenstra T, Corro-Ramos I, Hamerlijnck D, van Voorn G, Ghabri S. Four Aspects Affecting Health Economic Decision Models and Their Validation. Pharmacoeconomics. 2022;40(3):241-248.
21. Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes version 2.0: a practical guide for HTA bodies for legitimate benefit package design. Nijmegen (NL): Radboud University Medical Center; 2021: https://www.radboudumc.nl/getmedia/17a96fdb-553b-4e68-81ab-4d8d9a7f9ff1/Guide-on-EDPs-2nd-version.aspx. Accessed 2022 Mar 31.
22. Faulkner SD, Sayuri Ii S, Pakarinen C, et al. Understanding multi-stakeholder needs, preferences and expectations to define effective practices and processes of patient engagement in medicine development: A mixed-methods study. Health Expect. 2021;24(2):601-616.
23. Xie RZ, Malik ED, Linthicum MT, Bright JL. Putting Stakeholder Engagement at the Center of Health Economic Modeling for Health Technology Assessment in the United States. Pharmacoeconomics. 2021;39(6):631-638.
24. Cardoso F, Wilking N, Bernardini R, et al. A multi-stakeholder approach in optimising patients' needs in the benefit assessment process of new metastatic breast cancer treatments. Breast. 2020;52:78-87.
25. PCORI. Rare Disease and Research Engagement (RaRE). 2022; https://www.pcori.org/research-results/2020/rare-disease-and-research-engagement-rare#project_summary. Accessed 2022 Mar 31.
26. Recommendations on the required capabilities for patient engagement. Brussels (BE): PARADIGM; 2020: https://imi-paradigm.eu/PEtoolbox/pe-capabilities-bw.pdf. Accessed 2022 Mar 31.
27. Cavaller-Bellaubi M, Faulkner SD, Teixeira B, et al. Sustaining Meaningful Patient Engagement Across the Lifecycle of Medicines: A Roadmap for Action. Ther Innov Regul Sci. 2021;55(5):936-953.
28. Wale JL, Thomas S, Hamerlijnck D, Hollander R. Patients and public are important stakeholders in health technology assessment but the level of involvement is low - a call to action. Res Involv Engagem. 2021;7(1):1.
29. O'Flaherty M, Lloyd-Williams F, Capewell S, et al. Modelling tool to support decision-making in the NHS Health Check programme: workshops, systematic review and co-production with users. Health Technol Assess. 2021;25(35):1-234.
30. Angelis A, Kanavos P, Phillips LD. ICER Value Framework 2020 Update: Recommendations on the Aggregation of Benefits and Contextual Considerations. Value Health. 2020;23(8):1040-1048.
31. Manufacturer engagement guide. Boston (MA): Institute for Clinical and Economic Review; 2022: https://icer.org/wp-content/uploads/2022/01/ICER_Mfr_Engagement_Guide_102220.pdf. Accessed 2022 Mar 31.
32. Ali SR, Bryce J, Smythe C, et al. Supporting international networks through platforms for standardised data collection-the European Registries for Rare Endocrine Conditions (EuRRECa) model. Endocrine. 2021;71(3):555-560.
33. Implementing adaptive pathways - decision points and resource implications. Brussels (BE): ADAPT SMART Consortium; 2017: https://www.infographic.adaptsmart.eu/sites/adaptsmart/files/AS%20Deliverable%20D3.02.pdf. Accessed 2022 Mar 31.
34. Tummers M, Kvaerner K, Sampietro-Colom L, et al. On the integration of early health technology assessment in the innovation process: reflections from five stakeholders. Int J Technol Assess Health Care. 2020;36(5):481-485.
35. Patient and Community Advisory Committee meeting summary: September 15, 2020 and October 27, 2020. Ottawa (ON): CADTH; 2020: https://www.cadth.ca/sites/default/files/corporate/corp_committees/patient-community-advisory-committee/meetings/Fall%202020%20PCAC.pdf. Accessed 2022 Oct 6.
36. Raising awareness on managing competing interests in a multi-stakeholder environment: guidance to patients and engaging stakeholders. Brussels (BE): PARADIGM; 2020: https://imi-paradigm.eu/PEtoolbox/COI-Guidance-tools.pdf. Accessed 2022 Mar 21.
37. Rasburn M, Crosbie H, Tonkinson A, et al. Innovative Patient Involvement During Covid-19: Keeping Patients at the Heart of HTA. Front Med Technol. 2021;3:793119.
38. Patient engagement in early dialogues: tools and resources for HTA bodies. Brussels (BE): PARADIGM; 2020: https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fhtai.org%2Fwp-content%2Fuploads%2F2020%2F10%2FPARADIGM.docx&wdOrigin=BROWSELINK. Accessed 2022 Mar 31.
39. Clearfield E, Miller V, Nadglowski J, et al. coreNASH: Multi-stakeholder Consensus on Core Outcomes for Decision Making About Nonalcoholic Steatohepatitis Treatment. Hepatol. 2021;5(5):774-785.
40. Tejwani V, Chang HY, Tran AP, et al. A multistakeholder Delphi consensus core outcome set for clinical trials in moderate-to-severe asthma (coreASTHMA). Ann Allergy Asthma Immunol. 2021;127(1):116-122.e117.
41. James Lind Alliance. About us. 2022; https://www.jla.nihr.ac.uk/about-the-james-lind-alliance/. Accessed 2022 Oct 6.
42. CADTH. 2022 health technology trends to watch. 2022; https://www.cadth.ca/2022-health-technology-trends-watch-0. Accessed 2022 Oct 6.
43. Ronquest NA, Paret K, Gould IG, Barnett CL, Mladsi DM. The evolution of ICER's review process for new medical interventions and a critical review of economic evaluations (2018-2019): how stakeholders can collaborate with ICER to improve the quality of evidence in ICER's reports. J Manag Care Spec Pharm. 2021;27(11):1601-1612.