Our team of 20+ experts from across Europe and the US attended, spoke, and presented at ISPOR EU this year. Scroll down to see who from RTI Health Solutions attended and what contributions we made.
HTA Considerations for Large Language Models in Healthcare
Leonard C, Unsworth H, Warttig S, Gildea L, Mordin M, Ling C.
Caoimhe Leonard, MSc
Research Associate, Value and Access
RTI Health Solutions
Regulators and Health Technology Assessment bodies have struggled to keep pace with the rapidly evolving technology of large language models. These generative AI models are trained on extremely large data sets to be effective in natural language processing tasks. The latest generation of these models can perform healthcare-related tasks, including text analysis and summarisation, diagnostic assistance, answering medical queries, and image captioning. They have the potential to improve data handling, process automation, service quality, care personalisation, and shorten time to diagnoses.
NICE first published the Evidence Standards Framework for digital health technology evaluations in 2019 and updated it in 2022 to cover more current types of artificial intelligence. The Framework provides a set of standards, aimed to inform purchasing decisions in the NHS and to guide developers in generating evidence for their technologies. It was designed for digital health technologies that use AI and are data driven or have fixed or adaptive machine learning algorithms. Among these are AI image analysis, AI decision support, and health-related chatbots. However, large language models with healthcare applications are not covered by the Framework, as they were not available in 2022.
In our research, we consider the complexity and requirements for evaluating large language models and suggest updates to the Evidence Standards Framework to help HTA bodies and developers of digital health technologies meet standards to successfully approach HTA.
Machine Learning and Artificial Intelligence for Supporting Systematic Reviews: A Systematic Review of Recent Methodological Developments and Recommendations for Implementation.
Marcano Belisario J, Lunan M, Hawe E, Thurairajah S.
Speaker:
Michaela Lunan-Taylor, PhD
Senior Associate, Evidence Synthesis
RTI Health Solutions
Literature reviews (systematic, targeted or narrative) are a cornerstone of health economics and outcomes research and evidence-based decision making. However, the volume of scientific data is ever increasing, with the average systematic literature review (or SLR) estimated to take over 15 months to complete. Machine learning and AI are being hailed as the answer to streamlining literature reviews through reduced timelines and lower consumption of research resources. However, these methods need evaluation to ensure continued robustness and appropriate integration into existing workflows.
In our research, we identified recent evidence of the use of machine learning and AI to support systematic reviews. Barriers to the uptake included a lack of regulatory guidance, costs, training requirements, user-friendliness of tools, and transparency concerns.
Further research is needed on the best approaches to maximise recall of AI. There is also a need to expand the data sources used for training machine learning and AI tools. Given the high expectations of AI, performance assessment against current practice, rather than perfection, may be appropriate to enable efficiency and increased uptake of these methods.
