AI for Notetaking

Context
 

Health care providers spend a significant amount of time managing health records and taking notes to document patient history, physical examinations, test results, referral reports, and other administrative tasks.³⁶ Because these data are often unstructured, health care providers have to spend excessive time on notetaking. This increased time negatively impacts their workflows and workloads, potentially resulting in burnout.³⁷ Furthermore, when health care providers are documenting patient information on paper or computers during consultations, they tend to have decreased eye contact with their patients.³⁸ This can create communication barriers and reduce overall patient satisfaction. The considerable time additionally spent on documentation outside of direct patient interactions can feel tedious or as an increased workload, leading to feelings of burnout and professional dissatisfaction for health care providers.³⁸

Definition
 

AI-powered notetaking applications use advanced technologies, such as automatic speech recognition and natural language processing (which enables machines to understand, interpret, and generate human language), to transcribe conversations between patients and health care providers and to generate clinical notes.³⁹ These technologies work by automatically creating notes from conversations, which simplifies the notetaking process, and by converting patient data into structured information from unstructured sources. Health care providers can then edit, review, and sign the generated note.³⁸ 

Examples

AI scribes use machine learning technologies to create written summaries of conversations between patients and health care providers.²¹ These are capable of producing conversation transcripts, medical notes, and referral letters. 

• Using AI scribes significantly reduced the time spent on administrative tasks: a reported 69.5% reduction in laboratory settings and an average of 3 hours less per week in routine practice settings in Ontario.²¹ Primary care providers using AI scribes have reported reduced administrative burden, lower cognitive load, and less after-hours work. They also noted improved efficiency, increased job satisfaction, and enhanced quality of care.²¹
• A 10-week pilot study also demonstrated that AI scribes could decrease the time physicians spend on documentation during appointments. The authors of the study noted that this technology is capable of generating high-quality clinical notes, facilitating more meaningful interactions with patients, and reducing the workload that often extends beyond office hours.⁴⁰ Patients reported feeling comfortable with AI scribes and observed that health care providers spent less time on their computers.⁴⁰
• Current studies and reports indicate that AI scribes are imperfect and can have errors or omissions that require a review of notes by HCPs.⁴¹

Other AI tools are used for notetaking
 

• PhenoPad is an open-source interface for clinical notetaking that captures both free-form notes and standardized phenotypic data (e.g., information captured on a tablet by clinicians) through various methods, such as speech recognition, natural language processing, and handwriting recognition._³⁸
• Tali AI technologies integrate AI scribes, medical dictation, and medical information retrieval to enhance clinical documentation and streamline clinical workflows. 

Potential Positive Impacts 
 

• AI for notetaking could reduce the administrative burden by streamlining notetaking, minimizing errors, and enhancing overall efficiency in health care systems.⁴²
• Implementing AI for notetaking could potentially decrease the time spent on documentation, thereby enhancing patient-provider interactions.^43,44
• Accurate notetaking could enhance health outcomes and improve patient experiences.⁴⁴
• By consolidating patient data into high-quality medical notes, health care providers could gain a comprehensive view of the patient’s medical history, which may lead to better access to critical information for care teams.⁴⁵
• It can also improve health care provider satisfaction, potentially reducing burnout rates.⁴⁵

Additional Insights 
 

The workshop participants mentioned that AI for notetaking is essential for successfully implementing other AI technologies. High-quality clinical notes and structured data are critical for facilitating discussions between patients and health care providers about their concerns, laying the groundwork for disease detection and diagnosis, treatment optimization, and other AI technology categories discussed in the 2025 Watch List.

The workshop participants also emphasized efficiency gains associated with AI for notetaking because clinical notetaking consumes a significant amount of health care providers’ time. Most health care providers, whether in primary care, specialized fields, or other areas, need to complete clinical notes based on conversations with patients, physical examinations, and other information. As a result, there was considerable enthusiasm and demand among health care providers about adopting AI technologies for notetaking. Furthermore, some participants pointed out that various AI technologies for notetaking are already being used in health care in Canada, which could help alleviate health care provider burnout. A crucial insight shared at the workshop was that using AI for notetaking may be a safer early adoption of AI in health care than other applications (e.g., diagnosis and treatment).

AI can enhance notetaking efficiency, but accuracy is critical in health care because errors may lead to significant risks in diagnosis and treatment. However, using AI for notetaking can lead to errors, such as AI hallucinations (instances when AI generates distorted or inaccurate information).⁴⁶ In the context of notetaking, AI tools may record events that did not happen or leave out important information. AI may have difficulties with various languages and may not accurately document physical examinations.⁴¹ To prevent errors and AI hallucinations, and to ensure inclusivity in various languages, AI tools should incorporate strong error-checking measures and support multilingual capabilities. This includes providing comprehensive user training, ensuring that clinicians review AI-generated documents, and having developers focus on producing accurate AI outputs. Further research is needed to explore the trade-offs between administrative gains (i.e., efficiency) and the risks of bias or errors.⁴⁷

Additionally, there is a need to integrate AI for notetaking into electronic health records to assist health care providers in eliminating the need to copy and paste clinical notes from various applications.⁴⁸ The integration of AI and electronic health records is a challenging endeavour, potentially requiring coordinated efforts from multiple teams. If AI-generated notes can be securely integrated into electronic health records, it will further enhance workflow efficiency and reduce manual entry errors. 

AI Tools to Accelerate and Optimize Clinical Training and Education

Context
 

Clinical training and education enable HCPs to gain the knowledge and skills necessary for diagnosing and effectively treating patients. A common approach in medical education for health care providers emphasizes knowledge retention and relies heavily on memorizing evidence, procedures, and guidelines.^31,49 Using AI tools to accelerate and optimize clinical training and education may significantly enhance personalized learning, real-time feedback, skill development, and objective, automated assessment.⁵⁰ This could potentially transform the current medical education system, including both medical school and continuing medical education.⁵¹

Integrating AI into clinical training and education offers the potential to reduce health care costs, enhance the quality of care, and broaden access to care by empowering health care providers with advanced technological AI tools.^52,53 This includes incorporating AI-powered learning resources within the medical curriculum and continuing medical education as well as training health care providers to understand and effectively apply AI tools in diagnosis, treatment, and care delivery. These 2 areas are interconnected because it is necessary to understand the particulars of AI technology to make the best use of AI in clinical education.

In addition to AI tools supporting education, HCPs require training and education that equips them with the necessary skills to effectively implement AI into clinical practice. This could include understanding the language of AI and developing the technical skills to operate AI-driven tools by gaining knowledge of how these tools work and familiarity with basic concepts and principles behind AI technologies.49 Additionally, training may need to address other considerations related to AI, such as maintaining critical thinking abilities when using AI tools.⁴⁹

Both the Royal College of Physicians and Surgeons of Canada and the College of Family Physicians of Canada acknowledge the importance of AI in health care. For example, the Royal College of Physicians and Surgeons of Canada has made recommendations regarding implementing AI and digital technologies in residency training and health care delivery.⁵⁴ The recommendations emphasize the potential impacts of AI on both clinical practice and medical education, not just AI-specific skills. For example, the recommendations propose introducing a new discipline focusing on clinical informatics to equip physicians with AI tools for practice, encouraging collaboration with medical schools in Canada to promote AI through MD and PhD programs, and fostering “clinical innovators” as emerging careers in AI-driven health care.⁵⁴ The College of Family Physicians of Canada has also published a statement supporting AI research and development in family medicine and primary care.⁵⁵ 

Definition

AI tools to accelerate and optimize clinical training and education could summarize available evidence for physicians, medical students, and patients, providing general background knowledge and the latest evidence regarding interventions.⁵⁶ AI technologies should support upskilling and reskilling for both health care providers and patients.⁵⁷ 

Examples
 

• OpenEvidence is a language model specifically designed for medicine to aggregate and synthesize clinically relevant evidence in formats that are understandable and accessible, enabling more evidence-based decision-making and improving patient outcomes.⁵⁸ The tool has demonstrated significant accuracy in answering the US medical licensing examination questions. Health care providers and learners can set up an account for unlimited, free access to the tool. This tool offers a distinct advantage over other models by providing citations for its responses, allowing users to validate the information quickly. However, it was designed for targeted point-of-care clinical use, so its short responses may make it less useful as a comprehensive information resource.⁵⁸
• ChatGPT, a generative LLM, presents numerous opportunities for enhancing clinical training and education. Although some AI tools do not explicitly mention ChatGPT, they may use ChatGPT to generate their response. ChatGPT can be used to create virtual patient simulations and quizzes for medical students.⁵⁹ It can also critique simulated doctor-patient communications, summarize research articles, and generate a curriculum for health professionals. However, when using ChatGPT for medical education, it is important to use proper prompting and to be aware of the issue of AI hallucinations, particularly when ChatGPT fabricates references or contents.⁵⁹
• AI-VSP (Artificial Intelligence Virtual Simulated Patients) can be used in clinical teaching as a complementary learning tool. This technology represents an advancement in health care education, providing a powerful tool for training future HCPs.⁶⁰ By combining AI and virtual reality, the technology can create immersive, interactive, and personalized learning experiences that improve clinical skills and enhance decision-making abilities.⁶⁰ 

Potential Positive Impacts 
 

• AI could enhance the learning experience and improve training efficiency by summarizing large amounts of information and reducing the training burden on health care systems and easing the research burden on individual HCPs.⁵³
• AI technologies may help address health care resource crises by promoting innovative solutions and encouraging curiosity among HCPs by providing opportunities for personalized educational materials, novel solutions, and data-driven insights. AI could facilitate the exploration and adoption of other technologies and support more efficient resource allocation and problem-solving strategies.⁶¹
• Training health care providers to effectively use AI in their practice and incorporating these technologies into clinical training and medical education could ultimately improve the quality and efficiency of patient care and contribute to positive health outcomes.⁵³ 

Additional Insights 
 

During the workshop, participants highlighted how AI can serve as a tool to enhance clinical training and medical education overall. For example, one workshop participant mentioned that trainees have already begun using ChatGPT to support their clinical education, help them prepare presentations, and facilitate their learning processes.

The discussion also touched on the need for health care providers to receive education on how to effectively use AI technologies in their clinical practice within the health care system (training in AI or AI as a training subject). This education is essential for successfully implementing other technologies, including AI Technology 1: Notetaking, AI Technology 3: Disease Detection and Diagnosis, AI Technology 4: Disease Treatment, and AI Technology 5: Remote Monitoring. For instance, health care providers need to understand what AI is, how to interact with AI systems, how to use them to solve problems, and how to critically appraise AI models. The dual role of AI as both a training subject and as a tool to accelerate clinical training highlights the importance of integrating these 2 aspects.

Participants emphasized the importance of involving patients in this educational process, recognizing that AI has the potential to enhance patient engagement. AI technologies could offer more accessible and personalized materials for patients as well. Educating patients about AI tools in health care could encourage them to take more responsibility for their health and improve proactive self-management skills. Health care providers should actively support patients and caregivers to help facilitate the process. 

AI for Disease Detection and Diagnosis

Context
 

Disease detection and diagnosis are essential for timely treatment. Early detection and effective treatment are the most important solutions to reduce the death rates caused by chronic diseases.⁶² Health care providers typically need to consider and interpret various pieces of information, including clinical manifestations, physical examinations, and other relevant data, making the diagnosis process quite complex. Given the dynamic and changing environment of the health care system and the limited time that HCPs are in clinical practice, making accurate disease detection and diagnosis can become a cognitively challenging task.⁶³ AI technologies offer potential advantages in supporting this process. As of August 2024, the US FDA had authorized approximately 950 medical devices that use AI or machine learning.⁶⁴ Most of these devices are designed to assist in the detection and diagnosis of treatable diseases.⁶⁴ The top 5 medical specialties using AI technologies are radiology (e.g., Overjet Image Enhancement Assist), cardiology (e.g., EchoGo Heart Failure 2.0), neurology (e.g., BrainSee), hematology (e.g., AI-4510 Urine Particle Analysis System), and gastroenterology and urology (e.g., EndoScreener).⁶⁴  

Definition
 

AI for disease detection and diagnosis refers to using AI technologies, such as machine learning models, to assist health care providers in improving disease detection and diagnosis based on various data, such as medical images, physical examinations, family history, environmental factors, and dietary habits.⁶⁵ Radiology has been leading the way in AI for disease detection and diagnosis.⁶⁶ In recent years, there has been an explosion of AI technologies for analyzing medical images (e.g., radiology, pathology) to make faster and more accurate diagnoses.  

Examples
 

• ASIST-TBI was developed to quickly identify traumatic brain injuries by screening the CT scans of patients with head injuries in the emergency department. If the model indicates the need for surgery, the physician can consult a neurosurgeon directly, bypassing the wait for a radiologist review. The model showed accurate prediction for neurosurgical intervention with an area under the receiver operating characteristic curve of approximately 0.90, with accuracy, sensitivity, and specificity all exceeding 80%.⁶⁷ 

• LumeNeuro uses machine learning techniques to detect neurodegenerative brain diseases at an early stage by screening for retinal protein biomarkers. This technology is low cost and noninvasive, using polarimetric imaging to identify retinal amyloid deposits without dyes. The machine learning models developed by the researcher from the University of Waterloo can predict thioflavin positivity with high accuracy, sensitivity, and specificity, which is an indicator of amyloid presence and potentially Alzheimer disease.⁶⁸ 

   

Potential Positive Impacts 
 

• AI for disease detection and diagnostics may improve health systems by enhancing diagnostic accuracy and making advanced diagnostic tools accessible. (e.g., AI tools could detect patterns that human health care providers might miss).⁶⁹ However, one workshop participant highlighted that AI technologies might increase the demand for diagnostic tests (e.g., lab workload), which could increase the burden on our health care system. The Advisory Group experts recognized that leaving conditions undiagnosed over several years can negatively impact health care resources, which often leads to a need for more invasive tests, specialized care, and greater costs. It is important to find a balance between early disease diagnosis and the risk of overdiagnosis and misdiagnosis (e.g., false positives and false negatives), with the primary goal of minimizing potential harm.
• AI could enhance care pathways by identifying diseases or conditions earlier, which may reduce the waiting time for interventions or enable health care providers to assess and manage patients more efficiently.⁶⁹
• A recent randomized controlled trial conducted in the US found that the LLM ChatGPT Plus alone demonstrated higher performance in diagnostic reasoning compared with physicians, even when the LLM was available to them. However, when physicians used the LLM as a diagnostic aid, it did not statistically significantly enhance their clinical reasoning or reduce time spent per case compared with using traditional resources, such as UpToDate or Google.⁷⁰ The study only focused on diagnostic reasoning, but did not focus on other critical clinical skills (e.g., patient interaction or data collection) and the authors noted that LLMs should not be used for autonomous diagnosis without physician oversight. Although LLM shows promise in diagnostic reasoning, further research and development are needed to determine their real-world impacts on patient care.

Additional Insights 
 

During the workshop, participants agreed that there is a high demand for AI in disease detection and diagnosis. Progress in machine learning models, particularly deep learning models, has made these AI tools for disease detection and diagnosis more accurate. Early disease detection and diagnosis support disease prevention, which keeps people healthy and out of the queue for intensive medical care. In addition, a vast amount of data are available for training these models in disease detection and diagnosis. However, participants noted that this category includes a broad range of AI technologies, related to both radiology and other medical areas . These technologies may have different timelines for significantly impacting the health care system. Concerns about the readiness of the health care system to implement AI technologies for disease detection and diagnosis were raised during the discussions.

Using AI technologies to enhance disease detection and diagnosis could negatively impact health care system capacity due to increased demand for follow-up testing and interventions, potentially exceeding the current health care system capacity in Canada. There is a need to apply appropriate methods to identify the right populations for further diagnostic testing to mitigate these negative impacts. For example, colorectal cancer screening was suggested for high-risk populations (i.e., 15-year risk of colorectal cancer greater than 3%) while it was suggested against for low-risk populations (i.e., 15-year risk of colorectal cancer less than 3%) based on absolute risk reductions at population level.⁷¹ In addition to traditional methods for identifying high-risk populations, AI technologies using advanced machine learning models may help detect the right populations more accurately and efficiently for further screening, reducing the screening burden on our health care system. Another method to reduce the burden of screening or testing is to systematically use incidental data, often referred to as “opportunistic screening.”⁷² For instance, researchers have developed an AI-powered approach for detecting pancreatic cancer, which can accurately identify and classify pancreatic lesions using noncontrast CT scans that are routinely performed for other clinical purposes (i.e., not for screening purposes).⁷³ 

AI for Disease Treatment

Context
 

Clinical treatments, including pharmacological and nonpharmacological interventions, are a critical component of health care and are central to the clinical decision-making process. Disease treatment is the process of selecting the most appropriate and effective treatment based on available evidence, individual patient clinical needs, patient values, and other factors (e.g., costs).⁷⁴ This approach aims to achieve the best clinical outcomes while minimizing unnecessary interventions and side effects. Treatment is closely related to health care resource allocation and clinical results. Health care providers require various data and information to make informed decisions about treatment options. 
 

Definition
 

AI technologies for disease treatment offer a new way for patients to access the most appropriate and effective treatment, complementing traditional in-person health care and digital options such as telemedicine. The potential roles of AI in treatment optimization include:

• Identifying optimal treatment plans: AI can assist health care providers in determining the best medication, dosage, and treatment plan for each patient.⁷⁵ This involves considering potential drug interactions and customizing the treatment plan according to the patient’s unique genetic profile and medical history.⁷⁶ Additionally, AI has the potential to continuously update treatment plans based on new evidence and patient responses, which could enhance the overall patient experience.⁷⁷
• Assisting the triage process (especially in the emergency department): AI has the potential to enable earlier intervention. It may enhance health care decision-making by improving discrimination capabilities and predictive accuracy, leading to better risk assessments. This could help assess the need for hospitalization and optimize resource allocation.⁷⁸

Examples
 

• Kaia Health is a digital therapeutics company offering accessible, evidence-based treatments for disorders such as musculoskeletal pain, chronic obstructive pulmonary disease, and osteoarthritis. Using machine learning, it delivers interventions to help patients self-manage their conditions. Kaia Health is a member of the Digital Therapeutics Alliance, a nonprofit organization focused on advancing digital therapeutics.
• Wysa is an AI mental health chatbot for stress, anxiety, depression, self-care, and sleep disorders.⁷⁹ It offers a conversational AI tool that provides mental health support, guiding users through both cognitive behavioural therapy programs and on-demand assistance with a therapist.⁷⁹ The platform supports individuals experiencing subclinical symptom levels and helps them establish proactive prevention routines. The technology is especially beneficial because an AI chatbot can be available 24 hours a day.

• Valence Labs offers AI tools and software for drug discovery, such as LOWE — an LLM-orchestrated workflow engine designed to execute complex drug discovery workflows using natural language. It aims to provide an easy-to-use tool for drug discovery. LOWE may serve as a foundation model that accurately represents or simulates the biological and chemical aspects of drug discovery. It could assist people in formulating hypotheses, learning from results, and designing and executing experiments for hypothesis testing.⁸⁰  

   

Potential Positive Impacts 
 

• AI for disease treatment could support a more efficient and sustainable health care system, enhance health care provider and patient satisfaction, and improve overall population health by focusing on the most appropriate and effective treatment.⁸¹

• Using AI technologies to optimize treatment plans, health care providers could improve clinical outcomes, minimize adverse effects, reduce unnecessary interventions, and promote cost-effective care.⁸²

   

Additional Insights 
 

During the workshop, participants engaged in a similar discussion about using AI for disease detection and diagnosis. Patients need the most effective treatment for their condition. Using AI for treatment allows for assessing personalized treatment plans rather than using one-size-fits-all strategies. It also helps mitigate the health care human resource crisis, particularly for primary care providers in the community,⁸³ by providing physician decision support and improving operational efficiencies and patient self-management. However, the health care system may require significant resources to incorporate AI for treatment and other technology categories included in this Watch List, such as financial investment, training for HCPs, and ongoing technical support and updates.⁵¹ Concerns about the readiness of the health care system to implement AI technologies for treatment optimization were raised during the workshop discussions. These concerns included insufficient funding, disparities in technology adoption across geographic regions in Canada, and other issues discussed in the 2025 Watch List (e.g., privacy and data security and data availability, quality, and bias).

AI for Remote Monitoring

Context

Remote monitoring uses various biomedical sensors to collect health-related data outside of hospitals, typically in patients’ homes.⁸⁴ Health care providers can access these data wirelessly to make informed decisions about patient care, such as heart rate, respiration rate, temperature, blood pressure, and oxygen saturation. AI-powered analytics or machine learning algorithms can then process the collected data to identify risk factors and patterns, predict potential health issues, and provide clinicians with actionable insights beyond data. This system enables patients to maintain their usual activities while being monitored, potentially reducing health care costs, the inconvenience of in-person visits, and traffic congestion associated with hospital or clinic visits.⁸⁴ 

Definition
 

AI for remote monitoring refers to using AI technologies to collect, analyze, and interpret patient health data remotely and provide real-time data to health care providers in other locations to ensure appropriate and timely interventions without frequent in-person visits.⁸⁴ AI could use different types of data for remote monitoring, including information from wearable devices, smart home sensors, and smartphones. For instance, AI can track heart rate, blood pressure, respiratory rate, temperature, and physical activity.⁸⁴ These data can be collected in real time to identify any deviations from typical patterns or specific thresholds. AI for remote monitoring combines remote monitoring data and health care providers’ clinical judgment with machine learning algorithms. Potential roles of AI for remote monitoring include:
 

• generating alerts and notifications for health care providers, allowing for timely interventions and reducing the risk of adverse events^84,85
• predicting potential health issues or adverse events based on the analysis of both historical and real-time data, enabling proactive interventions^84,85 and, because AI algorithms can learn over time, their predictive ability may improve.⁸⁴ 

Examples
 

• AlayaCare offers software solutions that include remote patient monitoring, clinical documentation, and patient and family portals. These solutions empower care providers, particularly for home care providers, to achieve better health outcomes through AI technologies and data insights. According to research conducted by AlayaCare, its technology with machine learning could improve event predictions by 11% while reducing overdiagnoses by 54%. A clinical study in Canada reported that implementation of the AlayaCare program reduced both the number and cost of emergency department visits and hospitalizations for patients with chronic obstructive pulmonary disease or chronic heart failure.⁸⁶ Specifically, when the technology was implemented for 3 months, the number of emergency department visits was reduced by 68% and hospitalizations decreased by 35% compared to baseline before using the technology._⁸⁶ The average cost of emergency department visits fell from $243 at baseline to $67, and the average cost of hospitalizations dropped from $3,842 to $1,399 during the 3-month period.⁸⁶
• Coughy uses AI-based sound analysis technology to analyze digital audio biomarkers to assist patients and health care providers in making smarter, faster, and more informed decisions. It offers an AI-powered real-time remote cough monitoring solution that provides objective measurements for tracking chronic coughs. Patients can record their cough sounds using smartphones or smartwatches. Health care providers can monitor these recordings remotely and in real time.

Potential Positive Impacts 
 

• AI-powered remote monitoring could increase monitoring capabilities to expand access to health care by facilitating continuous, real-time monitoring of patient health and minimizing the need for frequent in-person visits, especially for those living in remote and rural areas.⁸⁷
• AI-powered remote monitoring could enable patients to receive medical and health care in their homes, reducing the need for hospitalization, allowing more efficient allocation of medical resources, and saving costs for both patients and health systems.⁸⁸ However, this is closely related to the accuracy of the AI tools.⁸⁸ If AI tools produce false positives or false negatives or overdiagnoses, it could increase costs or impose a greater burden on patients and the health care system. AI for remote monitoring is closely linked to AI Technology 3: Disease Detection and Diagnosis discussed in this Watch List.
• AI for remote monitoring could impact health care human resources by improving efficiency and allowing medical staff to concentrate on their core medical tasks.⁸⁸ By analyzing monitoring data, AI algorithms can evaluate patient flow, resource utilization, and staffing patterns, enabling better resource allocation.⁸⁸
• AI-powered remote monitoring could use algorithms that analyze large, real-time datasets to identify patterns and trends that allow for adjustments to treatment plans as necessary, resulting in more dynamic and responsive care.⁸⁸ Implementing AI for remote monitoring can enhance the quality of life for patients and help prevent potential adverse outcomes.⁸⁸

Additional Insights 
 

The workshop participants highlighted the importance of including AI for remote monitoring in the 2025 Watch List for several reasons. There are abundant data available from various wearable devices, and increasingly advanced machine learning models are being developed for data analysis. The shift in demographics toward an aging population and the growing prevalence of chronic diseases and comorbidities are generating increasingly large health care datasets. This increases the need for AI technologies in remote monitoring, especially for patients in rural and/or remote areas and underrepresented populations, such as Indigenous communities. Additionally, AI for remote monitoring can improve community-based and home-based care by prioritizing the needs of patients, caregivers, and health care providers, thus enhancing clinical outcomes and quality of life. However, implementing AI technologies for remote monitoring must consider challenges such as limited internet access in rural and/or remote areas, the populations’ technical literacy, and technical issues, particularly for older adults who may experience issues if technologies are not intuitive or do not accommodate their needs.

The accuracy and quality of these measurements are crucial, and it is important to be aware of any AI limitations in measurement. For example, studies have shown that the accuracy of pulse oximetry can be decreased in patients with darker skin tones.⁸⁹ Future research is needed to comprehensively assess the accuracy of these measures that contribute to AI technologies. 

Privacy and Data Security

AI in the health care system can learn patterns from large dynamic multidimensional datasets that include patient, provider, and health system data.⁶⁶ Because these databases contain personal health information that is used to make recommendations or predictions for patient care, there is concern about how this information can remain private and secure (i.e., safeguarding sensitive medical information to ensure it remains confidential and in secure locations) and how exactly it will be used (i.e., will its use be beneficial or harmful to patients).^27,66 

Examples of recent AI privacy and data security concerns in Canada: 
 

• A 2024 survey of physicians in Canada showed that 21% were confident about AI and patient confidentiality, whereas 79% were either not confident or unsure.⁹¹
• Canada ranks 10th in number of security breaches worldwide. From 2015 to 2023, there were at least 14 reported major cyberattacks on Canadian hospitals, labs, and health networks, including blocking services, using ransomware to lock access to personal health information, and compromising personal health information by removing it from health systems and sharing it illegally.^92,93
   

In the global market, not all AI technologies are designed to be applied in Canada. As described in the CDA-AMC report on the implementation of AI-enabled medical devices and other digital health technologies,³⁰ Canadian legislation for the private sector’s collection, disclosure, and use of personal information for commercial and for-profit endeavours falls under the Personal Information Protection and Electronic Documents Act (PIPEDA);⁹⁴ for the public health care sector (e.g., hospitals, long-term care facilities), it is under local provincial and territorial laws.^94,95 This includes jurisdictional-specific legislation, such as the Personal Health Information Privacy and Access Act in New Brunswick; the Personal Health Information Acts in British Columbia, Newfoundland and Labrador, Nova Scotia, and Manitoba; and the Freedom of Information and Protection of Privacy Act and the Personal Health Information Protection Act in Ontario.^94-96

It may be unclear to what extent AI technologies, such as scribes, comply with these legislative protections; therefore, it may be difficult for HCPs, patients, and decision-makers to know what is legally and ethically required to safeguard patient, provider, and health system information. For example, some publicly available AI scribe technologies that HCPs currently use to summarize conversations with patients may store data securely within Canada but their use by HCPs is still not regulated in Canada. Some local provincial colleges of physicians and surgeons (e.g., College of Physicians and Surgeons of British Columbia) have advised physicians to be cautious and risk-aware when using these tools.^41,96,97 There are other technologies (e.g., ChatGPT) that patients may use to understand their personal health information that are also not regulated in Canada and do not comply with security or privacy laws in the US (i.e., The Health Insurance Portability and Accountability Act [HIPAA]) where the data are stored.^96,98

Some patients may have concerns about privacy and consent when using AI. A 2024 scoping review⁹⁹ of 37 studies on patient perspectives on the use of AI in health care found that patients were concerned about the use of AI, such as informed consent, regulation, and trustworthiness. Patients were interested in knowing how AI tools were being applied, how they were developed, and whether their data would be used anonymously. There were also questions about who they could approach if errors were made.⁹⁹ As the use of AI becomes more integrated into the health care system in Canada, it will be important to consider privacy concerns, whether patients have fully consented to include their data in AI databases and understand the potential uses of their data in AI algorithms, and what different care pathways could look like for patients who opt in or out of AI processes.

Workshop participants stated that there is a strong need for privacy guardrails related to the use of AI in health care. They highlighted how this issue is related to AI Issue 4: Data Sovereignty and Governance because how data are managed can set the stage for its security. They mentioned how establishing clear privacy and security measures can create trust and help build support for the adoption of new AI technologies. An example of trust in action is Quebec’s Bill 64, An Act to modernize legislative provisions as regards the protection of personal information which received royal assent in 2021. This legislation requires that patients know how medical decisions using their personal information were made (i.e., if they were made exclusively using automated processes such as AI or in conjunction with physicians) and gives them the right to request to have these decisions reviewed, including review by a human if the decision was fully automated.^100,101 
 

Solutions can include: 
 

• proactive adoption of data privacy and security protocols for encryption, secure storage, access control, data anonymization and de-identification, and data transmission⁵¹
• frameworks and guidance to train HCPs and patients on using AI securely with a focus on patient care⁵¹
• compliance with local privacy laws⁵¹
• policies to allow patients to withdraw or grant informed consent about AI tools being used for their health, have access to their data, and have information about how AI uses their data.⁵¹ 

Liability and Accountability

Issues of liability (legal responsibility) and accountability (moral, legal, procedural, or organizational responsibility) arise from the use of AI in health care.^28,102 The use of AI in health care can take many forms and may include interpretation of data by AI and/or a human (i.e., the HCP) at various stages of the AI feedback loop which is known as “human-in-the-loop.”

HCPs do not control how a particular AI system’s functionality was developed, the decisions it makes, or the recommendations it provides. As a result, it can be difficult for HCPs to understand how an AI system came to certain conclusions based on their input.¹⁰² This challenge relates to both explainability and transparency in AI. Explainability refers to how well an AI system’s reasoning, validation, and reliability can be communicated to users.³¹  

However, AI systems can be so complex that they are not easily explainable or understandable for nontechnical users. They are sometimes referred to as “black boxes” because their processes are opaque. Therefore, HCPs may be using AI in their practice but not have the expertise to comprehend how the AI works or makes decisions.^31,103  

There are different ways that humans can work with AI systems. There are some approaches that employ human-in-the-loop AI in which humans actively provide feedback on AI outputs that the AI learns from and incorporates into future predictions; this is an example of collaborative decision-making between AI and humans.¹⁰⁴ Other approaches may use predefined algorithms designed by AI developers and engineers and the static AI outputs are then checked by humans. In the case of health care, an HCP may check an AI application’s output to determine whether it will be useful and safe for patients.¹⁰⁴ In either approach, the design of AI systems that include human involvement is complex given the variations of human understanding of AI functions.

Where does the responsibility for the actions of AI lie, especially in the event AI makes errors or causes patient harm in ways that are potentially unknown or hidden to HCPs?¹⁰² If HCPs use an AI system’s suggestions and a patient is harmed, who would be held legally liable — AI developers, the health care system, or HCPs?¹⁰³ The Canadian legal system does not have clear-cut answers to these questions because broad legislation for AI regulation is still being discussed in parliament (i.e., Bill C-27, Digital Charter Implementation Act, 2022).^101,105 Therefore, there will be challenges in dealing with these issues in the coming years.²⁸

Examples of recent AI liability and accountability concerns in Canada include the following: 
 

• Outside of health care, AI chatbots have come under scrutiny. In 2022, Air Canada was liable for damages relating to an AI chatbot promising a discount that did not exist to a customer.¹⁰⁶
• Several organizations have joined Canada’s Voluntary Code of Conduct on the Responsible Development and Management of Advanced Generative AI Systems.¹⁰⁷
• On November 12, 2024, Canada launched the Canadian Artificial Intelligence Safety Institute to invest $50 million over 5 years to support AI research, including cybersecurity. This is linked to AI Issue 1: Privacy and Data Security and with managing risks to the development of AI that can be a danger to human insight.^108,109
   

Workshop participants expressed that both developers and users of AI need to be clear about accountability. Also, when HCPs use AI, it is currently still viewed as something that the HCP is ultimately responsible for. They expressed that it may not be appropriate to place this burden on HCPs and patients who are the end-users of AI and not experts on AI development and regulation. These concerns overlap with concerns raised about overreliance on AI by clinicians and health systems (Appendix 3).

Ways that developers can work in collaboration with health care providers, patients, and HCPs in a shared decision-making process to manage issues such as explainability, intelligibility, transparency, accountability, responsibility, and liability include: 
 

• minimizing liability risks through agreements or predefining who is liable in certain scenarios²⁸
• establishing disclosure requirements²⁸
• improving hospital policies that help HCPs use AI tools safely²⁸
• providing guidance and support for HCPs on using AI to make decisions within health care system and legislative policies⁵¹
• providing guidelines on how HCPs can communicate with patients about AI use and how to incorporate recommendations from AI (i.e., informed consent)²⁸
• ensuring algorithm transparency so that AI systems are explainable to end-users such as HCPs and patients.⁵¹

Data Availability, Quality, and Bias

To be useful, AI in health care will require a high volume of data to train algorithms or generate new information. Data availability, quality, and bias are closely related and are factors that must be considered for its use. 
 

• Data availability: The reliable access to and use of data without interruptions to performance and functionality, including considerations for data validation and integrity, storage and retrieval of information, network responsiveness, and system performance.^110-112
• Data quality: The accuracy, reliability, completeness, and relevance of data that can lead to effective learning and decision-making by AI systems.¹¹³
• Data bias: The perpetuation of existing social biases by inaccurate and unreliable AI systems that can cause inequities, discrimination, and inaccurate decisions.¹¹⁴ AI bias can occur along the different stages of AI development — from data collection to model deployment and evaluation. This can include data bias (unrepresentative, skewed, and inaccurate data being put into AI systems), algorithmic bias (errors in machine learning algorithms because of how the data were coded and trained), or user bias (who gets to use AI and how they interpret AI outputs to make decisions).^114-116
   

There are already concerns about the quality and bias of AI. One study found that AI chatbots that use LLMs propagated inaccurate and harmful race-based medical content, furthering existing biases;, had different responses for the same clinical questions;, and provided incorrect responses about already debunked or outdated information depending on when and how the information databases were updated.^117,118 AI can easily perpetuate incorrect notions about underserved communities that experience inequities because it may learn incorrect ideas about how health conditions are currently understood and treated by the health care system and HCPs. Examples of this can include making assumptions about 1 group of people having higher rates of a disease, forming recommendations based on datasets that are misrepresentative of those affected by the health condition (e.g., missing data from equity-deserving groups), providing predictions with a one-size-fits-all approach when some groups may have different outcomes and needs, and using proxy variables based on correlations in models that are inaccurate (e.g., conflating an individual’s neighbourhood with their race, their health care spending with their need for complex care, their income with their knowledge, or their spending patterns with their medical conditions).^116,119,120

Examples of the concerns about data availability, quality and bias include:
 

Availability: Canadian health data are currently fragmented across jurisdictions and health systems. Although 93% of physicians in Canada use an electronic medical record in their local context, there have been challenges with coordination of care, including scheduling referrals, sharing information, and receiving reports from other HCPs.^121,122 This lack of harmonization within local systems and across the country can make it difficult to establish large central datasets with information from a variety of sources. This results in challenges when clinically and technically validating AI models that require seamless inputs of large volumes of data. Leaders across Canada have been addressing this issue in the following ways: 
 

• Canada Health Infoway and jurisdictional leaders have developed the Shared Pan-Canadian Interoperability Roadmap, which is a framework that promotes interoperability (i.e., communication, collaboration, and connection between HCPs, facilities, and health systems across geographies). The application of this framework can guide the seamless flow of information between different parts of the health care system.¹²³ This sharing of information can be used for patient care, health system planning, and research.¹²³
• In June 2024, the Parliament of Canada established Bill C-72, the Connected Care for Canadians Act. In practice, patients and providers would be able to easily access patient data and health information and technology vendors would be prevented from “data blocking” (interfering with the access or exchange of electronic health information) or incur financial penalties if its software does not meet policy requirements, which allows for patient health to remain a priority over AI vendor interests.¹²⁴ In this way, health data can be safely shared and accessed across systems and HCPs to promote timely and coordinated patient care, facilitate easier access and exchange of data, and drive health system innovation.^125,126 

Quality: Poor quality data (i.e., have errors or are imprecise, unvalidated, incomplete, or unreliable) can occur when the training data used to teach AI and build algorithms does not structurally match the new inputs it receives for processing.^127-129 This can cause direct harm to patients when AI systems use this incorrect data to make decisions about patient health and treatment. The decisions can be harmful, discriminatory, and ultimately may infringe on human rights or right to receive health care (e.g., being denied care based on AI outputs).^127-129

Bias: The data that AI uses should be representative of the population it will serve. Systemic biases in health care have historically omitted or misrepresented various equity-deserving groups based on factors such as sex, gender, race, and disability.¹³⁰ Existing health care data, ways that medical technologies are used, and current human health care practices can lead to AI systems perpetuating biases and exacerbating disparities, which can cause misdiagnoses, fatal outcomes, and issues with generalizability.¹³¹ Algorithmic bias has already been observed in current uses of AI that show racial bias against Black communities and perpetuate debunked assumptions about races being biologically different from one another. Some clinical algorithms have been harmful by falsely assuming that individuals who are Black have different muscle mass than those from other racial groups. Other algorithms have used health care spending as a proxy for care, leading to the flawed assumption that patients who are Black do not need as much care as those who are white because they have historically spent less on health care and they need to be sicker to be recommended for care.^132-135 There is evidence that medical devices such as pulse oximeters, scalp electrodes, and thermal thermometers are biased against groups with higher melanin concentrations in the skin which can compound potential harm to patients when combined with biased AI algorithms.^132-135 
  

AI algorithms become biased if they do not include data from diverse populations, their data are influenced by human subjectivity, their design is not regulated from inception, or they repeat inequities from historically discriminatory human practices.¹³¹ If equity-deserving groups are not included in datasets and models are not trained appropriately, algorithms will not be able to recognize patterns for these groups or be able to diagnose or treat these patients. Biases need to be actively prevented during the early stages of AI development because they can be difficult to recognize later on.¹³¹ For example, some studies have shown that there are differences in how chest pain is reported by women and how women are diagnosed with heart attacks.^136,137 Because symptoms were historically documented based on men, coronary symptoms in women are often labelled as “atypical” and can be missed or misdiagnosed by HCPs. If AI models use historical electronic medical record data based on the “typical” symptoms in men, they may not catch cardiac symptoms in women that require further testing.^136,137 This can harm women who need life-saving treatment but may have to wait longer because their symptoms were not recognized.^136,137  

The methods for preventing algorithm and other biases for new technologies are extremely complex and may require different equity-based approaches. For data and algorithm biases used to train AI, the first step is to ensure the appropriate collection of diverse data in a comprehensive data collection before deciding the appropriate approach for using — or not using — these data in AI models.¹³⁰

Workshop participants noted that this issue is related to AI Issue 1: Privacy and Data Security and AI Issue 4: Data Sovereignty and Governance. They expressed that data are a foundational part of how AI technologies work and that this needs to be set right from the beginning. High-quality data are essential, otherwise there can be harmful, irrelevant, and biased data going into and out of AI systems on a widespread scale (referred to by participants as “garbage in, garbage out”). 
 

• Participants highlighted there are gaps in our current health care system and the existing data are of poor quality; there is a lack of information on social demographics, patient-reported experiences, and economic factors. It is difficult to measure how algorithms can be biased across race, language, and many other factors. Participants suggested that people know these factors are important; however, they are not well measured and thus de-emphasized. It is also unclear whether the health care system should wait for validated thresholds for patient data that are not well measured or continue to use AI in its current state with the information that is available to take advantage of this emerging technology. For example, data from patients with rare diseases or those living with disabilities are very limited. Workshop participants highlighted that AI technology is highly developed and progressing faster than the health care system’s capacity to produce high-quality datasets.
• Participants discussed how there is already a combination of biases in existing data (e.g., from medical research) that can give rise to potentially toxic feedback loops that perpetuate the status quo. Biased information is being put into an AI “black box,” and it is hard to identify how the data are being used to make decisions or where the flaws are. This black box issue is referred to as opacity and is discussed as part of AI Issue 2: Liability and Accountability. Workshop participants also noted that electronic medical records are optimized for health care transactions (lab and medication information) and billing documentation but not to understand data at the level that research studies would use (e.g., including longitudinal data; documenting risk factors, socioeconomic status, education levels).¹³⁸
• Participants indicated that there may be a delay between clinical practice and AI development. As an example, the Ontario Renal Network and the Ontario Association of Medical Laboratories recommended the removal of a race-based variable in the equation for estimated glomerular filtration rate.¹³⁹ This change was disseminated through a memo to HCPs in Ontario and not through more widely available evidence sources, such as clinical practice guidelines or randomized controlled trials which are developed by clinicians and researchers; this makes it unclear if the developers of the original AI algorithm were aware of the practice modifications at the same time that the HCPs stopped adjusting the rate for patients. This example speaks to the need for strong partnerships between the health care system and AI developers to ensure that when practice standards are made by frontline workers who use AI tools, AI developers are also aware.  
• Furthermore, race-based data are not routinely collected in Canada; therefore, it is unclear how our health care system would be able to implement this so that patients would be able to give appropriate consent for the use of this information in AI models in scenarios that require knowledge about demographic information and whether the data could be collected without measurement bias.

Solutions can include:

Increasing data availability by

• collaborating and sharing health information across Canada under the Shared Pan-Canadian Interoperability Roadmap  

Improving data quality with:

• legislation to correct AI source data such as patient records¹⁴⁰
• automated error correction methods¹⁴⁰  
• allowing patients to access or correct errors in data¹⁴⁰

Mitigating algorithmic bias by

• framing the problem for AI prediction models correctly (diversity in the development team and design of the AI model including assumptions)¹²⁰
• combining multiple datasets to capture diversity and representation¹²⁰
• identifying sources of bias based on populations, settings, and demographics and managing bias in the preprocessing of data, such as standardized data collection (e.g., how it is measured and labelled) and the processes for how missing data are handled⁵¹
• reducing bias when models are developed and validated (e.g., assessing training data, applying real-world testing, assessing generalizability and reproducibility when using models in different settings, investigating whether bias adjustments are appropriate)¹²⁰
• postimplementation monitoring and managing of AI models to assess bias and user feedback (e.g., review and correct data, manage historical biases)^119,120
• establishing bias and fairness guidelines⁵¹ 

Data Sovereignty and Governance

The use of large datasets in AI systems has led to discussions about ownership and management of data. Data sovereignty refers to the rights a group of people have to control their own data, including how it is collected, stored, and interpreted.¹⁴¹ Data governance defines who has the authority and control to manage the data.¹⁴² Designing AI systems and their datasets requires working together with equity-deserving groups so they have control over their data and consent over how AI-enabled technologies interpret their data and generate new information from it. The goal is not just to collect data from these groups but to actively work to represent the data correctly, allow equity-deserving groups to have autonomy in how the data are used, and work toward a reduction of health inequities to dismantle structural racism.
 

Some communities have created frameworks to address the ownership of and the rights to their health data; examples include:
 

As a result of colonialism, the government in Canada has historically collected, held, and destroyed data from Indigenous Peoples about their health care, demographics, and Residential School records.¹⁴³ The Access to Information Act, which has mechanisms to allow individuals and corporations to request access to information held by the federal government,¹⁴⁴ continues to have barriers for Indigenous Peoples in how their data are stored, preserved, and archived.¹⁴³ Indigenous data sovereignty is the fundamental right of Indigenous Peoples to control data about their lands, communities, and cultures, which are principles described in
 

• the United Nations Declaration on the Rights of Indigenous Peoples¹⁴⁵
• CARE Principles for Indigenous Data Governance (collective benefit, authority to control, responsibility, ethics) from the Global Indigenous Data Alliance¹⁴⁶
• Canada’s Tri-Agency Research Data Management Policy¹⁴⁷
• OCAP principles for First Nations communities (ownership, control, access, possession) from the First Nations Information Governance Centre¹⁴⁸
• Inuit Qaujimajatuqangit, and the National Inuit Strategy on Research from the Inuit Tapiriit Kanatami and Inuit Qaujisarvingat National Committee^149-151
• OCAS principles for Métis governance practices (ownership, control, access, stewardship) from the Manitoba Métis Federation.^152,153

The COVID-19 pandemic revealed existing disparities due to structural and institutional anti-Black racism: Black communities were disproportionately exposed to COVID-19, experienced more infections, had lower rates of screening, and had lower uptake of COVID-19 vaccines.^154,155 In Ontario, these inequities have resulted in calls for the responsible collection of race-based data; however, this must be done appropriately.^156,157
 

• One group working on this is the Black Health Equity Working Group in Ontario that developed the EGAP (engagement, governance, access, and protection) framework for the sovereignty of data for Black communities so they have control over and can make decisions related to their data collection, use, management, and analysis.¹⁵⁴
   

In the context of AI, these frameworks can allow equity-deserving groups the opportunity to ensure that AI correctly understands information about them and generates outputs that can help improve the health of their communities, rather than perpetuating known historical biases. Ways of referring to equity-deserving groups, as well as the language they use to understand and communicate their health needs, are also evolving. AI technologies must stay up-to-date with these ways of knowing.

At our workshop, participants expressed the need for trust and governance of data especially for pan-Canadian clinical decision-making including a structure on how to use AI data in the health care system, clarity on who owns the data, and guidance for users of these systems. They also noted that this issue along with alignment with societal values, sets the stage for other issues such as AI Issue 1: Privacy and Data Security and AI Issue 3: Data Availability, Quality, and Bias.

Solutions can include:
 

• policies to empower patients to be involved in the use of their data, including control, access, and interpretation⁵¹
• participatory approaches to AI development between equity-deserving groups and AI developers to establish control of data at the beginning of the AI loop.¹⁵⁸ 

Environmental Costs

Climate change is causing floods, droughts, and heat waves that affect the Earth’s biodiversity as well as human health — and often the most vulnerable people are the most affected.¹⁵⁹ Although AI can provide potential benefits in the health care sector, it can also affect the environment negatively in several ways:
 

• AI systems use high amounts of energy because the data centres they require run on electricity. These centres also need water for cooling, a process that is increasingly difficult to maintain given rising worldwide temperatures.⁶ This infrastructure contributes to carbon emissions as algorithms train and become more advanced.^6,160 As of October 2024, there were 239 data centres in Canada and the number was increasing.¹⁶¹ In Ontario, electricity demand is projected to increase 75% by 2050 because of electric vehicle manufacturing and AI data centres; by 2035, 16 large data centres are projected to be in service.¹⁶²
• Rare earth-derived metals are being used to develop the hardware for AI (e.g., fuel cells, insulation, capacitors), its energy sources (i.e., batteries, devices), and its miniaturization for portability.⁶ Extraction of these metals harms the environment and humans; the process is dangerous, results in toxic waste, and is not sustainable because most of the materials are not recycled.⁶
• The physical devices that run AI do not last forever and the eventual electronic waste is often discarded in resource-poor settings and landfills or is incinerated in pits. These sources of pollution expose animals and humans to toxic waste.⁶
   

Workshop participants expressed that the environment should be top of mind so that the health care system does not contribute further harm to the planet, noting that this issue was related to AI Issue 2: Liability and Accountability. They highlighted there were not enough discussions about the environmental costs of AI, which can sometimes be an afterthought, and noted the need for environmental advocacy. Other participants acknowledged that AI could potentially reduce environmental costs in some cases (e.g., helping to optimize energy use). Participants were worried about devastating forest fires in Canada in 2023 and flooding in Europe in 2024 due to climate change. They were also concerned about the private sector’s role in purchasing nuclear reactors to fund AI; for example, Microsoft, Google, and Amazon have recently made nuclear energy deals to meet the high demands of AI use (e.g., data centres with routers, severs, cooling devices).¹⁶³ If AI in health care is here to stay, developers and the health care system need to work together to balance the expected benefits to patients with the potential harms to the environment to ensure AI technologies are sustainable.

Solutions can include:¹⁶⁰
 

• building energy-efficient models
• establishing green computing systems
• integrating routine life cycle assessments for appropriate ecodesign
• optimizing how data are managed and stored
• implementing workflows that reduce waste and energy consumption
• using low-carbon energy solutions
• practising sustainability with regards to AI hardware and the disposal of electronic waste
• increasing awareness among the health care sector, HCPs, AI developers, policy-makers, patients, researchers, and sustainability experts.