Evaluations of Virtual Care

This Environmental Scan aims to:

• identify methods, standards, and guidelines being used in Canada and other countries for evaluating virtual care (with a focus on economic evaluation and outcomes-based evaluation), specifically regarding interactions between health care providers and patients.
• identify Canadian and international evaluations of virtual care that address interactions between health care providers and patients, and the economic aspect of virtual care.
• summarize the methodologies and other relevant information (e.g., inputs, outcomes, measures of value, considerations, lessons learned) from the identified evaluation methods, standards, guidelines, and completed evaluations.

For the purpose of this work, the focus is on interactions between physicians/health teams and patients, either synchronously or asynchronously. Other digital health interventions or devices (e.g., remote monitoring) are beyond the scope of this scan.

Key Messages

• A literature search informed this Environmental Scan and identified 11 evaluations of virtual care in primary care health settings and 7 publications alluding to methods, standards, and guidelines (referred to as evaluation guidance documents in this report) being used in various countries to evaluate virtual care in primary care health settings. The majority of included literature was from Australia, the US, and the UK, with 2 evaluation guidance documents published by the Heart and Stroke Foundation of Canada.
• Evaluation guidance documents recommended using measurements that assess the effectiveness and quality of clinical care including safety outcomes, time and travel, financial and operational impact, participation, health care utilization, technology experience including feasibility, user satisfaction, and barriers and facilitators or measures of health equity.
• Evaluation guidance documents specified that the following key decisions and considerations should be integrated into the planning of a virtual care evaluation: refining the scope of virtual care services; selecting an appropriate meaningful comparator; and identifying opportune timing and duration for the evaluation to ensure the evaluation is reflective of real-world practice, allows for adequate measurement of outcomes, and is comprehensive, timely, feasible, non-complex, and non–resource-intensive.
• Evaluation guidance documents highlighted that evaluations should be systematic, performed regularly, and reflect the stage of virtual care implementation to encompass the specific considerations associated with each stage. Additionally, evaluations should assess individual virtual care sessions and the virtual care program as a whole.
• Regarding economic components of virtual care evaluations, the evaluation guidance documents noted that costs or savings are not limited to monetary or financial measures but can also be represented with time. Cost analyses such as cost-benefit and cost-utility estimates should be performed with a specific emphasis on selecting an appropriate perspective (e.g., patient or provider), as that influences the benefits, effects, and how the outcome is interpreted.
• Two identified evaluations assessed economic outcomes through cost analyses in the perspective of the patient and provider. Evidence suggests that, in some circumstances, virtual care may be more cost-effective and reduces the cost per episode and patient expenses (e.g., travel and parking costs) compared to in-person care. However, virtual care may increase the number of individuals treated, which would increase overall health care spending.
• Four identified evaluations assessed health care utilization. The evidence suggests that virtual care reduces the duration of appointments and may be more time-efficient compared to in-person care. However, it is unclear if virtual care reduces the use of medical resources and the need for follow-up appointments, hospital admissions, and emergency department visits compared to in-person care.
• Five identified evaluations assessed participation outcomes. Evidence was variable, with some evidence reporting that virtual care reduced attendance (e.g., reduced attendance rates) and other evidence noting improved attendance (e.g., increased completion rate and decreased cancellations and no-show rates) compared to in-person care.
• Three identified evaluations assessed clinical outcomes in various health contexts. Some evidence suggested that virtual care improves clinical outcomes (e.g., in primary care with integrated mental health services, symptom severity decreased) or has a similar effect on clinical outcomes compared to in-person care (e.g., use of virtual care in depression elicited similar results with in-person care).
• Three identified evaluations assessed the appropriateness of prescribing. Some studies suggested that virtual care improves appropriateness by increasing guideline-based or guideline-concordant antibiotic management, or elicits no difference with in-person care.