Analysis of Clinician and Patient Factors and Completion of Telemedicine Appointments Using Video

Key Points

Question  Which patient and clinician factors are associated with a successful or failed video visit?

Findings  This quality improvement study of 137 846 video visits showed an overall 90% success rate. Patient rather than clinician factors were more systematically associated with successful completion of video visits, and clinician comfort with technology was associated with successful video visits or conversion to telephone visits.

Meaning  The findings suggest that, as policy makers consider expanding telehealth coverage and hospital systems focus on investments, consideration of patient support, equity, and friction should be kept in the forefront.

Importance  Telemedicine provides patients access to episodic and longitudinal care. Policy discussions surrounding future support for telemedicine require an understanding of factors associated with successful video visits.

Objective  To assess patient and clinician factors associated with successful and with failed video visits.

Design, Setting, and Participants  This was a quality improvement study of 137 846 scheduled video visits at a single academic health system in southeastern Wisconsin between March 1 and December 31, 2020, supplemented with patient experience survey data. Patient information was gathered using demographic information abstracted from the electronic health record and linked with block-level socioeconomic data from the US Census Bureau. Data on perceived clinician experience with technology was obtained using the survey.

Main Outcomes and Measures  The primary outcome of interest was the successful completion of a scheduled video visit or the conversion of the video visit to a telephone-based service. Visit types and administrative data were used to categorize visits. Mixed-effects modeling with pseudo R² values was performed to compare the relative associations of patient and clinician factors with video visit failures.

Results  In total, 75 947 patients and 1155 clinicians participated in 137 846 scheduled video encounters, 17 190 patients (23%) were 65 years or older, and 61 223 (81%) patients were of White race and ethnicity. Of the scheduled video encounters, 123 473 (90%) were successful, and 14 373 (10%) were converted to telephone services. A total of 16 776 patients (22%) completed a patient experience survey. Lower clinician comfort with technology (odds ratio [OR], 0.15; 95% CI, 0.08-0.28), advanced patient age (66-80 years: OR, 0.28; 95% CI, 0.26-0.30), lower patient socioeconomic status (including low high-speed internet availability) (OR, 0.85; 95% CI, 0.77-0.92), and patient racial and ethnic minority group status (Black or African American: OR, 0.75; 95% CI, 0.69-0.81) were associated with conversion to telephone visits. Patient characteristics accounted for systematic components for success; marginal pseudo R² values decreased from 23% (95% CI, 21.1%-26.1%) to 7.8% (95% CI, 6.3%-9.4%) with exclusion of patient factors.

Conclusions and Relevance  As policy makers consider expanding telehealth coverage and hospital systems focus on investments, consideration of patient support, equity, and friction should guide decisions. In particular, this quality improvement study suggests that underserved patients may become disproportionately vulnerable by cuts in coverage for telephone-based services.

Although telemedicine has been practiced for decades, its necessity reached new heights during the COVID-19 pandemic. Prior to this pandemic, direct-to-consumer telemedicine use was driven by a relatively small segment of the health care consumer population.^1-3 However, during the COVID-19 pandemic, the rapid expansion of telemedicine in 2020 was facilitated by legislative and executive changes during the public health emergency that removed the “originating site” provision and increased restriction of in-person visits. With shortages of personal protective equipment and uncertainty regarding viral transmission, particularly with high viral transmission levels in several regions in the United States, health care shifted to virtual visits at a quick pace.⁴ Health care organizations rapidly deployed and scaled up virtual visits in efforts to adjust access to care, which meant that a majority of patients were experiencing telemedicine and virtual health care for the first time.^5-7

The rapid transition to virtual health care raised important questions concerning equity of access and the design of health care experiences⁸ and introduced an array of new technical challenges for stakeholders, including patients, clinicians, and health care systems. Telemedicine requires access to a smartphone or a computer, navigation of several screens to access the visit, and a stable internet connection that allows for fluent conversation and examination. Patients and physicians reported difficulties with platforms, accessing visits, and quality connections.⁶ These friction points are more likely to challenge less socioeconomically advantaged patients, which may contribute to health disparities.⁹ Understanding and exploring the difficulties that arose during the public health emergency is vital to address ongoing accessibility problems and provide adequate support for a sustainable medium of health care delivery in the future.

As governments, payers, and systems plan for telemedicine at the culmination of the public health emergency, an understanding of the limitations of telemedicine in reaching patients is critical to guide decisions and investments.^10,11 Prior studies have reported that patients at a relative socioeconomic disadvantage are less likely to schedule video visits as opposed to audio-only visits.^12-14 Using hierarchical regression analysis techniques, Rodriguez et al¹² also identified that much of the variability in choosing the modality for telemedicine rested with clinicians rather than with patients. In the present study, we sought to identify factors that were associated with the inability to complete a video visit at both the patient and clinician levels to better understand where variability resided.

This quality improvement study was conducted between March 1 and December 31, 2020, at Froedtert & the Medical College of Wisconsin Health Network, an academic health system based in southeastern Wisconsin. All patients scheduled for a video-based visit during this period were included across primary and specialty care. The health system concurrently used 2 video visit platforms during this period. The first was integrated into the electronic health record (EHR) and patient portal. The second was a nonintegrated solution that delivered a link to patients through a text message to join the visit through a web-based video-conferencing platform. Visits that were not successful using the integrated solution could use the nonintegrated solution as a backup. Clinics generally chose a preferred option that was the default mode for all visits in that department. Reporting of this study followed the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guideline.¹⁵ The institutional review board of the Medical College of Wisconsin approved the study and granted approval of a waiver of the Health Insurance Portability and Accountability Act authorization requirements at 45 CFR 164. No one received compensation or was offered any incentive for participating in this study.

Video visits were considered a success if the visit service was completed. Video visits were considered a failure if they resulted in a change to a telephone-based service. Both EHR and billing data were used to determine the video visit outcome. For the purposes of this analysis, patient cancellations (other than to reschedule as a same-day telephone service) and no-show visits were excluded.

We obtained information about patients through the EHR, including age at visit, race and ethnicity, and their insurance or payer. Patient median income was estimated from the US Census Bureau.¹⁶ High-speed internet availability was determined through Federal Communications Commission data linked to block-level data.¹⁷ Clinician information was obtained from the clinician’s record within the EHR and included physician specialty and the number of prior successful video visits.

Patient experience data were available for a subset of patients who responded to after-visit surveys. The survey data included patient ratings of how well the visit met their needs, their perception of how comfortable the clinician was with technology, and information about the type of device used to access the visit.

Descriptive statistics were summarized to characterize patients, clinicians, and visits. The weekly number of appointments by platform and the video visit success rates (the number of successful video visits across the total number of visits during the week) were depicted to show the video visit success rates with time. We applied logistic, multiple mixed-effects regressions for visit outcome (success vs failure) to 2 samples, including only the first visit for each patient (analysis 1) and only visits from patients with more than 1 visit (analysis 2). We used a mixed-effects model to account for the correlated data among visits by the same clinician and patient. In the mixed-effects models, potential confounding factors associated with successful video visits included patient characteristics (eg, age, race and ethnicity, medical insurance, geocode-based median income, comorbidity, the device used for the visit, and the number of prior successful video visits) and clinician factors, including title and specialty, perceived comfort with technology (as assessed by patient survey), and the number of previously successful video visits. We also considered clinical location as a potential factor.

We first selected statistically significant factors from univariate analysis and developed the final model using composite criteria, including stepwise variable selection based on Akaike information criteria, residuals, and reliable standard error estimates.^18-20 Bernstein polynomials and spline curves were used to approximate the log odds of video visit success for the number of prior video visit successes to allow for the detection of a nonlinear association with prior successful video visits without specifying cutoff points. The final model included random intercept effects for clinicians (analysis 1) and random intercept effects for clinicians and patients (analysis 2).

To address whether patient or clinician factors were relatively more strongly associated with the successful video visit, we compared the final model given in the eTable in the Supplement with reduced models, which included only patient factors and clinician factors from the final model produced in analysis 2, respectively, in terms of pseudo R² values.^18,21 The pseudo R² value is the proportion of the variance explained by the fixed effects only (marginal) and mixed effects (conditional) over the total variance consisting of the 3 variance components of fixed effects, random effects, and error. The pseudo R² values range from 0 to 1, similar to the ordinal R² value. A decrease in the pseudo R² value after removing patient or clinician factors indicates that those respective factors may be associated with the variation in successful video visits in the final model. We used 500 bootstrap resamples with replacement to calculate 95% CIs for the pseudo R² value.

All analyses were conducted using R, version 4.0.3 (R Project for Statistical Computing). A 2-sided value of P < .05 was considered statistically significant.

In total, 75 947 unique patients were included in the analysis (Table 1). Of the included patients, 17 190 (23%) were 65 years or older, 10 272 (14%) were African American or Black, 233 (<1%) were Alaska Native or American Indian, 1540 (2%) were Asian, and 61 223 (81%) were White. A total of 29 588 patients (39%) were Medicare or Medicaid beneficiaries, and 43 357 patients (57%) were under managed care programs. Among all included patients, 16 776 (22%) with completed visits responded to a patient experience survey.

Clinicians (n = 1155) represented a broad array of specialties, including 240 (21%) in primary care areas and 340 (29%) in internal medicine subspecialties (Table 2). As judged by their patients through survey responses, a majority of clinicians were assessed to have a high degree of comfort with technology (801 of 930 respondents [86%] strongly agreed).

We analyzed 137 846 billable scheduled video visits from March through December 2020 (Table 3). Of them, 123 473 (90%) were successful, and 14 373 (10%) were converted to telephone services. The clusters of clinicians and patients were not nested but partially overlapped. The median cluster size of clinicians (the number of all types of visits per clinician) was 50 (IQR, 11-152). Nearly 60% of the visits were the patient’s only visit within the study period. Among patients with more than 1 visit, the median number of visits was 3 (IQR, 2-5). Video visits were split between both the EHR or portal integrated solution (77 073 visits; 62%) and the nonintegrated solution (46 400 visits; 38%) (Figure).

Because analyses 1 and 2 yielded similar results, we reported analysis 1 (Table 4). Conditional on scheduling a video visit, multivariate modeling indicated that patient income ($75 001-$213 000: odds ratio [OR], 1.18; 95% CI, 1.06-1.30) and patient use of a tablet or a laptop (OR, 1.41; 95% CI, 1.14-1.74) were associated with successful visits, whereas lower clinician comfort with technology (OR, 0.15; 95% CI, 0.08-0.28), advanced patient age (66-80 years: OR, 0.28; 95% CI, 0.26-0.30), racial and ethnic minority status of patients (Black or African American: OR, 0.75; 95% CI, 0.69-0.81), and using the nonintegrated video visit solution (OR, 0.57; 95% CI, 0.53-0.61) were associated with conversion to telephone visits (Table 4). We found that the learning curves in log odds for a successful video visit were similar for both patients and clinicians (eFigures 1 and 2 in the Supplement).

We compared the final model with reduced models to determine whether patient or clinician factors were most strongly associated with the findings. The marginal pseudo R² values were 23.3% (95% CI, 21.1%-26.1%), and the conditional pseudo R² values were 42.3% (95% CI, 40.0%-44.5%). Those values decreased to 7.8% (95% CI, 6.3%-9.4%) for the marginal pseudo R² values and to 29.4% (95% CI, 27.3%-31.3%) for the conditional pseudo R² values when excluding patient factors in the model. Whereas the marginal R² value did not change (R² = 22.6%; 95% CI, 20.4%-25.4%), the conditional R² value decreased to 26.4% (95% CI, 24.0%-29.0%) when excluding clinician factors. That elimination of patient factors significantly decreased the marginal pseudo R² value, indicating that the patient factors were dominantly and systematically associated with video visit outcomes. Although the elimination of clinician factors decreased the conditional pseudo R² value compared with the elimination of patient factors, it was not statistically significant.

Examining the roles of demographic characteristics, patient and clinician associated factors, and technology learning curves provides insight into the successes and failures of telemedicine. Clinicians were associated with some variability as a part of the equation, especially those working remotely, with poor network or with Wi-Fi network dropped connections, or those learning how to manage new equipment and workflows. However, this study showed that most of the variability in successful or failed video visits was associated with patient characteristics vs clinician characteristics, particularly regarding sociodemographic characteristics and age.

Sociodemographic characteristics, including internet connectivity, technology literacy, educational level, and technology support, are critical to the success of a video visit. Internet connection with sufficient bandwidth to facilitate a video visit is often a hurdle for various populations.²² One-fourth of rural households do not have access to broadband internet; the digital divide is also present in urban communities, emphasizing the necessity of more inclusive internet access.⁹ Video communication yields higher patient understanding and satisfaction compared with only telephone communication.²³

In addition to differences by sociodemographic status, we observed differences by age, with patients older than 65 years being more likely to convert to a telephone visit. Older individuals may face more technology barriers, may have visual or movement disorders that make computing more difficult (especially on smaller devices), or may simply be more casual users of the internet.²⁴ Despite those assumptions, individuals who are older likely have a higher need for virtual care associated with transportation challenges to and from appointments or other impairments or chronic ailments that make leaving the house difficult.²⁵ In addition, telemedicine services may enable family members or caregivers, particularly those at a distance, to meaningfully participate in visits and to be a part of health care decision-making.

As patients and clinicians in the study population became more comfortable with technology, distinct learning curves were found in both user categories. The existence of a learning curve suggests that there are modifiable telemedicine program components, such as technical support or training, that may reduce video visit failures. Previous studies have shown that effective clinician training in telemedicine increases clinician confidence not only in using medical technology but in educating patients in how to have a successful video visit.²⁶ The patients’ learning curve was notably longer to achieve a successful video visit. This outcome may result from the irregular cadence of virtual care visits that does not allow for persistent learning or repetition, whereas clinicians perform multiple video visits on a single day. Future patient and clinician education on best virtual visit practices and patient accessibility to the internet or technology may help smooth experiences and shorten learning curves.

With the rapid implementation of virtual telemedicine during the COVID-19 pandemic, it can be safely assumed that several traditionally in-person medical encounters will be virtual for the foreseeable future.²⁷ As such, examining factors that are associated with the successes and failures of video visits provides insights into avenues to alleviate technical and user issues. Owing to connectivity problems, video visits are not always viable, especially for specific populations, such as those with limited digital access and literacy; this issue may worsen health inequity. A future focus for policy makers should consider inclusion of telephonic services as a form of reimbursable telemedicine. Permanent expansion of low-cost or free broadband internet for at-risk populations is also critical. For health care systems, it will be imperative to improve the ease of use of telemedicine as well as to provide support for patients to access such services.

Although this study provided insights into video visit failures, limitations exist. Data were collected between March 1 and December 31, 2020, at the height of the COVID-19 pandemic. As such, further studies should focus on longer periods of time. The population examined was limited to patients residing in the Midwest and receiving care at a single institution, although it was an institution covering academic and community settings as well as urban and rural patients; future studies should assess telemedicine failures throughout the United States. We included data for a subset of patients completing a patient experience survey; respondents were more likely to be older and less socioeconomically disadvantaged, and response bias may be present. When we removed these variables from the survey, the associations for the remaining variables were similar, suggesting that overall results were not biased. Different technology approaches may yield different results (eg, different modalities), although our approach incorporated both an EHR-integrated and stand-alone option for clinics to use. In addition, video visit failures were determined by EHR and visit coding data—the limitation of this method could underestimate the number of true issues experienced. In late September 2020, the technology used for video visits was upgraded and continued to progress over time. In addition, clinicians who had a failed video visit scheduled with the integrated solution had a second chance to conduct the visit through the nonintegrated strategy, and this practice would not be reclassified in the EHR; thus, the finding that the integrated solution was superior to the stand-alone solution may be misleading. This practice may explain some of the higher success rates with visits scheduled using the integrated platform, although issues such as network connectivity are unlikely to be materially better with another platform.

This quality improvement study found that patient factors, including sociodemographic characteristics and age, were dominantly and systematically associated with the success or failure of a video visit. As policy makers debate expanding telehealth coverage and hospital systems focus on investments, the consideration of patient support, equity, and friction should be kept in the forefront to ensure that underserved patients are not left to fall further behind. Underserved patients may become disproportionately challenged by decreases or cutbacks surrounding insurance coverage or reimbursement for telephone-based services, threatening to worsen health care disparities. Coverage of telephonic services may improve accessibility and equity across the age and ability spectrum.^12,28,29 A broader understanding of the variability, associated factors, and learning curves for telemedicine may help guide the next phase of optimization and refinement of telemedicine as a primary medium for health care.

Accepted for Publication: September 5, 2021.

Published: November 4, 2021. doi:10.1001/jamanetworkopen.2021.32917

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Crotty BH et al. JAMA Network Open.

Corresponding Author: Bradley H. Crotty, MD, MPH, Collaborative for Healthcare Delivery Science, Center for Advancing Population Science, Medical College of Wisconsin, 8701 Watertown Plank Rd, Milwaukee, WI 53226 (bcrotty@mcw.edu).

Author Contributions: Drs Crotty and Somai had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Crotty, Dong, Decker, Mortensen, Holt, Winn, Laud, Somai.

Acquisition, analysis, or interpretation of data: Crotty, Hyun, Polovneff, Dong, Mortensen, Holt, Winn, Laud, Somai.

Drafting of the manuscript: Crotty, Hyun, Polovneff, Mortensen, Holt, Laud, Somai.

Critical revision of the manuscript for important intellectual content: Crotty, Polovneff, Dong, Decker, Mortensen, Holt, Winn, Laud, Somai.

Statistical analysis: Hyun, Polovneff, Dong, Mortensen, Winn, Laud, Somai.

Administrative, technical, or material support: Winn, Somai.

Supervision: Crotty, Decker, Somai.

Conflict of Interest Disclosures: Dr Crotty reported being an advisor for Buoy Health outside the submitted work. Dr Winn reported receiving personal fees from Takeda outside the submitted work. Dr Laud reported receiving grants from the National Institutes of Health and from the US Department of Defense, US Department of the Navy as part of a subcontract from the National Marrow Donor Program outside the submitted work. No other disclosures were reported.