Just Say No to Paper Back-ups

Scientific rationale for optimizing electronic clinical outcome assessments in clinical trials

Capturing clinical trial assessments electronically provides higher quality and more reliable and accurate datasets than capturing this data on paper. Recognition of such improvement in data quality has resulted in an increased adoption of electronic clinical outcome assessments (eCOA).

The benefits of capturing high quality clinical trial data via eCOA has been widely recognized for more than a decade. Researchers have documented significant improvements in patient protocol compliance and data quality, a reduction of missing data and data ‘noise,’ and most importantly, increased study power with fewer patients.¹

However, some study teams who are new to eCOA occasionally raise questions about whether to use paper forms as a back-up in case of electronic data collection failure. As a result, some study teams request that provisions for a paper back-up system be in place.  

In this article, evidence will be presented illustrating that paper back-up provisions are unnecessary and do not provide the desired benefits. Paper back-ups produce little to no value and can call into question the overall quality of the clinical trial data being captured. In cases where a back-up solution is essential, alternative electronic approaches are recommended.

Electronic Device Failure? Not Likely.

Electronic device failure or device loss occurs very infrequently – historically, less than one percent of the time.² And, technology advancements are expected to further reduce this occurrence.³ Most provisioned devices are theft-adverse, meaning they are electronically “locked down” and have no other function beyond the collection of trial data. Thus, the probability of needing any type of back-up solution is very low.

In the event that an electronic device fails, most providers have multiple back-up electronic systems in place. They also have recommendations for minimizing any statistical impact, including:

• Provisioning for a back-up electronic device at high-enrollment sites

• Storing additional devices at depot locations in countries with long customs import lead times to enable speedy device replacement

• Overnight shipping of replacement devices to most locations.  

In most cases, the speed by which any device failure can be remedied with a back-up device provides an effective strategy for maintaining the consistent electronic capture of high-quality data in clinical trials.

Instances of missing data that cannot be readily addressed by a replacement electronic back-up device solution comprise a small fraction of the less than 1 percent of the total number of these cases of device failure or loss. In the case that every data point is deemed to be critical to determinations of efficacy and/or patient safety, such as primary endpoint in a Phase III study for a PRO-based labeling claim in a trial with a small patient sample, the recommended back-up solution beyond replacement devices is to have a web-based or interactive-voice response (IVR)-based electronic system in place. Using an alternate electronic data capture method will maintain high data quality should technology failures occur. Additionally, missing data can be recovered through imputation analysis. Clinical trial protocols commonly contain a statistical plan for handling missing data and what imputation analysis will be used to cover these rare potential occasions.  

FDA Guidenace on Quality Issues with Paper Collection

The quality of data collected on paper is markedly inferior to electronic data capture. Capturing clinical trial data on paper is not recommended by regulators and should be considered undesirable. Given the very limited instances in which device failure or loss occurs, the speed at which such events can be resolved by one or more of the above referenced electronic solutions, and the significantly reduced likelihood of the data collected on paper being accurate, the value of including paper as a back-up provision for any study is very limited.

The standard for the quality of data collected that is suitable for regulatory submissions have been defined by the FDA to include the following five criteria: attributable, legible, contemporaneous, original and accurate (abbreviated by the acronym ALCOA).⁴ Data collected electronically maps to these criteria, whereas data collected via paper does not.

Electronic data capture additionally provides a modality that fulfills several other criteria outlined in the FDA PRO Guidance⁵, including prescribed questionnaire order, required responses to items, limiting missing data, and verifiable time and date stamps. One of the most important aspects addressed via eCOA is real-time data collection that follows clinical trial design. The FDA PRO Guidance is strong in asserting that data must be collected in a contemporaneous manner:

“If a patient diary or some other form of unsupervised data entry is used, we plan to review the clinical trial protocol to determine what steps are taken to ensure that patients make entries according to the clinical trial design and not, for example, just before a clinic visit when their reports will be collected.”

Hidden (and Often Undisclosed) Cost of a Paper Back-up System

At a glance, paper back-ups may seem like an easy and cost-effective solution, but this notion is incorrect. In addition to being ineffective, paper back-ups represent a costly and time-consuming approach for little to no actualized gain ― and the potential for negative impact on trial outcomes. Further, additional overhead is required for a paper back-up provision in a protocol:

• Protocol amendments and planning: Use or inclusion of paper data collection modality must be outlined in and/or amended to a clinical study protocol.  

• Licensing: Licensing for paper forms must be put in place for standardized assessments.

• Translations: Additional costs may be incurred for translation of paper versions.

• Proof of equivalence: If data will be submitted to regulatory authorities for PRO-based labeling claims, evidence of the equivalence of the paper to the electronic version must be provided. This will most commonly include a cognitive debrief between the two modes.

• Data integration: The plan for collecting forms, data verification and cleaning, and entry into the electronic system must be documented for each study.

• Red zones: Collection of any type of VAS scale, e.g., patient and physician global impressions of change and other symptom ratings, is highly suspect to artifact on paper. Drawing a line exactly perpendicular to the page and ensuring that all copy machines print a VAS scale to the same resolution is essentially impossible to implement ― even for a short duration. Similarly, flagging symptoms such as pain or bleeding on a printed body diagram is prone to error.  

• Loss of paper back-up data: It is likely that at least 50 percent of the data captured on the paper back-ups will be unusable due to missing data, illegible or illogical entries, multiple items checked for a single selection question, subjects writing additional information on the form, and missing dates and times.

In addition to these overhead costs, there is a likely tendency for paper forms to be repeatedly used for more than a paper back-up once it is introduced. Once paper forms enter a study ― even if for a legitimate instance ― there is a recognized temptation for site personnel to subsequently use paper instead of an electronic device. There is no way to disengage paper once it enters a study; sufficient amounts of paper can cloud any potential actual signal.

Conclusion

The quest to avoid missing data should focus on the evidence at hand, which is that eCOA devices fail, are damaged, or are lost less than 1 percent of the time. Having a back-up device strategy in place reduces this to a fraction of that one percent, so an electronic back-up system is recommended. Adding a paper back-up provision does not ensure that any fraction will be fully captured; at most half of the 1 percent are viable. Paper back-up provisions represent a strategy of diminishing returns and should be considered unnecessary.


References:

1. Byrom B; Tiplady B. ePRO: electronic solutions for patient-reported data. 2010. Routledge. 297 pages. Print.
2. Data on File, ERT
3. National Academy of Sciences. 1988. Globalization of Technology: International Perspectives. Proceedings of the Sixth Convocation of The Council of Academies of Engineering and Technological Sciences. Book. Janet H. Muroyama and H. Guyford, Editors. ISBN: 0-309-49578-9, 224 pages.
4. Food and Drug Administration, FDA. 2010. Guidance for Industry: Electronic Source Documentation in Clinical Investigations.
5. Food and Drug Administration, FDA. 2009. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.