 |
|
Costs and Outcomes of PAPNET Secondary Screening Technology for Cervical Cytologic Evaluation
A Community Hospital's Experience
Gregory L. Brotzman, MD;
Susan Kretzchmar, MD;
David Ferguson, MD;
Mark Gottlieb, PhD;
Colleen Stowe, CT(ASCP)
Arch Fam Med. 1999;8:52-55.
ABSTRACT
| |
Objective To determine the effectiveness of and costs associated with semiautomated rescreening of Papanicolaou smears with negative findings at a community hospital.
Design A prospective study of 1200 Papanicolaou smear slides with negative findings using the PAPNET screening system (Neuromedical Systems, Incorporated, Suffern, NY).
Setting Community hospital laboratory.
Patients Patients with negative findings on Papanicolaou smears who agreed to have their smears reviewed using PAPNET.
Interventions None.
Main Outcome Measures Results of rescreening and resources involved in processing the PAPNET review.
Results Screening with PAPNET identified 8 patients with atypical squamous cells of undetermined significance (ASCUS) that were not diagnosed on initial screening, yielding a false-negative rate in our laboratory of 0.7% for ASCUS. No low- or high-grade squamous intraepithelial lesions were identified. Based on our laboratory processing 6000 Papanicolaou smears a year, at $19 per slide, it would cost our laboratory $102,600 for PAPNET review of all smears with negative findings. In contrast, the estimated cost to have another cytotechnologist review all such smears manually would cost $11,977. The rate of changed diagnoses in the PAPNET group was similar to the rate in our standard rescreening of 10% of all smears with negative findings. Mean turnaround time for a PAPNET screen was 13.9 days, compared with 3.9 days for manual review.
Conclusions For a laboratory with a low percentage of smears with abnormal findings, a quality cytotechnologist and pathologist, and required quality assurance standards in place, PAPNET may not improve appreciably the pick-up rate for missed cervical lesions, and may add significantly to the cost and turnaround time of cytologic evaluation of cervical smears.
INTRODUCTION
A PAPANICOLAOU smear of the uterine cervix is a commonly performed procedure. Each year, results of millions of Papanicolaou smears are interpreted by cytotechnologists across the country.1 Although a pathologist reviews all slides with abnormal results, they do not usually review the slides with the results interpreted as negative by the cytotechnologist. To address the concern of quality assurance with cervical cytologic screening, Clinical Laboratory Improvement Amendments (CLIA) regulations require 10% of Papanicolaou smears with negative findings to be rescreened by a pathologist or a senior cytotechnologist.2 This still leaves 90% of these smears that are at risk for a missed diagnosis of squamous intraepithelial lesions (SIL) or carcinoma. Due to screening or interpretation error, it is expected that occasional lesions will be missed, with the literature showing wide variations in the estimated incidence of false-negative rates ( 5%-25%).3-4 There are several factors that may interfere with proper screening and interpretation of smear results, such as obscuring inflammation, air-drying artifact, or blood from menstruation. In addition, failure of the clinician to obtain a representative sample may contribute to the chances of missing a lesion.3
PAPNET was developed by Neuromedical Systems, Incorporated, Suffern, NY, to address the issue of cervical cytologic screening and false-negative findings.5-10 PAPNET is a semiautomated cervical cytologic screening method that uses computer imaging and a neural-network type of artificial intelligence to review slides that previously have been interpreted as having negative findings. PAPNET identifies 128 of the most abnormal areas on each slide it reviews and records these on a data storage tape. These tapes are then reviewed by a cytotechnologist on a high-resolution color monitor. The results are interpreted by the cytotechnologist as negative or requiring review of the glass slide. On review of the glass slide, the findings are triaged as negative or as abnormal and requiring pathologist review. The pathologist reviews the slide and determines the final diagnosis.
PAPNET received US Food and Drug Administration approval in November 1995 as a secondary screening device for cervical cytologic smears, and it has been marketed heavily in medical and lay publications. PAPNET is not approved for primary screening or for known abnormal findings on smears. Most studies of the efficacy of PAPNET have been conducted in large academic centers or with selected populations of women known to be at higher risk. Studies show that PAPNET will detect 0.2% to 2.8% of previously undetected lesions, with most of these lesions being atypical squamous cells of undetermined significance (ASCUS).7-9
To evaluate PAPNET's usefulness in a community hospital laboratory, we conducted a prospective study of 1200 Papanicolaou smear slides with negative findings (1197 patients, 3 with 2-slide smears). We sought to determine what the false-negative rate of our cytotechnologist (C.S.) was in the PAPNET-reviewed cases, to assess the actual costs to the laboratory and patient associated with using this technology in a community hospital setting, and to determine if there is a greater yield using PAPNET technology in a community hospital setting compared with that of previous studies at large academic institutions.
A method to evaluate the quality of a laboratory is the false-negative fraction. The false-negative fraction is defined as the number of false-negative findings divided by the number of false-negative plus true-positive findings.11 The false-negative fraction at our laboratory is 6.0% (5%-10% indicates good performance).
MATERIALS AND METHODS
The cytotechnologist (C.S.) and 2 pathologists (S.K. and D.F.) from the laboratory of Columbia Hospital, Milwaukee, Wis, attended a 5-day PAPNET training session at Neuromedical Systems. The cytotechnologist performed all the PAPNET review during the study. Initial screening and manual review of 10% of slides with negative findings were performed in the usual fashion. The study protocol obtained approval from the institutional review board, and we accepted consecutive smears with negative findings from physicians and patients who consented to participate in the study at no charge to the clinic or patient. The population served by our hospital laboratory is diverse, with a relatively low rate of abnormal findings on smears (5% of all smears being ASCUS or worse). These 1200 smears were collected during a 6-month period and were sent to PAPNET for imaging in batches of approximately 60 smears. Records were kept of time spent by the cytotechnologist on clerical tasks associated with sending slides to PAPNET for imaging and of time spent in review of PAPNET data tapes and associated review of glass slides. The pathologists kept records of time spent reviewing PAPNET cases triaged for pathologist review. Preliminary reports of negative findings were sent out after the initial screen, including the following comment: "This case is pending PAPNET review." After PAPNET review was completed, a final report was sent, including documentation of PAPNET secondary screening results. All findings determined to be abnormal using PAPNET-assisted rescreening were correlated with subsequent results of biopsies and Papanicolaou smears. Since the false-negative fraction of the PAPNET cohort could not be determined (PAPNET does not accept smears known to have positive findings), we instead determined the false-negative rate for the cohort. False-negative rates were calculated by taking the total number of PAPNET-associated slides with additional abnormal findings divided by the total number of Papanicolaou smears.
RESULTS
Of the 1200 slides sent for PAPNET review, the cytotechnologist reviewed the data tapes and triaged 37 smears (3.1%) for review by the pathologist due to abnormal findings. The pathologist interpreted 8 results as ASCUS and 29 as negative. Low-grade SIL was favored in all of the results interpreted as ASCUS. None were interpreted as high-grade SIL. Two of the 8 cases of ASCUS were initially interpreted as abnormal findings by the cytotechnologist. On further pathologist review, the findings were interpreted as negative and sent to PAPNET. After PAPNET review, these findings were triaged as abnormal and requiring pathologist review. The second reviewing pathologist in both cases gave a final diagnosis of ASCUS. This yielded a cytologist and pathologist false-negative rate in our laboratory of 0.7% for ASCUS.
In our study, 8 of the 1200 smears with initially negative findings reviewed with PAPNET had a change in the diagnosis to ASCUS, favoring low-grade SIL. A summary of these cases is provided in Table 1. Data were available up to 13 months following the conclusion of this study, with all subsequent findings of follow-up Papanicolaou smears or biopsies being reviewed. Six of the 8 patients with ASCUS underwent follow-up studies. Of these 6 patients, 3 patients had negative results of repeated Papanicolaou smears, 2 patients had findings indicative of ASCUS, and 1 patient had findings indicative of low-grade SIL. Two of the 6 patients underwent cervical biopsy and/or endocervical curettage, with findings interpreted as negative, and 1 patient underwent an endocervical curettage with findings suggestive but not diagnostic of human papillomavirus. A subsequent endocervical curettage yielded negative results.
|
|
|
|
Comparison of Papanicolaou Smear Interpretations Before and After PAPNET Review*
|
|
|
Based on our sample experience, PAPNET review of 1000 manually screened, comparable samples with negative findings would yield from 2.9 to 13.1 additional cases of ASCUS (based on the exact 95% binomial confidence interval about the false-negative rate of 0.7%). The number of missed cases of SIL identified would be 3.1 or fewer (based on the 1-sided 97.5% binomial confidence interval associated with the 0 false-negative detection rate for SIL).
Clerical time required to log, prepare, and ship the 1200 slides for review by PAPNET was 20.3 hours. In addition, 37.8 hours were required by the cytotechnologist to review and interpret the PAPNET results, including review of glass slides. Approximately 35 minutes (0.6 hours) of pathologist time was required to review PAPNET cases.
At our institution, the middle range of salary plus benefits for a cytotechnologist is $22.18/h; for a laboratory aide, $11.72/h. In our study, based on the total time and labor expenditures, we calculated clerical and cytotechnologist costs at approximately $1.00 per case for reviewing PAPNET data. The current PAPNET charge to the laboratory is $18.00 per slide. This would result in $19.00 of extra charges for the usual single-slide Papanicolaou smear (this charge is to the laboratory, not the patient; the laboratory is responsible for recouping these costs when billing the patient). When the laboratory considers pricing of Papanicolaou smears, other factors must be considered in the bottom line, including variable reimbursement by third-party payers, capitated contracts, billing costs, costs of quality assurance activities, and nonpayment by some patients.
The screening rates of cytotechnologists vary widely.12 Using 10 slides per hour as an acceptable screening rate for an average cytotechnologist,2 review costs $2.22 per slide. This cost is similar to what has been recently reported.11 With 5400 smears with negative findings at a cost of $19 per slide, it would cost our laboratory $102,600/y for PAPNET review of all smears. In contrast, to have another cytotechnologist review all of these smears manually would cost $11,977/y. This would yield a marginal cost of $7832 to $35,379 per case of ASCUS detected with PAPNET.
The 1200 slides were shipped and processed with PAPNET in 20 batches collected in the hospital laboratory from March 11 to August 28, 1996. On average, each batch was collected during 7.8 days (SD, 2.5 days). The longest collection period was 12 days, whereas the shortest was 3 days. On average, each batch consisted of 60 slides (SD, 13 slides). The delay between the time that a batch of slides was completed for forwarding to PAPNET and the completion by the cytotechnologist reviewing the PAPNET data tapes was 6.2 days (SD, 2.1 days). The average time from the date of the specimen collection to final diagnosis of a PAPNET review during the study was 13.9 days (SD, 3.4 days). The average turnaround time for a non-PAPNET review in our laboratory during the study period was 3.9 days.
COMMENT
The Papanicolaou smear is a very effective screening tool for cervical cancer, but it has its limitations. Over the past decade, there has been increasing pressure to improve the quality of the Papanicolaou smear screening program in the United States. As a result of this pressure, CLIA 88 was implemented (Public Law 100-578). It included limits on the number of smears screened per day by cytotechnologists and minimum rescreening requirements of smears with negative findings; it prohibited payments per case screened by cytotechnologists and home screening.2 More recently, attention has turned to automated systems for rescreening of smears with negative findings, including PAPNET. Several studies have shown PAPNET to be effective as a secondary screening method.4-8,10
The clinical significance of a finding of ASCUS on a Papanicolaou smear is uncertain. Based on the very short-term follow-up of additional patients with ASCUS identified in our study, there did not appear to be much benefit to these patients, but there was the potential for increased interventions with repeated Papanicolaou smears, colposcopy with biopsy, and the associated costs and patient discomfort. Studies have quoted underlying rates of cervical intraepithelial neoplasia with ASCUS found on Papanicolaou smears as being variable (10%-40%).13
Using PAPNET technology, a negative diagnosis was changed to ASCUS in 0.7% of cases in our study sample. The usual procedure in our laboratory, as mandated by CLIA, is to rescreen at least 10% of smears with negative findings. In 1996, we rescreened on average 13% of these smears, including targeted (high-risk) and random rescreenings. Examples of targeted rescreening include patients with previous Papanicolaou smears with abnormal findings or previous treatment for cervical intraepithelial neoplasia. With this rescreen policy, 0.9% of initial diagnoses were changed in our laboratory. In our patient population, there is a low incidence of abnormal findings on smears. We interpret results of approximately 6000 smears a year. Of these 6000 smears, 90.6% have negative findings or show benign cellular changes; 6.6%, ASCUS; 2.1%, low-grade SIL; 0.6%, high-grade SIL; 0.1%, atypical glandular cells of undetermined significance; and 0.6%, unsatisfactory findings. As would be expected, given the low incidence of abnormalities, there were few changed diagnoses in the PAPNET study group. The percentage of changed diagnoses in the PAPNET group was similar to the percentage of changed diagnoses in our standard 10% rescreening group.
In our laboratory, the PAPNET study population was comparable to manual secondary screening for identification of missed lesions. The major impediment to 100% rescreening of Papanicolaou smears with negative findings has been the associated costs. In today's cost-conscious medical environment, a secondary screening method must be efficacious and cost-effective to be widely accepted. At its current price, PAPNET is not cost-effective compared with manual rescreening. We believe that, after using the system, if costs decrease, PAPNET would be a useful adjunct to quality assurance of cytopathological studies. PAPNET could provide a rapid means for a senior cytotechnologist to review the work of less experienced cytotechnologists. PAPNET may also enhance identification of rare high-grade cells that could be missed during primary review of smears by cytotechnologists. Finally, our cytotechnologist appreciated the security PAPNET provided as a check on her interpretations of Papanicolaou smears as having negative findings.
We limited analysis to costs to the laboratory and patient costs of Papanicolaou smears. It was beyond our scope to calculate additional costs outside the laboratory, such as repeated Papanicolaou smears, colposcopies, or treatment methods associated with false-negative and false-positive findings on Papanicolaou smears. The follow-up of the 8 patients with ASCUS was very short, and any statement as to the long-term consequences of identifying these individuals is uncertain. Another limitation was that we did not directly measure costs of having an additional cytotechnologist rescreen all of our smears with negative findings. Rescreening might lead to cytotechnologist overcall of abnormalities and result in additional cost or repeated smears, colposcopy, or other such evaluations. We did not include indirect costs to the patient, such as time lost from work, anxiety, or patient preferences for follow-up of the additional cases of ASCUS identified on PAPNET review of our slides with negative findings.
For a laboratory with a low percentage of abnormal findings on smears, a quality cytotechnologist and pathologist, and required quality assurance standards in place, PAPNET may not improve appreciably on the pick-up rate for missed cervical lesions, while adding substantially to the cost and turnaround time of cervical smears.
AUTHOR INFORMATION
Accepted for publication October 27, 1997.
This research was supported by a grant from the Scientific Advancement and Education Fund, Columbia Hospital, Milwaukee, Wis.
Reprints: Gregory L. Brotzman, MD, Columbia Family Care Center, 210 W Capitol Dr, Milwaukee, WI 53212 (e-mail: brotzman{at}mcw.edu).
From the Department of Family and Community Medicine, Medical College of Wisconsin (Drs Brotzman and Gottlieb), and the Department of Clinical Pathology (Drs Kretzchmar and Ferguson) and Cytotechnology Laboratory (Ms Stowe), Columbia Hospital, Milwaukee, Wis.
REFERENCES
| |
1. Kurman RJ, Henson DE, Herbst AL, Noller KL, Schiffman MH. Interim guidelines for management of abnormal cervical cytology: the 1992 National Cancer Institute Workshop. JAMA. 1994;271:1866-1869.
FREE FULL TEXT
2. Rules and regulations: cytology, 57 Federal Register 42-43(1992).
3. Koss L. The Papanicolaou test for cervical cancer detection: a triumph and a tragedy. JAMA. 1989;261:737-743.
FREE FULL TEXT
4. Sherman M, Kelly D. High-grade SILS and invasive carcinoma following the report of three negative Papanicolaou smears: screening failures or rapid progressions? Mod Pathol. 1992;5:337-342.
FULL TEXT
|
ISI
| PUBMED
5. Mango L. Computer-assisted cervical cancer screening using neural networks. Cancer Lett. 1994;77:155-162.
FULL TEXT
|
ISI
| PUBMED
6. Boon M, Kok L. Neural network processing can provide means to catch errors that slip through human screening of Pap smears. Diagn Cytopathol. 1993;9:411-416.
PUBMED
7. Ashfaq R, Liang Y, Saboorian M. Evaluation of PAPNET system for rescreening of negative cervical smears [published correction appears in Diagn Cytopathol. 1995;13:188]. Diagn Cytopathol. 1995;13:31-36.
FULL TEXT
| PUBMED
8. Slagel D, Zalenski S, Cohen M. Efficacy of automated cervical cytology screening. Diagn Cytopathol. 1995;12:26-30.
9. Buckner S, Tellado M, Stevens A, O'Leary T. PapNet-assisted rescreening of cervical smears: effectiveness and cost. Acta Cytol. 1997;41:1544. Abstract 6.
10. Mango L. Neuromedical Systems, Inc. Acta Cytol. 1996;40:53-59.
PUBMED
11. Kelsey JL, Whittemore AS, Evans AS, Thompson WD. Methods in Observational Epidemiology. 2nd ed. New York, NY: Oxford University Press; 1996:343.
12. Bishop J. The cost of production in cervical cytology: comparison of conventional and automated primary screening systems. Am J Clin Pathol. 1997;107:445-450.
PUBMED
13. Sedlacek T. Management guidelines for follow-up of atypical squamous cells of undetermined significance (ASCUS). Colposcopist. 1996;27:1-8.
RELATED ARTICLES
The Archives of Family Medicine Continuing Medical Education Program
Arch Fam Med. 1999;8(1):23-25.
FULL TEXT
Reducing Mortality Due to Cervical Cancer: PAPNET Fails the Test
Joy Melnikow and James Nuovo
Arch Fam Med. 1999;8(1):56-57.
FULL TEXT
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
Reducing Mortality Due to Cervical Cancer: PAPNET Fails the Test
Melnikow and Nuovo
Arch Fam Med 1999;8:56-57.
FULL TEXT
|
