Purpose To develop and evaluate domain-specific and pretrained bidirectional encoder representations from transformers (BERT) models in a transfer learning task on varying training dataset sizes to annotate a larger overall dataset. Materials and Methods The authors retrospectively reviewed 69 095 anonymized adult chest radiograph reports (reports dated April 2020-March 2021). From the overall cohort, 1004 reports were randomly selected and labeled for the presence or absence of each of the following devices: endotracheal tube (ETT), enterogastric tube (NGT, or Dobhoff tube), central venous catheter (CVC), and Swan-Ganz catheter (SGC). Pretrained transformer models (BERT, PubMedBERT, DistilBERT, RoBERTa, and DeBERTa) were trained, validated, and tested on 60%, 20%, and 20%, respectively, of these reports through fivefold cross-validation. Additional training involved varying dataset sizes with 5%, 10%, 15%, 20%, and 40% of the 1004 reports. The best-performing epochs were used to assess area under the receiver operating characteristic curve (AUC) and determine run time on the overall dataset. Results The highest average AUCs from fivefold cross-validation were 0.996 for ETT (RoBERTa), 0.994 for NGT (RoBERTa), 0.991 for CVC (PubMedBERT), and 0.98 for SGC (PubMedBERT). DeBERTa demonstrated the highest AUC for each support device trained on 5% of the training set. PubMedBERT showed a higher AUC with a decreasing training set size compared with BERT. Training and validation time was shortest for DistilBERT at 3 minutes 39 seconds on the annotated cohort. Conclusion Pretrained and domain-specific transformer models required small training datasets and short training times to create a highly accurate final model that expedites autonomous annotation of large datasets.Keywords: Informatics, Named Entity Recognition, Transfer Learning Supplemental material is available for this article. ©RSNA, 2022See also the commentary by Zech in this issue.
A standardized, holistic artificial intelligence (AI) curriculum is necessary to meet trainee demand and prepare them for effective AI tool use in their future practice.
To evaluate the inter-reader reproducibility and prognostic accuracy of the Liver Imaging Reporting and Data System (LI-RADS) treatment response algorithm (LR-TR) at the time of initial post-treatment evaluation following drug-eluting beads transarterial chemoembolization (DEB-TACE) for hepatocellular carcinoma (HCC). This retrospective study included patients with HCC who underwent first-line DEB-TACE between January 2011 and December 2015. Six readers (three fellowship-trained radiologists and three radiology trainees) independently assessed lesion-level response in up to two treated lesions per LR-TR and modified Response Evaluation Criteria in Solid Tumors (mRECIST)-target criteria, as well as patient-level response per mRECIST-overall criteria, on the initial post-treatment CT/MRI. Inter-reader agreement was calculated by Fleiss’ multi-reader κ. We tested whether LR-TR, mRECIST-target, and mRECIST-overall response were associated with overall survival using Kaplan–Meier and Cox proportional hazard model analyses. A total of 82 patients with 113 treated target lesions were included. Inter-reader agreement was moderate for LR-TR and mRECIST-overall (κ range 0.42–0.57), and substantial for mRECIST-target (κ range 0.62–0.66), among all three reader-groups: all readers, experienced readers, and less-experienced readers. LR-TR and mRECIST-target response were not significantly associated with overall survival regardless of reader experience (P > 0.05). In contrast, mRECIST-overall response was significantly associated with overall survival when assessed by all readers (P = 0.02) and experienced readers (P = 0.03), but not by the less-experienced readers (P = 0.35). Although LR-TR algorithm has moderate inter-reader reproducibility, it alone may not predict overall survival on the initial post-treatment CT/MRI after first-line DEB-TACE for HCC.
Pelvic floor disorders are common and can negatively impact quality of life. Imaging of patients with pelvic floor disorders has been extremely heterogeneous between institutions due in part to variations in clinical expectations, technical considerations, and radiologist experience. In order to assess variations in utilization and technique of pelvic floor imaging across practices, the society of abdominal radiology (SAR) disease-focused panel on pelvic floor dysfunction developed and administered an online survey to radiologists including the SAR membership. Results of the survey were compared with published recommendations for pelvic floor imaging to identify areas in need of further standardization. MRI was the most commonly reported imaging technique for pelvic floor imaging followed by fluoroscopic defecography. Ultrasound was only used by a small minority of responding radiologists. The survey responses demonstrated variability in imaging utilization, patient referral patterns, imaging protocols, patient education, and interpretation and reporting of pelvic floor imaging examinations. This survey highlighted inconsistencies in technique between institutions as well as potential gaps in knowledge that should be addressed to standardize evaluation of patients with pelvic floor dysfunction.
Introduction Liver segmentation and volumetry have traditionally been performed using computed tomography (CT) attenuation to discriminate liver from other tissues. In this project, we evaluated if spectral detector CT (SDCT) can improve liver segmentation over conventional CT on 2 segmentation methods. Materials and Methods In this Health Insurance Portability and Accountability Act–compliant institutional review board–approved retrospective study, 30 contrast-enhanced SDCT scans with healthy livers were selected. The first segmentation method is based on Gaussian mixture models of the SDCT data. The second method is a convolutional neural network–based technique called U-Net. Both methods were compared against equivalent algorithms, which used conventional CT attenuation, with hand segmentation as the reference standard. Agreement to the reference standard was assessed using Dice similarity coefficient. Results Dice similarity coefficients to the reference standard are 0.93 ± 0.02 for the Gaussian mixture model method and 0.90 ± 0.04 for the CNN-based method (all 2 methods applied on SDCT). These were significantly higher compared with equivalent algorithms applied on conventional CT, with Dice coefficients of 0.90 ± 0.06 (P = 0.007) and 0.86 ± 0.06 (P < 0.001), respectively. Conclusion On both liver segmentation methods tested, we demonstrated higher segmentation performance when the algorithms are applied on SDCT data compared with equivalent algorithms applied on conventional CT data.
To assess the frequency and indications for use of oral water-soluble contrast challenge as a diagnostic test for small bowel obstruction in four regions of the USA. We distributed a 9-question web-based survey to the abdominal section heads of academic radiology departments throughout the USA (N = 97). The questions pertained to use of water-soluble contrast for management of small bowel obstruction. Descriptive statistics and Fisher’s exact tests were used for data analysis. The overall response rate was 46%. Eighty percent of the responding hospitals had more than 500 beds in operation. Water-soluble contrast challenge was considered standard of care for management of non-operative small bowel obstruction in 60% of the responding radiology departments. The majority of the responding departments (41%) performed 2–8 contrast challenge studies per month on average. The most frequent indication for the study was distinguishing partial vs complete bowel obstruction. Eighty percent of the responding radiologists believed that the contrast challenge is useful for management of small bowel obstruction. Overall, there was no statistically significant difference in frequency and indication for use of water-soluble contrast challenge based on geographic location. The water-soluble contrast challenge was considered standard of care for non-operative management of small bowel obstruction in majority of the academic radiology departments represented in this survey. Surgeons were referring clinicians in every case. The most common clinical indication for the study was distinguishing partial versus complete small bowel obstruction.
AI applications have the potential to become an essential element of care provided by radiology departments, particularly when handling tedious or time-consuming tasks. Safety-net hospitals must carefully consider the need for and potential impact of these applications and seek value beyond merely meeting increased imaging volumes. Caseload prioritization and improvements to reporting times, patient safety, and communication are important objectives for AI applications. Assessing and implementing AI solutions by an enterprise governance team will help maintain focus and ensure alignment with the hospital's mission needs, financial realities, and strategic goals.
Hematuria evaluation remains a common problem, particularly in patients who smoke and are at risk for urothelial tumors. Lifetime surveillance of the urothelium is often required once urothelial cancer is diagnosed. Computed tomography urography (CTU) has exquisite sensitivity and specificity for identification of renal and urothelial lesions. The examination is well accepted by patients and physicians. Possible harms include radiation exposure and contrast-induced nephropathy. MR imaging is also an accurate test, but requires longer exam times, and may not demonstrate stones. We present the technical and interpretation skills required to use MR urography and CTU effectively.
Purpose To develop recommendations for magnetic resonance (MR) defecography technique based on consensus of expert radiologists on the disease-focused panel of the Society of Abdominal Radiology (SAR). Methods An extensive questionnaire was sent to a group of 20 experts from the disease-focused panel of the SAR. The questionnaire encompassed details of technique and MRI protocol used for evaluating pelvic floor disorders. 75% agreement on questionnaire responses was defined as consensus. Results The expert panel reached consensus for 70% of the items and provided the basis of these recommendations for MR defecography technique. There was unanimous agreement that patients should receive coaching and explanation of commands used during MR defecography, the rectum should be distended with contrast agent, and that sagittal T2-weighted images should include the entire pelvis within the field of view. The panel also agreed unanimously that IV contrast should not be used for MR defecography. Additional areas of consensus ranged in agreement from 75 to 92%. Conclusion We provide a set of consensus recommendations for MR defecography technique based on a survey of expert radiologists in the SAR pelvic floor dysfunction disease-focused panel. These recommendations can be used to develop a standardized imaging protocol.
Acute appendicitis is the most common abdominal surgical emergency in the United States with approximately 250,000 cases annually. Computed Tomography (CT) has emerged as the most accurate diagnostic test to triage these patients for emergent surgery. Although the radiology search pattern is prioritized to detect an inflamed appendix, not all appearances equate to a typical surgical appendicitis. There are a select set of atypical pathologies involving the appendix that have subtle differences on CT, but can have catastrophic complications if treated with emergent appendectomy. This paper will review the spectrum of CT appearances and clinical management for typical and atypical appendiceal pathologies.
Determining the clinical impact of imaging exams at the enterprise level is problematic, as radiology reports historically have been created with the content meant primarily for the referring provider. Structured reporting can establish the foundation for enterprise monitoring of imaging outcomes without manual review providing the framework for assessment of utilization and quality. Ultrasound (US) for deep vein thrombosis evaluation (DVT) is an ideal testbed for assessing this functionality. The system standard template for Doppler US for extremity venous evaluation for DVT was updated with a discrete fixed picklist of impression options and implemented system wide. Template utilization and interpretive outcomes were actively monitored and use reinforced as part of standard clinical practice. From January 1, 2017 to December 31, 2017, 9111 US exams for DVT were performed with 8997 utilizing structured reporting (98.75%). Of those in the structured reporting group, 1074 (11.79%) were positive for any type of DVT with 732 (8.03%) reported as Acute/New above the knee. Positive rates for any type of DVT were 10.29% emergency department, 14.17% inpatient, and 13.20% outpatient. While being the lowest positive rate, the emergency department had the highest overall volume of exams. Structured reporting for DVT US assessment outcomes can be implemented with a very high rate of radiologist adoption and adherence providing accurate determination of positive rates, month by month, in differing patient locations. Structured elements can be used to automatically trigger downstream processes; in our institution, this will alert providers in the EHR if the patient does not receive anticoagulation within 2h of a positive test. This lays the foundation for effective enterprise assessment of imaging outcomes forming the basis of future quality and safety initiatives on optimizing health system resource utilization.
HomeRadiologyVol. 292, No. 1 PreviousNext Reviews and CommentaryFree AccessEditorialPublication Bias in Radiology: How Does It Happen and What Is the Cost?Julia R. Fielding Julia R. Fielding Author AffiliationsFrom the Department of Radiology, UT Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390-9316.Address correspondence to the author (e-mail: [email protected]).Julia R. Fielding Published Online:May 28 2019https://doi.org/10.1148/radiol.2019190985MoreSectionsPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked In See also the article by Cherpak et al in this issue.IntroductionPublication of scientific studies without a clear hypothesis, study design, or appropriate statistical analysis is not uncommon. This has been reported in multiple disciplines, including radiology. Over-generalization of results, or spin, can contribute to a positive conclusion. Studies with positive results are more commonly published than are those with negative results (1,2). This is publication bias.Bias becomes critically important when studies are incorporated into meta-analyses, systematic reviews, and clinical guidelines. Incorporation of flawed studies with inaccurate or overstated positive results can lead to recommendation of inappropriate therapies.In this issue of Radiology, Cherpak and colleagues (3) investigate publication bias based on the design of conference abstracts presented at the Radiological Society of North America (RSNA) Annual Meeting. Specifically, they assessed whether reporting measures of diagnostic accuracy, in this case sensitivity and specificity, improved the likelihood of full-text publication.Of the abstracts presented at the 2011 or 2012 RSNA Annual Meeting, 405 were selected for the study. Cherpak et al reviewed these abstracts for first author, presentation type (oral or poster), study design (prospective or retrospective), sample size, clinical subspecialty, imaging modality, and highest reported sensitivity and specificity. Beginning in 2018, the authors worked backward to identify which abstracts were published within 5 years after presentation. This information was obtained primarily from a review of the online databases. Of the 405 included abstracts, 288 (71%) were published. Cherpak et al used multivariate analysis and found no bias based on reporting diagnostic accuracy. The only characteristic that showed a significant association with full-text publication was presentation type. Oral presentations were more likely to be published than posters. Absence of publication bias based on abstract design has been reported in the diagnostic fields of ophthalmology, stroke, and dementia.Why do so many abstracts presented at the RSNA Annual Meeting get published? Cherpak and colleagues hypothesize that the RSNA Annual Meeting is one of the largest and most prestigious meetings in the world and attracts the best scientific work. Editors of the major radiology journals may be biased toward publication. However, many physician scientists may reserve their highest quality abstract submissions for subspecialty meetings focusing on specific populations, diseases, or modalities. At these venues, the work is reviewed and discussed by reviewers and colleagues with similar interests. Some radiologists now work as part of multidisciplinary teams. Abstracts based on this work are often submitted to national or international meetings focusing on clinical trials. High publication rates and absence of publication bias based on RSNA abstract acceptance may not be generalizable to other meetings or publications.Because the data in this study were primarily obtained with manual review, many features of accepted abstracts could not be assessed. How much published research was performed at large academic medical centers? Without mentors, biostatisticians, funding, and protected time, it is extremely difficult to produce high-quality work. A small number of the abstracts accepted for publication had industry support. As this model may become the norm, a larger sample size may be required to identify bias. It has been reported that female radiologists have fewer first or last author publications (4). There may be a gender disparity in the rate of acceptance of abstracts by female authors. If it were possible to obtain any of this information, many additional hypotheses could be generated and tested.Cherpak and colleagues suggest that unpublished work is research waste. While that may be true in most cases, there is still room for the single-site observational study. Review of an abstract by a colleague may generate an idea for a larger better-designed study. Research collaborators are often identified at meetings. Focus groups, like those at the Society of Abdominal Radiology, are a good way to combine ideas and data to develop scientifically sound studies with the rigor to become practice guidelines. Finally, presentation of scientific work is a skill. Trainees and junior staff benefit from designing an organized and polished presentation.There have been many attempts to improve the quality of published diagnostic accuracy–based research. Perhaps the best-known tool to improve reporting quality is the Standards for Reporting of Diagnostic Accuracy (STARD) guidelines (5). In 2003, a list of 25 items was developed by an expert panel of methodologists, scientists, and editors. This was in response to the publication of many studies with poor methods. Incorporation of the STARD guidelines into a publication is designed to help a reader judge the applicability of findings, assess the validity of conclusions and recommendations, and identify presence of bias. These guidelines were modified in 2015 to include 30 statements. Over 200 journals, including Radiology, have accepted these guidelines as requirements for submission of research. It has been reported that STARD adherence is associated with increased citation rates in higher-impact journals (6).However, physician scientists and journal editors in all disciplines have been slow to adopt these guidelines (7). The ever-increasing demands of clinical work make it difficult for a part-time scientist to design and complete a high-quality study. Absent a biostatistician, many studies are underpowered and do not demonstrate accurate results. Radiology now mandates a STARD checklist be included with all submissions. Adoption of the guidelines has come with positive and negative changes. Although published research is now more rigorous, studies can be difficult to interpret. It can be hard to identify reviewers with the required expertise who are willing to spend the time to produce an accurate and nuanced review. Most importantly, many journal subscribers do not find complex studies clinically relevant. Practicing radiologists must be experts in the diagnosis of multiple diseases using many modalities. Time spent on each case is precious. Increasingly, they choose to obtain continuing medical education credits in the most efficient way. This often consists of online review and attendance at commercially produced meetings.The discipline of radiology is changing. Research is more heavily based on data transfer, decision analysis, and the development of machine learning. High-quality studies are essential to maintain the vibrancy of our field. However, during this transition, it is crucially important to keep our most important skill, contribution to patient care, at the forefront of our education.Disclosures of Conflicts of Interest: J.R.F. disclosed no relevant relationships.References1. Treanor L, Frank RA, Salameh JP, et al. Selective citation practices in imaging research: are diagnostic accuracy studies with positive titles and conclusions cited more often? AJR Am J Roentgenol 2019 Apr 17:1–7; [Epub ahead of print]. Crossref, Medline, Google Scholar2. Ochodo EA, de Haan MC, Reitsma JB, Hooft L, Bossuyt PM, Leeflang MM. Overinterpretation and misreporting of diagnostic accuracy studies: evidence of “spin”. Radiology 2013;267(2):581–588. Link, Google Scholar3. Cherpak LA, Korevaar DA, McGrath TA, et al. Publication bias: association of diagnostic accuracy in radiology conference abstracts with full-text publication. Radiology 2019;292:120–126. Link, Google Scholar4. McDonald JS, McDonald RJ, Davenport MS, et al. Gender and radiology publication productivity: an examination of academic faculty from four health systems in the United States. J Am Coll Radiol 2017;14(8):1100–1108. Crossref, Medline, Google Scholar5. STARD statement: The STAndards for Reporting of Diagnostic accuracy studies. http://www.stard-statement.org. Published 2008. Accessed April 28, 2019. Google Scholar6. Dilauro M, McInnes MD, Korevaar DA, et al. Is there an association between STARD statement adherence and citation rate? Radiology 2016;280(1):62–67. Link, Google Scholar7. Korevaar DA, van Enst WA, Spijker R, Bossuyt PM, Hooft L. Reporting quality of diagnostic accuracy studies: a systematic review and meta-analysis of investigations on adherence to STARD. Evid Based Med 2014;19(2):47–54. Crossref, Medline, Google ScholarArticle HistoryReceived: Apr 30 2019Revision requested: May 6 2019Revision received: May 7 2019Accepted: May 8 2019Published online: May 28 2019Published in print: July 2019 FiguresReferencesRelatedDetailsCited ByAnalysis of conference abstracts of prosthodontic randomised-controlled trials presented at IADR general sessions (2002–2015): a cross-sectional study of the relationship between demographic characteristics, reporting quality and final publicationJunshengChen, YubinCao, MeijieWang, XueqiGan, ChunjieLi, HaiyangYu2020 | BMJ Open, Vol. 10, No. 2Accompanying This ArticlePublication Bias: Association of Diagnostic Accuracy in Radiology Conference Abstracts with Full-Text PublicationMay 28 2019RadiologyRecommended Articles Should Artificial Intelligence Tell Radiologists Which Study to Read Next?Radiology: Artificial Intelligence2021Volume: 3Issue: 2Diversity in Our Workforce: An Urgent Need to Do MoreRadiology2022Volume: 305Issue: 3pp. 648-649No Time for Complacency: Near-term Impact of the COVID-19 Pandemic on Author Gender in RadiologyRadiology2021Volume: 300Issue: 1pp. E308-E309Improving Imaging Care for Diverse, Marginalized, and Vulnerable Patient PopulationsRadioGraphics2018Volume: 38Issue: 6pp. 1833-1844Arterial Spin Labeling Perfusion of the Brain: Emerging Clinical ApplicationsRadiology2016Volume: 281Issue: 2pp. 337-356See More RSNA Education Exhibits Imaging Evaluation of Vascular Cognitive Impairment: Small Vessel Disease is Not a Small ProblemDigital Posters2019A Review Of Diversity In RadiologyDigital Posters2021Progress And Reversion In Mild Cognitive Impairment: An 8-year Follow-up MRI StudyDigital Posters2021 RSNA Case Collection Normal Pressure HydrocephalusRSNA Case Collection2021Left atrial myxomaRSNA Case Collection2020Diffuse idiopathic pulmonary neuroendocrine cell hyperplasiaRSNA Case Collection2020 Vol. 292, No. 1 Metrics Altmetric Score PDF download
Background: Previously reported dual-energy CT methods for detecting noncalcified gallstones have reduced accuracy for gallstones smaller than 9 mm. Purpose: To develop a dual-energy CT method for differentiating isoattenuating gallstones from bile and compare it with previously reported dual-energy CT methods by using a prospective ex vivo phantom reader study. Materials and Methods: From May 2017 to May 2018, gallstones were collected from 105 patients (34 men; mean age, 51 years; age range, 18-84 years) undergoing cholecystectomy and placed inside 120-mL vials containing ox bile. The vials were placed inside a water-filled phantom and were scanned with dual-layer dual-energy CT. Thirty isoattenuating gallstones (4.3-24.7 mm in diameter)were evaluated. Conventional CT images, virtual noncontrast images, and monoenergetic images at 200 and 40 keV were created.Segmented images were created by using a two-dimensional histogram of Compton and photoelectric attenuation. Six readers evaluated the presence of isoattenuating gallstones in each image. Intra-and interreader agreement was measured by using percentage agreement, diagnostic performance was evaluated by using mean area under the receiver operating characteristic curve (AUC) estimates and pairwise comparisons, and the agreement of gallstone sizes measured at pathologic examination with those measured on segmented images was compared by using Bland-Altman analysis. Results: For all gallstones, segmented images provided the highest mean intrareader (88.1%) and interreader (88.2% and 93.6%) agreements for all readers and reading sessions and the highest overall AUC (0.99; 95% confidence interval [CI]: 0.97, 1.00; adjusted P < .02 for all). For gallstones larger than 9 mm, no significant difference was found between the segmented and monoenergetic AUCs (all P > .94, adjusted P > .05 for all). For gallstones measuring 9 mm or smaller, the segmented images had the highest overall AUC (0.99; 95% CI: 0.97, 1.00; adjusted P < .01 for all). The mean difference in stone sizes was -0.6 mm, with limits of agreement from 2.6 to -3.8 mm. Conclusion: Segmented images from Compton and photoelectric attenuation coefficients improve detection of isoattenuating gall-stones compared with previously reported dual- energy CT methods.(C) RSNA, 2019
With the spread of positron emission tomography/magnetic resonance (PET/MR), the question of comparability of studies becomes important. We aim to determine whether PET/MR and PET/computed tomography (PET/CT) are comparable for the case of cervical cancer. Fifteen cervical cancer patients identified by either a radiation oncologist or an oncologic surgeon had both PET/MR and PET/CT performed for initial staging within 3 weeks. We then compared the results both quantitatively (measuring standardized uptake values [SUVs] on visible lesions) as well as qualitatively (having radiologists and nuclear medicine physicians interprets the results). While interpretations between PET/MR and PET/CT varied in many cases, SUVs of primary lesions were similar to within 25% in all but one case, and correlation coefficient was 0.92. Maximum SUV ranged between 4.9 and 25.2 for PET-MR and between 5.8 and 30.4 for PET-CT for primary tumors and between 1.5 and 18.8 for PET-MR and between 1.8 and 20.8 for PET-CT for nodes. However, clinical reads often varied significantly between PET/MR and PET/CT. This suggests that SUV is similar on PET/MR and PET/CT although the differing anatomic modalities available for correlation may make the difference in terms of qualitative interpretation.
Purpose Spectral detector computed tomography (SDCT) is a new CT technology that uses a dual-layer detector to perform energy separation. We aim to assess 3 clinical concepts using a phantom model: noise profile across the virtual monoenergetic (VME) spectrum, accuracy of iodine quantification, and virtual noncontrast (VNC) reconstructions' ability to remove iodine contribution to attenuation. Methods Six vials containing varying concentrations of iodinated contrast (0–6 mg/mL) diluted in water were placed in a water bath and scanned on an SDCT scanner. Virtual monoenergetic (40–200 keV at 10-keV increments), iodine–no-water, and VNC reconstructions were created. Attenuation (in Hounsfield units [HU]), VME noise at each energy level, CT-derived iodine concentration, and VNC attenuation were recorded. Results Virtual monoenergetic noise was improved at all energies compared with conventional images (conventional, 9.8–11.2; VME, 7.5–9.5). Noise profile showed a slightly higher image noise at 40 keV, but was otherwise relatively flat across the energy spectrum. On iodine–no-water reconstructions, measured varied from actual iodine concentration by ±0.1 mg/mL (SD, 0.16–0.36). Virtual noncontrast attenuation was within 5 HU of water attenuation at all iodine concentrations. Conclusion Reconstructions of SDCT show lower VME image noise, accurate iodine quantification, and VNC attenuation values within 5 HU of expected in a phantom model.