Full text of DOI:10.1007/s10140-001-0192-5

Emergency Radiology (2002) 9: 55–59
DOI 10.1007/s10140-001-0192-5
ORIGINAL ARTICLE
Yair Safriel Æ Salvatore Sclafani Æ Brian Gale
Divyang Patel Æ David Gordon
Comparing the interpretations of CT pulmonary angiograms
by attending and resident radiologists: can residents identify
life-threatening pulmonary emboli in hospitalized patients?
Published online: 20 February 2002
ꢀ ASER 2002
Abstract Objective: CT pulmonary angiography is now arteries. Interobserver agreement was poorest (K<0.05)
often the first-line investigation for pulmonary emboli. in the left interlobar, left lower lobe lateral basal segment,
When these studies are performed after hours in teaching right lower lobe superior segment, and left lower lobe
hospitals, they are often initially interpreted by trainees. It superior segment branches. None of the patients expired
is of great significance whether the interpretations of due to pulmonary emboli. Conclusion: Most life-threat-
trainees and certified radiologists with regard to the eningpulmonary emboli requiringurgent treatment are
presence of pulmonary emboli on CT pulmonary angio- the more central emboli. This study demonstrates that
grams correspond, because of the morbidity and mortal- trainees and certified radiologists can make similar con-
ity of both the condition and its treatment. Material and clusions regarding these central pulmonary emboli in
methods: Twenty-five consecutive CT pulmonary angio- hospitalized patients and that preliminary interpretations
grams (CTPAs) of hospitalized patients were viewed at by trainees should not therefore adversely affect patient
lungand soft tissue windows both on a workstation and care.
on hard copies, at the observers’ discretion. Each CTPA
Keywords CT angiography Æ Emboli Æ Pulmonary Æ
Education
was divided into 28 arterial zones based on pulmonary
anatomy (including the subsegmental arteries), giving a
total of 700 arterial zones, and analyzed retrospectively
and independently by two cross-sectional imaging spe-
cialists and four residents. Each arterial segment was
rated with regard to pulmonary embolus as either high,
intermediate, or low probability or not visualized. The
kappa (K) test, which tests for interobserver agreement,
was used for statistical analysis. Results: At the time of the
scan all patients were hospitalized for underlyingcondi-
tions. Of the 25 patients studied, 9 were referred from the
ICU, 7 experienced severe acute shortness of breath and
respiratory failure, 5 were post-partum women, 2 had had
a recent stroke, 1 patient had antithrombin III deficiency,
and 1 had a diagnosis of breast cancer. The incidence of
pulmonary emboli was 44%. For the main pulmonary
arteries interobserver agreement was good (K=0.61) and
for the segmental pulmonary arteries it was fair
(K=0.26). For the subsegmental arteries interobserver
agreement was poor (K=0.16). The zones where inte-
robserver agreement was greatest (K>0.4) were the left
main, left lower lobe, and the right main pulmonary
Introduction
Pulmonary embolism accounts for up to 100,000 deaths
per year in the United States [1, 2]. Traditionally pul-
monary angiography has been the diagnostic and de-
finitive study of choice for pulmonary emboli (PE). Even
though empiric anticoagulation therapy carries a higher
morbidity and mortality than pulmonary angiography
[3], only 15% of patients ultimately proceed to formal
angiography [4, 5] due to the definite although minimal
risk and cost associated with it [6, 7]. Thus, the potential
of CT as a noninvasive study in its ability to both screen
for and detect particularly the acute, life-threateningPE
has great promise.
There have been numerous studies of the sensitivity,
specificity, and interobserver variability associated with
CT pulmonary angiography (CTPA) for the detection of
acute PE [3, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20],
but none of these studies addressed the reality that in
many teachinghospitals, only radioloigsts-in-training
(RITs) are present to provide an opinion after hours,
with a review the followingmorningby a certified ra-
diologist. Thus, RITs are regularly called to interpret a
CTPA for the presence of an acute PE, and treatment
Y. Safriel (&) Æ S. Sclafani Æ B. Gale Æ D. Patel Æ D. Gordon
Department of Radiology,
State University of New York at Brooklyn (Downstate),
New York, USA
E-mail: safriel@hotmail.com

56
decisions are made on the basis of their interpretation. inability to remain motionless in the scanner. This
As the treatment of PE has its own morbidity and yielded a final group of 25 studies. The age range was
mortality [3], it is of great clinical significance that cor- 23–81 years with a mean age of 53.04 years. In 18 cases a
rect interpretations are made by anyone interpreting correlative study was performed where the overall rate
CTPAs for the presence of PE.
of PE was 44%. In the remaining7 cases no correlative
study was performed, either because the referringphy-
sicians were confident of the diagnosis provided by
CTPA or due to patient-related factors.
Materials and methods
There was a wide variety of patients all of whom were
either inpatients or emergency room patients. Nine of
the studies were referred from the ICU (seven with un-
derlyingchronic obstructive pulmonary disease and two
with end-stage renal disease). Seven patients were re-
ferred from the emergency room with severe acute
shortness of breath and respiratory failure. From the
floors, two patients had a recent stroke and were bed
ridden, one had antithrombin III deficiency, and one
had breast cancer. Five post-partum patients were
referred from the gynecology department.
Six observers at a single center participated in the study. Two are
radiologists who have been certified by the American Board of
Radiology for 30 and 6 years. The remaining observers were RITs,
two of whom had 3 years of training, one of whom had 2 years of
training, and one was in his first year of training. The reviewers
examined the CTPAs of 25 consecutive patients retrospectively.
Patients recruited were all in-patients (this included emergency room
patients). All patients were under clinical suspicion of acute PE and
were referred by the clinical service. The observers were aware of the
age and sex of the patient but were unaware of the clinical findings.
An institutional protocol issued to all certified radiologists and
RITs upon joiningthe department includes a standard CTPA for
PE protocol. We did not provide any specific trainingor instruction
over and above that given to all trainees in the interpretation of
CTPAs. Prior to participatingin the study observers were advised
to perform a review of the vascular and lobar anatomy of the lung.
All scans were performed on a GE CTi scanner (General
Electric Medical Systems, Milwaukee, Wis.). A non-contrast-en-
hanced spiral CT of the chest at a collimation of 5 mm for the
mediastinum and 7 mm for the rest of the lungwas performed first.
A time–density curve was then acquired usingenhancement of the
right or left pulmonary arteries with 20 ml noniodinated contrast
material at 2–3 ml per second via an automatic injector (Medrad,
Pittsburgh, Pa.). CT angiography was then undertaken in a caudal-
to-cranial direction at 3 mm collimation, a pitch of 1.6–2, 230–
240 mA and 120 kV with 130 ml nonionic contrast material at an
injection rate and delay determined by the time–density curve.
CTPAs were viewed at lungand soft tissue windows both on the
CT console and on hard copies at the observers’ discretion. Each
CTPA was divided into 28 arterial zones based on pulmonary
anatomy for analysis: right and left main pulmonary artery (PA),
right and left upper lobe PA, segmental arteries of the right (an-
terior, posterior, superior) and left (anterior, apicoposterior) upper
lobe PA, right and left interlobar PA, right middle lobe PA, lin-
gular (considered a lobe) branch of left PA, segmental arteries of
the lingula (superior, inferior) and right middle lobe (medial, lat-
eral), right and left lower lobe PA, segmental arteries of the right
(anterior, posterior, medial, lateral, superior) and left (posterior,
lateral, superior, anteromedial) lower lobe PA. This format was
based on several prior studies [3, 19].
None of the patients died within four weeks of the
completion of the study.
The results are represented in Table 1.
A total of 700 arterial zones were analyzed. In ac-
cordance with Altman [21], overall agreement was good
for the main pulmonary arteries, fair for the lobar and
interlobar arteries, and poor for the segmental arteries.
The zones where interobserver agreement was great-
est (K>0.4) were the left main, left lower lobe, and right
main pulmonary arteries. Interobserver agreement was
poorest (K<0.05) in the left interlobar, left lower lobe
lateral basal segment, right lower lobe superior segment,
and left lower lobe superior segment branches.
Both certified radiologists and RITs reported similar
percentages of nonanalyzable arteries (8.2% and 8.0%
respectively). Segmental arteries were more likely to be
considered nonanalyzable than lobar or interlobar ar-
teries, which in turn were more likely to be nonanalyz-
able than the main pulmonary arteries.
There was also little difference in the number of in-
termediate probability zones between RITs (2.4%) and
certified radiologists (2.6%).
Both residents (RITs) and attending(certified) radi-
ologists cited flow artifacts and breathing as the greatest
obstacle encountered when interpretingCTPAs. Both
groups also noted confusion with respect to the evalu-
ation of the left interlobar artery. In addition, residents
also cited difficulties with some of the anatomical
aspects of cross-sectional imaging of certain zones and
differentiatinglymph nodes from flow defects (Fig. 1).
Each arterial segment was rated for pulmonary embolus as ei-
ther high, intermediate, or low probability or not visualized.
An additional correlative study was requested on each patient.
This took the form of scintigraphy, calf venous ultrasound, or
fluoroscopic angiography.
For statistical analysis the arterial zones were divided into three
regions. Region 1 included the main pulmonary arteries, region 2
represented the lobar and interlobar arteries, and region 3 included
the segmental arteries. The kappa test, which tests for interobserver
agreement, was used for statistical analysis. As per Altman [21],
interobserver agreement was considered maximal for K=1. K=0
denotes a result equivalent to a chance guess (50%). Statistical
analysis was conducted by the Center for Scientific and Academic
Computingat the State University of New York at Brooklyn.
Discussion
This study attempted to recreate the clinical environ-
ment where a certified radiologist’s word is often final
and confirmatory studies are requested on a minority of
patients. Thus, in evaluatingour results, importance was
Results
Twenty-seven patients were enrolled in the study, 2 of attached to the findings of the certified radiologist as
whom were excluded due to renal insufficiency and compared with the trainee. Several studies have already

57
Table 1. Means and standard
deviations of kappa by arterial
region
Region
No.
Mean
Standard
deviation
95% Confidence
intervals
Interobserver
agreement grade [21]
Lower
Upper
1
2
16
64
144
224
0.61
0.26
0.16
0.22
0.24
0.25
0.24
0.27
0.48
0.19
0.12
0.18
0.74
0.32
0.20
0.25
Good
Fair
Poor
Fair
3
Total
been undertaken on the detection rate of PE by CTPA as
compared with a gold standard, providing specificities,
sensitivities, and interobserver agreement for this tech-
nique [3, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. Our
study does not aim to provide accuracy data for the use
of CTPA in the detection of PE, but rather to fill a void
in the literature not answered in earlier reports by ad-
dressingthe real-world situation in regard to the trainee
and to guide teaching institutions in the use of CTPA
after hours.
A factor that reduced our overall kappa and the
kappa for region 2 in particular was confusion regarding
the left interlobar artery. There is widespread disagree-
ment on whether the left lung(havingtwo lobes) actu-
ally has a left interlobar artery or rather just a lower and
upper lobe artery. This was noted by observers to have
caused confusion, and the negative effect is further
proved by the fact that the segments on either side of the
left interlobar artery (the left main and left lower lobe
pulmonary arteries) were the arteries with the highest
kappa values.
Fig. 1. Areas such as that marked by the arrowhead proved the
downfall for residents who labeled this as an embolus. This patient
went on to have a normal VQ scan
Our study showed lower interobserver agreement
than do other studies published in the literature. The
only other study [19] comparinginterobserver agreement
for the detection of PE usingCTPAs utilized six ob-
servers with an overall kappa of 0.56 on an arterial zone
basis and a kappa of 0.47 for the segmental arteries
(comparable to region 3 in the present study). All six
observers in that study were certified radiologists with
further fellowship trainingand there was no differenti-
ation between outpatients and inpatients. Before the
start of that study, all observers were given a refresher
course on interpretingCTPAs. In our study we specifi-
cally did not provide a refresher course so as to ap-
proximate the clinical situation more closely.
Furthermore, our kappa value was hurt by the relatively
large proportion of ventilated and critically ill patients
since many more scans are suboptimal in these patients,
but they are the type of patients on whom scans are
likely to be performed urgently after hours.
Several other studies also tested interobserver agree-
ment for the detection of PE on CTPAs, but all used few
observers or did not employ statistical methods to
quantify interobserver agreement. Using two observers,
Van Rossum et al. [3] obtained an overall kappa of 0.77
for the detection of acute PE on CTPAs but did not
calculate a kappa for a zone to zone comparison or
provide figures on the differences between regions. An-
other study by the same authors [22] yielded an overall
kappa of 0.85, while another study [11] resulted in a
kappa of 0.61. Two more studies [23, 24], which ap-
peared in abstract form only, provided overall kappas of
0.72 and 0.87 respectively for the detection of acute PE
on CTPAs. Bergin et al. [18], in determining the sensi-
tivity and specificity of chronic PE detection on CTPAs
in a predominantly outpatient setting, found a kappa of
0.66 for central disease (comparable to region 1 in this
study) and kappas of 0.31 and 0.34 for lobar (region 2)
and segmental (region 3) emboli respectively with two
observers.
Several studies in the past have shown that accuracy
in detectingacute PE on CTPAs increases with experi-
ence. Van Rossum et al. [25] showed that accuracy of
CTPA interpretation improves with the experience of
the observer. This held true even amongst certified
radiologists specializing in cross-sectional imaging [19].
We believe our interobserver agreement was weaker
than in other studies as four of our six observers were
RITs, some with as little as 1 year of postgraduate ra-
diological experience. Nevertheless, we believe this to be
a strength rather than a weakness as it approximates
more closely the situation after hours in a teaching
hospital. Our results should not be interpreted as
showingthat residents are as good as certified radiolo-
gists in the interpretations of CTPAs. Rather, residents

58
6. Stein PD, Athansoulis C, Alvi A, et al (1992) Complication and
validity of pulmonary angiography in acute pulmonary embo-
lism. Circulation 85:462–468
7. Zuckerman DA, SterlingKM, Oser RF (1996) Safety of pul-
monary angiography in the 1990’s. J Vasc Interv Radiol 7:199–
205
8. Dalen JE (1993) When can treatment be withheld in patients
with suspected pulmonary embolism? Arch Intern Med
153:1415–1418
9. Remy-Jardin M, Remy J, Wattinne L, et al (1992) Central
pulmonary thrombo-embolism: diagnosis with spiral volumetric
CT with the single breath hold technique – comparison with
pulmonary angiography. Radiology 185:381–387
10. GargK, Welsh CH, Feyerabend AJ, et al (1998) Pulmonary
embolism diagnosis with spiral CT and ventilation–perfusion
scanning – correlation with pulmonary angiographic results or
clinical outcome. Radiology 208:201–208
11. Mayo JR, Remy-Jardin M, Muller ML, et al (1997) Pulmonary
embolism: prospective comparison of spiral CT with ventila-
tion–perfusion scintigraphy. Radiology 205:447–452
12. Remy-Jardin M, Remy J, Deschildre F, et al (1996) Diagnosis
of pulmonary embolism with spiral CT: comparison with pul-
monary angiography and scintigraphy. Radiology 200:699–706
13. Drucker EA, Rivitz SM, Shepard JO, et al (1998) Acute pul-
monary embolism: assessment of helical CT for diagnosis.
Radiology 209:235–241
are adequate in providingpreliminary interpretations
until an attendingradiologist arrives.
In interpretingour results, the clinical presentation of
emboli to different levels of the pulmonary circulation is
important. The immediately dangerous and imminently
lethal emboli are the larger, more central emboli [8, 26],
whereas emboli to the smaller arteries tend to cause
chronic symptoms.
Previous studies on the ability of RITs to evaluate CT
scans of the head for life threateningconditions noted
that, while there were some discrepancies which occa-
sionally involved major injury, overall patient care did
not suffer [27, 28]. This was also the case in our study
since none of the patients died when the CTPAs were
originally interpreted by RITs in the course of their
regular duties.
Our detection rate for PE was somewhat higher than
those in recent multicenter trials [17]. We believe that
this was due to our patient population, which was made
up entirely of hospitalized patients of whom a significant
number where in the ICU or were post-partum.
Even though our patient base was relatively small,
because of the nature of the patient population we be-
lieve it to be large enough to provide an adequate
number of positive cases. This population size also
compares favorably to that in previous studies [26]
which formed the basis of patient care decisions for
several years.
14. Sostman HD, Layish DT, Victor FT, et al (1996) Prospective
comparison of helical CT and MR imaging in clinically sus-
pected acute pulmonary embolism. J Magn Reson Imaging
6:275–281
15. Pruszczyk P, Torbiki A, Pacho R, et al (1997) Noninvasive
diagnosis of suspected severe pulmonary embolism: trans-
esophageal echocardiography vs spiral CT. Chest 112:722–728
16. Herold CJ, Hahne J, Ghaye B, et al (1999) Prospective evalu-
ation of pulmonary embolism: diagnostic performance of spiral
CT angiography in the ESTIPEP trial. Radiology 213(P):127
17. van Strijen MJ, de Monye W, Kieft GJ, et al (1999) Diagnosis
of pulmonary embolism with spiral CT: a prospective cohort
study in 617 consecutive patients (ANTELOPE trial). Radiol-
ogy 213(P):127
Conclusions
As our study showed moderate to good interobserver
agreement for the detection of central and lobar acute
pulmonary emboli, and as these are the emboli which are
more likely to cause immediate death, we believe that
our study shows that radiologists-in-training can iden-
tify life-threateningpulmonary emboli on CT anigo-
grams of hospitalized patients. However, our results do
not show that residents are equal to attendingradiolo-
gists in the interpretation of CTPAs, and therefore early
attendingreview is essential.
18. Bergin CJ, Sirlin CB, Hauschildt JP, et al (1997) Chronic
thromboembolism: diagnosis with helical CT and MR imaging
with angiographic and surgical correlation. Radiology 204:695–
702
19. Chartand-Lefebvre C, Howarth N, Lucidarme O, et al (1999)
Contrast enhanced helical CT for pulmonary embolism
detection: inter and intra observer agreement among radiol-
ogists with variable experience. AJR Am
172:107–112
J Roentgenol
20. Safriel Y, Zinn H (2002) CT pulmonary angiography in the
detection of pulmonary emboli: a meta-analysis of sensitivities
and specificities. Clin Imaging 26:101–105
21. Altman DG (1992) Practical statistics for medical research.
Chapman and Hall, London
22. van Rossum AB, Pattynama PM, Mallens WM, et al (1998)
Can helical CT replace scintigraphy in the diagnostic process in
suspected pulmonary embolism? A retrolective–prolective co-
hort study focusingon total diagnostic yield. Eur Radiol 8:90–
96
23. Herold C, Remy-Jardin M, Grenier PH, et al (1998) Prospective
evaluation of pulmonary embolism: initial results of the Euro-
pean Multicenter Trial (ESTIPEP) (abstract) Radiology
209(P):299
References
1. National Institute of Health Consensus Conference (1986)
Prevention of venous thrombosis and pulmonary embolism.
JAMA 256:744–749
2. Gupta A, Frazes CK, Ferguson JM, et al (1999) Acute pul-
monary embolism: diagnosis with MR angiography. Radiology
210:353–359
3. van Rossum AB, Pattynama PMT, Tjin ER, et al (1996) Pul- 24. Baghaie F, Remy-Jardin M, Remy J, et al (1998) Diagnosis of
monary embolism: validation of spiral CT angiography in 149
peripheral acute pulmonary emboli: optimization of the spiral
patients. Radiology 201:467–470 CT acquisition protocol (abstract). Radiology 209(P):299
4. Henschke CI, Mateescu I, Yankelevitz DF (1995) Changing 25. van Rossum AB, van Erkel AR, van Persijn van Meerten EL,
practice pattern in the workup of pulmonary embolism. Chest
107:940–945
5. van Beek EJ, Buller HR, Brandjes DP, et al (1994) Diagnosis of
et al (1998) Accuracy of helical CT for acute pulmonary
embolism: ROC analysis of observer performance related to
clinical expertise. Eur Radiol 8:1160–1164
clinically suspected pulmonary embolism: a survey of current 26. Goodman LR, Curtin JJ, Mewissen MW, et al (1995) Detection
practice in teachinghospitals. Neth J Med 44:50–55 of pulmonary embolism in patients with unresolved clinical and

59
scintigraphic diagnosis: helical CT versus angiography. AJR 28. Roszler MH, McCarroll KA, Rashid T, Donovan KR, Kling
Am J Roentgenol 164:1369–1374
GA (1991) Resident interpretation of emergency computed
tomographic scans. Invest Radiol 26:374–376
27. Wysoki MG, Nassar CJ, Koenigsberg RA, Novelline RA, Faro
SH, Faerber EN (1998) Head trauma: CT scan interpretation
by radiology residents versus staff radiologists. Radiology
208:125–128