Full text of DOI:10.1080/10903120290938896

PERFORMANCE MEASUREMENTS IN EMERGENCY MEDICAL SERVICES
James Dunford, MD, Robert M. Domeier, MD, Thomas Blackwell, MD, Gregory Mears, MD,
Jerry Overton, MPA, Edgardo J. Rivera-Rivera, MD, Robert Swor, DO
ABSTRACT
medical services; performance meas-
urements; performance indicators;
standards.
component of the health care sys-
tem, emergency medical services
With the strong encouragement of
leading health care agencies, business
principles are being implemented
throughout health care, including
emergency medical services (EMS).
The reason is simple—quality of care
can be enhanced by incorporating the
management concepts of continuous
quality improvement (CQI). The CQI
process couples carefully identified,
measurable performance indicators
with information systems to monitor,
analyze, and trend data. Benchmarking
outcomes with other EMS systems
allows the identification of “best prac-
tices” and the evolution of standards.
Emergency medical services profes-
sionals must actively participate with
the broader health care community in
creating performance measurements to
ensure that high-quality care is deliv-
ered consistently. Key words: continu-
ous quality improvement; emergency
(
EMS) administrators must devel-
PREHOSPITAL EMERGENCY CARE
op CQI strategies to ensure confi-
dence in the quality of their sys-
tems.
2002;6:92–98
Practices common to American
business more than 30 years ago
are increasingly being applied to
the health care industry today. The
public now expects the quality and
consistency of health care to be
evaluated by using performance
measures analogous to those used
by traditional service industries,
such as retail stores, automobile
In 1998, The President’s Ad-
visory Commission on Consumer
Protection and Quality in the
Health Care Industry offered gen-
eral recommendations for quality
measurement and reporting in
4
health care (Table 1). Keeping
these basic recommendations in
mind, EMS administrators must
now begin to select performance
measurements that reflect their
essential services and benchmark
outcomes; they must also champi-
on “best practices” and promote
the evolution of evidence-based
standards of care. Quality can no
longer be assumed—it must be
managed. Further, the adage “you
can only manage what you can
measure” is truer today than ever
before.
manufacturers, and hotels.
A
recent Institute of Medicine publi-
cation titled Crossing the Quality
Chasm stated that the U.S. health
care system is “broken” and advo-
cated for fundamental change
through the implementation of
continuous quality improvement
1
(
CQI) activities. Growing public
Received May 11, 2001, from the City of San
Diego EMS and Department of Emergency
Medicine, UCSD Medical Center (JD), San
Diego, California; Saint Joseph Mercy
Hospital (RMD), Ann Arbor, Michigan;
Mecklenburg EMS Agency, Carolinas
Medical Center (TB), Charlotte, North
Carolina; North Carlina State EMS and
Department of Emergency Medicine,
University of North Carolina (GM), Chapel
Hill, North Carolina; Richmond Ambulance
Authority (JO), Richmond, Virginia;
Saginaw Cooperative Hospitals, Saginaw/
Tuscola Medical Control Authority (EJRR),
Saginaw, Michigan; and EMS Programs,
William Beaumont Hospital (RS), Royal
Oak, Michigan. Revision received August
concern over the alarming rates of
preventable medical errors has
reinforced the need for processes
that can be used to identify and
CONTINUOUS QUALITY
IMPROVEMENT IN EMERGENCY
MEDICAL SERVICES
Although quality itself may seem
elusive, in 1990 the Institute of
Medicine defined quality in health
2
correct systemic error. The quality
of health care can be defined and
measured with a degree of scientif-
ic accuracy comparable to that of
most analytic measures used in
3
clinical medicine. As an integral
TABLE 1. Recommendations for Quality
Measurement and Reporting in Health Care*
•
•
•
Identify sets of quality measures for standardized reporting by each sector of the health
care industry.
3
0, 2001; accepted for publication
September 17, 2001.
Develop a framework for quality measurement and reporting and standardization of
quality reporting.
Presented at the Turtle Creek Conference
III, Dallas, Texas, April 29–May 1, 2001.
Support the development of quality measures to enhance the ability to evaluate and
improve health care.
Supported by an unrestricted educational
grant from Wyeth-Ayerst Laboratories.
•
•
Determine stable mechanisms for quality measurement and reporting.
Address correspondence and reprint re-
quests to: James Dunford, MD, EMS Med-
ical Director, City of San Diego, UCSD Med-
ical Center, 200 West Arbor Drive, #8676,
San Diego, CA 92103. e-mail: <jdunford@
ucsd.edu>.
Ensure that information about health care quality is valid, reliable, comprehensible, and
available to the public.
*Source: The President’s Advisory Commission on Consumer Protection and Quality in the Health Care
Industry. Quality First: Better Health Care for All Americans; Final Report to the President of the United
States. Washington, DC: U.S. Government Printing Office, March 1998.
92

Dunford et al. PERFORMANCE MEASUREMENTS IN EMS
93
care as “the degree to which health
services for individuals and popu-
lations increase the likelihood of
desired health outcomes and are
consistent with current profession-
clear that a process for continuous-
ly evaluating the quality of EMS
(one that also allowed for rapid
insertion of improvements into the
system) was required. The chal-
lenge remains to identify the most
important aspects of care and to
develop a method to measure and
TABLE 2. Emergency Medical
Services Attributes Recommended
by the National Highway
Traffic Safety Administration for
Inclusion in Continuous Quality
Improvement Programs*
5
al knowledge.” Until recently,
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Clinical care
measuring quality in health care
often involved quality assurance
techniques. Although it provides
useful information, quality assur-
ance is typically static, retrospec-
tive, implemented by manage-
ment, and clinician-focused. Chart
audits can identify protocol devia-
Communications systems
Education systems
Emergency medical services research
Evaluation
Human resources
Information systems
Integration of health services
Legislation and regulation
Medical direction
Prevention
Public access
Public education
System finance
1
5,16
monitor them.
To develop a foundation and
framework, NHTSA funded a five-
year project called the Emergency
Medical Services Outcomes Project
(EMSOP). The EMSOP is designed
to identify: 1) conditions that
should take precedence in EMS
outcomes research; 2) risk adjust-
ment measures for these priority
conditions; and 3) outcome meas-
ures for these priority conditions.
6
tions or individual or event-based
7
–9
failures
but lack the ability to
maintain and enhance perform-
ance. As the value of CQI (devel-
oped in the late 1940s by W.
Edwards Deming for business
*
Source: National Highway Traffic Safety
Administration. Agenda for the Future (DOT HS
08 441). Washington, DC: U.S. Department of
8
Transportation, August 1996.
1
0
17
management ) was recognized by
the health care industry in the mid-
In EMSOP phase I, Maio et al.
identified “relief of discomfort” as
the clinical indicator with the
greatest potential benefit for imple-
mentation in a CQI process. The
authors endorsed the use of “trac-
er” conditions (those with high fre-
quency and a high potential to ben-
1
980s, the means of measuring
cated organization for the alloca-
tion and coordination of emer-
gency medical personnel, proto-
quality shifted from the inspection
approach of traditional quality
assurance to the analytical ap-
proach of CQI. The CQI method
emphasizes organizational systems
and processes rather than individ-
ual behavior as targets for data col-
lection, analysis, and improve-
ment. By evaluating specific indi-
cators of performance, assessing
the processes associated with these
performances, and implementing
solutions to inadequate systems,
CQI both addresses current prob-
lems and prevents future occur-
rences by resolving the problem
source.
2
0
cols, equipment, and supplies.
The report concluded that to oper-
ate in an efficient manner, EMS
systems must have comprehensive
assessment programs in which cur-
rent processes are reviewed and
improved as necessary. To assess
the effectiveness of an EMS system,
administrators should gather data
relevant to current standards or
expectations to validate existing
processes or identify areas for
change. Any redesign or imple-
mentation of a new process should
also be evaluated with relevant
performance measures. In the EMS
Agenda for the Future, NHTSA iden-
tified 14 attributes that should be
included in a CQI program (Table
1
8
efit from medical care) as indices
of an EMS system’s overall effec-
tiveness. In EMSOP phase II, Spaite
1
9
et al. described the need for dif-
ferent EMS outcome models based
on the severity and therapeutic
time dependency of particular con-
ditions. For example, the impact of
prehospital care on nontraumatic
cardiac arrest, airway obstruction,
anaphylaxis, respiratory arrest,
and possibly severe trauma can be
analyzed with outcome measures
such as survival or change in phys-
iologic status. Outcome analyses
for conditions with less time-criti-
cal elements (e.g., painful extremi-
ty injury) are better modeled by
defining and measuring delivery of
discrete “units of service,” such as
pain relief and patient satisfaction.
By 1992, the Joint Commission
on Accreditation of Healthcare
Organizations (JCAHO) was
strongly encouraging health care
organizations to implement CQI as
a means to more accurately pro-
vide information regarding the
needs of patients and other stake-
holders. A modified JCAHO hospi-
tal model was suggested for use in
2
0
2). They also emphasized the
value of input from local health
care professionals and the commu-
nity in developing relevant CQI
indicators and suggested that out-
come categories, such as cost–effec-
tiveness, specific interventions
analysis, and systems readiness,
could be disseminated in a report-
card format to inform the public
and EMS providers of the status of
their system. It was also recom-
mended that legislative safeguards
1
1
EMS, and the American College
DESIGNING AND
IMPLEMENTING CONTINUOUS
QUALITY IMPROVEMENT
In the 1996 landmark publication
EMS Agenda for the Future, NHTSA
defined an EMS system as a dedi-
1
2
of Emergency Physicians, the
National Association of EMS
1
3
Physicians,
and the National
Highway Traffic Safety Admini-
1
4
stration (NHTSA) endorsed CQI
for EMS. By the mid-1990s, it was

94
PREHOSPITAL EMERGENCY CARE JANUARY / MARCH 2002 VOLUME 6 / NUMBER 1
TABLE 3. Evaluation Recommendations for the California Emergency
Medical Services (EMS) Authority Proposed by a Technical Advisory
Team of the National Highway Traffic Safety Administration (NHTSA)*
refers to the features of effective
performance measurement tools;
they should be specific, measura-
ble, action-oriented, relevant, and
•
•
•
•
Develop a comprehensive, medically directed statewide quality improvement program
to evaluate patient care processes and outcomes.
27
timely.
Develop a statewide integrated information system that will have the ability to moni-
tor, evaluate, and elucidate EMS and trauma care in California.
Structural assessment consists of
the straightforward evaluation of
the environment in which care is
Ensure the design capability for linkages of the statewide integrated information system
to other public and private data systems.
1
3
delivered.
Examples include
facilities, equipment, and provider
Allocate personnel and resources to implement the statewide integrated information
system, including necessary technical assistance, materials, and funding to local EMS
agencies.
1
1,13,25
training and knowledge base.
A structural indicator could also be
used to monitor EMS staff creden-
•
Enforce the use of a uniform prehospital data set consistent with NHTSA Uniform
Prehospital Data Set. Mandate submission of an agreed on, timely, limited, uniform, com-
mon language data set from the local EMS agencies to the California EMS Authority.
1
1
tialing. Structural indicators are
the most difficult element to relate
2
5
•
•
•
Seek ways to improve the number of completed care records that are delivered to the
emergency department staff on patient arrival, with a goal of 98% compliance.
to outcomes.
Process assessment typically
analyzes the actions taken by both
the provider and the patient and is
evaluated by indicators of discrete
steps in requesting and providing
care. A process indicator, for exam-
ple, could consist of measuring the
appropriate steps for administer-
ing bronchodilator therapy to a
patient with reactive airway dis-
ease. Although process does not
necessarily affect outcome, the def-
inition could be expanded to
include the repeatable sequence of
steps performed by all levels of the
EMS system to effect a good out-
Request that NHTSA conduct the Leadership Workshop on Quality Improvement for
EMS systems in California.
Write and help shepherd through the legislative process legislation to ensure the con-
fidentiality and non-discoverability of EMS and trauma records and EMS provider pro-
tection while participating in EMS quality improvement activities.
•
Promote and support decreasing barriers to performing EMS research in California.
Vigorously develop a statewide human subjects review approval process for out-of-hos-
pital research.
*Boldface indicates a priority recommendation. Source: Bass RR, Conn AK, Dawson D, et al. State of
California: An Assessment of Emergency Medical Services. Report of the National Highway Traffic
Safety Administration Technical Assistance Team, August 23–26, 1999. Available at: http://www.emsa.
ca.gov/vision/nhtsarpt99.asp.
2
3
be established to protect EMS med-
ical directors and field personnel
from liability resulting from a
review of unexpected or unusual
situations. In follow-up publica-
tions, NHTSA proposed CQI tem-
plates and techniques to imple-
tions. The team compared current
standards, present status, and
future goals in ten essential areas
of EMS and prioritized recommen-
dations for improvement (Table 3).
As a result, a series of voluntary
statewide performance indicators
2
5
come.
Outcome measures, both final
and intermediate, assess the effect
of care on the overall health, satis-
faction, knowledge, and behavior
of the patient. Outcome indicators
should assess whether the integrat-
ed elements of the system function
2
1
24
ment these recommendations.
is currently being evaluated.
The companion document EMS
Agenda for the Future: Implementa-
tion Guide offers examples of the
tools, infrastructure, and strategic
partnerships necessary to accom-
plish short, intermediate, and long-
STANDARDS, PERFORMANCE
INDICATORS, AND
1
7,25
to achieve the desired effect.
An
example of an outcome indicator
would be the consistent attainment
of pain relief through the appropri-
ate use of parenteral analgesics.
Benchmarking entails the use of
a structured method to quantita-
tively compare processes or prod-
ucts with the goal of identifying
current best practices. Benchmarks
may eventually become best prac-
tices or industry standards. How-
ever, the superiority of one practice
over another may depend on the
BENCHMARKING
Although the terminology for
assessing EMS systems varies, the
definitions provided in Table 4 are
commonly used. Measures of sys-
tem performance are traditionally
based on three elements—struc-
2
2
term EMS Agenda objectives.
While NHTSA continues to
address EMS quality improvement
at a national level, individual states
have begun analyzing their current
quality improvement processes as
well. As a component of its Vision
for EMS project, the California
EMS Authority, in concert with the
Office of Traffic Safety, invited a
technical assistance team from
NHTSA to evaluate its EMS pro-
gram and make recommenda-
1
1,13,25
ture, process, and outcome.
The JCAHO refers to “dimensions
of performance,” which encom-
passes measures of efficacy, appro-
priateness, availability, timeliness,
effectiveness, continuity of care,
safety, efficiency, and respect and
2
5,28
criteria used to test it.
For
example, best practices might dif-
fer when evaluated for cost–effec-
2
6
caring.
The acronym SMART

Dunford et al. PERFORMANCE MEASUREMENTS IN EMS
95
tiveness rather than patient out-
come. With patients increasingly
expecting service and value both in
and outside the health care indus-
try, it may be appropriate to con-
sider adopting certain EMS bench-
marks from other service indus-
tries. In particular, there are
many parallels between EMS and
the hotel and restaurant industries
regarding customer service, tele-
phone protocol use, and staff train-
ing.
Also, to achieve uniformity in
diverse EMS systems, standards
and performance indicators may
need to be defined by state or fed-
eral legislators.
TABLE 5. Quality Performance
Indicators Proposed by the
International Association
of Fire Fighters*
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Call processing
Turnout time
Travel time
Staffing
Recently, significant efforts to
implement quality measurement
tools have been made by fire serv-
ices, including the International
Association of Fire Fighters (IAFF),
the International Association of
Fire Chiefs, the National Fire
Protection Association (NFPA),
and the Commission of Fire
Accreditation. The IAFF has devel-
oped a pilot set of quality indica-
2
8
Deployment
Road structure coverage capability
Patient care protocol compliance
Patient outcome
Defibrillation availability
Extrication capability
Employee illness and injury
Employee turnover
Quality program
PRACTICALITIES OF
STANDARDS AND
PERFORMANCE INDICATORS
System user opinion
Multi-casualty event response plan
2
9
tors (Table 5) and a three-part
pilot performance measurement
*
These performance indicators are currently
being tested in cities across the United States.
Adapted from: Moore L. Prehospital EMS
System Performance Measures: Quality Indicat-
ors. Washington, DC: International Association
of Fire Fighters, 2001.
3
0
Until recently, EMS physicians and
administrators usually set EMS
standards, often by consensus
without the benefit of research or
instrument.
Each indicator is
defined, measurement methods are
discussed, existing standards are
referenced, and EMS system goals
are proposed. The IAFF perform-
ance measurement tool is currently
being tested in more than 250 U.S.
cities to assess the EMS perform-
ance goals contained in the recent-
ly approved NFPA document
(NFPA 1710) on standards for the
organization and deployment of
fire suppression operations, emer-
gency medical operations, and spe-
cial operations to the public by
2
5
publication NFPA 1720 set mini-
mum criteria for response times,
staffing, and quality assurance for
all paid and volunteer fire depart-
ments. A companion NFPA 1221
publication sets dispatch stan-
dards, recommending, for exam-
ple, that calls be answered at the
public service answering point
within 30 seconds and subsequent-
ly dispatched within 60 seconds.
Finally, the Commission of Fire
Accreditation is assessing the feasi-
bility of establishing a commission
whose purpose would be to
validation of indicators. Several
nongovernmental organizations,
including the American Society for
Testing and Materials, the
Commission on Accreditation of
Ambulance Services, and the Joint
Review Committee on Education
Programs for Emergency Medical
Technicians and Paramedics, have
developed voluntary standards.
The impact of these voluntary stan-
dards has been limited largely
because of a lack of validation.
3
1
career fire departments.
The
NFPA 1710 document and its sister
TABLE 4. Terms Commonly Used in Assessment
accredit
fire-department-based
of Emergency Medical Services Systems
EMS systems.
Structure—measure that assesses whether organizational resources and arrangements are in
place to deliver health care, such as the number, type, and distribution of medical person-
nel, equipment, and facilities; the interrelated components forming a system.
Nowhere is the task of measur-
ing quality more difficult than in
creating valid structural indicators
of an EMS provider’s clinical com-
petency. In 1998, the Milwaukee
County EMS system described
minimum biennial paramedic per-
formance standards using counts
of patient contacts, runs as team
leader, endotracheal intubations,
defibrillations, intravenous line ini-
tiations, and medications adminis-
Process—interrelated series of events, activities, actions, mechanisms, or steps that trans-
form inputs into outputs; the system of task, workflow, information flow, and other inter-
dependencies that produce some specific outputs or results; how work is done, how out-
puts or results are achieved, and how value is provided to the business or customer.
Outcome—measure that indicates the result of the performance (or nonperformance) of a
function(s) or process(es).
Benchmarking—structured method of measuring processes and products against each other
to identify and understand the characteristics of superior performance (“best practices”).
Customer—recipient or beneficiary of the output of work effort or purchaser of products;
can be either internal or external to the organization and must be satisfied with the outputs
of the work effort.
3
2
tered. In Pittsburgh, EMS admin-
istrators found it valuable to ask
paramedics to select quality indica-
Performance indicator—measurable characteristic of a product, service, or process that best
represents quality and customer satisfaction.
3
3
tors. In addition to traditional
indicators of performance, such as
success rates of procedures, proto-
Practice—technique or method used to perform a step in a particular process.
Standard—quantitative marker for comparing acceptable performance.

96
PREHOSPITAL EMERGENCY CARE JANUARY / MARCH 2002 VOLUME 6 / NUMBER 1
col deviations, and response times,
the paramedics’ suggestions in-
cluded crew and equipment ap-
pearance and internal satisfaction.
Performance measures in EMS
are likely to improve service deliv-
ery when providers both under-
stand and endorse them. Joyce et
al. demonstrated system-wide
improvements in 13 of 19 perform-
ance indicators when results of
these indicators, including protocol
compliance, time elements of
response, and triage and transport
elements, were routinely circulated
to providers and training was
offered.
dence-based EMS outcomes data
exist.
to service industries must be
adapted to federal, state, and local
EMS needs. At each level, EMS
leaders must begin the process of
defining the structural, process,
and outcome performance meas-
ures critical to their organizations.
They should consider what the
patients value and create methods
to ensure that goals are being met.
Each performance indicator must
be linked to an essential aspect of
excellence, and each EMS provider
contributing to that indicator must
understand its significance. Those
responsible for administering EMS
systems must ensure the budget
needs of the information systems
and personnel required to support
this process. As part of the CQI
cycle, the results of initial research
efforts must be effectively shared
with providers and the public. The
ultimate result of defining valid
EMS performance measurements
today will be the ability to demon-
strate the value and quality of EMS
care tomorrow.
3
7
For the CQI process to work,
data must be reliable, uniform, and
3
8–40
accurate.
A number of EMS
systems have demonstrated the
value of integrated computer-
assisted quality assurance pro-
grams in performance evalua-
3
4
4
1–43
tions.
Computer analysis can
help personnel objectively focus on
deficiencies and strengths, as well
as compare and suggest improve-
ments in specific performance indi-
4
4
cators. Computerized data collec-
tion and analysis have also been
extended to evaluation of system
costs and resource allocation for
Traditional public health out-
come indicators measure the “6
Ds”: death, disease, disability, dis-
comfort, dissatisfaction, and desti-
tution. Emergency medical servic-
es systems have infrequently used
all these outcomes, instead focus-
ing most often on survival.
Unfortunately, other than survival
4
1
quality improvement programs.
CONCLUSION
Measuring the quality of care and
implementing processes to im-
prove this care have become a top
priority for many EMS systems
across the United States. The
accepted quality management
tools that have long been integral
3
5
after sudden cardiac arrest or
3
6
major trauma, little or no evi-

Dunford et al. PERFORMANCE MEASUREMENTS IN EMS
97
CONSENSUS PRESENTATION
ported by multiple paramedics on
each incident.
must be established and accepted.
Variations in EMS systems and
environments must be considered
when setting standards and apply-
ing performance measurements.
Because EMS lacks an oversight
organization such as the JCAHO to
drive the CQI agenda, EMS med-
ical directors, administrators, and
providers must take the initiative
to develop and evaluate their own
performance measurements to
meet the expectations of the com-
munity for quality care.
The first recommendation from
the consensus group was to use the
term performance measurements
rather than performance standards
for EMS systems. There is a pauci-
ty of literature available in this area
to support a choice of standards.
Although some standards have
been published, they lack the
desired evidence-based support
that should be a requirement for
determining current or future stan-
dards. The consensus participants
overwhelmingly agreed that all
EMS systems should be using some
form of performance measure-
ments to set, evaluate, and imple-
ment standards of care. One of the
first steps in beginning this process
is to support EMS data collection
through the establishment of
enabling legislation at the state
level. The participants also sup-
ported the position that such legis-
lation should provide confidential-
ity for CQI review to protect these
data and allow the process to work
without fear of liability.
Performance measures are also
affected by changes in structural
variables. For example, first-
responder defibrillation capability
is an important structural compo-
nent, but the widespread imple-
mentation of public-access defibril-
lation programs might change that
someday. A challenge to EMS
medical directors in the era of cost–
effectiveness is ensuring that per-
formance
measurements
are
grounded by sound clinical deci-
sions. The consensus group also
reinforced the need to consistently
apply appropriate statistical meth-
ods to all CQI activities.
References
1. Committee on Quality of Health Care in
America, Institute of Medicine. Crossing
the Quality Chasm: A New Health
System for the 21st Century. Washing-
ton, DC: Institute of Medicine, 2001.
Available at: http://books.nap.edu/cat-
alog/10027.html?onpi_newsdoc030101.
CATEGORIES OF PERFORMANCE
MEASUREMENTS
2. Kohn LT, Corrigan JM, Donaldson MS
The consensus participants dis-
cussed several areas in which per-
formance measurements should be
established; these included regula-
tion and policy standards, resource
management, human resources
and training, transportation, facili-
ties, communications, medical
direction, trauma systems, and
evaluation. Within each of these
categories, numerous areas were
identified that could be targeted
for specific performance measure-
ments, such as dispatch intervals,
peripheral intravenous success
rates, response times for adult car-
diac arrest, ambulance diversion,
preventable deaths, chart docu-
mentation, and patient satisfaction.
(
eds). Committee on Quality of
Healthcare in America. Institute of
Medicine. Washington, DC: National
Academy Press, 1999.
3. Chassin M, Galvin R. Statement on
Quality of Care. National Roundtable on
Health Care Quality. September 16,
1
998. Washington, DC: Institute of
DEVELOPING PERFORMANCE
MEASUREMENTS
Medicine. Available at: http://www.
nap.edu/catalog/9439.html.
4
. The President’s Advisory Commission
on Consumer Protection and Quality in
the Health Care Industry. Quality First:
Better Health Care for All Americans;
Final Report to the President of the
United States. Washington, DC: U.S.
Government Printing Office, March
1998.
Although the goal of choosing,
using, and evaluating performance
measurements to achieve bench-
marks and eventually best practice
guidelines is worthwhile, some of
the basic needs of the process are
still missing. Definitions within
EMS often vary widely, affecting
the validity of within-system and
cross-system comparisons of out-
comes. For example, the beginning
and end of the response time inter-
val are not consistent across sys-
tems. Urban, suburban, rural, and
frontier EMS systems each require
specific performance measure-
ments to ensure consistency. The
level of staff training and experi-
ence varies, as some systems are
staffed by basic emergency medical
technicians, while others are sup-
5. Lohr KN (ed). Medicare: A Strategy for
Quality Assurance, Vol. I. Institute of
Medicine. Washington, DC: National
Academy Press, 1990.
6
. Salerno SM, Wrenn KD, Slovis CM.
Monitoring EMS protocol deviations: a
useful quality assurance tool. Ann
Emerg Med. 1991;20:1319-24.
CONCLUSION
Although discussion of specific
performance measurements was
beyond the scope of the consensus
group given the time constraints,
the participants strongly recom-
mended that EMS systems develop
performance measurements. To
provide consistency and allow for
comparisons, uniform definitions
7
. Palumbo L, Kubincanek J, Emerman C,
et al. Performance of a system to deter-
mine EMS dispatch priorities. Am J
Emerg Med. 1996;14:388-90.
8. Slovis CM, Carruth TB, Seitz WJ, et al. A
priority dispatch system for emergency
medical services. Ann Emerg Med.
1985;14:1055-60.
9. Stratton SJ. Triage by emergency med-
ical dispatchers. Prehosp Disaster Med.

98
PREHOSPITAL EMERGENCY CARE JANUARY / MARCH 2002 VOLUME 6 / NUMBER 1
1
992;7:263-9.
0. Deming WE. Out of the Crisis.
Cambridge, MA: Massachusetts
Services (EMS) Systems. Contract
Prehosp Emerg Care. 1998;2:335-6.
33. Greenberg MD, Garrison HG, Delbridge
TR, et al. Quality indicators for out of
hospital emergency medical services:
the paramedics’ perspective. Prehosp
Emerg Care. 1997;1:23-7.
34. Joyce SM, Dutkowski KL, Hynes T.
Efficacy of an EMS quality improvement
program in improving documentation
and performance. Prehosp Emerg
Care.1997;1:140-4.
35. Eisenberg M, Horwood BT, Cummins
RO. Cardiac arrest and resuscitation: a
tale of 29 cities. Ann Emerg Med.
1990;19:179-86.
36. Shackford SR, Mackersie RC, Hoyt DB,
et al. Impact of a trauma system on out-
come of severely injured patients. Arch
Surg. 1987;122:523-7.
37. Callaham M. Quantifying the scanty sci-
ence of prehospital emergency care.
Ann Emerg Med. 1997;30:785-90.
38. Braun O. EMS system performance: the
use of cardiac arrest timelines. Ann
Emerg Med. 1993;22:52-61.
39. Cummins RO, Chamberlain DA,
Abramson NS, et al. Recommended
guidelines for uniform reporting of data
from out of hospital cardiac arrest, the
Utstein style. Ann Emerg Med.
1991;20:861-74.
40. Spaite D, Benoit R, Brown D, et al.
Uniform prehospital data elements and
definitions: a report from the uniform
prehospital emergency medical services
data conference. Ann Emerg Med.
1995;25:525-34.
41. Holliman CJ, Swope G, Mauger L, et al.
Comparison of two systems for quality
assurance of prehospital advanced life
support services. Prehosp Disaster Med.
1993;8:303-10.
42. Stewart RD, Burgman J, Cannon GM,
Paris PM. A computer-assisted quality
assurance system for an emergency
medical service. Ann Emerg Med.
1985;14:25-9.
43. Valenzuela TD, Keeley KT, Criss EA, et
al. Implementation of a computerized
management information system in an
urban fire department. Ann Emerg Med.
1989;18:573-8.
44. Swor RA, Hoelzer M. A computer-
assisted quality assurance audit in a
multiprovider EMS system. Ann Emerg
Med. 1990;19:286-90.
1
DTNH 22-95-C. Washington, DC:
National Highway Traffic Safety
Administration, July 1997. Available at:
http://www.nhtsa.dot.gov/people/inj
ury/ems/leaderguide/index.html.
22. National Highway Traffic Safety Ad-
ministration. Agenda for the Future.
Implementation Guide. (DOT HS 808
711). Washington, DC: U.S. Department
of Transportation, 1998.
23. Bass RR, Conn AK, Dawson D, et al.
State of California: An Assessment of
Emergency Medical Services. Report of
the National Highway Traffic Safety
Administration Technical Assistance
Team, August 23–26, 1999. Available at:
http://www.emsa.ca.gov/vision/nht
sarpt99.asp.
24. California Statewide EMS System Eval-
uation Project. Standards and Guide-
lines for Statewide EMS System
Evaluation. Final Project Report. Special
Project EMS #8039, 1998–2000.
25. Moore L. Measuring quality and effec-
tiveness of prehospital EMS. Prehosp
Emerg Care. 1999;3:325-31.
Institute of Technology, Center for
Advanced Engineering Study, 1986.
1. Dagher M, Lloyd RJ. Developing EMS
quality assessment indicators. Prehosp
Disaster Med. 1992;7:69-74.
2. Polsky SS, Culhane DE, McDowell RM,
et al (eds). Continuous Quality
Improvement in EMS. Dallas, TX:
American College of Emergency
Physicians, 1992.
1
1
1
3. Swor RA, Rottman SI, Pirrallo RG, Davis
E
(eds). Quality Management in
Prehospital Care. St. Louis, MO: C. V.
Mosby, 1993.
1
4. National Highway Traffic Safety
Administration. EMS Outcomes Evalua-
tion: Key Issues and Future Directions.
Proceedings from NHTSA Workshop on
Methodologies
Morbidity Outcomes
Washington, DC: U.S. Department of
Transportation, April 11–12, 1994.
for
Measuring
in EMS.
1
5. Delbridge TR, Bailey B, Chew JL, et al.
EMS agenda for the future: where we
are…where we want to be. Ann Emerg
Med. 1998;31:251-63.
26. Horak B. Strategic Planning in Health-
care: Building a Quality Based Plan
Step-by-Step. New York, NY: Quality
Resources, 1997.
27. Harbour JL. The Basics of Performance
Measurement. New York, NY: Quality
Resources, 1997.
1
6. Spaite DW, Criss EA, Valenzuela TD, et
al. Developing a foundation for the eval-
uation of expanded-scope EMS: a win-
dow of opportunity that cannot be
ignored. Ann Emerg Med. 1997;30:791-6.
7. Maio RF, Garrison HG, Spaite DW, et al.
Emergency Medical Services Outcomes
Project I (EMSOP I): prioritizing condi-
tions for outcomes research. Ann Emerg
Med. 1999;33:423-32.
8. Kessner DM, Kalk CE, Singer J.
Assessing health quality: the case for
tracers. N Engl J Med. 1973;288:189-94.
9. Spaite DW, Maio R, Garrison HG, et al.
Emergency Medical Services Outcomes
Project (EMSOP) II: developing the
foundation and conceptual models for
out-of-hospital outcomes research. Ann
Emerg Med. 2001;37:657-63.
1
28. Clarke RW, Sucher TO. Benchmarking
for the competitive marketplace.
J
Ambulatory Care Manage. 1999;22:72-8.
29. Moore L. Prehospital EMS System
Performance Measures: Quality Indicat-
ors. Washington, DC: International
Association of Fire Fighters, 2001.
1
30. Moore L. Prehospital EMS System
Performance Measures: Measurement
Instrument. Washington, DC: Inter-
national Association of Fire Fighters, 2001.
31. National Fire Protection Association.
Standard for the Organization and
Deployment of Fire Suppression,
Emergency Medical Administration
Operations, and Special Operations to
the Public by Career Fire Departments.
National Fire Protection Association,
Document No. 1710. Available at:
http://www.nfpa.org.
1
2
0. National Highway Traffic Safety
Administration. Agenda for the Future
(DOT HS 808 441). Washington, DC:
U.S. Department of Transportation,
August 1996.
2
1. Eastham J, Champion HR, Bass R, et al
(eds). A Leadership Guide to Quality
32. Pirrallo R. Establishing biennial para-
medic experience benchmarks [letter].
Improvement for Emergency Medical