Full text of DOI:10.1177/01632780022034499

IMPLEMENTATION OF
THE METHADONE
TREATMENT QUALITY
ASSURANCE SYSTEM
Findings From
Evaluation
Ducharme, Luckey
&
the Health Professions
/ March 2000
/
METHADONE TREATMENT
This article describes the development and
large-scale testing of the Methadone Treat-
ment Quality Assurance System (MTQAS).
MTQAS originated as a NIDA (National Insti-
tute on Drug Abuse) funded study to assess the
feasibility and utility of implementing a
performance-based feedback reporting sys-
tem for narcotic addiction treatment pro-
grams. After several years of research to iden-
tify a set of valid, reliable, and clinically
useful indicators of program performance
using patients’ in-treatment outcomes,
MTQAS was tested in a feasibility study with
more than 70 methadone treatment clinics in
seven states. For each of nine quarters, par-
ticipating clinics received a report summariz-
ing their patient outcomes; reports also
included summary data permitting compari-
sons across states, across clinics within a
state, and within a clinic over time. In addi-
tion, MTQAS included a case-mix adjustment
strategy that permitted fair comparisons of
performance among programs with system-
atically different patient caseloads. The struc-
ture and operation of MTQAS, selected analy-
ses of the data, and lessons for future
performance measurement systems are
discussed.
the Feasibility Study
LORI J. DUCHARME
JAMES W. LUCKEY
Westat
AUTHORS’ NOTE: The Methadone Treat-
ment Quality Assurance System Project was
conducted in two phases by the Research Tri-
angle Institute under Contract No. N01DA-
9-8532 from the National Institute on Drug
Abuse (NIDA). Dr. Luckey was project direc-
tor during Phase II of the feasibility study;
Dr. Dorynne Czechowicz was the NIDA
government project officer throughout. The
opinions expressed herein do not necessarily
reflect those of NIDA or any other federal
agency. Contributions made to this project by
Dr. Czechowicz, the members of the advisory
panels, the Center for Substance Abuse Treat-
ment, the National Association of State Alco-
hol and Drug Abuse Directors, and the partici-
pating state methadone authorities and treat-
ment providers are gratefully acknowledged.
EVALUATION & THE HEALTH PROFESSIONS, Vol. 23 No. 1, March 2000 72-90
© 2000 Sage Publications, Inc.
72
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
73
ethadone is one of the most often studied and most highly
regulated medical therapeutics (Molinari, Cooper, &
M
Czechowicz, 1994; Rettig & Yarmolinsky, 1995). Narcotic addiction
treatment programs in the United States are regulated by the Food and
Drug Administration (FDA), the Drug Enforcement Agency, and
State Methadone Authorities. Although the effectiveness of metha-
done in reducing heroin use and improving productive behavior is well
established (Rettig & Yarmolinsky, 1995), questions have been raised
about whether some narcotic addiction treatment programs are pro-
viding adequate treatment, even in light of this unique three-tiered
regulatory oversight system (Ball & Corty, 1988; General Accounting
Office [GAO], 1990b). Questions have also been raised about the
effectiveness of federal and state oversight methods, which focus
largely on compliance with process-based standards. Compliance
with present regulations does not guarantee a minimal or acceptable
level of program quality or patient outcomes (GAO, 1990b). Much
recent attention has been paid to developing potential alternative
approaches to the oversight of narcotic addiction treatment, so that the
structure, process, and performance of treatment facilities could be
systematically measured and compared to a national or state-specific
set of norms or accepted standards.
In 1990, the GAO recommended that the National Institute on Drug
Abuse (NIDA) and the FDA examine the use of performance stan-
dards for narcotic addiction treatment (GAO, 1990a). Subsequently,
NIDA initiated an effort to develop and test a performance-based
reporting and feedback system for methadone treatment programs.
Such a system would serve a quality assurance or monitoring function
for federal, state, and local oversight agencies while also helping pro-
grams improve their treatment services and patient outcomes. This
article reviews the methods and findings of a recent feasibility study of
the Methadone Treatment Quality Assurance System (MTQAS).
The MTQAS project involved the development of a prototype
performance-based feedback and reporting system and a test of the
feasibility of its implementation in a sample of clinics. Specifically,
the project had five goals:
1. Define performance indicators, including patient outcomes, that may
serve as the basis for a performance measurement system;
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

74
Evaluation & the Health Professions / March 2000
2. determine what information must be gathered about patients to fairly
compare treatment outcomes among programs by adjusting for case
mix (i.e., to separate the contribution of patient characteristics from
program performance);
3. investigate operational problems that may arise if such a system were
implemented;
4. determine how performance feedback should be structured so that it
would be of the greatest utility to programs; and
5. examine how performance-based monitoring might be used to modify
the federal regulatory structure for methadone treatment programs.
Due to recent proposals for changes in the federal regulation of
methadone treatment, the purpose, goals, and role of MTQAS are of
equal or greater importance today than when the project was first con-
ceived. A national test of a proposed accreditation-based oversight
system for ensuring quality of care is currently under way with two
major accrediting bodies. Likewise, changes in the health care field,
along with the federal government’s increased focus on measuring
performance and results, reinforce the need for systems that provide
outcomes-based accountability. Notably, in 1993, Congress passed
the Government Performance and Results Act (GPRA), which held all
federal programs accountable for providing concrete evidence of
progress toward their objectives. Consistent with the GPRA require-
ments, the Office of National Drug Control Policy (ONDCP) pub-
lished Performance Measures of Effectiveness: A System for Assess-
ing the Performance of the National Drug Control Strategy,
1998-2007, which outlined ONDCP’s 10-year strategic plan to con-
front drug abuse in the United States and to measure progress toward
its goals (ONDCP, 1998). Included in this plan are measurable targets
specific to the effectiveness and impact of substance abuse treatment.
Thus, the development of MTQAS represented a timely response to
this heightened sensitivity toward assessing and improving the per-
formance of a critical component of the drug abuse treatment field.
MTQAS provided a valuable tool for narcotic addiction treatment
clinics to demonstrate their performance relative to a normative
standard, using reliable and clinically useful measures of in-treatment
outcomes—that is, measures of change in patient drug use and other
behaviors during the course of their treatment episode. Most existing
performance measurement or outcomes monitoring systems for sub-
stance abuse treatment programsrelyonmethodsthat compare patient
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
75
status at admission and discharge; pre-post comparisons are then used
to assess treatment effectiveness and program performance. However,
such measures are inadequate for methadone treatment providers,
where retention in treatment (often for a number of years) is a frequent
and desired outcome. Instead, MTQAS assessed program perfor-
mance by tracking the progress of individual patients at regular inter-
vals subsequent to treatment intake. Patient outcomes were then
aggregated within clinics on a quarterly basis, and clinic-specific
feedback reports were produced that permitted comparisons across
treatment providers on 17 different dimensions. Using these reports,
outliers could be identified and targeted for increased monitoring or
technical assistance.
PROJECT PHASES
MTQAS was conducted in two phases. Phase I (1989-1995) was
devoted to the system’s initial design, which included identification of
measures, a pretest of the measures, and pilot-testing of the proce-
dures. Part 1 of Phase I assessed the reliability and clinical utility of
the instruments in a sample of patients from five methadone treatment
programs. Results of the analyses performed on data from Part 1 dem-
onstrated that overall, test-retest reliability of the items on each instru-
ment was very good (0.703). Review of reliability scores, in conjunc-
tion with clinicians’ assessments of the utility of each item, led to
revisions in the instruments used for a small-scale pilot-test of the
system.
Part 2 of Phase I assessed whether a limited number of clinics (25)
could implement this prototype performance-based system under a
controlled set of conditions. Staff from all of the participating clinics
received extensive training; data collection was monitored via weekly
telephone calls to each clinic by the research team; and participating
clinicians were paid to collect data on patient outcomes for a 3-month
period. This small-scale pilot-test provided evidence that programs
could collect the data necessary for a performance measurement sys-
tem if they were given at least some degree of external support or
supervision. Using data obtained during this phase of the project, the
research team also developed a case-mix adjustment methodology for
use inreportingcomparative program outcomes(Phillips et al., 1995).
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

76
Evaluation & the Health Professions / March 2000
During this lengthy design phase, input was obtained from a broad
range of constituents to guide the development of the system. The pro-
ject’s Internal Advisory Panel included representatives of numerous
federal agencies involved in methadone treatment; a separate External
Advisory Panel included treatment providers, researchers, state offi-
cials, and representatives from several national professional organiza-
tions with an interest in substance abuse treatment policy and services.
At the conclusion of Phase I, a proposal for the further development
and broader implementation of MTQAS in a second phase was un-
animously supported by the two advisory panels and by the clinic
directors who had participated in the Phase I pilot-test. Anticipating
pressure from managed care organizations, treatment providers rec-
ommended that measures of patient satisfaction and health care utili-
zation be added to the various drug use outcomes already included in
the MTQAS instrumentation. Items unnecessary for measuring pro-
gram performance or for case mix-adjustment analyses were elimi-
nated, and streamlined instruments were produced for the Phase II
feasibility study.
Phase II (July 1995 to March 1998) was designed to provide a feasi-
bility test of MTQAS under realistic conditions with a group of
community-based methadone treatment providers. Along with the
noted changes in the instrumentation, the Phase II feasibility test
moved MTQAS to a research-provider partnership model. Participat-
ing providers collected all the necessary data and, in turn, were given
feedback reports each quarter. Unlike Phase I, Phase II did not include
reimbursement to participating programs for their data collection
efforts; instead, programs were asked to adopt MTQAS as part of their
standard record-keeping procedure. This expectation further moved
the project from the research realm to a full demonstration of the sys-
tem’s real-world operation and utility. The specific objectives of the
feasibility test were to
1. determine whether a performance-based system could be implemented
in methadone treatment programs on an ongoing basis and identify any
operational problems with such a system (implementation), and
2. assess whether performance feedback alone, or in combination with
technical assistance, could be used to guide changes in clinic
processes or procedures to enhance the quality of care provided (qual-
ity improvement).
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
77
Eight states were selected and invited to participate in Phase II of
MTQAS. Selection of these states was based on several considera-
tions, including the likelihood of statewide implementation, indica-
tions of support from the treatment provider associations, availability
of staff to handle data processing responsibilities, the size of the
methadone treatment service delivery system, and geographic distri-
bution. Seven states (Arizona, Colorado, Georgia, Massachusetts,
North Carolina, Pennsylvania, and Washington) agreed to participate
in the project. A total of 103 methadone treatment clinics were operat-
ing in these seven states at the start of the project; of these, 90 clinics
initially indicated that they would become involved in the feasibility
study. Overall, 64 clinics (or 71% of the 90 sites indicating initial
interest) actively participated in the MTQAS data collection activities
throughout Phase II. Active participants included both public and pri-
vate (for-profit and not-for-profit) methadone treatment programs.
There were several reasons why the feasibility study relied on a
purposive sample of states rather than a random sample of clinics
nationwide. First, given the important role of state regulations in the
oversight of narcotic addiction treatment programs, it was believed
that states would likely have responsibility for MTQAS or a similar
performance-monitoring system if it were to become permanently
operational. By having states assume full responsibility for day-to-
day oversight of data collection and processing, and by involving state
staff in the distribution and review of clinic feedback reports, Phase II
allowed for a realistic assessment of the operation and utility of
MTQAS. Second, a state-based implementation also controlled for
variations in regulatory and funding patterns that may have contributed
to any systematic differences observed in clinics’structure, operation,
and treatment effectiveness from state to state. Third, implementing
MTQAS on a statewide basis in a small number of states made it feasi-
ble to coordinate the project’s data collection needs with information
available in each state’s management information system (MIS),
thereby reducing burden on clinic staff whenever possible. Finally,
policymakers at the state level were key constituents for MTQAS,
because they saw its potential for assisting them in responding to
increasing pressure to develop and maintain outcomes-based moni-
toring systems for substance abuse services.
The actual data-collection activities were the responsibility of
counselors in each of the participating clinics. Three separate forms
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

78
Evaluation & the Health Professions / March 2000
were used. For each newly admitted patient, staff completed an initial
assessment form requiring about 15 minutes to administer. This form
collected demographic information suitable for inclusion in case
mix-adjustment analyses, as well as baseline behaviors against which
future reassessments could be compared. Counselors subsequently
conducted periodic reassessments every 3 months for a patient’s first
year in treatment and every 6 months thereafter. The periodic reassess-
ments required about 10 to 15 minutes to administer and collected
information on patients’self-reported drug use and other behaviors, as
well as urine test results abstracted from patient charts. The third
instrument was a discharge form that provided information on the date
and reason for discharge, as well as most recent urine test results, for
patients who left treatment during the feasibility study. A brief sum-
mary of the data collection procedures and the variables used in
MTQAS is provided in Table 1.
Data collection and processing during the feasibility test required a
collaborative effort at several levels. Counselors in each of the clinics
conducted patient reassessments and completed paper forms, which
were then forwarded to a central office in each state for processing.
Each quarter, state staff scanned each clinic’s MTQAS forms into a
database, checked for scanning and coding errors, resolved problems
by contacting clinics for clarification, and compiled a data file. States
then forwarded the data files to the research team for analysis. Project
staff built analytic files aggregating data from all seven states (averag-
ing 5,000 patient reassessments each quarter), matched the periodic
reassessment data with admission profiles obtained in previous quar-
ters, and then prepared individualized clinic feedback reports showing
aggregate in-treatment outcomes for patients assessed during the
quarter, along with comparisons to other clinics participating in the
project.
MEASURING VARIATION IN PROGRAM PERFORMANCE
As noted above, the MTQAS project grew out of concerns about the
effectiveness of the federal regulation of methadone treatment and the
apparent variability in quality across methadone treatment programs
(GAO, 1990b). MTQAS included a feedback reporting mechanism to
identify variations in clinic performance. Customized feedback
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
79
TABLE 1
Methadone Treatment Quality Assurance System
Data Collection Procedures and Variables
Instrument
Timing of Data Collection
Variable Measured
Demographics
Initial
At admission
assessment
Use of opiates, cocaine, alcohol, and
other drugs in past 3 months (self-report)
Injection drug use in past 3 months
(self-report)
Physical health status
Mental health status
Hospital stays and emergency room use
(past 3 months)
Employment status
Criminal behavior (past 3 months)
Use of opiates, cocaine, alcohol, and
other drugs in past 3 months (self-report)
Recent use of opiates, cocaine, and
other drugs (urinalysis results)
Physical health status
Periodic
reassessment
Every 3 months for first
year in treatment; every
6 months thereafter
Mental health status
Hospital stays and emergency room use
(past 3 months)
Employment status
Criminal behavior (past 3 months)
Satisfaction with treatment
Retention in treatment (days)
Optional items (at discretion of participat-
ing states): use of benzodiazepines, use
of marijuana, methadone dose level, take-
home privileges, social support/social
networks, HIV risk behaviors
Date of discharge
Status at discharge (successful completion
versus other reasons for discharge)
Recent drug use (last urinalysis result)
Discharge
form
At discharge
reports, distributed to the clinics each quarter, provided comparisons
between clinics within states and across the entire set of MTQAS clin-
ics. Comparisons were also provided within clinics over time, in the
form of graphs depicting quarter-by-quarter results for 17 different
in-treatment outcomes. Because lengthoftime intreatment isstrongly
associated with outcomes, separate analyses were provided for patients
in treatment less than 1 year (shorter-term) and patients in treatment
for more than 1 year (longer-term). Both unadjusted and case mix-
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

80
Evaluation & the Health Professions / March 2000
TABLE 2
Methadone Treatment Quality Assurance System (MTQAS)
Quarterly Feedback Report: Sample Page for Hypothetical Clinic
Case-Mix-Adjusted Outcomes for Your Clinic, Quarter 5 (July-September 1997)
Comparison Group
All MTQAS Clinics
Clinics in Your State
(N = 2,979 patients)
(N = 679 patients)
Information from patient records
Urine test results
Positive for methadone
Negative for opiates
a
a
a
a
Negative for cocaine
a
a
Retained in treatment
Patients’ self-reported information
Self-reported drug use
No heroin use
+
+
a
a
No use of other opiates
No cocaine use
a
Injection drug use
No injection drug use
Satisfaction with treatment
Did not need additional services
a
a
a
a
a
Mostly or always treated fairly by staff
a
a. Indicates an outcome typical of MTQAS clinics, controlling for case-mix.
+ Indicates a significantly better outcome than expected, given the clinic’s case-mix (at p <
.001).
Indicates a significantly worse outcome than expected, given the clinic’s case-mix (at p <
.001).
adjusted outcomes were provided. Tables reporting case mix-adjusted
outcomes illustrated how each clinic’s outcomes compared to a statis-
tically computed “average clinic,” taking patient characteristics
(demographics and status at intake) into account.
Table 2 provides an example page from the MTQAS quarterly feed-
back report, using case mix-adjusted outcomes. The table is intended
to be illustrative, showing outcomes for a hypothetical clinic during
the fifth quarter of the feasibility study. In this portion of the feedback
report, the clinic’s outcomes are compared to all other MTQAS clinics
providing data in the quarter and to all other treatment programs in the
clinic’s state. The outcomes include objective and self-reported
patient information (i.e., in-treatment outcomes). Clinics were com-
pared on (a) the proportion of patients whose urine tests indicated use
of methadone, and abstention from opiates and cocaine, throughout
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
81
the entire reporting quarter; (b) the proportion of patients who were
retained in treatment; (c) the proportion of patients self-reporting
abstinence from heroin, other opiates, cocaine, and injection drug use;
and (d) the proportion of patients reporting satisfaction with the pro-
gram (in terms of services received and fair treatment by staff ).
The quarterly report was designed to allow clinics to identify easily
the particular outcomes on which they were performing above or
below other clinics. As shown in Table 2, clinic ratings were denoted
with a series of symbols to indicate performance that was within the
normative range, significantly better than average, or significantly
worse than average, with appropriate adjustments for case mix. Very
stringent statistical criteria were used to identify outlier clinics
(alpha = .001). Thus, a clinic director reading the report shown in
Table 2 would see that his or her clinic had significantly better-than-
expected outcomes for treatment retention in the quarter and signifi-
cantly worse-than-expected outcomes for heroin and cocaine use. The
director would also note that there was insufficient variation in
patients’assessments of how they were treated by staff to permit com-
parisons with other clinics. (Lack of variation was not an uncommon
occurrence on the treatment satisfaction variables, as patient ratings
were usually highly skewed toward favorable assessments.) On all
other outcomes, this clinic performed on par with other MTQAS
clinics.
During the feasibility test, we found that the data collected through
MTQAS could effectively be used to differentiate between clinics on
various in-treatment outcomes. Each quarter, a small number of clin-
ics had significantly poorer outcomes compared to the group as a
whole. Figure 1 portrays the distribution of all MTQAS-participating
clinics on one outcome (negative urine tests for opiates) for Quarter 5,
with outcomes for shorter-term patients (12 or fewer months in treat-
ment) and longer-term patients (more than 12 months in treatment)
shown separately. For this quarter, 68 clinics provided sufficient data
to receive a report. Of these 68 clinics, 18 reported that less than 40%
of their shorter-term patients tested negative for opiates throughout
the quarter, and another 12 clinics reported fewer than 50% testing
negative. These were the clinics with the poorer outcomes. Among the
best-performing clinics, 3 reported over 90% of their shorter-term
patients testing negative for opiates, and another 11 had more than
80% testing negative throughout the quarter.
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

82
Evaluation & the Health Professions / March 2000
Figure 1: Variation in Outcomes Across Clinics in Quarter 5 (N = 69): Negative Urine
Tests for Opiates by Time in Treatment
Variability in clinic outcomes was also observed with patients in
treatment for more than 1 year. As is typical, longer-term patients had
better outcomes: only 4 clinics had fewer than 50% of their longer-
term patients consistently testing negative, whereas 31 clinics had at
least 81% of their longer-term patients test negative throughout the
quarter. This type of analysis is another example of how outlier clinics
could be identified for each of the in-treatment outcomes measured in
MTQAS.
Identification of outlier clinics was an important component of the
MTQAS methodology. The measures were sensitive enough to iden-
tify true outliers on any given outcome, as well as to detect significant
changes in clinic performance over time. Throughout the feasibility
test, a single clinic or set of clinics was rarely identified as a poor per-
former on more than one or two outcomes in any single quarter. For
example, in Quarter 5, four clinics had poorer-than-average outcomes
for opiate use (measured using urinalysis data), three had poor out-
comes on self-reported opiate use, four had poor outcomes for cocaine
use (urinalysis data), and four had poor outcomes on self-reported
cocaine use. Fourteen different clinics were represented in these 15
poor-performance scores for the quarter. Thus, the MTQAS method-
ology was able to differentiate performance on a number of independ-
ent dimensions, thereby providing indications of where improve-
ments were most needed in a given clinic. By tracking outcomes over
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
83
Figure 2: Variation of Outcomes Across States: Negative Urine Tests for Opiates (Quar-
ter 5) by Time in Treatment
NOTE: MTQAS = Methadone Treatment Quality Assurance System.
time, MTQAS was able to identify dimensions on which certain pro-
grams were consistently performing above or below average.
MTQAS allowed for the identification of poor-performing clinics and
the nature of their specific weaknesses; with this information, partici-
pating states could then target these clinics for technical assistance,
closer monitoring, or other strategies aimed at improving the quality
of care provided.
Because MTQAS was designed as a state-based system, it was also
possible to assess variations in clinic performance across states. This
was an important feature of the system, given the project’s goal of pro-
viding a potential alternative approach to federal oversight of narcotic
addiction treatment programs. Analyses of the data indicated notable
differences in outcomes across states, and MTQAS was effective in
identifying this variation. For example, Quarter 5 data showed sub-
stantial variation across each of the seven states on average clinic
outcomes for opiate use as measured by urinalysis results. These
state-by-state comparisons are portrayed in Figure 2. (States are not
identified.)
As shown, clinics in State B had the lowest percentage of patients
testing negative for opiates (averages of 35% of shorter-term patients
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

84
Evaluation & the Health Professions / March 2000
and 64% of longer-term patients), whereas clinics in State E had the
highest overall outcomes (averages of 75% of shorter-term patients
and 86% of longer-term patients). Similar variation among states was
found for other outcomes. The ability of MTQAS to detect such differ-
ences is important because state regulatory and funding environments
can vary greatly, with important influences on the organization and
delivery of treatment services at the clinic level. Likewise, there may
be important geographic variations in employment, drug availability,
and other potential correlates of patient outcomes. Because MTQAS
incorporated a state-based design and a standard set of measures, such
interstate comparisons are possible. The ability of the system to differ-
entiate among states contributed to its value as a performance-
measurement system for narcotic addiction treatment programs.
A particular advantage of MTQAS is that it defined a standardized
set of procedures for measuring and analyzing in-treatment outcomes.
If clinics share a common methodology for collecting and recording
data, this information can be aggregated across clinics within a state,
as well as across states. Thus, MTQAS is consistent with ongoing
efforts to build national data systems that permit interstate compari-
sons of treatment effectiveness. In addition, the feedback reporting
process provided individual clinics a means of incorporating ongoing
analysis of patient outcomes into their continuous quality improve-
ment (CQI) activities. Finally, the feedback reports gave clinics and
states concrete evidence of clinic performance relative to other clinics.
In this way, the MTQAS system helped to identify performance norms
for this important segment of the health care field.
PARTICIPANTS’ ASSESSMENT OF MTQAS
Clinic directors, counselors, and state regulatory agencies were the
key stakeholders in MTQAS. Counselors had primary responsibility
for collecting MTQAS data from their patients at regular (3- and
6-month) intervals. State staff assumed responsibility for data pro-
cessing. Clinic directors, program staff, and state regulatory staff were
the primary audiences for the quarterly feedback reports. Each of
these groups responded positively to their participation in the
MTQAS feasibility study.
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
85
State staff notedseveral benefits from theirparticipationinthe proj-
ect. Specifically, MTQAS allowed them to (a) establish collaborative
working relationships with clinics statewide, (b) increase and improve
regular communication with clinics, (c) begin obtaining data from pri-
vate clinics, (d) respond to the information needs of key stakeholders
(e.g., state legislatures), and (e) demonstrate the effectiveness of
methadone treatment. Several states also noted that the MTQAS proj-
ect represented one of the first attempts to gather statewide outcome
data for a segment of their substance abuse treatment services system;
MTQAS thereby provided a model for future efforts with other sub-
stance abuse treatment modalities or other human services. MTQAS
also represented the first time that many clinics had been asked to col-
lect in-treatment outcomes on their patients. Whereas substance abuse
treatment programs have traditionally relied on patient data gathered
at admission and discharge, MTQAS was specifically designed to assess
the progress of patients in methadone treatment programs, where
retention in treatment is a desired outcome.
Staff from participatingstates andclinicsalsoprovided a number of
recommendations for improvements to the system. Specifically, clini-
cians requested more interpretive guidance in the quarterly feedback
reports and brief descriptions of how items were scored or analyzed.
In addition, some clinicians suggested that more outcomes (e.g., alco-
hol use, depression, and anxiety) should be added to the feedback
reports. Clinicians also suggested rewording some of the measures,
including the response categories for patients’employment status and
satisfaction with the treatment received. Additional recommendations
for refinements to the system included (a) fully automating the data
collection and processing activities, (b) integrating the MTQAS data
into clinics’administrative databases and state management informa-
tion systems, and (c) adding a component to track patients’status after
discharge. Notably, clinicians rarely called for reductions in the scope
of the system. In fact, most of the suggested refinements stemmed
from clinicians’ genuine interest in learning more about the data and
their desire for informationtofacilitate the use ofthe findings for qual-
ity improvement purposes.
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

86
Evaluation & the Health Professions / March 2000
LESSONS FROM THE FEASIBILITY TEST
For the research team, MTQAS provided lessons about factors that
are critical to the successful implementation of any performance
measurement system, particularly systems designed to be used col-
laboratively by regulatory or funding agencies, substance abuse treat-
ment providers, and researchers. At a strategic level, leadership from
participating states, treatment provider associations, or both, is a pre-
requisite for success. End-user support is essential to establish a sense
of ownership of the data among participating facilities. At the same
time, developers of performance measurement systems must be able
to recognize and respond to the temporal and regional variability in
clinics’ operating environments (including funding, regulatory, and
market conditions) that may affect the implementation and operation
of the system.
Within participating clinics, one or more staff members must be
identified to take responsibility for the day-to-day operation of the
system, and they must have the organizational support necessary (e.g.,
time, resources) for effective oversight. From a technical standpoint,
clinics must find creative ways to integrate data collection into their
ongoing activities; with MTQAS, clinics had to create their own
reminder systems to ensure completion of patient reassessments on
the appropriate schedule. Finally, technical expertise in data manage-
ment at the state level is essential to the effective functioning of a sys-
tem such as MTQAS.
MTQAS provided several examples of the benefits of collaboration
between the research and practice communities. At the same time, the
project highlighted the need for better communication between the
two groups. The primary example of this disconnect was the quarterly
feedback reports. Although the reports were intended to serve as tools
for clinics’CQI efforts, their potential was not always realized among
the clinics participating in the feasibility study. Efforts were made to
produce reports that were user friendly, but clinicians reported diffi-
culty interpreting the data. This difficulty stemmed from a lack of
understanding of how particular measures were scored and computed,
as well as confusion about the case mix-adjustment methodology
underlying key sections of the report. A Guide to the Quarterly Feed-
back Reports, produced early in the project, helped alleviate this prob-
lem somewhat by providing clinicians with a source of information
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
87
about how the reports were generated. But for MTQAS to function as
a quality improvement tool, clinicians had to not only understand the
reports, but translate the findings into action. Feedback received from
participating clinics indicated that more focused technical assistance
may be needed to achieve this goal.
Perhaps the most important lesson learned from the feasibility test
was the value of a central data clearinghouse. The strength of MTQAS,
and a necessity in any performance measurement system, was its abil-
ity to provide comparative data for all participating clinics. In the fea-
sibility test, clinics submitted paper forms to a central state office; the
states in turn scanned the forms into a data file, which was forwarded
to the research team. The project staff then aggregated data from all
seven states and built quarterly data files within which individual
patients could be linked over time. This central data clearinghouse
was essential for producing interstate data comparisons, and it pro-
vided a sizable set of patient and clinic records against which to com-
pare participating programs.
The data clearinghouse was funded by NIDA (and, for two addi-
tional quarters, by the Center for Substance Abuse Treatment) for the
duration of the feasibility study. With the end of the project, the cen-
tralized data analysis capabilities were disbanded. Without a data
clearinghouse, data analysis and reporting must be undertaken by
oversight agencies within states. However, this likely would limit the
comparative base to only those clinics under the purview of a given
state. Massachusetts has developed its own MTQAS-like system to
monitor the performance of its clinics, and it continues to provide
quarterly feedback reports (including case mix-adjusted compari-
sons) to clinics throughout the state. However, many other states lack
the personnel and analytic resources necessary to implement and
maintain such a system.
At present, the narcotic addiction treatment field does not have a
national performance monitoring system. Three current movements
in the field may build on the features of MTQAS or benefit from the
lessons of the feasibility test. Efforts are under way at the federal level
to implement an accreditation-based oversight system for methadone
treatment programs. Because accreditation standards require the col-
lection and use of patient outcome data for performance measure-
ment, an accreditation system could provide the impetus for the devel-
opment of national or state databases for benchmarking purposes. The
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

88
Evaluation & the Health Professions / March 2000
Center for Substance Abuse Treatment has encouraged the incorpora-
tion of MTQAS data elements into commercially available computer
software packages for patient record-keeping, but a central data
repository does not now exist. The development and maintenance of a
central data clearinghouse as part of such a system would greatly
enhance its market appeal. Finally, there is continued state and federal
interest in national substance abuse treatment-outcome studies and
the development of standardized data collection systems permitting
interstate comparability. Such interest is consistent with the clearing-
house concept that was so central to MTQAS.
CONCLUSIONS
The MTQAS project was successful in achieving all five of its
intended goals. During the initial design phase, a reliable, valid, and
clinically useful set of performance indicators was defined and tested
(Goal 1). A case mix-adjustment methodology was developed and
tested to ensure that apparent between-clinic differences were attrib-
utable to treatment programming, and not to the characteristics of
patients served; this methodology permitted fair comparisons of clinic
performance (Goal 2). Activities undertaken during the feasibility test
demonstrated that it was possible to implement a performance mea-
surement system in real-world, community-based methadone treat-
ment programs and identified a number of conditions necessary for
successful implementation (Goal 3). Feedback reports were devel-
oped and distributed each quarter, allowing each clinic to compare its
outcomes to all other MTQAS clinics and to itself over time (Goal 4).
Finally, as efforts at the federal level continue to move toward an
accreditation-based oversight system for methadone treatment,
MTQAS has become recognized as a useful tool for meeting accredi-
tation standards pertaining to performance measurement and CQI
(Goal 5).
The MTQAS project achieved its goals largely through collabora-
tion among federal agencies and through a partnership between the
research and practice communities. Throughout the development and
testing of MTQAS, input was sought from a diverse set of constituents
including researchers, representatives from federal and state agencies
involved in methadone treatment, national professional organizations,
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

Ducharme, Luckey / METHADONE TREATMENT
89
and, most notably, the treatment providers who were the intended
users of the system. Developing the final design for MTQAS was an
iterative process that involved obtaining feedback and recommenda-
tions from these groups at each stage. As a result of this process,
MTQAS in able to respond to the needs of states and clinics and can
serve as a model for future developments in the field of performance
measurement.
The NIDA-funded MTQAS feasibility test concluded in mid-1998.
Since that time, several of the participating states and clinics have
adapted or adopted it for permanent use. Indeed, the strongest evi-
dence of the value of MTQAS is the degree of interest shown by the
narcotic addiction treatment field. At present, Massachusetts has per-
manently incorporated MTQAS into its substance-abuse-treatment
outcome monitoring system and has required the participation of its
methadone treatment programs via amendments to its regulations.
The state of North Carolina has incorporated the MTQAS measures
into its central data system. And the state of Georgia is planning to
continue the feedback-reporting process with all of its clinics. In addi-
tion, the research team has responded to numerous requests for infor-
mation about MTQAS from treatment providers and researchers,
located primarily in the United States but also in Canada, New Zea-
land, and the United Kingdom. At present, the Center for Substance
Abuse Treatment continues to explore ways to provide information
and technical assistance to states and clinics wishing to implement the
MTQAS methodology.
In addition to achieving the specific goals of the feasibility test,
MTQAS offers additional realized and potential benefits to the field
that extend well beyond the scope of the project. Through the use of
in-treatment outcomes, MTQAS responds to the unique structure of
methadone treatment programs, where retention in treatment is a
desired outcome. As a result, clinics using MTQAS have access to
information that other outcomes monitoring systems often set aside in
favor of pre-post treatment designs. In addition, MTQAS provides a
mechanism for differentiating between clinics using a standard set of
measures and accounting for differences in patient severity; as such, it
serves a valuable quality-assurance function for states and other agen-
cies. As a proven system for measuring and monitoring outcomes of
methadone treatment programs, MTQAS could be useful for justify-
ing the benefits of treatment dollars spent, achieving compliance with
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015

90
Evaluation & the Health Professions / March 2000
state licensure or other requirements, and developing a set of norms
for treatment outcomes. Finally, because it is consistent with several
key accreditation standards and goals, MTQAS can be a valuable tool
for improving the quality of care in narcotics addiction treatment pro-
grams nationwide.
REFERENCES
Ball, J. C., & Corty, E. (1988). Basic issues pertaining to the effectiveness of methadone mainte-
nance treatment. In C. G. Leukefeld & F. M. Tims (Eds.), Compulsory treatment of drug
abuse: Research and clinical practice (Research Monograph, 86, DHHS Publication No.
ADM 88-1578, pp. 178-191). Rockville, MD: National Institute on Drug Abuse.
General Accounting Office. (1990a). Drug abuse research on treatment may not address current
needs (GAO/HRD-90-114). Washington, DC: Author.
General Accounting Office. (1990b). Methadone maintenance: Some treatment programs are
not effective, greater federal oversight needed (GAO/HRD-90-104). Washington, DC:
Author.
Molinari, S. P., Cooper, J. R., & Czechowicz, D. (1994). Federal regulation of clinical practice in
narcotic addiction treatment: Purpose, status, and alternatives. Journal of Law, Medicine,
and Ethics, 22, 231-239.
Office of National Drug Control Policy. (1998). Performance measures of effectiveness: A sys-
tem for assessing the performance of the drug control strategy, 1998-2007. Washington, DC:
Author.
Phillips, C., Hubbard, R., Dunteman, G., Fountain, D., Czechowicz, D., & Cooper, J. (1995).
Measuring program performance in methadone treatment programs using in-treatment out-
comes: An illustration. Journal of Mental Health Administration, 22, 214-225.
Rettig, R. A., & Yarmolinsky, A. (Eds.). (1995). Federal regulation of methadone treatment.
Washington, DC: National Academy Press.
Downloaded from ehp.sagepub.com at SIMON FRASER LIBRARY on June 5, 2015