Full text of DOI:10.1177/109442802237117

1OA0Ri.m1G1aAn7-7NS/1ImZ0iA9th4T4eI2Ot8aN0l.A2/2LC3R7O1EN1SD7EUACRTCIHNGMSETTUHDOIDES OF DECISION MAKING
METHODOLOGICAL RESOURCES
Conducting Studies of Decision
Making in Organizational Contexts:
A Tutorial for Policy-Capturing and
Other Regression-Based Techniques
LYNDA AIMAN-SMITH
STEVEN E. SCULLEN
STEVE H. BARR
North Carolina State University
Policy-capturing, conjoint analysis, and related techniques are all regression-
based methods used in various areas of organizational research to determine the
importance people attach to cues when they make decisions. Despite the wide-
spread use of those methods, the organizational research literature lacks an inte-
grated tutorial for researchers who are interested in studying decision making but
who have little experience with the techniques required for this type of research.
The authors use empirical findings and suggestions from a number of sources to
offer a step-by-step tutorial covering the effective design, execution, analysis, in-
terpretation, and reporting of policy-capturing studies.
Many organizational studies scholars engage in decision-making or judgment task
research. Researchers in marketing seek to understand consumers’ purchasing deci-
sions. Strategy researchers are interested in the factors that affect executives’decisions
about strategic initiatives for their firms. Organization behavior/human resources
(OB/HR) researchers study personal and organizational characteristics that influence
personnel selection, job choice, turnover, and other aspects of the employment rela-
tionship. In each case, the researcher’s goal is the same—to determine which pieces of
information are most influential in determining decisions.
Given the similarity of these goals, it is not surprising that scholars from different
disciplines use similar methodologies to conduct this type of decision-making
research. Each of the organizational disciplines, and others outside the realm of orga-
nizational studies, uses a regression-based method for studying decision making. The
terms that refer to this same basic methodology, however, vary across disciplines. In
marketing it is called conjoint analysis, stated preference models, or trade-off analysis
(e.g., Cattin & Wittink, 1982; Green & Srinivasan, 1990). Researchers on problem-
solving with computer and information systems use the term protocol analysis
Organizational Research Methods, Vol. 5 No. 4, October 2002 388-414
DOI: 10.1177/109442802237117
© 2002 Sage Publications
388

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 389
(Mackay, Barr, & Kletke, 1992). In environmental and social policy research, the
terms contingent preference, contingent choice, or analytical hierarchical process
models are often used (e.g., Adamowicz, Louviere, & Williams, 1994; MacKenzie,
1993). Strategy and OB/HR researchers generally use the term policy-capturing (e.g.,
Barr & Hitt, 1986; Rynes & Lawler, 1983)—the specific technique we discuss in this
tutorial. Despite the differences in terminology, actual differences in the techniques
are minimal. Priem and Harrison (1994) have given a general descriptive comparison
of several of these research approaches.
In this article, we address two basic problems in regression-based methods of orga-
nizational decision-making research. One is that terminology differences have
obscured the methodological similarities across disciplines, thus needlessly limiting
many methodological contributions. For example, many OB/HR researchers who do
policy-capturing research are unaware of how they may be informed by conjoint anal-
ysis research—or that conjoint analysis or any of the other related techniques even
exist. Therefore, one goal of this article is simply to draw attention to similarities in
research techniques across organizational disciplines and the resulting potential for
researchers in any of those disciplines to benefit from methodological advances in
other areas.
The primary problem we address is that researchers who are not yet experienced in,
but who consider doing, policy-capturing research do not have convenient access to a
single, simple source that addresses specifically how to deal with the range of issues
they are likely to encounter. Published articles offer little help because they tend to
focus on problems relevant to their specific research projects. These articles usually do
not discuss in detail the general design, execution, and analysis issues the researchers
faced or how and why the issues were resolved in a particular way. Thus, the would-be
policy-capturing researcher must wade through volumes of work (e.g., Carroll &
Johnson, 1990; Cooksey, 1996; Stewart, 1988), much of it rather long and/or technical.
This assumes, of course, that the researcher is aware of these sources. As indicated
above, much of this work is published with titles or keywords that may not be familiar
to most researchers outside the authors’specific discipline. These factors are likely to
frustrate researchers, especially the inexperienced, and might even discourage them
from doing this type of research at all.
In light of this, our main goal in this article is to provide, in a source readily avail-
able to organizational researchers, a condensed “how-to” guide—a primer or global
tutorial—that addresses design, execution, and analysis in policy-capturing and
related types of research. Space prohibits an attempt to provide a complete guide for all
types of decision-making research. Instead, we draw from published research and from
our own experience and the experiences of other researchers to make specific recom-
mendations to help identify and deal with key issues in each step of a policy-capturing
study. We include a flow sheet, a sequential process checklist of choices, to facilitate
the process (see Appendix). For readers wanting more detail and for those seeking
more advanced treatments of a particular topic, we suggest a number of supplementary
readings. Where appropriate, we point out differences of opinion in terms of what con-
stitutes best practice.
In sum, this article serves as a guide to what methodological research has shown to
be effective practice in policy-capturing studies. It is directed primarily toward readers
who are familiar with research methods and statistical analyses in general but have
limited expertise in policy-capturing research. Although the article is not targeted to

390 ORGANIZATIONAL RESEARCH METHODS
experts in policy-capturing research, we hope they will also find it useful as a refresher
or as a general reference.
Before proceeding, we note that we use the term policy-capturing in this article. We
emphasize, however, that the material we present here applies to conjoint analysis and
the other regression-based decision-making methods mentioned earlier. We begin
with a brief overview of policy-capturing and its place in the bigger picture of deci-
sion-making research methodologies. We then examine the issues and problems and
provide recommendations regarding the design, execution, analysis, and interpreta-
tion of a policy-capturing study.
Overview: History and
General Nature of Policy-Capturing
We note that policy-capturing and related methods are a subset of the different types
of decision-making research being done today. Readers can see Highhouse (2001) for
a discussion of the larger realm of modern decision-making research techniques.
Policy-capturing is a regression-based decision-capturing methodology in which
participants are asked to make decisions in response to a series of decision or problem-
solving scenarios presented by the researcher. The researcher regresses the decision
outcomes on the values of one or more cues embedded in the scenarios and then uses
the resulting regression weights to make inferences about the respondents’ judgment
policies.
Policy-capturing evolved from Egon Brunswik’s probabilistic “lens” model,
wherein respondents use proximal cues from the environment and interpret them to
develop judgements about physical or social reality (Cooksey, 1996; Hammond,
1966). In this model of individual perception, the process of cause and effect is
described as a lens. The environment scatters information; the individual takes note of
only a limited amount of that information and then recombines that information to pro-
ject a view of the environment. Brunswik’s ideas moved into research on human judge-
ment in the 1950s (Brehmer & Brehmer, 1988). In the following decades, a flurry of
work on probabilistic learning, research on modeling human judgement with linear
multipleregression, and socialpsychology ledto numerous judgment analysis studies.
Policy-capturing is an example of what Carroll and Johnson (1990) called
weighted-additive or input-output models. Weighted-additive models express deci-
sion outcomes as mathematical functions of situational attributes. Weighted-additive
models differ from process models, which concern the underlying psychological theo-
ries of decision making and may seek ways to improve the decision process. Weighted-
additive models of policy-capturing and related techniques, like traditional experi-
mental manipulations, are designed to explain existing judgments and to predict future
decision outcomes. Traditional experimental manipulations and policy-capturing
have their own relative strengths and weaknesses.
Primary strengths of policy-capturing relative to traditional experimental manipu-
lations include the use of systematic sampling of stimuli and control over confounding
(Carroll & Johnson, 1990; McGrath, 1982). By presenting decision makers with
appropriately sampled stimuli (discussed later), the researcher can focus on particular
stimuli or combinations of stimuli and, by repeating those stimuli, assess the reliability
of the data. Policy-capturing studies also provide a relatively high degree of control
over confounding (i.e., have the ability to rule out competing explanations for results).

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 391
They also provide a reasonable level of precision of measurement and, depending on
the specific design used, generalizability of results from sample to population (Carroll &
Johnson, 1990; McGrath, 1982). The use of within-subjects designs results in some
specific statistical advantages, which we outline in a later section.
Probably the most significant weakness of policy-capturing designs is the potential
lack of realism. This includes the deficiencies and the artificiality associated with
using hypothetical situations and “paper” people (Gorman, Clover, & Doherty, 1978;
Hobson & Gibson, 1983). That is, policy-capturing studies do not and cannot simulate
all of the richness of the social and environmental contexts in which real-life decisions
are made; nor can they provide respondents with all of the information that actual deci-
sions makers would have at their disposal. Policy-capturing methodology is artificial
in the sense that real-life decision-making situations often involve one decision made
after consideration of a single set of conditions, not a large number of decisions made
in a short period of time under constantly changing sets of conditions. Similarly, real
decision makers typically do not rely on information handed to them in a set of short
scenarios. Managerial decisions usually are affected by contextual factors present in
ongoing decision processes.
Thus, policy-capturing is like all other research methods in that its strengths and
weaknesses make it well suited for some purposes and less so for others (McGrath,
1982). Selecting one method over another involves, in part, a comparison of those
methods’ strengths relative to the nature of the research question to be answered.
Experimental manipulations, for example, might be superior to policy-capturing in
terms of placing a decision maker in a particular context, one in which information
becomes available in more natural ways (e.g., through experiences, observation, and
interaction with others). On the other hand, gains made through experimental manipu-
lation are offset by losses, such as limitations in the number of decisions that can be
made and uncertainty as to how situational factors may have influenced the decision
maker.
In this article, we assume that the researcher has compared the potential choices of
methodology and determined that policy-capturing is the method of choice. In the
upcoming sections, we discuss steps to maximize the strengths and minimize the
weaknesses inherent in each phase of a policy-capturing research study. We give more
attention to areas that are relatively ambiguous and/or likely to be problematic, such as
study design, and less to those that are already familiar to most organizational
researchers, such as interpreting regression results. Below, we discuss each of the five
steps in a policy-capturing study: study design, study execution, analysis, interpreting
results, and reporting results.
Step 1: Study Design
Three design issues should be addressed early in policy-capturing research. The
first is the nature of the research question—whether the research is intended to exam-
ine individual decision outcomes (i.e., the outcomes of decisions made by specific
individuals) or general tendencies in outcomes aggregated over many decision mak-
ers. The second issue is representativeness—the selection of appropriate subjects and
the design of appropriate scenarios and measures. The third issue is experimental
design, including techniques for determining the required numbers of respondents and
scenarios.

392 ORGANIZATIONAL RESEARCH METHODS
Nature of the Research Question:
Idiographic, Nomothetic, or Both?
Policy-capturing designs can address two related, but distinct, types of research
questions: idiographic and nomothetic. Idiographic research examines individual
decision-making outcomes. That is, how does a particular individual combine multi-
ple pieces of information when making a choice? For example, How does a given
recruiter use information in résumés to decide which candidates he or she will inter-
view? Which types of information influence that recruiter’s choices, and to what
extent? Because idiographic research examines the policies of individual decision
makers, it can be used not only to understand individual tendencies but also to deter-
mine whether there are important differences in the decision policies of individuals
within or between specific groups of interest. As we discuss in more detail later, it can
also be used in a more exploratory manner to determine whether there are identifiable
clusters of individuals with distinct policy types.
Nomothetic questions differ from idiosyncratic questions in that nomothetic ques-
tions focus on the factors that predict the overall tendencies in decisions aggregated
across many decision makers. Although some researchers (e.g., Hobson & Gibson,
1983) argue that policy-capturing is by definition research on individual policies,
much of organizational policy-capturing research seeks to answer more general (i.e.,
nomothetic) research questions. Examples include the following: Do managers tend to
use both job-relevant and job-irrelevant information when evaluating job applicants?
What kinds of applicant information best predict whom recruiters in general will elect
to interview (see Hitt & Barr, 1989; Thoms, McMasters, Roberts, & Dombkowski,
1999)?
We note that it is possible, using advanced techniques such as hierarchical linear
modeling (HLM) (Bryk & Raudenbush, 1992) or SAS PROC MIXED (SAS Institute,
1998), for researchers to design policy-capturing studies that simultaneously assess
both idiographic and nomothetic decision policies. Details of such studies are beyond
the scope of this article, but see Hurt, Maver, and Hofmann (1999) for an example of
policy-capturing research using HLM.
Our first recommendation is that researchers determine explicitly at the earliest
stage of their study design whether their research questions or hypotheses are
idiographic, nomothetic, or both. Idiographic and nomothetic questions are not just
conceptually different; they differ in terms of their data collection and analysis
requirements as well as in the interpretation of results. Specific requirements for each
type of design are examined in more detail in the Experimental Design section below.
Because the two types of questions have different requirements and yield qualitatively
different types of information about decisions, researchers must be sure that their
designs are consistent with the nature of their research questions.
Representativeness
Representativeness is the extent to which the policy-capturing design captures
important elements of real-life judgment situations. To develop highly representative
studies, researchers must use an appropriate sample of respondents and create an envi-
ronment in which the cues, tasks, and decision outcomes are consistent with those in
naturally occurring decision situations. Thus, representativeness concerns focus on

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 393
designing appropriate scenarios, constructing useful measures, and determining
whom the respondents should be.
The first representativeness issue is that the scenarios and cues be reasonably realis-
tic. Making scenarios realistic requires that the most important hypothesized cue vari-
ables be included and that the values of those cues be reasonable. No controlled
research can include all of the variables that may affect real-life decision making but,
at the same time, scenarios that fail to include factors that significantly affect decision
outcomes are likely to yield seriously biased results (Hoffman, 1960). Any generaliza-
tions a researcher makes can relate only to the set of cues used. A researcher designing
a policy-capturing study can use extant literature, qualitative data from interviews,
focus groups, or information from the popular press as guides to determining which
variables, and which of their levels or values, are likely the most important in influenc-
ing decisions. The number of cue variables that can be accommodated is another key
design issue and is discussed in a later section.
Many scholars believe that it is important that the cues take on realistic values.
Their argument is that there is little to be learned from asking people to make judg-
ments about situations that are unlikely to ever arise. Cooksey (1996) suggested sam-
pling the characteristics of real decision situations to ensure that the distributions (i.e.,
means, standard deviations, ranges) of the cue values are realistic. For numeric cue
variables, researchers often use three values—one that is near the true mean, one that is
approximately one standard deviation above the mean, and one that is approximately
one standard deviation below the mean. Cable and Judge (1994) used that method with
data on starting pay of recent engineering graduates to develop high and low salary
cues for a policy-capturing study. Larger deviations can be used to represent greater
magnitudes of cue levels. Larger deviations, however, may not be representative of the
distribution of the cue in organizations, and this larger deviation can create demand
effects.
The researcher should take care to ensure that combinations of cue values in each sce-
nario are realistic. Hair, Anderson, Tatham, and Black (1992) illustrated this with an
example of a scenario in which consumers are asked about the likelihood that they would
buy a car that features both a high-powered engine and outstanding fuel economy—a
combination that does not exist. Hair et al. suggested that scenarios of this sort are
unrealistic and should normally be avoided.
As an example of realistic cues, Zacharakis and Meyer (2000) examined venture
capitalists’funding decisions using disguised but actual venture business cases to cre-
ate scenarios and cues. These researchers included real information about market size,
product superiority and protection, and competition. They asked a consulting expert
venture capitalist to screen out any scenarios containing a combination of cues that did
not seem feasible.
It is worthwhile to note that some authors do advocate the use of unusual combina-
tions of cue values, at least under some circumstances. Carroll and Johnson (1990)
argued that if two or more cue variables are strongly correlated (positively or nega-
tively) in the real world, and if respondents are asked only to judge situations with typi-
cal combinations of cue values, then it will not be possible to determine which of those
variables actually drive decision making. By presenting unusual combinations of val-
ues, the researcher might gain insights that would not otherwise be available.
The next representativeness issue concerns the choice of a metric and a format for
the decision outcome measure. Measures should normally be constructed in the same

394 ORGANIZATIONAL RESEARCH METHODS
Performance Appraisal Diary
Employee: K. Jones
Job title: VTL Operator
Times late for work in past year: 4
Total gears produced to standard: 1,894
Times absent in past year: 5
Total rework: 321
Total scrap: 21
Actively participates on departmental quality improvement team
Anticipates bottlenecks and takes preventive measures
Performance ratings (circle the number that best describes this individual)
Basic work requirements (attendance, punctuality, etc.)
1
Poor
2
3
4
5
6
6
6
6
7
Adequate
Excellent
Results (quantity and quality ofwork produced)
1
Poor
2
3
4
5
7
Adequate
Excellent
Working with others (communication, cooperation, teamwork, etc.)
1
Poor
2
3
4
5
7
Adequate
Excellent
Self-management (working independently, taking initiative, etc.)
1
2
3
4
5
7
Poor
Adequate
Excellent
Overall job performance (total performance considering all aspects of job)
1
2
3
4
5
6
7
Poor
Adequate
Excellent
Overtime work and extra pay. Overtime pays 150% ofan individual’s regular wage. Because
this is a chance to earn extra money, and because there is not much overtime available,
employees like to have a chance to work overtime. What is the probability you would offer
this individual overtime when it next becomes available?
0
.25
.50
.75
1.0
Not at all
Certain
Merit raises: Our organization gives annual raises based on performance. It is your
responsibility to determine pay raises. The range approved for this year is 0% to 6%.
Circle the percentage pay raise this employee will receive.
0%
1%
2%
3%
4%
5%
6%
Figure 1: Example Scenario Used in a Policy-Capturing Study on Performance Ratings
units of measure and in formats that respondents would use in a naturally occurring
decision situation. For example, if judgments are typically made in terms of the per-
centage likelihood that an event will occur, given a certain set of circumstances, then
the measures should be expressed as percentages. If decisions are normally expressed
in dollars, then judgments should be expressed in dollars. The researcher should also
present respondents with measures that are formatted in a way that is representative of
the actual judgment task. Figure 1 illustrates a measure that was used in an unpub-
lished study by the first author to record judgments in a policy-capturing study involv-
ing performance appraisals. Note that the response format is similar to what an experi-
enced rater might expect to use in a real appraisal situation.
If Likert-type scales are used for the outcome measure, determining the best num-
ber of scale points can be a bit more complex. On one hand, research (Cicchetti,
Showalter, & Tyrer, 1985) suggests that reliability tends to plateau at approximately 5
to 7 scale points. Thus, 5- or 7-point scales can normally be expected to exhibit satis-
factory variability and reliability (see also Nunnally, 1978). On the other hand, Russell
and Bobko (1992) and Russell, Pinto, and Bobko (1991) have shown that using
“coarse” scales (i.e., scales with a small number of points) for decision outcomes can

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 395
have significant effects on R2 values, especially if interaction effects are included in the
model. Russell et al. argued that response scales with only a few points may result in
lost information because they force respondents to “squeeze” their latent dependent
response to fit onto the discrete overt response scale (p. 262). They concluded that it is
important for response scales to contain a sufficient number of options to allow for the
operational measurement of the entire theoretical response domain.
One means for accomplishing this is to determine, from the number of distinct lev-
els of the independent variables, the number of points necessary to represent all combi-
nations of values for the independent variables. This number will depend on the type of
model (additive, multiplicative, etc.) employed. Another possibility is to use a depend-
ent scale format in which respondents place a mark on a line segment. Measuring the
distance from the end of the segment to the mark effectively creates a continuous
dependent measure. See Russell et al. (1991) for more detailed explanations of these
techniques.
Our recommendation is that researchers, and especially those who plan to examine
interaction effects in their policy-capturing studies, adopt one of the techniques
described by Russell et al. (1991) to determine the number of points to be included in
their response scales. A failure to do so can result in a loss of statistical power, due to
compressed variance, and resultant problems with the interpretability of results.
Researchers often use 5- to 7-point Likert-type scales in cases where they do not exam-
ine interaction terms. We note, however, that the Russell et al. results indicate that the
number of dependent variable scale points can affect R2 values in these studies too.
Therefore, researchers are probably best advised to use the same techniques in all pol-
icy-capturing studies. We also note that the appropriate scaling of the dependent vari-
able requires sufficiently detailed descriptions or examples for attaching meanings to
the scale points and careful attention to ensuring that the range and distribution of out-
comes are realistic (Wedell, Parducci, & Lane, 1990).
Although having a representative sample could be seen as a general research
methods issue, we note its importance in studies of decision making. In a well-
designed policy-capturing study, the respondents’ experience level and familiarity
with the judgment task are matched to the group to which the researcher wants to gen-
eralize results. Research on decision support systems shows that novices and experts
approach decision making differently (Hardiman, Dufresne, & Mestre, 1989; Mackay
et al., 1992; Polson & Jeffries, 1982). Shanteau and Stewart (1992) discussed the need
to use expert respondents for tasks requiring expert judgment. For example, if the
research concerns how recruiters use information on résumés, then recruiters should
be used as respondents. Shanteau and Stewart and others (e.g., Barr & Hitt, 1986) have
cautioned against using naïve respondents (usually students) for such tasks and then
extrapolating results to professionals and managers. Of course, if the research con-
cerns what college students consider when deciding whether to accept a job offer, then
college students are appropriate respondents.
In summary, our recommendations with regard to representativeness are that
researchers provide an appropriate sample of respondents with scenarios and decision
options that those respondents will perceive as being realistic representations of actual
decision situations. This requires considering what types of cue variables must be
included, the distributions of values to be allowed, and the combinations of values that
will be allowed in individual scenarios. Issues such as the number of scenarios and the

396 ORGANIZATIONAL RESEARCH METHODS
orthogonality of the cue variables must also be considered before final design deci-
sions aremade. Theseissues are discussedin the next section on experimental design.
Experimental Design
After deciding the nature of the research question, addressing representativeness,
identifying the appropriate type of subjects, and designing appropriate scenarios and
measures, the researcher must address numerous issues in experimental design before
being able to collect policy-capturing data.
Number of Cues and Number of Scenarios. One of the major trade-offs in policy-
capturing research is the number of cue variables to be presented in each scenario ver-
sus the total number of scenarios that respondents will be asked to consider. The desire
for representative richness might tempt the researcher to include in the scenarios all of
the variables that may be relevant, but this can lead to two problems. First, respondents
might be faced with more information than they need or could use in making the deci-
sion. Second, adding cues can quickly make the research design unwieldy.
Research indicates that people’s cognitive limits may be exceeded if the number of
independent variables or cues is more than seven plus or minus two (Brehmer &
Brehmer, 1988; Miller, 1956). Beyond that, studies with more than five cues per sce-
nario may be unrealistic in terms of the number of scenarios that would be required per
respondent. This is especially true if an orthogonal design is used (see below).
Values of Cues. In determining the number of cues to be included in the scenarios,
the researcher must also consider the number of values that will be used for each cue.
There will generally be an inverse relationship between the number of cues and the
number of values per cue. That is, the greater the number of values per cue, the smaller
the number of cues that can be included, and vice versa. Most studies have either two or
three levels of cue values. The values may be categorical or numeric. Some categorical
cues (e.g., gender) are naturally dichotomous. Those with three or more values are
called polychotomous cues. One method for attaching values to numeric cues, using
real data and having values with the mean and standard deviation, was described ear-
lier under the Representativeness section. Others will be discussed in the Step 3: Anal-
ysis section.
Cue Ranges. The distribution of each cue should reflect the distribution that occurs
in the real world. Researchers have noted that when respondents are presented with
scenarios where some variables have narrow ranges and others have wide ranges,
respondents may focus on the variables with the wider ranges (Highhouse, Luong, &
Sarkar-Barney, 1999). Therefore, the ranges should be about the same across all cues
to avoid an induced effect; we suggest using a –1 to 1 SD range. Where possible, the
cues in scenarios should be consistent in the number of levels (e.g., have cues with two
or three levels but do not create scenarios with some cues having two and others having
three). Of course, this will not always be reasonable. Some cues are limited to two val-
ues (e.g., gender), but others may be better represented by using more values. Supple-
menting written information about cues with pictures, bar charts, graphs, or tables
enriches the information presentation. For a thorough discussion of attribute ranges,
readers are referred to Highhouse et al. (1999).

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 397
Demand Effects. Possibly counterbalancing the need to keep the number of cue
variables relatively small is the possibility that respondents will be influenced by
demand characteristics. That is, with a small number of cue variables, respondents
may quickly determine the nature of the research and could possibly react by attempt-
ing either to (a) confirm what they believe the research hypothesis to be, (b) disconfirm
that hypothesis, or (c) create a positive impression of themselves. Shimp, Hyatt, and
Snyder (1991) discussed these possibilities at some length and conclude that the risk of
bias from demand artifacts is often overemphasized. One possible means for reducing
the effects of demand characteristics is to include a few bits of extraneous information
in each scenario (Carroll & Johnson, 1990). Drawing on an earlier example of research
asking, “Which types of information influence that recruiter’s choices, and to what
extent?” the researcher could include extraneous changing bits of information like
hobbies, hometown, high-school attended, musicality, and so on, in each résumé.
Another approach is a posttask assessment, either questionnaire or interview, to deter-
mine respondents’level of “guessing” and to gather more information about the nature
of their choices.
Correlated or Uncorrelated Cues. It is important to consider the correlational rela-
tionships between the cues as well. Most policy-capturing researchers have used
orthogonal designs because they yield the most stable and unambiguous estimates of
the regression coefficients (Karren & Barringer, 2002 [this issue]; Neter, Kutner,
Nachtsheim, & Wasserman, 1996). Ensuring orthogonal cues is usually accomplished
by fully crossing all values of each cue with the values of each of the others. There are
two drawbacks with crossed designs: representativeness and practicality. As we have
noted, many researchers following the work of Brunswick stress representativeness in
the research, and in the real world, variables are usually correlated (e.g., horsepower
and fuel economy or quality and cost). Researchers must weigh the statistical advan-
tages of an orthogonal design against the dangers inherent in presenting respondents
with unrepresentative scenarios. The more realistic the researcher wants the scenarios
to be, the more the researcher must consider using a nonorthogonal design, even
though the interpretation of the resulting regression coefficients will be less straight-
forward. Interpretation issues are addressed in more detail in the Step 4: Interpreting
Results section of this article.
A practical consideration is that a fully crossed design can be unwieldy for the
researcher and exhausting for the respondent. Fully crossing all of the cue variables
results in a situation in which the number of scenarios grows dramatically if either (a)
more cue variables are added to the scenarios or (b) the number of values per cue is
increased. For a study with four cue variables, each with three levels, a completely
crossed design would require 34 = 81 scenarios. Adding a fifth cue variable would
increase the required number of scenarios to 35 = 243. Similarly, if the original design
were changed from three cue variables to four (still with four values each), then the
number of scenarios required would be 44 = 256. In both of these examples, the number
of scenarios required is at least tripled.
Thus, researchers face a critical trade-off between the number of cues per scenario
and the total number of scenarios that will be required. Effective design requires
enough scenarios and cues to yield realistic and stable estimates; having too many sce-
narios will result eventually in respondent boredom or fatigue. These are particularly

398 ORGANIZATIONAL RESEARCH METHODS
important issues for studies in which all respondents are asked to judge all scenarios
(Graham & Cable, 2001). The researcher must determine the optimal combination of
number of cue variables and number of values per variable by considering their effects
on the number of required scenarios for the study at hand.
The Length and the Complexity of the Scenarios. Scenarios should be presented in
readable language (or easily viewable format if pictures or videos are used) and should
be sufficiently comprehensive that respondents will take the task seriously. At the
same time, they should not contain so much information that they take too long to read
or view. A potential problem with longer scenarios is that they may introduce unin-
tended confounds. The researcher must try to strike a balance, with enough informa-
tion to make the scenarios realistic and interesting but not so much as to create an
undue risk of respondent fatigue or boredom. This trade-off may be complicated fur-
ther by the issue of whether extraneous information should be included to reduce
demand effects.
One way to ensure that the scenarios and cues are appropriate is to begin scenario
development by surveying or interviewing a small group of people who have expertise
in the relevant subject matter, then pilot testing scenarios with small samples of
appropriate respondents. Adequate pilot testing should help develop scenarios that
are suitable for use in the study. Figure 1 illustrates a sample scenario, and Cooksey
(1996) has provided several others that researchers could use as aids for designing
their own scenarios.
Clearly, the required number of scenarios increases with the number of cues investi-
gated, but researchers have not reached consensus on how the minimum acceptable
number of scenarios should be determined. Some recommendations are based on a
power analysis framework. Cohen and Cohen (1983) suggested that respondents
should be presented with at least 10 scenarios per cue variable, whereas others (Stewart,
1988; Tabachnick & Fidell, 1996) believe that a scenarios-to-variables ratio of 5:1 is
sufficient.
Stewart (1988) published a series of useful curves and tables and urged researchers
to calculate the number of scenarios and cues based on their task, their respondents, the
maximum acceptable standard error level, and the amount of multicollinearity in their
data. Stewart’s data suggest that a minimum of 25 to 40 scenarios are needed and that
this number could grow to 100 or more if multicollinearity among cues is high (i.e., in
excess of .8 to .9). Cooksey (1996) took a somewhat simpler approach to this question.
He advised researchers to determine the minimum number of scenarios by considering
(a) the maximum number of cues that can be included without creating a cognitive
overload situation for respondents and (b) the required ratios of the number of scenar-
ios to the number of cues. Cooksey argued that unless multicollinearity among cues is
high, a 5:1 ratio of scenarios to cues provides a minimally sufficient number of scenar-
ios. Stewart and Cooksey agreed that too many scenarios is better than too few and that
researchers should be conservative in determining the minimum number of scenarios
that will be acceptable for their research.
Another approach would be to “backward engineer” and take into account fatigue,
boredom, and other factors and determine up front the total number of scenarios that
would be reasonable to have respondents manage. Determining early in the research
design the total number of scenarios that are acceptable is an iterative way to work
backwards to determine if the number and level of cues are appropriate. Doing this

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 399
would force researchers to prioritize the number and cue levels of variables to be stud-
ied in their research.1
Start-Up Effects. Rather than assuming that the regression weight for each cue is
constant across all observations or scenarios for a single subject, our experience sug-
gests there are start-up effects. Subjects will need some start-up opportunity to become
accustomed to reading or viewing and responding to scenarios. In a sense, the respon-
dents are learning how to participate in the research. In prior data collected in a number
of policy-capturing studies using written scenarios, we examined the regressions of
each individual, and noted that the regressions became stable (e.g., the R2 and error
terms became reasonably consistent) after each respondent had read between 8 and 12
scenarios. Therefore, researchers should recognize this start-up learning phenome-
non. Researchers can do a practice trial to acquaint the respondent with the process
(include extra scenarios for this start-up process) or closely examine the data and use
the data only after the regressions have become stable. Techniques to deal with this are
presented in the Step 4: Interpreting Results section.
Minimizing Fatigue or Boredom. Cooksey (1996) argued that it is preferable for
respondents to be able to complete their tasks in a single sitting without being overly
taxed and that the maximum number of written scenarios that most respondents can
judge adequately under those conditions is typically less than 100. Graham and Cable
(2001) compared policy-capturing data from a full and a fractional design. In frac-
tional designs, respondents are each given a subset of the scenarios. Graham and Cable
concluded that fractional designs may represent a viable alternative approach to
answering nomothetic questions. They concluded that the fractional design gave sub-
stantially the same results as did the fully crossed design but was less fatiguing and
resulted in subjects’ being more willing to participate in future research. The draw-
backs of fractional designs are that individual equations are much less stable (hence
not appropriate for idiographic questions), more respondents are needed, and it is diffi-
cult to draw conclusions about higher order interactions. See Neter, Wasserman, and
Kutner (1990) for a comprehensive discussion of fractional designs.
Number of Respondents Required. Hobson and Gibson (1983) argued that the ratio
of scenarios to raters is often insufficient in policy-capturing research and that this
insufficiency can lead to unstable estimates of regression coefficients with large stan-
dard errors. We point out that the number of respondents required depends on the
research question. If it is idiographic, then the number of scenarios per respondent is
more important than the number of respondents. In fact, meaningful idiographic
research can be done with only one respondent. For such studies, the number of sce-
narios reviewed by each respondent must be sufficient to yield stable estimates of
regression coefficients for that individual. If the research question is nomothetic,
which is often the case in organizational research, then the number of judges is
always important.
For nomothetic research, power analysis procedures (Cohen, 1988) can be used to
determine the number of judges required to reach a desired level of power for a specific
statistical test. As Howell (1992) observed, each respondent in this type of study
makes a series of judgments, and this makes the data more reliable and the standard
errors much smaller than they would be if each respondent contributed only a single
observation. Cohen and Cohen (1983) referred to designs in which each participant

400 ORGANIZATIONAL RESEARCH METHODS
responds to each scenario as a subjects-by-conditions (S × C) design. They point out
that in S × C designs, effect sizes and hence power are based on the proportion of
within-subject variance (rather than total variance) that is accounted for by the condi-
tions (scenarios). Therefore, unless respondents are very inconsistent in their judg-
ments, policy-capturing studies can be expected to yield larger effect sizes than more
traditional survey research designs. Hence, “nomothetic comparisons in a [policy-
capturing] study can achieve a power level equivalent to traditional versions of the test
(which use data based on single observations) with fewer observations per group”
(Cooksey, 1996, p. 134). See Cohen and Cohen (1983, chap. 11) for details.
Posttask Data Collection. As part of the research design, we recommend including
a posttask process to ask some open-ended questions to find out more about how the
respondents viewed the study, to determine their awareness of cue emphasis, and to
find out why they emphasized some cues over others. This could also be a valuable
opportunity for researchers to examine the extent to which respondents understand
and can articulate their own decision policies (see Reilly & Doherty’s [1992] research
on self-insight in decision processes). Posttask data collection will take more time and
effort, but the payoff in terms of richness when the researcher wants to interpret and
write up the results makes it worthwhile.
Recommendations. First, scenarios in a full factorial design should include no more
than five cues unless the researcher chooses to include extraneous cues to reduce
demand effects based on “obviousness.” Two or three values per cue should be suffi-
cient for most of these full factorial designs. Of course, in some studies, more than five
cues are required to include all the information required for a decision. In these cases,
more complex (fractional or incomplete block) designs will be required. Again, Neter
et al. (1990) have provided a full discussion of fractional designs.
We recommend that if the research uses written scenarios, the total number of sce-
narios be between approximately 50 and 80. As shown above, 81 scenarios will allow
for a completely crossed study design with four variables and three values for each.
This recommendation provides for at least 10 scenarios per cue variable (Cohen &
Cohen, 1983), provided that the number of cues does not exceed eight. In the example
of four variables with three values each, 81 scenarios would yield more than 20 obser-
vations per cue. Recall, too, that prior research (Brehmer & Brehmer, 1988; Miller,
1956) indicated that more that four or five cue variables may result in cognitive over-
load for respondents.
The issue of the number of scenarios should also include consideration of the issues
of reliability and respondents’ learning how to engage in the research. Despite the
importanceofreliability in validity (Karren& Barringer, 2002), empiricalstudieshave
seldom included estimates of judgment reliability. We recommend that researchers
include 4 or 5 duplicate scenarios within the full set and that these be used to estimate
test-retest reliability (for an example, see Cable and Judge, 1994). Policy-capturing
reliability is discussed more fully in Karren and Barringer (2002). Duplicate scenarios
shouldberemovedfrom the response setbefore analysis (Cooksey, 1996). We reiterate
that in the process of acclimating the respondents, our experience indicates about 8 to
12 written scenarios are likely to be required to reach stable regressions.
The number of cues, the number of values per cue, and the correlations between the
cues will all affect the number of scenarios needed. If representativeness issues require

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 401
the use of more than five cues or more than four values per cue, the number of scenarios
required may be more than the respondents are able or willing to judge effectively. In
these cases, the researcher should break the judgment task into two or more sessions or
use fractional factorial or incomplete block designs.
The nature of the research question determines the number of respondents needed.
In idiographic studies, it is not the number of respondents that is important but using an
appropriate set of judges with a large number of scenarios. For nomothetic studies, use
power analysis for S × C designs as recommended by Cohen and Cohen (1983) to
determine the number of respondents. We encourage researchers to include a qualita-
tive follow-up as part of their research design. We now turn to factors that should be
considered in the second step of policy-capturing research—study execution. Given
that organizational researchers will be familiar with most of these issues, our treatment
of the study execution step is brief.
Step 2: Study Execution
In this section, we focus on the procedures and instruments used in data collection.
First, respondents should be informed about the purpose of the study. Researchers
should be sure that respondents understand the nature of the task they are asked to
complete, including the nature of any assumptions that should be made. This might
include discussing the decision-making context and the source and quality of the cue
information. For example, the researcher might instruct respondents as follows:
Assume that you are head of a small private firm (e.g., 50 employees) that makes elec-
trical circuit breakers. The firm has just developed a working prototype of a new tech-
nology that can distribute electrical power throughout a building without wires. This
technology means enormous company growth potential, depending on several fac-
tors, including possible health hazards stemming from the technology and the market
forecasts for new construction.
Respondents will be asked to make decisions about increasing the company’s produc-
tion capabilities to tap this new market. Relevant cues must be provided, and the nature
of the sources of that information should be discussed. For example, respondents
wouldprobablytreatanOccupational Safety and Health Administration assessment of
the health risk differently than one that came from a small-scale research project spon-
sored by a rival trade association. For results of a policy-capturing study to be repre-
sentative of real decision-making situations, it is important that respondents be made
aware of the nature and the supposed source of the information they are asked to con-
sider. Respondents’assessments of the quality, validity, and value of that information
will then be reflected in their judgments.
Although information is often presented to respondents in written scenarios, there
is no need for presentations to be limited to that format. Other formats, such as audio
tapes, videos, pictures, bar charts, graphs, or tables may be more effective (Gorman
et al., 1978). The important point is that respondents fully understand both the judg-
ment task they are asked to undertake and the information presented to them in each
judgment situation.
Administration is probably most appropriately done in a single sitting of about an
hour or less. Research suggests that people can do approximately 40 to 75 written sce-
narios per hour, depending on complexity (Cooksey, 1996). Thus, the 50 to 80 written

402 ORGANIZATIONAL RESEARCH METHODS
scenarios that we have recommended will probably require about 1 hour. Researchers
using videotape scenarios may find they need to schedule multiple data collection
times. We suggest that researchers use pilot-testing sessions to estimate the length of
time that will be required for participation in the study. If the anticipated time is much
longer than 1 hour, a rest period or a fractional design may be appropriate.
In sum, we have three major recommendations for the execution phase. First, be
sure that respondents understand the nature of the judgment task and the assumptions
that should be made. Second, provide cue information in an accessible, full-context
informational format. Third, structure administration so the respondents will not be
required to spend much more than about 1 hour without having a rest period to avoid
fatigue or boredom effects.
Step 3: Analysis
Although multiple regression is familiar to organizational researchers, the analysis
of policy-capturing data requires two special considerations. The first concerns how
cue values will be coded. The second is that alternatives to the typical ordinary least
squares (OLS) regression models will likely be required. We examine the implications
of both issues in this section.
Coding Independent Variables in Policy-Capturing
The independent variables in a policy-capturing analysis may be categorical or
numeric. Perhaps the most common choice for representing categorical variables is to
use indicator (dummy) variables, but there are two other possibilities that could also be
considered: allocatedcoding and effect coding. Because most organizational research-
ers are already familiar with dummy coding, we will focus most of our comments on
the other two coding systems. For a full explanation of dichotomous and
polychotomous dummy coding, see Cohen and Cohen (1983).
For ordinal cue variables, researchers might be tempted to consider using allocated
coding (Neter et al., 1996). As an example of allocated coding, the values 1, 2, and 3
could be assigned to represent, respectively, low, medium, and high degrees of uncer-
tainty in a firm’s competitive environment. Neter et al. (1996) have pointed out, how-
ever, that allocated coding presents conceptual difficulties. These center around the
assumption, inherent in allocated coding, that the difference between low uncertainty
(1) and medium uncertainty (2) is equal to the difference between medium and high (3)
uncertainty. Because that assumption may be difficult to support, Neter et al. recom-
mended instead that researchers use either dummy variables or the related technique of
effect coding.
To use dummy variables with ordinal cues, the researcher chooses one level of the
variable as a baseline (e.g., low uncertainty in the previous example) and then uses
dummy variables to represent each of the other levels of the variable. In this example,
one dummy variable would represent medium uncertainty and the other would repre-
sent high uncertainty. This technique requires no assumptions about equality of the
differences between levels. The computed regression coefficients for medium and
high uncertainty in this example index the effects of those two uncertainty levels, rela-
tive to the low uncertainty condition, with all of the other regressors held constant.

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 403
Effect coding is similar to dummy coding. But rather than giving the baseline cate-
gory a value of 0 on all of the indicator variables (as would be done with dummy vari-
ables), the researcher assigns the baseline category a value of –1 on all indicator vari-
ables. For the nonbaseline categories, values of 0 and 1 are assigned in exactly the
same way as in dummy coding. Because values are assigned differently in effect cod-
ing than in dummy coding, the meanings of the regression coefficients are also differ-
ent. Whereas coefficients from dummy coding represent differences between the base-
line category and each of the categories represented by a dummy variable, coefficients
in effect coding represent differences between each category and the grand mean
across all categories. Neter et al. (1996) discussed this difference in interpretations in
some detail.
Turning now to the numeric cue variables, some studies use specific values (e.g.,
$10 million in sales), and others use ranges of values (e.g., number of employees is less
than 100, 100 to 999, or 1,000 or more). In cases where specific values are used, we
recommend that researchers use those values in computing their regression equations.
This will maintain a meaningful metric for their results. In the annual sales example
above, entering the sales (in millions of dollars) into the regression will result in
regression coefficients that indicate how million-dollar variations in sales affect
assessments of value, other factors being held constant.
Where ranges are used to represent numeric variables, dummy or effect coding
again offer the advantages over allocated coding described above. The choice between
dummy and effect coding should be based on the ease of interpretation of the results.
Dummy coding is appropriate where it is most meaningful to make comparisons to a
baseline group or a baseline cue value. Effect coding is appropriate where it is compar-
isons of each group or cue value to the grand mean of all groups or values that are most
meaningful.
Selecting the Proper Regression Model
There are a number of multiple regression techniques (e.g., OLS, generalized least
squares [GLS], logistic regression) that might be used in policy-capturing research,
and it is important to choose the one that is appropriate for the specific study. Two char-
acteristics of a policy-capturing study can influence that choice. The first is whether
the dependent variable is categorical or numeric. The second characteristic, in studies
where the dependent variable is numeric, is that the viability of the four basic assump-
tions of multiple regression must be considered before the proper regression technique
can be selected. Each of these issues is discussed below.
RegressionWithaCategorical Dependent Variable. If the dependent variable is cat-
egorical, the researcher should avoid OLS regression and instead use an alternative
model, typically logistic regression. Logistic regression yields equations that can be
used to compute (using values of the cue variables) the probability that the dependent
variable falls into a particular category. Thus, logistic regression is well suited to prob-
lems where the dependent variable is measured on a nominal rather than an ordinal
scale. OLS regression is unsuitable for problems of this type, especially if there are
three or more possible values of the dependent variable and there is no natural ordering
to the values. Another advantage of logistic regression for this type of problem is that it

404 ORGANIZATIONAL RESEARCH METHODS
provides unbiased estimates of the standard errors of the regression coefficients,
whereas OLS does not. Space limitations do not allow further discussion of logistic
regression techniques or the interpretation of their results here. Readers needing fuller
explanation are referred to texts such as Neter et al. (1996) or Tabachnick and Fidell
(1996). The major statistical packages, such as SAS and SPSS, have routines for con-
ducting logistic regression.
Regression With a Numeric Dependent Variable. If the dependent variable is
numeric (e.g., a Likert-type scale), then the second important study characteristic con-
cerns the four basic regression assumptions of linearity, normality, homoscedasticity,
and independence of residuals or observations. Statistics texts have comprehensive
discussions of the nature and the implications of these assumptions. Linearity is criti-
cal and can be checked by inspecting bivariate scatterplots of the relationships
between the dependent variable and each of the independent variables or plots of resid-
uals against each independent variable. Concerning normality, evidence suggests that
judgments in policy-capturing research tend to be normally distributed, especially if
there are approximately 50 or more scenarios per judge and if the judgment scale has a
reasonable number of possible values (Cooksey, 1996). Thus, it is unlikely that the
data in a sufficiently large and well-designed policy-capturing study will seriously
violate the assumption of normality.
It is the homoscedasticity and independence of residuals assumptions that are most
likely to be violated in policy-capturing research (Cooksey, 1996). The problem cre-
ated by violation of these assumptions is that the standard errors and the variance of the
error terms may be badly underestimated (Neter et al., 1996). As a consequence, confi-
dence intervals and significance tests may be inaccurate. It is therefore important for
researchers to determine whether their data are consistent with the homoscedasticity
and independence assumptions. Methods of detecting and dealing with violations of
these assumptions are discussed below.
Design characteristics may affect the likelihood that heteroscedasticity or
nonindependence will be present in the data. In policy-capturing designs, whether
idiographic or nomothetic, each respondent contributes multiple responses to the anal-
ysis. Hence, the observations are not independent. In idiographic studies, the idiosyn-
crasies of the individual respondent pervade the observations used to compute that
respondent’s policy equation. As an example, a respondent could become increasingly
harsh or lenient while progressing through the scenarios. If so, the residual from one
scenario could be related to (predictable from) the residual from the previous scenario.
The data in such a case do not meet the assumption of independence of residuals.
In nomothetic studies, multiple judgments from multiple judges may be combined
into a single data set from which one regression equation is computed. Nonindepen-
dence could be present in the aggregated data set for the same reasons that were dis-
cussed in the case of idiographic studies as well as the fact that each respondent’s sub-
set of observations could have its own unique characteristics. Some respondents may
give systematically lower judgments than do others, for example. Thus, idiosyncrasies
in the policies of individual respondents can lead to correlated errors in the residuals
resulting from a regression equation based on all respondents’ choices.
Nonindependence of residuals, or autocorrelation, can be checked by computing
the Durbin-Watson d statistic. Most of the major statistical packages have this capabil-
ity. If the d statistic indicates that autocorrelation is not present, then the researcher

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 405
may proceed with OLS regression. A d statistic that indicates the presence of
autocorrelation is a signal that the researcher should not use OLS regression, because
that procedure is likely to yield downwardly biased estimates of the standard errors of
the regression.
Researchers (e.g., Cable & Judge, 1994) who encounter autocorrelation usually
indicate that the GLS regression model should be used in their data. We agree with that
assessment. GLS and OLS regression both yield unbiased estimates of the regression
coefficients; however, in the presence of autocorrelation, OLS presents biased esti-
mates of the standard errors of those coefficients. GLS regression presents unbiased
estimates of the standard errors (Neter et al., 1996). Therefore, GLS regression is pre-
ferred in these cases.
Based on questions we have received from other researchers, we believe that orga-
nizational researchers are aware of the need for GLS regression but are not familiar
with the GLS model or with how statistical packages can be used to compute GLS esti-
mates. Published policy-capturing research often indicates that the authors used GLS
regression, but seldom does it describe specifically how this was done.
In SAS (SAS Institute, 1998), there are two types of GLS regression packages that
are appropriate, one for idiographic studies and one for nomothetic studies. GLS
regression can be performed with both PROC AUTOREG and PROC TSCSREG. The
SAS PROC AUTOREG computes the Durbin-Watson d statistic to determine whether
autocorrelation is present and gives both OLS and GLS estimates of the regression
coefficients. This is suitable for idiographic studies where all responses in a given
analysis come from a single respondent and the researcher wants to determine whether
some factor (e.g., changing standards or fatigue) has resulted in autocorrelated
responses. If there is evidence of autocorrelation, then the GLS estimates should be
preferred. If there is no such evidence, then OLS estimates may be used.
In nomothetic data, autocorrelation is likely in the responses from each individual
decision maker. Heteroscedasticity is more probable as well, because each decision
maker’s set of responses may have its own distinct distributional properties. The added
complexity of these data sets can be handled by statistical procedures that are used to
analyze panel data. Panel data sets represent multiple observations of multiple entities,
often in the form of cross-sectional data involving the same objects being measured at
several points in time (time series). Although the context in policy-capturing is some-
what different, the general nature of the data is the same.
In SAS, PROC TSCSREG (time-series cross-sectional regression) as well as
PROC MIXED are appropriate for this type of data (SAS Institute, 1998). STATA, a
software package often used by organizational strategy researchers, also has a proce-
dure, XTGLS (cross-sectional time series generalized least squares), that is appropri-
ate for panel data (STATA, 1999). SPSS version 11 offers a mixed-models procedure
that does this type of analysis (SPSS, 2001). These packages allow for the possibility
that observations of the same object may be autocorrelated and that each set of obser-
vations may exhibit a different degree of variability (i.e., that they may be
heteroscedastic). Again, the Durbin-Watson d statistic is presented along with both
OLS and GLS estimates of the regression coefficients and their standard errors.
Because both OLS and GLS provide unbiased estimates of the regression coefficients,
the researcher may find only small differences between the two types of estimated
coefficients. Again, however, differences in the standard errors between OLS and GLS
models have implications for the amount of expected sample-to-sample variability of

406 ORGANIZATIONAL RESEARCH METHODS
estimated regression coefficients, as well as for any statistical tests based on those
standard errors. Biases will be introduced by using the wrong model. In the case of
autocorrelated residuals, it is the GLS statistics that should be reported.
Nomothetic analyses are appropriate when researchers use policy-capturing to
determine differences in policy-capturing between two known groups. HLM (Bryk &
Raudenbush, 1992) uses GLS estimation and is well suited for analysis of data where
individuals are nested within identifiable groups. SAS PROC GLM can also be used
for GLS group comparisons using ANOVA. For an example using this technique, see
Rynes and Miller (1983).
When researchers have not identified a priori discrete groups but want to determine
if there are statistically defensible groupings of respondents, with each group having a
unique decision process, cluster analytic techniques can be used. For examples, see
Graves and Karren (1992) or Powell and Mainiero (1999). For further insight on use of
nomothetic clustering, see Cooksey (1996).
Stated briefly, our recommendations with regard to analysis concern determining
an appropriate method for coding both categorical and numeric variables and also
selecting the proper regression model. For coding categorical variables, we recom-
mend either dummy or effect coding. Numeric variables may be coded either by actual
values or by ranges (again using dummy or effect coding systems). Choosing the cor-
rect regression model requires first determining whether the dependent variable will
be categorical (use logistic regression) or numeric (use OLS or GLS techniques,
depending on the validity of the four regression assumptions).
Step 4: Interpreting Results
We make four recommendations for the interpretation step of a policy-capturing
study. First, researchers must be aware and acknowledge that their models cannot
include all of the variables that influence decision making. Nonetheless, if the cues are
well chosen and the study is properly designed, policy-capturing results in useful,
although incomplete, models of policy-capturing processes (Hoffman, 1960).
Second, researchers must take care to match their interpretation to the type of anal-
ysis (i.e., idiographic or nomothetic) they have done. The meaning of regression coef-
ficients is different in idiographic research designs than in nomothetic designs.
Regression coefficients in idiographic designs are useful for understanding the deci-
sion processes of individual respondents. Nomothetic designs, on the other hand, are
of relatively little value for understanding or predicting an individual’s choices but can
be useful for predicting behavior or choices in the aggregate.
Third, the researcher should examine the data carefully for any indication that start-
up or fatigue effects may be present. We suggest using a pilot group to test, then run-
ning all the regressions for each individual respondent, and then examining the change
in error terms. The logic is that it will take a few scenarios for the regression weights to
become stable. At some point, in our experience often between the 8th and 12th sce-
narios, the error terms usually stabilize. Often the error terms for the regression equa-
tions will begin to increase again toward the end of the scenarios, indicating fatigue or
boredom. A researcher can use the information from the pilot to better set up the actual
data collection and analysis. Alternately, a researcher can set up a set of “practice tri-
als” to acquaint the respondents with the task to minimize start-up effects. If adequate
pilottestingorpracticetrialsare done, start-up and fatigue effects should be minimal.

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 407
Finally, researchers should know the degree of multicollinearity in their data and
consider its effects on their results. The general goal in any type of policy-capturing
research is to assess the importance of each cue variable. This can be done with either
unstandardized or standardized regression weights. Unstandardized weights have the
advantage of being expressed in the metric of the cue variables. Unfortunately, how-
ever, the sizes of the coefficients are dependent on the particular metrics used, and
therefore unstandardized regression coefficients are not directly comparable. With the
dollar metric, for example, the size of the regression coefficient depends greatly on
whether the unit of measure is dollars, thousands of dollars, millions of dollars, and so
on. The coefficient will be much larger if the independent variable is expressed in mil-
lions of dollars than if it is expressed in thousands of dollars.
Standardized weights are often used to deal with this problem. With standardized
weights, all variables are, in effect, measured in standard deviation (SD) units. Hence,
a standardized regression coefficient indicates the expected change, in SD units, in the
outcome variable that is associated with a change of one SD in the cue variable, with
the values of all other cue variables held constant. Placing all variables on the same
metric (i.e., SD units) improves the comparability of regression coefficients across
variables.
The weights are still not directly comparable, however, unless the cues are uncorre-
lated (orthogonal). If the cues are uncorrelated, then the variance in the dependent vari-
able that is associated with or explained by each cue is unique to that particular cue.
Hence, the importance of each cue is reflected in the magnitude of its regression coeffi-
cient. This, of course, is the reason for the attractiveness of the fully crossed designs
discussed earlier.
The interpretation of regression coefficients associated with correlated cues is less
straightforward, especially if the intercorrelations between cues are .9 or higher. Esti-
mates of regression coefficients become unstable under conditions of high
multicollinearity. Their standard errors are inflated, and the magnitudes of the esti-
mates can vary greatly depending on the exact set of cue variables included. If multiple
sets of respondents are used, estimates can also differ dramatically across samples. Inter-
preting results from data where the cues are intercorrelated must, therefore, be done with
great care. See Hair et al. (1992) for a more detailed description of this problem.
In summary, researchers must be aware of and acknowledge that policy-capturing
models cannot include all of the variables that influence decision making. Researchers
should match interpretation to their idiographic or nomothetic research questions.
Researchers should examine regression equations for start-up and fatigue effects and
use appropriate stable regressions for interpretation. And finally, interpreting data
with multicollinearity must be done carefully. An additional suggestion is that
researchers use qualitative information gathered in posttask interviews to enrich their
interpretation of results.
Step 5: Reporting Results
When reporting a policy-capturing study, enough detail must be included so the
reader can understand the context and process of the research and be able to extend the
research if desired. Cooksey (1996) proposed a more comprehensive list than we
include here. In our experience, when reporting, it is important to do the following:

408 ORGANIZATIONAL RESEARCH METHODS
1. Describe the process used for producing the scenarios so that an interested researcher
could replicate the research.
2. Describe the cue variables and cue values, including their measurement and distribu-
tional properties and intercorrelations.
3. Describe the judgment scale and its measurement and distributional properties. If the
judgment scale is categorical, define each of the categories and report the frequency of
responses for each.
4. Discuss the method for selecting judges and include a description of their familiarity
with the judgment task involved in the study.
5. If the study is idiographic, then report the policy characteristics (i.e., cue weights, mul-
tiple R, and standard errors) and the distributions of the judgments for each respondent.
Cooksey (1996) argued that such an idiographic table should be included even for
nomothetic studies.
Conclusion
Policy-capturing can be a powerful tool for understanding the relative importance
of various types of information to decision makers. To tap the full potential of this
method, researchers must understand the principles of sound practice and apply them
to each step of a policy-capturing study. Given the volume of empirical policy-capturing
research, the organizational literature has offered surprisingly little comprehensive,
yet accessible, guidance geared specifically to problems of study design, study exe-
cution, analysis, interpretation, and reporting results for researchers using policy-
capturing. This article fills that void.
We believe the article makes two types of contributions. At a basic level, it draws
attention to the similarities in decision-making research across various disciplines.
The bigger contribution is that it answers a number of questions about policy-captur-
ing using recommendations based on extant literature and empirical research. Clearly,
the design and execution of any policy-capturing study involves a series of trade-offs,
such as deciding on the number of cues, cue levels, scenarios, and respondents. Where
we were not able to give much space to more advanced questions, we have directed
readers to more detailed sources of information. In the article, we focused attention on
questions that should be explicitly addressed in any policy-capturing study. For exam-
ple, Is the study idiographic or nomothetic, and what implications does that have for
designing the study and interpreting its results? Should cues be correlated or orthogo-
nal, andwhy?Howmight a brief follow-up to collect qualitative information enrich the
interpretation of the data? These are examples of questions that researchers, especially
those who are inexperienced with this method, might not have considered.
It is our hope that this article will improve the practice and use of policy-capturing
research by serving as a convenient, central source of information about this research
method and by focusing attention on the keys to effectiveness in each step of a study.
The consistent use of sound research principles should enhance the validity of policy-
capturing research and advance knowledge in all of the fields in which it is used.

Appendix
Flow Sheet of Policy-Capturing: Comparing Idiographic and Nomothetic Process Flow
Step and Major Questions
Step 1: Study design
Idiographic
Nomothetic
What kinds ofresearch questions are appropri - Questions about how individuals make decisions. Questions about how people in general make deci-
ate for policy-capturing? Example: Do interviewers within Company A use sions. Example: In general, do interviewers from
different decision-making policies about whom to Company A use different decision criteria than
interview? interviewers from Company B?
What do I need to do to make the research more Choose respondents who have expertise in the sit- Choose respondents who have expertise in the sit-
representative ofreal-lief situations?
uation. Use real data for value ranges. Format the uation. Use real data for value ranges. Format the
scenarios so they are realistic. Pilot test.
scenarios so they are realistic. Pilot test.
How many cues (stimuli) can I include? How
many levels in the cues?
Five or fewer cues. Two or three levels. Have the
value ranges be similar in the levels. Ifyou decide value ranges be similar in the levels. Ifyou decide
on two levels, have all cues be at two levels. on two levels, have all cues be at two levels.
Five or fewer cues. Two or three levels. Have the
IfI use a scale of r the dependent variable, how
many points should it have?
Idiographic analysis might be interested as much Ifmain eef cts are ofmost interest, then 5- to 9-
in interactions as main effects. We suggest 10- to point scales should suffice.
20-point scales, depending on the interactions that
might be interesting. See Russel, Pinto, and Bobko
(1991) for further help.
What should the scenarios/stimuli look like?
How many scenarios?
Scenarios can be paper, audio, video, etc. Use
real values for the cues. Be cautious about having values for the cues. Be cautious about having
combinations that are inherently unrealistic (the
example ofthe high-power, incredibly fuel eco-
nomical car).
Scenarios can be paper, audio, video, etc. Use real
combinations that are inherently unrealistic (the
example ofthe high-power, incredibly fuel eco -
nomical car).
We suggest 50 to 80. Include 4 to 5 duplicate sce- We suggest 50 to 80. Include 4 to 5 duplicate sce-
narios to check reliability, but do not use those in
the final analysis. If the design needs more total
scenarios, either give a rest period or consider a
fractional design.
narios to check reliability, but do not use those in
the final analysis. If the design needs more total
scenarios, either give a rest period or consider a
fractional design.
(continued)

Appendix
Flow Sheet of Policy-Capturing: Comparing Idiographic and Nomothetic Process Flow
Step and Major Questions
Idiographic
Nomothetic
How do you manage demand effects?
Include some extraneous information in each sce- Include some extraneous information in each sce-
nario. As well, hold a postexperiment set ofinter-
views to gather qualitative data about respon-
dents’ guessing about the research and other
qualitative information.
nario. As well, hold a postexperiment set ofinter-
views to gather qualitative data about respon-
dents’ guessing about the research and other
qualitative information.
How many respondents do you need?
Because the question is about individual decision Use power analysis (Cohen, 1988), recognizing
making, the sample should reflect the question. that a smaller sample will probably yield larger
For example, the board ofdirectors ofa company. effect sizes than other types of research.
Step 2: Study execution
How do you set up the data collection?
Give complete context instructions. Each respon- Give complete context instructions. Each respon-
dent will be answering questions regarding a num- dent will be answering questions regarding a num-
ber ofscenarios. Keep the data collection process ber ofscenarios. Keep the data collection process
to approximately 1 hour. Pilot test thoroughly so
that you know problems in advance.
to approximately 1 hour. Pilot test thoroughly so
that you know problems in advance.
Step 3: Analysis
How do you code these levels ofcues?
For categorical variables, dummy coding or effect For categorical variables, dummy coding or effect
coding.
coding.
How do you do the analysis?
Run regressions on each individual. SAS PROC
TSCS (SAS Institute, 1998).
Pool data and run regressions. SAS PROC GLM;
PROC AUTOREG (SAS Institute, 1998). STATA
(1999). SPSS mixed models version 11 (SPSS,
2001).
Step 4: Interpretation
Regression coefficients in idiographic designs are Nomothetic designs answer questions about how
useful for understanding the decision processes of groups or people in general. Recognize and report
individuals. Recognize and report that you cannot that you cannot have included all the pertinent
have included all the pertinent variables and the
study is limited. Check for start-up and fatigue
effects by examining the data to see where the
variables and the study is limited. Check for start-
up and fatigue effects by examining the data to
see where the error terms are more stable. Ifyou

error terms are more stable. Ifyou have used
have used orthogonal cues, use standardized
orthogonal cues, use standardized coefficients. If coefficients. If you have used correlated cues
you have used correlated cues (which is most
likely to represent real situations), interpretation
becomes more complex. See Hair, Anderson,
Tatham, and Black (1992) for help.
(which is most likely to represent real situations),
interpretation becomes more complex. See Hair
et al. (1992) for help.
Step 5: Reporting
Report so that interested researchers could under- Report so that interested researchers could under-
stand your process and replicate and extend your stand your process and replicate and extend your
research. Report on how scenarios were con-
structed and how judges were selected. Report
research. Report on how scenarios were con-
structed and how judges were selected. Report
the policy characteristics (i.e., cue weights, multi- the policy characteristics for the grouped data.
ple R, and standard errors) and the distributions of
the judgments for each respondent.

412 ORGANIZATIONAL RESEARCH METHODS
Note
1. The authors thank an anonymous reviewer for this suggested approach to thinking about
the number of scenarios, cues, and cue levels.
References
Adamowicz, W., Louviere, L., & Williams, M. (1994). Combining revealed and stated prefer-
ence methods for valuing environmental amenities. Journal of Environmental Economic
Management, 26, 271-292.
Barr, S., & Hitt, M. (1986). A comparison of selection decision models in manager versus stu-
dent samples. Personnel Psychology, 39, 599-617.
Brehmer, A., & Brehmer, B. (1988). What have we learned about human judgement from thirty
years of policy-capturing? In B. Brehmer & C. R. B. Joyce (Eds.), Human judgement: The
SJT view. Amsterdam: North-Holland Elsevier.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchicallinear models. Newbury Park, CA:Sage.
Cable, D., & Judge, T. (1994). Pay preferences and job search decisions: A person-organization
fit perspective. Personnel Psychology, 47, 317-348.
Carroll, J. S., & Johnson, E. J. (1990). Decision research: A field guide. Newbury Park, CA:
Sage.
Cattin, P., & Wittink, D. (1983). Commercial use of conjoint analysis: A survey. Journal of Mar-
keting, 46, 44-53.
Cicchetti, D. V., Showalter, D., & Tyrer, P. J. (1985). The effect of number of rating scale catego-
ries on levels of interrater reliability: A Monte Carlo investigation. Applied Psychological
Measurement, 9, 31-36.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ:
Lawrence Erlbaum.
Cohen, J., & Cohen, P. (1983). Applied multiple regression-correlation analysis for the behav-
ioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
Cooksey, R. (1996). Judgement analysis: Theory, methods, and applications. San Diego, CA:
Academic Press.
Gorman, C., Clover, W., & Doherty, M. (1978). Can we learn anything about interviewing real
people from “interviews” of paper people? Two studies of the external validity of a para-
digm. Organizational Behavior and Human Performance, 22, 165-192.
Graham, M., & Cable, D. (2001). A comparison of full versus fractional factorial designs in pol-
icy-capturing studies. Organizational Research Methods, 4, 26-45.
Graves, L., & Karren, R. (1992). Interviewer decision processes and effectiveness: An experi-
mental policy-capturing investigation. Personnel Psychology, 45, 313-340.
Green, P., & Srinivasan, V. (1990). Conjoint analysis in marketing: New developments with im-
plications for research and practice. Journal of Marketing, 54, 3-19.
Hair, J., Anderson, R., Tatham, R., & Black, W. (1992). Multivariate data analysis with read-
ings. New York: Macmillan.
Hammond, K. (1966). Probabilistic functionalism: Egon Brusnwik’s integration of the history,
theory, and method of psychology. In K. Hammond (Ed.), The psychology of Egon
Brusnwik. New York: Holt, Rinehart and Winston.
Hardiman, P., Dufresne, R., & Mestre, J. (1989). The relation between problem categorization
and problem solving among experts and novices. Memory & Cognition, 17, 627-638.
Highhouse, S. (2001). Judgment and decision-making research: Relevance to industrial and
organizational psychology. In N. Anderson, D. Ones, H. K. Sinangil, & C. Viswesvaran
(Eds.), Handbook of industrial, work and organizational psychology (pp. 314-332). London:
Sage.

Aiman-Smith et al. / CONDUCTING STUDIES OF DECISION MAKING 413
Highhouse, S., Luong, A., & Sarkar-Barney, S. (1999). Research design, measurement, and ef-
fects of attribute range on job choice: More than meets the eye. Organizational Research
Methods, 2, 37-48.
Hitt, M., & Barr, S. (1989). Managerial selection decision models: Examination of configural
cue processing. Journal of Applied Psychology, 74, 53-61.
Hobson, C., & Gibson, F. (1983). Policy-capturing as an approach to understanding and improv-
ing performance appraisal: A review of the literature. Academy of Management Review, 10,
640-649.
Hoffman, P. (1960). The paramorphic representation of clinical judgement. Psychological Bul-
letin, 57, 116-131.
Howell, D. (1992). Statistical methods for psychology. Boston: PWS-Kent Publishing.
Hurt, J. L., Maver, J. A., & Hofmann, D. (1999). Situational and individual influences on judg-
ments of hostile environment sexual harassment. Journal of Applied Social Psychology, 29,
1395-1415.
Karren, R. J., & Barringer, M. W. (2002). A review and analysis of the policy-capturing method-
ology in organizational research: Guidelines for research and practice. Organizational Re-
search Methods, 5, 337-361.
Mackay, J., Barr, S., & Kletke, M. (1992). An empirical investigation of the effects of decision
aids on problem-solving processes. Decision Sciences, 23, 648-672.
MacKenzie, J. (1993). A comparison of contingent preference models. American Journal of Ag-
ricultural Economics, 75, 593-603.
McGrath, J. E. (1982). Dilemmatics: The study of research choices and dilemmas. In J. E.
McGrath, J. Martin, & R. A. Kulka (Eds.), Judgment calls in research (pp. 69-102). Beverly
Hills, CA: Sage.
Miller, G. (1956). The magical number seven, plus or minus two: Some limits on our capacity
for processing information. Psychological Review, 63, 81-97.
Neter, J., Kutner, M., Nachtsheim, C., & Wasserman, W. (1996). Applied linear regression mod-
els. Chicago: Irwin.
Neter, J., Wasserman, W., & Kutner, M. (1990). Applied linear statistical models: Regressional,
analysis of variance, and experimental design. Homewood, IL: Irwin.
Nunnally, J. (1978). Psychometric theory. New York: McGraw-Hill.
Polson, P., & Jeffries, R. (1982). Problem-solving as search and understanding. In R. J. Stern-
berg (Ed.), Advances in the psychology of human intelligence. Hillsdale, NJ: Lawrence
Erlbaum.
Powell, G., & Mainiero, L. (1999). Managerial decision making regarding alternative work ar-
rangements. Journal of Occupational and Organizational Psychology, 72, 41-56.
Priem, R., & Harrison, D. (1994). Exploring strategic judgement: Methods for testing the assump-
tions of prescriptive contingency theories. Strategic Management Journal, 15, 311-324.
Reilly, B., & Doherty, M. (1992). The assessment of self-insight in judgement policies. Organi-
zational Behavior and Human Decision Processes, 53, 285-309.
Russell, C. J., & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse
for comfort. Journal of Applied Psychology, 77, 336-342.
Russell, C. J., Pinto, J. K., & Bobko, P. (1991). Appropriate moderated regression an inappropri-
ate research strategy: A demonstration of information loss due to scale coarseness. Applied
Psychological Measurement, 15, 257-266.
Rynes, S., & Lawler, J. (1983). A policy-capturing investigation of the role of expectancies in
decision to pursue job alternatives. Journal of Applied Psychology, 68, 147-154.
Rynes, S., & Miller, H. (1983). Recruiter and job influences on candidates for employment.
Journal of Applied Psychology, 68, 147-154.
SASInstitute. (1998). SAS/ETS users guide, version7[Computersoftware]. Cary, NC:Author.
Shanteau, J., & Stewart, T. (1992). Why study expert decision making: Some historical perspec-
tives andcomments. OrganizationalBehavior andHuman Decision Processes, 53, 95-106.

414 ORGANIZATIONAL RESEARCH METHODS
Shimp, T. A., Hyatt, E. M., & Snyder, D. J. (1991). A critical appraisal of demand artifacts in
consumer research. Journal of Consumer Research, 18, 273-283.
SPSS. (2001). Product information sheet S11CMP-0901. In What’s new in SPSS 11.0 for Win-
dows. Chicago: Author.
STATA. (1999). STATA reference manual, release 6.0 [Computer software]. College Station,
TX: STATA Press.
Stewart, T. (1988). Judgement analysis: Procedures. In B. Brehmer & C. R. B. Joyce (Eds.), Hu-
man judgement: The SJT view. Amsterdam: North-Holland Elsevier.
Tabachnick, B., & Fidell, L. (1996). Using multivariate statistics. New York: HarperCollins
College.
Thoms, P., McMasters, R., Roberts, M., & Dombkowski, D. (1999). Resume characteristics as
predictors ofaninvitation tointerview. JournalofBusiness andPsychology, 13, 339-356.
Wedell, D., Parducci, A., & Lane, M. (1990). Reducing the dependence of clinical judgement on
the immediate context: Effects of number of categories and types of anchors. Journal of
Personality and Social Psychology, 58, 319-329.
Zacharakis, A., & Meyer, G. (2000). The potential of actuarial decision models: Can they im-
prove the venturecapitalinvestmentdecision?JournalofBusiness Venturing, 15, 323-346
Lynda Aiman-Smith (Ph.D., Purdue University) is an associate professor in the Department of Business
Management in the College of Management at North Carolina State University. Her current research inter-
ests include technology implementation issues, organizational culture, and managing technical profession-
als. She also is currently the associate director for the Center for Innovation Management Studies.
Steven E. Scullen (Ph.D., University of Iowa) is an assistant professor in the Department of Business Man-
agement at North Carolina State University. His current research interests include measurement issues re-
lated to performance appraisal, performance management, and 360-degree feedback.
Steve H. Barr (Ph.D., University of Iowa) is a professor in the Department of Business Management at North
Carolina State University. His current research interests include effects of decision support systems and
other aids on problem-solving and decision-making processes.