Testing models of decision making using confidence ratings in classification.

Classification implies decision making (or response selection) of some kind. Studying the decision process using a traditional signal detection theory analysis is difficult for two reasons: (a) The model makes a strong assumption about the encoding process (normal noise), and (b) the two most popular decision models, optimal and distance-from-criterion models, can mimic each other's predictions about performance level. In this article, the authors show that by analyzing certain distributional properties of confidence ratings, a researcher can determine whether the decision process is optimal, without knowing the form of the encoding distributions. Empirical results are reported for three types of experiments: recognition memory, perceptual discrimination, and perceptual categorization. In each case, the data strongly favored the distance-from-criterion model over the optimal model. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measures and interpretations of vigilance performance: evidence against the detection criterion

Operators' performance in a vigilance task is often assumed to depend on their choice of a detection criterion. When the signal rate is low this criterion is set high, causing the hit and false alarm rates to be low. With increasing time on task the criterion presumably tends to increase even further, thereby further decreasing the hit and false alarm rates. Virtually all of the empirical evidence for this simple interpretation is based on estimates of the bias measure beta from signal detection theory. In this article, I describe a new approach to studying decision making that does not require the technical assumptions of signal detection theory. The results of this new analysis suggest that the detection criterion is never biased toward either response, even when the signal rate is low and the time on task is long. Two modifications of the signal detection theory framework are considered to account for this seemingly paradoxical result. The first assumes that the signal rate affects the relative sizes of the variances of the information distributions; the second assumes that the signal rate affects the logic of the operator's stopping rule. Actual or potential applications of this research include the improved training and performance assessment of operators in areas such as product quality control, air traffic control, and medical and clinical diagnosis.     

Some more sensitive measures of sensitivity and response bias.

In this article, the author proposes a new pair of sensitivity and response bias indices and compares them to other measures currently available, including d′ and β of signal detection theory. Unlike d′ and β, these new performance measures do not depend on specific distributional assumptions or assumptions about the transformation from stimulus information to a discrimination judgment. With simulated and empirical data, the new sensitivity index is shown to be more accurate than d′ and 16 other indices when these measures are used to compare the sensitivity levels of 2 experimental conditions. Results from a perceptual discrimination experiment demonstrate the feasibility of the new distribution-free bias index and suggest that biases of the type defined within the signal detection theory framework (i.e., the placement of a decision criterion) do not exist, even under an asymmetric payoff manipulation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Continuous recollection versus unitized familiarity in associative recognition.

Recollection has long been thought to play a key role in associative recognition tasks. Evidence that associative recollection might be a threshold process has come from analyses of the associative recognition receiver operating characteristic (ROC). Specifically, the ROC is not as curvilinear as a signal detection theory requires. In addition, the ?-ROC is usually curvilinear, as a threshold recollection model requires, not linear, as a signal detection model requires. In Experiment 1, word pairs were strengthened at study, which yielded a curvilinear ROC and a linear ?-ROC (in accordance with signal detection theory). This result suggests that associative recognition performance was based on a continuous variable, one that likely consists of either unitized familiarity or continuous recollection. The remember–know procedure and an unexpected cued recall test suggested that the more curvilinear ROC in the strong condition was mainly due to increased recollection. In Experiment 2, word pairs were presented for an old–new recognition decision before being presented for an associative recognition decision. When pairs consisting of items not recognized as having been seen on the list were removed from the analysis, the ROC again became curvilinear, the ?-ROC again became linear, and most associative recognition decisions were associated with remember judgments. These findings suggest that the curvilinear ?-ROC often observed on associative recognition tests results from noise, as a mixture signal detection model assumes, and that recollection is a continuous process that yields a curvilinear ROC that is well characterized by signal detection theory. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An application of signal detection theory with finite mixture distributions to source discrimination.

A mixture extension of signal detection theory is applied to source discrimination. The basic idea of the approach is that only a portion of the sources (say A or B) of items to be discriminated is encoded or attended to during the study period. As a result, in addition to 2 underlying probability distributions associated with the 2 sources, there is a 3rd distribution that represents items for which sources were not attended to. Thus, over trials, the observed response results from a mixture of an attended (A or B) distribution and a nonattended distribution. The situation differs in an interesting way from detection in that, for detection, there is mixing only on signal trials and not on noise trials, whereas for discrimination, there is mixing on both A and B trials. Predictions of the mixture model are examined for data from several recent studies and in a new experiment. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

"Two-stage dynamic signal detection: A theory of choice, decision time, and confidence": Erratum.

Reports an error in "Two-stage dynamic signal detection: A theory of choice, decision time, and confidence" by Timothy J. Pleskac and Jerome R. Busemeyer (Psychological Review, 2010[Jul], Vol 117[3], 864-901). The name of the philosopher Charles Peirce was misspelled throughout as Charles Pierce. (The following abstract of the original article appeared in record 2010-14834-006.) The 3 most often-used performance measures in the cognitive and decision sciences are choice, response or decision time, and confidence. We develop a random walk/diffusion theory—2-stage dynamic signal detection (2DSD) theory—that accounts for all 3 measures using a common underlying process. The model uses a drift diffusion process to account for choice and decision time. To estimate confidence, we assume that evidence continues to accumulate after the choice. Judges then interrupt the process to categorize the accumulated evidence into a confidence rating. The model explains all known interrelationships between the 3 indices of performance. Furthermore, the model also accounts for the distributions of each variable in both a perceptual and general knowledge task. The dynamic nature of the model also reveals the moderating effects of time pressure on the accuracy of choice and confidence. Finally, the model specifies the optimal solution for giving the fastest choice and confidence rating for a given level of choice and confidence accuracy. Judges are found to act in a manner consistent with the optimal solution when making confidence judgments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Signal detection theory with finite mixture distributions: Theoretical developments with applications to recognition memory.

An extension of signal detection theory (SDT) that incorporates mixtures of the underlying distributions is presented. The mixtures can be motivated by the idea that a presentation of a signal shifts the location of an underlying distribution only if the observer is attending to the signal; otherwise, the distribution is not shifted or is only partially shifted. Thus, trials with a signal presentation consist of a mixture of 2 (or more) latent classes of trials. Mixture SDT provides a general theoretical framework that offers a new perspective on a number of findings. For example, mixture SDT offers an alternative to the unequal variance signal detection model; it can also account for nonlinear normal receiver operating characteristic curves, as found in recent research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A dynamic stimulus-driven model of signal detection.

Signal detection theory forms the core of many current models of cognition, including memory, choice, and categorization. However, the classic signal detection model presumes the a priori existence of fixed stimulus representations—usually Gaussian distributions—even when the observer has no experience with the task. Furthermore, the classic signal detection model requires the observer to place a response criterion along the axis of stimulus strength, and without theoretical elaboration, this criterion is fixed and independent of the observer's experience. We present a dynamic, adaptive model that addresses these 2 long-standing issues. Our model describes how the stimulus representation can develop from a rough subjective prior and thereby explains changes in signal detection performance over time. The model structure also provides a basis for the signal detection decision that does not require the placement of a criterion along the axis of stimulus strength. We present simulations of the model to examine its behavior and several experiments that provide data to test the model. We also fit the model to recognition memory data and discuss the role that feedback plays in establishing stimulus representations. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Perceptual separability, decisional separability, and the identification-speeded classification relationship

Three observers participated in several sessions of identification and speeded classification. Predictions from F. G. Ashby and W. T. Maddox's (1994a) response time (RT) theory of speeded classification were tested. A key theoretical concept is decisional separability (DS), that is, that the decision about the level of 1 component does not depend on the perceived value of the other component. The theory predicted that DS would hold in the filtering task but would be violated in the redundancy task, resulting in the redundancy gains. To summarize the results, (a) DS held in the filtering conditions, but not in the redundancy conditions; (b) redundancy gains occurred; (c) despite the redundancy gains, strong evidence was obtained that the stimulus components are perceptually separable; (d) 2 new models that each assumed RT decreases with the distance between the percept and the decision bound provided good accounts of the RT distributions and accuracy rates; and (e) the shift from identification to speeded classification influenced both perceptual and decisional processes.     

Perceptual separability, decisional separability, and the identification–speeded classification relationship.

Three observers participated in several sessions of identification and speeded classification. Predictions from F. G. Ashby and W. T. Maddox's (1994a) response time (RT) theory of speeded classification were tested. A key theoretical concept is decisional separability (DS), that is, that the decision about the level of 1 component does not depend on the perceived value of the other component. The theory predicted that DS would hold in the filtering task but would be violated in the redundancy task, resulting in redundancy gains. To summarize the results, (a) DS held in the filtering conditions, but not in the redundancy conditions; (b) redundancy gains occurred; (c) despite the redundancy gains, strong evidence was obtained that the stimulus components are perceptually separable; (d) 2 new models that each assumed RT decreases with the distance between the percept and the decision bound provided good accounts of the RT distributions and accuracy rates; and (e) the shift from identification to speeded classification influenced both perceptual and decisional processes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relation between signal detectability theory and the traditional procedures for measuring sensory thresholds: Estimating d' from results given by the method of constant simuli.

The theory of signal detectability assumes that the central effect of a stimulus varies because of physical and neural noise; consequently, the detection of a signal requires a central statistical decision procedure. Similar assumptions have been made by psychophysicists to explain the results of traditional threshold measurement procedures. The interrelations between signal detectability and threshold measures are discussed in relation to psychophysical statistical decision theory, and it is shown that (a) the false positive rate should be related to the Crozier ratio C = ΔΙ/?ΔΙ, and (b) it should be possible to use responses given in the method of constant stimuli to predict the value of d' that will be assigned to a given stimulus by a signal detectability procedure. Evidence supporting both predictions is reported, and the relation between threshold measures and "personality tests" is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Can decision criteria interchange locations? Some positive evidence.

In its simplest form, signal detection theory posits a single, invariant criterion for the yes/no detection task. But if paradigms with multiple criteria, such as the rating procedure, admit criterion variance, a new problem arises. If there are 2 or more response criteria on the decision axis and if these are each subject to random variation, the question arises as to whether a pair of criteria is constrained to maintain its nominal order at all times, or cause 2 criteria to change places. The theory of criterion setting developed by the 1st author and T. C. Williams (see record 1984-08523-001) is described, and it is shown that criterion-setting processes may produce criterion variance that is at least partly uncorrelated. Two alternative models for the organization of criteria in the rating procedure, the standard model and the distributed criterion model, are discussed. A prediction is derived that allows a test between the hypotheses that criteria are constrained to maintain a fixed order or that they may interchange locations on the decision axis. The results of 2 experiments, with 7 22–34 yr old observers (including the 2nd author), support the hypothesis that decision criteria may change places with one another. Alternative explanations are also considered. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The mirror effect and mixture signal detection theory.

The mirror effect for word frequency refers to the finding that low-frequency words have higher hit rates and lower false alarm rates than high-frequency words. This result is typically interpreted in terms of conventional signal detection theory (SDT), in which case it indicates that the order of the underlying old item distributions mirrors the order of the new item distributions. However, when viewed in terms of a mixture version of SDT, the order of hits and false alarms does not necessarily imply the same order in the underlying distributions because of possible effects of mixing. A reversal in underlying distributions did not appear for fits of mixture SDT models to data from 4 experiments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Should I go with my gut? Investigating the benefits of emotion-focused decision making.

Deliberative decision strategies have historically been considered the surest path to sound decisions; however, recent evidence and theory suggest that affective strategies may be equally as effective. In four experiments we examined conditions under which affective versus deliberative decision strategies might result in higher decision quality. While consciously focusing on feelings versus details, participants made choices that varied in complexity, in extent of subsequent conscious deliberation allowed, and in domain. Results indicate that focusing on feelings versus details led to superior objective and subjective decision quality for complex decisions. However, when using a feeling-focused approach, subsequent deliberation after encoding resulted in reduced choice quality. These results suggest that affective decision strategies may be more effective relative to deliberative strategies for certain complex decisions. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Measuring the effect of attention on simple visual search.

Set-size effects in visual search may be due to 1 or more of 3 factors: sensory processes such as lateral masking between stimuli, attentional processes limiting the perception of individual stimuli, or attentional processes affecting the decision rules for combining information from multiple stimuli. These possibilities were evaluated in tasks such as searching for a longer line among shorter lines. To evaluate sensory contributions, display set-size effects were compared with cuing conditions that held sensory phenomena constant. Similar effects for the display and cue manipulations suggested that sensory processes contributed little under the conditions of this experiment. To evaluate the contribution of decision processes, the set-size effects were modeled with signal detection theory. In these models, a decision effect alone was sufficient to predict the set-size effects without any attentional limitation due to perception. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Signal Detection Theory Analyses of Semantic Priming in Word Recognition.

Word recognition, semantic priming, and cognitive impenetrability research have used signal detection theory (SDT) measures to separate perceptual and postperceptual processes. In the D. Norris (1986) checking model and model simulation (D. Norris, 1995), priming alters only postperceptual word decision criteria: Stimulus-related priming reduces uncertainty, increasing sensitivity; stimulus-unrelated priming increases false alarms more than hits, reducing sensitivity. This work is cited as strong evidence that criterion changes can alter perceptual sensitivity and that SDT is inappropriate for investigating complex cognitive processes. The authors' current SDT ideal observer analysis of the model demonstrates that related priming does not directly alter sensitivity and that unrelated priming increases only false-alarm rate, reducing sensitivity. This analysis provides new perspectives on SDT concepts of complex decision processing. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The mirror effect and attention–likelihood theory: A reflective analysis.

The mirror effect refers to findings from studies of recognition memory consistent with the idea that the underlying "strength" distributions are symmetric around their midpoint separating studied and nonstudied items. Attention-likelihood theory assumes underlying binomial distributions of marked features and claims that old-item differences result from differential attention across conditions during study. The symmetry arises because subjects use the likelihood ratio as the basis for decision. The author analyzes the model and argues that one of the main criticisms (the complexity of the likelihood-ratio decision rule) is unwarranted. A further analysis shows that other distributions (the Poisson and the hypergeometric) can also produce a mirror effect. Even with the binomial distribution, a variety of parameter values can produce a mirror effect, and with the right combination of parameter values, differential attention across conditions is not necessary for a mirror effect to occur. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The regularities of recognition memory.

Describes 3 regularities in recognition memory with supporting data: the mirror effect, the order of receiver operating characteristic slopes, and the symmetry of movement of underlying distributions. The derivation of these regularities from attention/likelihood theory is demonstrated. The theory's central concept, which distinguishes it from other theories, is the following: Ss make recognition decisions by combining information about new and old items, the combination made in the form of likelihood ratios. The central role of the likelihood ratios extends the implications of signal detection theory for recognition memory. Attention/likelihood theory is fitted to data of 2 series of experiments. It is argued that the regularities require a revision of most current theories of recognition memory. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Hyperactive and hypoxic children: Signal detection, sustained attention, and behavior.

Two parameters from signal detection theory—perceptual sensitivity and decision criterion cutoff scores—were used in the analysis of vigilance performance of 30 hyperactive (mean age 8.5 yrs), 30 hypoxic (mean age 9 yrs), and 47 normal (mean age 8.8 yrs) children. Signal detection analyses of 3 Continuous Performance Test conditions indicated that with increasing age, Ss obtained significantly more hits, fewer false alarms, higher perceptual sensitivity, and responded with greater caution. Overall deficits in signal discrimination (perceptual sensitivity level) were obtained for both the hyperactive and hypoxic Ss when compared to normal age-mates. Whereas the hypoxic Ss demonstrated additional decrements in sustaining attention (sensitivity decrement over time), the hyperactive Ss were impaired by low-response caution, reflecting difficulty inhibiting impulsive responses. Possible differences in hyperactive sample characteristics across studies and time-related recovery factors following hypoxia are discussed. (43 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)