The Teenage Inventory of Social Skills: Development, reliability, and validity.

Describes the development and the evaluation of reliability and convergent and discriminant validity of a newly designed self-report questionnaire for the assessment of adolescent social competence: the Teenage Inventory of Social Skills (TISS). Two-week test–retest reliabilities for positive and negative behavior scales were .90 and .72; internal consistencies were .88. Convergent validity was assessed by comparing TISS scales with self-monitoring data, ratings by peers, and sociometric data. Discriminant validity was examined by investigating correlations between scores on the TISS and social desirability, SES, and another paper-and-pencil self-report instrument (Conflict Behavior Questionnaire) thought not to be necessarily related to adolescent social behavior. Results provided adequate evidence for both the convergent and discriminant validity of the TISS scales. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity concerns and usefulness of student ratings of instruction.

The validity of student rating measures of instructional quality was severely questioned in the 1970s. By the early 1980s, however, most expert opinion viewed student rating measures as valid and as worthy of widespread use. In retrospect, older discriminant-validity concerns were not so much resolved as they were displaced from research attention by accumulating evidence for convergent validity. This article introduces a Current Issues section that gives new attention to validity concerns associated with student ratings. The section's 4 articles deal, respectively, with (a) conceptual structure (are student ratings unidimensional or multidimensional?), (b) convergent validity (how well do ratings correlate with other indicators of effective teaching?), (c) discriminant validity (are ratings influenced by factors other than teaching effectiveness?), and (d) consequential validity (are ratings used effectively in personnel development and evaluation?). Although all 4 articles favor the use of ratings, they disagree on controversial points associated with interpretation and use of ratings data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater bias in performance ratings: Superior, self-, and peer ratings.

Leniency errors, halo effects, and differential dimensionality were explored in an analysis of superior, self-, and peer performance ratings of 107 managerial and 76 professional employees in a medium-sized manufacturing location, representing 95% of the managerial and professional staff. Self-ratings showed greater leniency effects than superior or peer ratings. A multitrait–multimethod (MTMM) analysis supported the presence of strong halo effect and significant convergent validity but not discriminant validity. The dimensional analysis supported the presence of strong halo effect. A statistical control procedure for the halo effect was developed that involved calculating residuals of the performance items, controlling for the "overall effectiveness" variance component in each item. The procedure did not reduce the significant halo effect, nor did it improve the nonsignificant discriminant validity in the MTMM analysis. It did, however, clarify the dimensional structure of ratings by superiors. Data from 4 previously published studies were also reanalyzed using the statistical control procedure. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Construct validity of multiple measures in teaching, research, and service and reliability of peer ratings.

The construct validity measures in teaching, research, and service were determined using a multitrait–multimethod matrix. The measures in research (research publications, research grants, research awards, and peer ratings in research) and three of the measures in teaching (student evaluations, teaching awards, and peer ratings in teaching) supported convergent and divergent validity. However, the three-pronged model in academia was not validated, and exploratory factor analysis identified 5 and possibly 6 domains of behavior (research, classroom teaching, writing about teaching, community service, writing about service, and internal service). For one measure, peer ratings, reliability varied across academic areas and was moderately high for only high-confident ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Initial validation of a computer-administered Addiction Severity Index: The ASI–MV.

The Addiction Severity Index—Multimedia Version (ASI–MV) is a CD-ROM-based simulation of the interview-administered Addiction Severity Index (ASI). Clients in treatment (N ?=?202) self-administered the ASI–MV to examine the test–retest reliability, criterion validity, and convergent–discriminant validity of the ASI–MV. Excellent test–retest reliability was observed for composite scores and severity ratings. Criterion validity, tested against the interviewer-administered ASI, was good for the composite scores. For severity ratings, variable agreement was observed between the ASI–MV and each interviewer, suggesting poor interrater reliability among interviewers. This conclusion was bolstered by a finding of superior convergent–discriminant validity for both composite scores and severity ratings compared to the standard ASI. The ASI–MV is a viable alternative to the expensive and potentially unreliable interviewer-administered version. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of race on peer ratings in an industrial situation.

Examined the effect of race on the peer ratings of 43 black and 50 white industrial employees recently exposed to a foreman-training program which included intensive human relations training. Contrary to previous studies, no race effect was found. In addition, almost all the requirements for convergent and discriminant validity between the races were met. Possible explanations for these results and implications for the use of peer ratings in integrated settings are discussed. (15 ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A multimethod factor analysis of behavioral and objective measures of psychopathology.

Evaluated the extent to which certain objective tests, ward behavior ratings, recorded verbalization scores, and interview behavior ratings possessed convergent and discriminant validity for 10 psychotic syndromes in a varied group of 125 psychotics drawn from 5 hospitals. A multimethod factor analysis was made of the 10 interview behavior and 12 ward behavior syndrome scores, and a similar analysis made of the entire set of 48 variables. The 1st analysis yielded 10 multimethod factors that corresponded closely to the original 10 interview syndromes. In the 2nd analysis, 12 interpretable factors were obtained, all of which exhibited convergent validity for 2 or 3 method sources. 1 or more objective test variables were meaningfully associated with each of the factors except 1. It is concluded that the convergent and discriminant validity of the syndromes is strongly supported. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Strangers' ratings of the five robust personality factors: Evidence of a surprising convergence with self-report.

Attempted to replicate and extend the results of Passini and Norman (1966), who found surprising evidence of convergent validity (i.e., significant correlations with the targets' self-ratings) in strangers' judgments of 5 broad personality factors. In the current study, 250 previously unacquainted Ss were run in small, same-sex groups of various sizes. Ss rated both themselves and their fellow group members on the same set of 20 bipolar trait scales used by Passini and Norman. Consistent with previous research, significant self–peer agreement correlations were obtained for Extraversion and Conscientiousness. Ratings of Agreeableness also showed significant convergent validity when a sufficient number of peers rated the target. More generally, self–peer agreement correlations tended to rise as the number of peer raters increased. Possible explanations for the validity of strangers' trait ratings are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives.

Although peer reviewing of writing is a way to create more writing opportunities in college and university settings, the validity and reliability of peer-generated grades are a major concern. This study investigated the validity and reliability of peer-generated writing grades of 708 students across 16 different courses from 4 universities in a particular scaffolded reviewing context: Students were given guidance on peer assessment, used carefully constructed rubrics, and were provided clear incentives to take the assessment task seriously. Distinguishing between instructor and student perspectives of reliability and validity, the analyses suggest that the aggregate ratings of at least 4 peers on a piece of writing are both highly reliable and as valid as instructor ratings while (paradoxically) producing very low estimates of reliability and validity from the student perspective. The results suggest that instructor concerns about peer evaluation reliability and validity should not be a barrier to implementing peer evaluations, at least with appropriate scaffolds. Future research needs to investigate how to address student concerns about reliability and validity and to identify scaffolds that may ensure high levels of reliability and validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Midterm feedback from students: Its relationship to instructional improvement and students' cognitive and affective outcomes.

Feedback from student ratings collected at midterm resulted in (a) more favorable student ratings at the end of the term, (b) better final examination scores, and (c) more favorable affective outcomes. Ss were the 931 students in 30 sections of a course in computer programming. Randomly selected instructors were given feedback from midterm ratings and met with the authors to discuss the ratings and strategies for improvement. Ss in the feedback sections were similar to other Ss in terms of pretest scores and midterm ratings. These findings and a review of the literature suggest that feedback from student ratings, particularly when coupled with a frank discussion of their implications with an external consultant, can be an effective intervention for improving teaching effectiveness. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Construct validity of multiple job performance measures using confirmatory factor analysis.

The construct validity of job performance ratings obtained from self, supervisors, and peers was assessed relative to performance data obtained from an objective test of task proficiency. Data were gathered from a sample of 256 first-term jet engine mechanics in the U.S. Air Force. Widaman's (1985) approach to assessment of convergent and discriminant validity of multitrait–multimethod data using confirmatory factor analysis to contrast alternative models was used. As expected, all analyses revealed support for a model with correlated performance traits and uncorrelated methods. Significant convergence among the three rating sources was found. In addition, convergence was found between task ratings and scores from the objective proficiency test. Significant discriminant validity was found in all of the analyses. Results revealed that self, supervisors, and peers can be equally valid as sources of performance information, although performance of some tasks may be more validly evaluated by one source than another. Needs for further research on the effects of task characteristics on the validity of ratings are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Construct validation of a strategy model of student self-regulated learning.

Using student interviews, teacher ratings, and achievement test outcomes, we validated a strategy model of student self-regulated learning as a theoretical construct. Forty-four male and 36 female high school students were asked to describe their use of 14 self-regulated learning strategies in six contexts, and their teachers rated these students for their self-regulated learning during class. Factor analyses of the teachers' ratings along with students' scores on a standardized test of mathematics and English revealed a single self-regulated learning factor that accounted for nearly 80% of the explained variance and two smaller factors that were labeled Student Verbal Expressiveness and Achievement. Students' reports of using self-regulated learning strategies during a structured interview correlated .70 with the obtained teachers' rating factor and were negatively related to the Student Verbal Expressiveness and Achievement factors. Our results indicate both convergent and discriminative validity for a self-regulated learning construct. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparability of interviewer, group, and individual interview ratings.

Compared interview trait ratings made by individual and groups of managers following exposure to videotaped interview playbacks with ratings made by experienced interviewers participating in 34 live interviews couched in a typical entry-level managerial employment setting. A convergent and discriminant validity analysis was made to assess the adequacy of the interview rating procedure. Mean ratings made by groups of managers were similar to mean ratings made by the interviewers. Agreement between the 2 dissimilar rater types using direct (face to face) and indirect (videotaped) stimuli indicated that perceptual distortion was now a strong factor. Individual managers' mean ratings suggested a leniency effect. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Construct validity of measures of college teaching effectiveness.

Compared evaluation form, student, colleague, trained observer, former student, and self-rating teacher assessments of 43 university instructors. Data show that student and former student ratings displayed substantially greater validity coefficients of teaching effectiveness than self-report, colleague, and trained observer ratings. Advantages of student rating techniques (i.e., greater exposure to instructor's teaching), various teaching assessment methods, and problems in the literature due to limitations of research approaches typically used are discussed. (27 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Aggregation, moderator variables, and the validity of personality tests: A peer-rating study.

A study with 85 college fraternity members investigated the effects of aggregation and moderator variables on the validity of personality tests. Aggregation over items and raters yielded an average self–peer correlation of .44 for ratings on 4 personality dimensions. The combination of social communication skill and self-knowledge produced significant moderating effects. Ss high on the Acting subscale and high on a composite of Private Self-Consciousness and Personal Identity scales had stronger agreement between self- and peer ratings than did those low on these measures. Results for 4 specific moderator variables (the importance, variability, and observability of each rating dimension) were in the predicted direction, although weaker than expected. It is concluded that aggregation and the use of moderator variables are both important techniques for obtaining convincing validity coefficients in personality research. (75 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Severity Indices of Personality Problems (SIPP-118): Development, factor structure, reliability, and validity.

This article describes a series of studies involving 2,730 participants on the development and validity testing of the Severity Indices of Personality Problems (SIPP), a self-report questionnaire covering important core components of (mal)adaptive personality functioning. Results show that the 16 facets constituted homogeneous item clusters (i.e., unidimensional and internally consistent parcels) that fit well into 5 clinically interpretable, higher order domains: self-control, identity integration, relational capacities, social concordance, and responsibility. These domains appeared to have good concurrent validity across various populations, good convergent validity in terms of associations with interview ratings of the severity of personality pathology, and good discriminant validity in terms of associations with trait-based personality disorder dimensions. Furthermore, results suggest that the domain scores are stable over a time interval of 14-21 days in a student sample but are sensitive to change over a 2-year follow-up interval in a treated patient population. Taken together, the final instrument, the SIPP-118, provides a set of 5 reliable, valid, and efficient indices of the core components of (mal)adaptive personality functioning. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cross-informant agreement for ratings for social skill and problem behavior ratings: An investigation of the Social Skills Improvement System—Rating Scales.

One of the most consistent findings in rating scale research with children and adolescents is the modest agreement among different informants’ ratings. The present study systematically explored patterns of agreement among teachers, parents/caregivers, and students in domains of social skills and problem behaviors using the Social Skills Improvement System—Rating Scales (SSIS–RS; F. M. Gresham & S. N. Elliott, 2008). Two subsamples from the normative sample of the SSIS–RS were used. The first sample of participants consisted of 168 students who had all 3 informants (parent, teacher, and self) complete the SSIS–RS scales, which was necessary to assess agreement across different raters. The second sample consisted of 164 students who had raters in a similar or same role (father–mother, teacher–teacher). The results replicated an extensive literature showing that cross-informant agreements for social skills and problem behaviors are weak to moderate. The current study invoked multitrait–multimethod logic to interpret the correlations among raters derived from different informants and showed that the convergent validity coefficients were consistently stronger than the discriminant validity correlations. Implications for assessment practices and future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A multitrait (ADHD–IN, ADHD–HI, ODD toward adults, academic and social competence) by multisource (mothers and fathers) evaluation of the invariance and convergent/discriminant validity of the Child and Adolescent Disruptive Behavior Inventory with Thai adolescents.

Confirmatory factor analysis was used with a multitrait (attention-deficit/hyperactivity disorder—inattention, attention-deficit/hyperactivity disorder—hyperactivity/impulsivity, oppositional defiant disorder toward adults, academic competence, and social competence) by multisource (mothers and fathers) matrix to test the invariance and convergent/discriminant validity of the 5-factor model between mothers’ and fathers’ ratings of Thai adolescents (Year 1: n = 872; Year 2: n = 903; Year 3: n = 700; Year 4: n = 984) with the Child and Adolescent Disruptive Behavior Inventory (G. L. Burns, T. Taylor, & J. Rusby, 2001). The results showed equality of like-item loadings, intercepts, and residuals, as well as like-factor variances, covariances, and means between mothers’ and fathers’ ratings within each of the 4 yearly samples. In addition, the between-parent factor correlations showed convergent and discriminant validity with the within-parent factor correlations, showing discriminant validity for each year as well. These results for Thai adolescents and similar results (G. L. Burns et al., 2008) for mothers’ and fathers’ ratings of Brazilian, Thai, and American children provide broader support for the construct validity of the scale. The confirmatory factor analysis invariance and convergent/discriminant validity procedure with multiple sources is considered to provide a much more sophisticated procedure to evaluate the construct validity of attention-deficit/hyperactivity disorder and oppositional defiant disorder rating scales than a single-source approach. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers.

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (1) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (2) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (3) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The development and validation of a test of emotional styles.

Developed the Test of Emotional Styles based on 3 dimensions of emotionality: responsiveness (intensity of affect), expressiveness (interpersonal communication of affect), and orientation (attitudes toward emotion). 3 measures of these dimensions (incomplete sentences, true-false, and forced choice) were analyzed in a multitrait-multimethod matrix. Construct validity was established by correlations with peer ratings of emotionality, experimentally induced affect, the Marlowe-Crowne Social Desirability scale, the Rotter Incomplete Sentences Blank, and the Mosher Forced-Choice Guilt Inventory. There was substantial support for convergent and discriminant validity, and the measures were unconfounded with adjustment or social desirability. Females scored significantly higher than males on all dimensions in a sample of 120 undergraduates. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)