Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation

The Internet has become increasingly popular as a way to administer self-report questionnaires, especially in the field of Internet delivered psychological treatments. Collecting questionnaire data over the Internet has advantages, such as ease of administration, and automated scoring. However, psychometric properties cannot be assumed to be identical to the paper-and-pencil versions. The aim of this study was to test the equivalence of paper-and-pencil and Internet administered versions of self-report questionnaires used in social phobia research. We analyzed data from two trials in which samples were recruited in a similar manner. One sample (N = 64) completed the paper-and-pencil version of questionnaires and the second sample (N = 57) completed the same measures online. We included the Liebowitz Social Anxiety Scale-self-assessment (LSAS-SR), the Social Interaction and Anxiety Scale (SIAS), and the Social Phobia Scale (SPS) as measures of social anxiety. Also included were the Montgomery Åsberg Depression Rating Scale-self-assessment (MADRS-S), the Beck Anxiety Inventory (BAI), and the Quality of Life Inventory (QOLI). Results showed equivalent psychometric properties across administration formats. Cronbach’s α ranged between 0.77 and 0.94. There was an indication of a somewhat higher construct validity when participants filled out questionnaires using paper-and-pencil. We conclude that the LSAS-SR, SIAS, and SPS can be administered via the Internet with maintained psychometric properties.     

Comparison of paper-and-pencil versus Web administration of the Youth Risk Behavior Survey (YRBS): Participation,data quality,and perceived privacy and anonymity

The Youth Risk Behavior Surveillance System (YRBSS) monitors priority health-risk behaviors among US high school students. To better understand the ramifications of changing the YRBSS from paper-and-pencil to Web administration, in 2008 the Centers for Disease Control and Prevention conducted a study comparing these two modes of administration. Eighty-five schools in 15 states agreed to participate in the study. Within each participating school, four classrooms of students in grades 9 or 10 were randomly assigned to complete the Youth Risk Behavior Survey questionnaire in one of four conditions (in-class paper-and-pencil, in-class Web without programmed skip patterns, in-class Web with programmed skip patterns, and “on your own” Web without programmed skip patterns). Findings included less missing data for the paper-and-pencil condition (1.5% vs. 5.3%, 4.4%, 6.4%; p < .001), less perceived privacy and anonymity among respondents for the in-class Web conditions, and a lower response rate for the “on your own” Web condition than for in-class administration by either mode (28.0% vs. 91.2%, 90.1%, 91.4%; p < .001). Although Web administration might be useful for some surveys, these findings do not favor the use of a Web survey for the YRBSS.     

Do millennial undergraduates’ views of writing differ when surveyed online versus on paper?

The purpose of this study was to test whether different survey delivery conditions made a difference in assessing college students’ practices, affect, and conceptions of academic and nonacademic writing. The delivery conditions represented combinations of three underlying factors: survey format (online versus paper-and-pencil), location (classroom, lab, home), and supervision (proctored or not). Participants (N = 268) were randomly assigned to one of three conditions: (1) a paper version of the survey administered in classrooms at scheduled proctored sessions; (2) an online version administered in scheduled proctored sessions in a computer lab; (3) an online version at a location and time of the students’ choice. The survey had 103 closed-ended and three open-response questions. Results showed different participation rates across conditions and more variability in time spent for the “home online” group. However, there were few differences by condition to substantive questions regarding the students’ practices, conceptions, and affective responses associated with writing. The only place where responses differed by condition was in response to the optional open-ended evaluation of the survey.     

Equivalence of paper-and-pencil and computerized self-report surveys in older adults

Research is lacking on the equivalence of paper-and-pencil and computer-administered surveys in older adult samples. In addition, few comparability studies have used best practices for examining equivalence. The current study investigated quantitative (means), qualitative (internal consistency and intercorrelations), and auxiliary (missing items, completion time, and comfort) equivalence for completing self-report surveys in paper-and-pencil or computerized conditions in an older adult sample. One hundred and eight older adults ages 60 and over were randomly assigned to a paper-and-pencil or computer condition and completed questionnaires assessing personality, social desirability, and computer self-efficacy. Results generally showed qualitative equivalence, with some notable differences for quantitative and auxiliary equivalence.     

Comparison of web-based versus paper-and-pencil administration of a humor survey

This study compared the measurement invariance of paper-and-pencil (PP) and web-based (WB) administration formats through a humor survey. Participants were 401 undergraduate students divided into four groups (A, B, C, and D), and each group completed one of the four testing conditions (group A: PP → PP, group B: PP → WB, group C: WB → PP, and group D: WB → WB). The WB and PP versions of the revised Multidimensional Sense of Humor Scale (Wang, Cheng, Liu, & Ho, 2011), which measure humor production, attitudes toward humor, and humor coping, were administered to the participants. The results indicated that both of the PP and WB survey formats were practically invariant. No significant differences across administration situations were found for humor production or humor coping. Interestingly, the mean score of the WB format was significantly higher than that of PP format on attitudes toward humor. The findings suggest that researchers should carefully examine the measurement invariance and the effect of characteristics of the construct on measurement results when using a WB-format instrument.     

Response effects and computer-administered questionnaires: the role of the entry task and previous computer experience

In a field experiment at a tourist attraction, a between-subject; design was used to compare paper-and-pencil survey responses against those collected from two computer-administered questionnaire ( CAQ) packages; one using a cursor movement answering protocol, and the second using number pad entry. Three response effects were identified for the groups completing computerized questionnaires: an increase in the range of responses to scale questions, a tendency to select only one choice in answering multiple response questions, and longer answers to open-ended questions. Response effects were shown to be as much a function of the specific computer entry task, as the computerized interviewing situation in general. This finding suggests that computer-collected data should not be compared with data from other methodologies or even different questionnaire software without qualification. Respondent's previous computer experience was shown to have a marginal effect on the size of response effects, and only for the cursor movement protocol. Those with more computer experience completed both computerized questionnaires slighty faster than those with less computer experience.     

Response effects and computer-administered questionnaires: the role of the entry task and previous computer experience

Abstract

In a field experiment at a tourist attraction, a between-subject; design was used to compare paper-and-pencil survey responses against those collected from two computer-administered questionnaire ( CAQ) packages; one using a cursor movement answering protocol, and the second using number pad entry. Three response effects were identified for the groups completing computerized questionnaires: an increase in the range of responses to scale questions, a tendency to select only one choice in answering multiple response questions, and longer answers to open-ended questions. Response effects were shown to be as much a function of the specific computer entry task, as the computerized interviewing situation in general. This finding suggests that computer-collected data should not be compared with data from other methodologies or even different questionnaire software without qualification. Respondent's previous computer experience was shown to have a marginal effect on the size of response effects, and only for the cursor movement protocol. Those with more computer experience completed both computerized questionnaires slighty faster than those with less computer experience.     

Physical activity levels among adult computer users

As computer use becomes prevalent at work and in leisure pursuits it presents a potential barrier to regular physical activity. However an individual’s confidence in overcoming such barriers may moderate the relationship between computer use and physical activity levels. This study examined the associations among computer use, self-efficacy in overcoming barriers to exercise, and physical activity levels among adult computer users. Participants (N = 615) were recruited through national email and newsletters announcements, and completed standardized self-report questionnaires online. Computer use was not associated with meeting physical activity guidelines (adjusted odds ratio, AOR = 0.63-1.00, n.s.). Among leisure computer users, the moderate efficacy group (AOR = 3.72, 95% CI = 1.59-8.69) and the high efficacy group (AOR = 5.31, 95% CI = 2.37-11.91) had higher odds of meeting physical activity guidelines compared to the low efficacy group. Among the work related computer users, the high efficacy group (AOR = 2.83, 95% CI = 1.18-6.77) had higher odds of meeting physical activity guidelines than the low efficacy group. These results suggest that barriers posed by high levels of computer use are surmountable. Efforts to increase efficacy in overcoming barriers to exercise may be more practical and have a greater impact on physical activity levels than trying to decrease computer use among adults.     

Adolescents’ use of Instant Messaging as a means of emotional relief

Instant Messaging (IM) plays a major role in online communication, whether through dedicated software or through chat integrated in a social network’s platform. IM-based online conversation enables private, synchronous, interpersonal communication while being invisible and possibly anonymous; facilitates self-disclosure and intimacy; and possesses advantageous features of expressive writing and social support. For adolescents, the use of IM is a legitimate, available, and free alternative vehicle for communicating with peers to ventilate negative emotions and to receive social support and advice. The present study examined effects of IMing friends on the emotional state of distressed adolescents through both pre-post (n = 100) analyses and comparison with an un-distressed group (n = 50). Dependent measures included self-report questionnaires, textual analysis, and expert judges’ evaluations of the conversations. Findings revealed that IM conversation significantly contributed to the well-being of distressed adolescents. In addition, participants’ level of introversion–extroversion moderated the degree of their perceived emotional relief, so that introverted participants profited from IMing more than did extraverts. The implications of these findings are discussed in the context of online communication theory, as well as the practical implementations for troubled adolescents.     

Paper-and-pencil versus computer-administered MMPIs: A comparison of patient's attitudes

Attitudinal variables contributing to patients' perceptions of computer-administered MMPIs were examined in a heterogeneous clinical sample. One hundred and eighty-five subjects were randomly assigned to computer or paper-and-pencil administration conditions in which they received both the MMPI and an attitude survey. In general, the computer administered MMPI was perceived as more interesting and less anxiety-provoking than the paper-and-pencil MMPI. Subjects in the computer group felt more positively toward the examiner administering test instructions and felt that the test went more quickly. Subjects in the computer and paper-and-pencil groups did not differ regarding their concerns about privacy and the disposition of test results. Results suggest that cyberphobia does not characterize the general clinical population and that patients' attitudinal and affective responses to computerized versions of the MMPI may enhance the utility of this assessment procedure.     

Youth independent living assessment: testing the equivalence of web and paper/pencil versions of the Ansell–Casey Life Skills Assessment

This study compared a paper-and-pencil and a web-based (i.e. Internet) version of the Ansell–Casey Life Skills Assessment (ACLSA) for mode equivalence. Youth (n=97) completed an age-appropriate ACLSA version in both formats, administered in a repeated measures, counterbalanced design. No significant main effects between modes of administration emerged. However, higher scores were achieved on the second administration regardless of format. These findings demonstrate the equivalence of Internet and paper-and-pencil formats for all three ACLSA levels and a predictable learning effect across the first and second testings.     

Reliability,equivalence and respondent preference of computerized versus paper-and-pencil mental health questionnaires

The purpose of this study was to examine the reliability, equivalence and respondent preference of a computerized version of the General Health Questionnaire (GHQ-12), Symptom Checklist (SCL-90-R), Medical Outcomes Study Social Support Survey (MOSSSS), Perceived Stress Scale (PSS) and Utrecht Coping List (UCL) in comparison with the original version in a general adult population. Internal consistency, equivalence and preference between both administration modes was assessed in a group of participants (n = 130) who first completed the computerized questionnaire, followed by the traditional questionnaire and a post-assessment evaluation measure. Test–retest reliability was measured in a second group of participants (n = 115), who completed the computerized questionnaire twice. In both groups, the interval between first and second administration was set at one week. Reliability of the PC versions was acceptable to excellent; internal consistency ranged from α = 0.52–0.98, ICC’s for test–retest reliability ranged from 0.58–0.92. Equivalence was fair to excellent with ICC’s ranging from 0.54–0.91. Interestingly, more subjects preferred the computerized instead of the traditional questionnaires (computerized: 39.2%, traditional: 21.6%, no preference: 39.2%). These results support the use of computerized assessment for these five instruments in a general population of adults.     

Hands-free administration of subjective workload scales: Acceptability in a surgical training environment

Introduction

Subjective workload measures are usually administered in a visual-manual format, either electronically or by paper and pencil. However, vocal responses to spoken queries may sometimes be preferable, for example when experimental manipulations require continuous manual responding or when participants have certain sensory/motor impairments. In the present study, we evaluated the acceptability of the hands-free administration of two subjective workload questionnaires - the NASA Task Load Index (NASA-TLX) and the Multiple Resources Questionnaire (MRQ) - in a surgical training environment where manual responding is often constrained.

Method

Sixty-four undergraduates performed fifteen 90-s trials of laparoscopic training tasks (five replications of 3 tasks - cannulation, ring transfer, and rope manipulation). Half of the participants provided workload ratings using a traditional paper-and-pencil version of the NASA-TLX and MRQ; the remainder used a vocal (hands-free) version of the questionnaires. A follow-up experiment extended the evaluation of the hands-free version to actual medical students in a Minimally Invasive Surgery (MIS) training facility.

Results

The NASA-TLX was scored in 2 ways - (1) the traditional procedure using participant-specific weights to combine its 6 subscales, and (2) a simplified procedure - the NASA Raw Task Load Index (NASA-RTLX) - using the unweighted mean of the subscale scores. Comparison of the scores obtained from the hands-free and written administration conditions yielded coefficients of equivalence of r = 0.85 (NASA-TLX) and r = 0.81 (NASA-RTLX). Equivalence estimates for the individual subscales ranged from r = 0.78 (“mental demand”) to r = 0.31 (“effort”). Both administration formats and scoring methods were equally sensitive to task and repetition effects. For the MRQ, the coefficient of equivalence for the hands-free and written versions was r = 0.96 when tested on undergraduates. However, the sensitivity of the hands-free MRQ to task demands (η_partial² = 0.138) was substantially less than that for the written version (η_partial² = 0.252). This potential shortcoming of the hands-free MRQ did not seem to generalize to medical students who showed robust task effects when using the hands-free MRQ (η_partial² = 0.396). A detailed analysis of the MRQ subscales also revealed differences that may be attributable to a “spillover” effect in which participants’ judgments about the demands of completing the questionnaires contaminated their judgments about the primary surgical training tasks.

Conclusion

Vocal versions of the NASA-TLX are acceptable alternatives to standard written formats when researchers wish to obtain global workload estimates. However, care should be used when interpreting the individual subscales if the object is to make comparisons between studies or conditions that use different administration modalities. For the MRQ, the vocal version was less sensitive to experimental manipulations than its written counterpart; however, when medical students rather than undergraduates used the vocal version, the instrument’s sensitivity increased well beyond that obtained with any other combination of administration modality and instrument in this study. Thus, the vocal version of the MRQ may be an acceptable workload assessment technique for selected populations, and it may even be a suitable substitute for the NASA-TLX.     

Aggression,impulsivity, and health functioning in a veteran population: equivalency and test–retest reliability of computerized and paper-and-pencil administrations

The present study examined the equivalency and test–retest reliability of two administration methods (paper-and-pencil and computerized) of the Aggression Questionnaire (AQ), Barratt Impulsiveness Scale-11 (BIS-11), and SF-36 Health Survey (SF-36) in a sample of 97 veterans. The two-week test–retest reliability for the conventional paper-and-pencil administrations of the AQ, BIS-11, and the SF-36 in the veteran population were generally reliable and comparable to the test–retest reliability of these measures noted in the research for other populations. Equivalence of the computer vs. paper-and-pencil administrations were examined and results indicated that for the AQ, ICCs for all four subscales and total score were substantially lower for the PC administration. For the BIS-11, ICCs were comparable with the exception of the Motor subscale, which was lower in the PC condition. For the SF-36, two of the domains were higher in the PC condition, with four more being comparable. Overall, the computerized administration of the BIS-11 and SF-36 was found to be equivalent to the “gold standard” paper-and-pencil administration of these measures and can be utilized successfully among veterans. The computerized administration of the AQ, however, was not found to be equivalent to the paper-and-pencil administration of the measure.     

Comparing actual and self-reported measures of Facebook use

Numerous studies exist examining how college students use Facebook and how this affects aspects of their college experience; however, all of these studies have relied on self-report measures of Facebook use. Research in other areas of human behavior has shown that self-report measures are substantially inaccurate when compared to actual behaviors. This study provides the first test of the criterion validity of measures of Facebook frequency by comparing self-reported time spent on the site and number of logins against actual usage as measured by computer monitoring software. A sample of 45 college students installed software that monitored their computer usage for 1 month. There was a strong positive correlation between self-reported and actual time spent on Facebook; however, there was a significant discrepancy between the two. Students spent an average of 26 min (SD = 30) per day on Facebook, significantly lower than the average of 145 (SD = 111) minutes per day obtained through self-report. There was a moderate relationship between number of logins and actual time spent on Facebook. Although there are some limitations of monitoring computer usage, researchers are encouraged to attempt to relate their self-report measures to actual behaviors in order to improve external validity.     

Is exposure to online content depicting risky behavior related to viewers' own risky behavior offline?

There are public and governmental concerns that social media may encourage risky behavior in the offline environment. Using international survey data from 412 young adults aged between 18 and 25 years of age (M = 21.20 years, SD = 2.31 years), this study demonstrates that there is a relationship between exposure to online content depicting risky behavior and users’ own offline risky behavior. This relationship was found for six behaviors: drug use, excessive alcohol use, disordered eating, self-harm, violence to others, and dangerous pranks. A borderline effect was found for two further behaviors: unprotected sex and sex with a stranger. The relationship between content depicting disordered eating and offline behavior was only significant for females; suggesting that female users may be more vulnerable to effects of viewing content depicting disordered eating habits, and/or use social media content to find material related to their existing behavior. No other gender moderation effects were found. The findings provide preliminary evidence that social media use may influence offline risky behavior in young adults.     

The effects of computer versus paper-and-pencil administration on measures of anger and revenge with an inmate population

A current issue in personality assessment is whether computer versions of personality tests are psychometrically equivalent to paper-and-pencil originals. A total of 52 male inmates were randomly assigned to computer or paper-and-pencil administration conditions. The following measures were administered to the inmates in either computer or paper-and-pencil formats: the trait section of the State-Trait Personality Inventory (Spielberger et al., 1979), the Anger Expression Scale (Spielberger et al., 1986), the Belief Scale (Malouff & Schutte, 1986), and the Vengeance Scale (Stuckless & Goranson, 1992). No mean or variance differences were found indicating that computer and paper-and-pencil versions of these instruments are essentially equivalent. Analyses which examined the effects of computer anxiety, preference for computer administration and social desirability also indicated equivalency. The findings of this study, although indicating the equivalency of a number of measures, do not support the use of computer versions of other paper-and-pencil instruments without prior demonstration of their equivalency with the population of interest. The pervasive social desirability contamination found in this study brings into question the validity of assessments of incarcerated individuals regardless of administration modality.     

Computer usage questionnaire: Structure, correlates, and gender differences

Computer usage, computer experience, computer familiarity, and computer anxiety are often discussed as constructs potentially compromising computer-based ability assessment. After presenting and discussing these constructs and associated measures we introduce a brief new questionnaire assessing computer usage. The self-report measure consists of 18 questions asking for the frequency of different computer activities and software usage. Participants were N = 976 high school students who completed the questionnaire and several covariates. Based on theoretical considerations and data driven adjustments a model with a general computer usage factor and three nested content factors (Office, Internet, and Games) is established for a subsample (n = 379) and cross-validated with the remaining sample (n = 597). Weak measurement invariance across gender groups could be established using multi-group confirmatory factor analysis. Differential relations between the questionnaire factors and self-report scales of computer usage, self-concept, and evaluation are reported separately for females and males. It is concluded that computer usage is distinct from other behavior oriented measurement approaches and that it shows a diverging, gender-specific pattern of relations with fluid and crystallized intelligence.     

Computerized versus standard personality measures: Equivalency, computer anxiety, and gender differences

Undergraduate students were administered the Beck Depression Inventory (BDI) and Purpose in Life Test (PIL) in either paper-and-pencil or computer-administered format. Prior to administration, all subjects completed the Computer Anxiety Rating Scale (CARS). State-Trait Anxiety Inventory (STAI), and the Phobos Mathematics Anxiety Inventory (PHOBOS) in their original paper-and-pencil formats. For those subjects who received the personality tests via computer administration, high CARS scores were associated with higher BDI scores and lower PIL scores, even when state, trait, and mathematics anxiety were partialled out. The results suggest that standardized normative distributions on personality tests may not be applicable to computerized personality tests (CPT).