Reliability,equivalence and respondent preference of computerized versus paper-and-pencil mental health questionnaires

The purpose of this study was to examine the reliability, equivalence and respondent preference of a computerized version of the General Health Questionnaire (GHQ-12), Symptom Checklist (SCL-90-R), Medical Outcomes Study Social Support Survey (MOSSSS), Perceived Stress Scale (PSS) and Utrecht Coping List (UCL) in comparison with the original version in a general adult population. Internal consistency, equivalence and preference between both administration modes was assessed in a group of participants (n = 130) who first completed the computerized questionnaire, followed by the traditional questionnaire and a post-assessment evaluation measure. Test–retest reliability was measured in a second group of participants (n = 115), who completed the computerized questionnaire twice. In both groups, the interval between first and second administration was set at one week. Reliability of the PC versions was acceptable to excellent; internal consistency ranged from α = 0.52–0.98, ICC’s for test–retest reliability ranged from 0.58–0.92. Equivalence was fair to excellent with ICC’s ranging from 0.54–0.91. Interestingly, more subjects preferred the computerized instead of the traditional questionnaires (computerized: 39.2%, traditional: 21.6%, no preference: 39.2%). These results support the use of computerized assessment for these five instruments in a general population of adults.     

Equivalence of paper-and-pencil and computerized self-report surveys in older adults

Research is lacking on the equivalence of paper-and-pencil and computer-administered surveys in older adult samples. In addition, few comparability studies have used best practices for examining equivalence. The current study investigated quantitative (means), qualitative (internal consistency and intercorrelations), and auxiliary (missing items, completion time, and comfort) equivalence for completing self-report surveys in paper-and-pencil or computerized conditions in an older adult sample. One hundred and eight older adults ages 60 and over were randomly assigned to a paper-and-pencil or computer condition and completed questionnaires assessing personality, social desirability, and computer self-efficacy. Results generally showed qualitative equivalence, with some notable differences for quantitative and auxiliary equivalence.     

A small-scale comparison of the equivalence of paper-and-pencil and computerized versions of student end-of-course evaluations

This study compared responses to two versions of an end-of-course evaluation instrument completed by graduate students: the traditional printed form completed using pencil and paper, and a microcomputer-based form that presented equivalent items and accepted student responses. A finding of no significant difference in favorableness of composite ratings between the two versions prompted the researcher to perform item-by-item analyses of the two instruments. These analyses revealed that ratings of the individual items on one instrument were highly correlated with the ratings of their matched corresponding items on the other instrument. The paper-and-pencil and computerized evaluation instruments were found to be of almost identically high reliability. Possible limitations of the findings and areas for future research are discussed.     

Mode comparison study on willingness to buy and willingness to pay for organic foods: paper-and-pencil versus computerized questionnaire

This contribution looks into the survey mode effect using a randomized trial comparing data from paper-and-pencil and computerized web questionnaire concerning the willingness to buy (WTB) and the willingness to pay (WTP) for two food products certified as organic. A survey questionnaire was filled in by 110 university students for each mode in thermal comfort lab conditions. The design enables to study measurement variance specifically attributable to the mode of questionnaire completion (i.e. mode effect) and presentation of products (i.e. stimulus effect). While the two questionnaires were as similar as possible, the paper-and-pencil version involved the actual presentation of a tetra pack package of organic orange juice of 750 ml and a paper package of organic spaghetti of 500 g, but the computerized version involved their video projection. As regards the difference in substantive results, the prospective consumers “subjects” seemed to be more willing to buy the organic orange juice when presented live than presented on video embedded in the computerized questionnaire, while only women were willing to pay more for orange juice for the paper-and-pencil mode. No difference was found for the organic spaghetti product. As regards the response quality, in contrast to previous studies respondents wrote fewer words to the open-ended question in the computerized than in the paper-and-pencil version of the questionnaire. In addition, the study shows that using video clips as replacement for physical product presentations when measuring WTB and WTP as important concepts in consumer preference research needs further testing and evaluation as respondents may not react to them in the same way.     

Assessing the equivalence of paper-and-pencil vs. computerized tests: Demonstration of a promising methodology

A computer-administered version of a test-attitudes screening instrument was examined for generalizability and evaluated against criteria for parallelism with the identical test in paper-and-pencil format. Implications for computerized testing are discussed.     

Task difficulty of virtual reality-based assessment tools compared to classical paper-and-pencil or computerized measures: A meta-analytic approach

Virtual reality-based assessment tools arise as a promising alternative for classic neuropsychological assessment with an increased level of ecological validity. Because virtual reality cognitive measures recreate tasks that resemble with the demands from the real world it is assumed that they require additional cognitive resources and are more difficult than classical paper-and-pencil or computerized measures. Although research has focused on comparing the performance obtained on virtual reality-based measures with classical paper-and-pencil or computerized measures, no meta-analysis has been conducted on this topic. Thirteen studies met our inclusion criteria: assessed any cognitive process using virtual reality and analogous classical or computerized assessment tools of the same process. Based on a random effects model, the results indicated a moderate effect size in favor of classical and computerized tests (g = −0.77) revealing an increased task difficulty in virtual reality. Overall, results from the current meta-analysis point out that cognitive performance obtained in virtual reality is poorer than the one in classical or computerized assessment which might suggest that tasks embedded in virtual reality have an increased level of complexity and difficulty and require additional cognitive resources.     

Aggression,impulsivity, and health functioning in a veteran population: equivalency and test–retest reliability of computerized and paper-and-pencil administrations

The present study examined the equivalency and test–retest reliability of two administration methods (paper-and-pencil and computerized) of the Aggression Questionnaire (AQ), Barratt Impulsiveness Scale-11 (BIS-11), and SF-36 Health Survey (SF-36) in a sample of 97 veterans. The two-week test–retest reliability for the conventional paper-and-pencil administrations of the AQ, BIS-11, and the SF-36 in the veteran population were generally reliable and comparable to the test–retest reliability of these measures noted in the research for other populations. Equivalence of the computer vs. paper-and-pencil administrations were examined and results indicated that for the AQ, ICCs for all four subscales and total score were substantially lower for the PC administration. For the BIS-11, ICCs were comparable with the exception of the Motor subscale, which was lower in the PC condition. For the SF-36, two of the domains were higher in the PC condition, with four more being comparable. Overall, the computerized administration of the BIS-11 and SF-36 was found to be equivalent to the “gold standard” paper-and-pencil administration of these measures and can be utilized successfully among veterans. The computerized administration of the AQ, however, was not found to be equivalent to the paper-and-pencil administration of the measure.     

The effects of computer versus paper-and-pencil administration on measures of anger and revenge with an inmate population

A current issue in personality assessment is whether computer versions of personality tests are psychometrically equivalent to paper-and-pencil originals. A total of 52 male inmates were randomly assigned to computer or paper-and-pencil administration conditions. The following measures were administered to the inmates in either computer or paper-and-pencil formats: the trait section of the State-Trait Personality Inventory (Spielberger et al., 1979), the Anger Expression Scale (Spielberger et al., 1986), the Belief Scale (Malouff & Schutte, 1986), and the Vengeance Scale (Stuckless & Goranson, 1992). No mean or variance differences were found indicating that computer and paper-and-pencil versions of these instruments are essentially equivalent. Analyses which examined the effects of computer anxiety, preference for computer administration and social desirability also indicated equivalency. The findings of this study, although indicating the equivalency of a number of measures, do not support the use of computer versions of other paper-and-pencil instruments without prior demonstration of their equivalency with the population of interest. The pervasive social desirability contamination found in this study brings into question the validity of assessments of incarcerated individuals regardless of administration modality.     

Notes on desirability and conditional lower previsions

We detail the relationship between sets of desirable gambles and conditional lower previsions. The former is one the most general models of uncertainty. The latter corresponds to Walley’s celebrated theory of imprecise probability. We consider two avenues: when a collection of conditional lower previsions is derived from a set of desirable gambles, and its converse. In either case, we relate the properties of the derived model with those of the originating one. Our results constitute basic tools to move from one formalism to the other, and thus to take advantage of work done in the two fronts.     

Insuring sample equivalence across internet and paper-and-pencil assessments

Internet-based data collection is becoming an increasingly popular method of conducting psychological research. A growing number of studies have investigated the equivalence of data collected over the Internet and data gathered using traditional assessment methods. Many of these studies, however, have had methodological limitations such as using non-equivalent comparison groups and inappropriate data analytic strategies. In this study, 213 undergraduate students were randomly assigned to complete attractiveness ratings via traditional paper-and-pencil methods or through an Internet site; data were analyzed using equivalency testing methodology. Results indicated that mean ratings of physical and sexual attractiveness between the two conditions were equivalent for the full sample. However, when examined separately by respondent gender, ratings of same gender targets were significantly higher for the paper-and-pencil methodology. Implications are discussed.     

The effects of hierarchical structure and visualization ability on computerized information retrieval

This study examined the effects of visualization ability on search time in databases with different hierarchical structures. It was designed to determine whether manipulation of the hierarchical structures of information could accommodate the needs of low‐visualization ability users. The task consisted of finding specific “target” files in each of the four different data structures that varied in depth of organization. The study found the expected effects of organizational structure and visualization ability on retrieval time from the database. It did not find any evidence of an interaction between the two variables on performance. The results suggest that individual differences in performance are the result of differences in perceptual speed and that altering the structure of the information in a database is not an effective way to accommodate to low‐visualization ability users.     

A usability comparison of PDA-based quizzes and paper-and-pencil quizzes

In the last few years, schools and universities have incorporated personal digital assistants (PDAs) into their teaching curricula in an attempt to enhance students’ learning experience and reduce instructors’ workload. One of the most common uses of PDAs in the classroom is as a test administrator. This study compared the usability effectiveness, efficiency, and satisfaction of a PDA-based quiz application to that of standard paper-and-pencil quizzes in a university course. Effectiveness was measured as students’ quiz scores and through a mental workload questionnaire; efficiency was the time it took students to complete each quiz; and satisfaction was evaluated using a subjective user satisfaction questionnaire. The study showed the PDA-based quiz to be more efficient, that is, students completed it in less time than they needed to complete the paper-and-pencil quiz. No differences in effectiveness and satisfaction were found between the two quiz types. Computer anxiety was not affected by the quiz type. For these reasons, as well as other advantages to both students (e.g., real-time scoring) and teachers (e.g., less time spent on grading), PDAs are an attractive test administration option for schools and universities.     

A usability comparison of PDA-based quizzes and paper-and-pencil quizzes

In the last few years, schools and universities have incorporated personal digital assistants (PDAs) into their teaching curricula in an attempt to enhance students’ learning experience and reduce instructors’ workload. One of the most common uses of PDAs in the classroom is as a test administrator. This study compared the usability effectiveness, efficiency, and satisfaction of a PDA-based quiz application to that of standard paper-and-pencil quizzes in a university course. Effectiveness was measured as students’ quiz scores and through a mental workload questionnaire; efficiency was the time it took students to complete each quiz; and satisfaction was evaluated using a subjective user satisfaction questionnaire. The study showed the PDA-based quiz to be more efficient, that is, students completed it in less time than they needed to complete the paper-and-pencil quiz. No differences in effectiveness and satisfaction were found between the two quiz types. Computer anxiety was not affected by the quiz type. For these reasons, as well as other advantages to both students (e.g., real-time scoring) and teachers (e.g., less time spent on grading), PDAs are an attractive test administration option for schools and universities.     

Response rate of bricklayers and supervisors on an internet or a paper-and-pencil questionnaire

It is unclear whether or not internet surveys yield response rates comparable to paper-and-pencil surveys for specific occupational groups, such as construction workers. The objective of this study was to examine the differences in response rates between a paper-and-pencil questionnaire and an internet questionnaire for two construction-related occupations: bricklayers and supervisors. In total 600 participants were randomly assigned to the internet questionnaire survey or to the conventional paper-and-pencil questionnaire survey. Undeliverable postal was excluded from the response rate. A total of 237/590 subjects responded, leading to a response rate of 40% across both arms of the study. Total response rate in the paper-and-pencil questionnaire group (45%, 131/293) was significantly higher (Chi-Square = 4.99, p = 0.025) than in the internet questionnaire arm (36%, 106/297). Among bricklayers, the response rate on the paper-and-pencil questionnaire (44%, 64/147) was significantly higher (Chi-Square 7.36, p = 0.007) than on the internet questionnaire (28%, 42/148). For construction supervisors response rates were not different for both arms of the study, 46% (67/146) on the paper-and-pencil questionnaire and 43% (64/149) on the internet questionnaire. Overall, a paper-and-pencil questionnaire is more effective in terms of response rate than an internet questionnaire in a random sample of two construction professions.Relevance to industryQuestionnaire surveys play an important role in gathering information on interventions aimed at reducing occupational risks and health effects for construction workers. The most effective survey mode should be explored before a large scale survey is carried out.     

The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments

Computer-based assessment (CBA) is yet to have a significant impact on high-stakes educational assessment, but the equivalence between CBA and paper-and-pencil (P&P) test scores will become a central concern in education as CBA increases. It is argued that as CBA and P&P tests provide test takers with qualitatively different experiences, the impact of individual differences on the testing experience, and so statistical equivalence of scores, needs to be considered. As studies of score equivalence have largely ignored individual differences such as computer experience, computer anxiety and computer attitudes, the purpose of this paper is to highlight the potential effects of these. It is concluded that each of these areas is of significance to the study of equivalence and that the often inconsistent findings result from the rapid changes in exposure to technology.     

Response effects and computer-administered questionnaires: the role of the entry task and previous computer experience

In a field experiment at a tourist attraction, a between-subject; design was used to compare paper-and-pencil survey responses against those collected from two computer-administered questionnaire ( CAQ) packages; one using a cursor movement answering protocol, and the second using number pad entry. Three response effects were identified for the groups completing computerized questionnaires: an increase in the range of responses to scale questions, a tendency to select only one choice in answering multiple response questions, and longer answers to open-ended questions. Response effects were shown to be as much a function of the specific computer entry task, as the computerized interviewing situation in general. This finding suggests that computer-collected data should not be compared with data from other methodologies or even different questionnaire software without qualification. Respondent's previous computer experience was shown to have a marginal effect on the size of response effects, and only for the cursor movement protocol. Those with more computer experience completed both computerized questionnaires slighty faster than those with less computer experience.     

Response effects and computer-administered questionnaires: the role of the entry task and previous computer experience

Abstract

In a field experiment at a tourist attraction, a between-subject; design was used to compare paper-and-pencil survey responses against those collected from two computer-administered questionnaire ( CAQ) packages; one using a cursor movement answering protocol, and the second using number pad entry. Three response effects were identified for the groups completing computerized questionnaires: an increase in the range of responses to scale questions, a tendency to select only one choice in answering multiple response questions, and longer answers to open-ended questions. Response effects were shown to be as much a function of the specific computer entry task, as the computerized interviewing situation in general. This finding suggests that computer-collected data should not be compared with data from other methodologies or even different questionnaire software without qualification. Respondent's previous computer experience was shown to have a marginal effect on the size of response effects, and only for the cursor movement protocol. Those with more computer experience completed both computerized questionnaires slighty faster than those with less computer experience.