Comparison of check-all-that-apply and forced-choice Yes/No question formats for sensory characterisation

The application of check-all-that-apply (CATA) questions for sensory product characterisation is gaining acceptance and popularity. This question format has been reported to be a quick and reliable means of gathering sensory profiles from consumers, concurrently with hedonic assessment. However, a limitation of CATA questions is that they do not encourage deep processing by respondents. Forced-choice questions, where respondents answer “yes” or “no” for each term, may encourage systematic processing and be useful when consumers undertake sensory profiling tasks. This research compared sensory profiles elicited by consumers using CATA questions or forced-choice Yes/No questions and contribute to ongoing investigations of CATA questions and related question formats with a view to developing guidelines for best practise. Across seven consumer studies with 600+ consumers and multiple product categories, consistent evidence was obtained that forced-choice Yes/No questions are associated with higher term citation frequencies. However, this did not consistently translate into greater product discrimination. Conclusions regarding similarities and differences amongst samples and the stability of sample and term configurations were generally independent of question format (i.e., whether the sensory data were elicited by CATA or forced-choice Yes/No questions). Overall, the comparison of CATA and forced-choice Yes/No questions for sensory characterisation suggested parity of the two question formats. This extended to consumers’ perceived difficulty and tediousness for completing the test. Regardless of question format, consumers, on average, perceived the tests as easy and not tedious.     

Comparison of Correspondence Analysis based on Hellinger and chi-square distances to obtain sensory spaces from check-all-that-apply (CATA) questions

Sample configurations from check-all-that-apply (CATA) questions are obtained using Correspondence Analysis (CA). Classical CA is based on chi-square distance, which has been reported to be strongly affected by infrequently selected terms. The Hellinger distance has been proposed as an alternative distance metric, and the aim of the present work was to compare product spaces from CATA questions obtained using CA based on chi-square and Hellinger distances. Data sets from 71 studies (5121 consumers), differing by product category, number of consumers, number of samples and number of terms included in the CATA question, as well as frequency of infrequently used terms, were analyzed. For each of the studies, frequency tables were input to CA based on chi-square and Hellinger distances. Sample and term configurations in the first two dimensions were compared using the RV coefficient. Furthermore, the stability of sample and term configurations for each type of distance was evaluated by simulating repeated experiments using a bootstraping resampling approach. Sample and term configurations obtained using Hellinger and chi-square distances were similar (average RV coefficients for sample configurations = 0.99; average RV coefficients between term configurations = 0.89). The stability of sample and term configurations were not largely affected by the type of distance used to analyze frequency tables. Results from the present work suggest that CA based on chi-square and Hellinger distances provide similar results. Contributing to guidelines for practitioners, this research therefore supports classical CA analysis as an acceptable approach to the analysis of sensory-specific CATA data.     

Visual attention by consumers to check-all-that-apply questions: Insights to support methodological development

The use by consumers of simplifying response strategies when answering check-all-that-apply (CATA) questions for sensory characterization cannot be eliminated. However, ways to motivate consumers to cognitively engage more in the processing of CATA questions and provide their full attention to the task may be welcome. In this context, the aim of the present work was to evaluate patterns of visual attention to CATA questions by consumers and use the insights gained to advance knowledge about CATA question design. A consumer study was carried out in which participants were asked to evaluate two product sets with five samples each (potato chips or plain crackers) and to answer a check-all-that-apply question composed of 20 sensory terms, which was presented on a computer screen. Half of the consumers received the CATA question with the terms in a fixed order, whereas for the other half of participants the order of terms was different for each sample (i.e., simulating within participants balancing of term presentation order). While consumers completed the CATA tasks their eye movements were recorded using a remote eye-tracker. Results showed that the first time that consumers read the CATA question (i.e., for sample 1), they processed the terms from left to right and from top to bottom. In general, they considered the whole list of terms and then selected those that applied to the sample. As the task progressed consumers changed how they answered the CATA question, making fewer and shorter eye fixations to complete the task. Presenting the terms in different order for each sample significantly increased the total number and duration of fixations, which indicates higher cognitive effort. Implications for the design of CATA questions and suggestions for further research are discussed.     

Check-all-that-apply (CATA) questions for sensory product characterization by consumers: Investigations into the number of terms used in CATA questions

In recent years methodological research into application of CATA questions has gained momentum. Yet, key questions for this approach remain unaddressed – how to generate the sensory terms that populate CATA questions and how many terms should be used. The second of these questions was addressed in seven consumer studies, involving a total of 735 consumers and five product categories (crackers, cheese, fruit-flavored drinks, chocolate, milk desserts). Sensory product characterizations elicited with “short” and “long” CATA questions (10–17 terms vs. 20–28 terms) were compared on a number of criteria such as frequency of CATA term use, product differences, spatial configurations (samples and terms) and task perceptions. Two strategies for generating “long” lists of CATA terms were examined: adding synonym terms to those already featuring on the “short” list (e.g., ‘hard’ and ‘firm’), and adding antonym terms to those already featuring on the “short” list (e.g., ‘hard’ and ‘not hard’ or ‘natural’ and ‘artificial’). Between-subjects experimental designs were used to compare product characterizations from “short” and “long” CATA questions. Results revealed that “short” and “long” lists of CATA terms generated largely similar results. In general, sample configurations were very similar, as were task perceptions. However, there were, at times, differences in frequency of CATA term use and term configurations, as well as instances where conclusions about sample differences depended on whether “short” or “long” CATA lists were used. Additionally, here was some evidence that CATA questions with “long” lists of synonym or antonym terms may cause a “dilution” effect of the responses. This fits expectations of idiosyncrasy in consumer perception/expression of sensory stimuli, but may be associated with reduced discriminatory ability of the CATA question. How to best balance these opposing considerations is deserving of further investigation.     

Stability of sample configurations from projective mapping: How many consumers are necessary?

Projective mapping for sensory characterisation with consumers has been used for a relatively short period of time, which suggests that the development of guidelines regarding best practices is strongly needed. The present work aims to provide an insight on the minimum number of consumers needed to reach stable sample configurations. Data sets from 21 different consumer studies, differing in product category, number of samples and degree of difference among them, were used to evaluate the influence of the number of consumers on the stability of sample configurations by means of a resampling approach. For each study, 1000 random subsets of different number of consumers were generated from the original data set. For each virtual panel, sample configurations were obtained using Multiple Factor Analysis. The agreement between them and the reference configurations (obtained with all the consumers) was evaluated through the RV coefficient, using the first two and the first four dimensions of the MFA. Results showed that the stability of sample configuration clearly depended on the degree of difference and type of differences among samples and the number of samples in the dataset. Across the 21 data sets analysed, results suggested that when working with widely different samples, 50 consumers seems as a safe recommendation of minimum consumer panel size to obtain reliable results with projective mapping. However, after any characterisation by projective mapping is completed, it is highly recommended to check, a posteriori, the reliability of the sample space configuration using a bootstrapping procedure.     

Recommendations for use of balanced presentation order of terms in CATA questions

Balancing presentation order of the terms of a CATA question between and within participants has been recommended to minimize the influence of primacy bias on consumer responses and maintain their attention throughout the task. Compared to balancing presentation order only between participants, this experimental factor may make the task more difficult/tedious for participants and shift their attention away from the sensory characterization task with possible detrimental effects for sample discrimination. The aim of the present work was to compare sensory product characterizations obtained using CATA questions in which presentation order of the terms was balanced between participants with those obtained using CATA questions in which presentation order of the terms was balanced both between and within participants. Nine studies with 1028 consumers involving different product categories were conducted. Between-subjects experimental designs were used in all studies. No major differences were found in the sensory product characterizations obtained using presentation orders balanced between participants (BB) and both between and within participants (BW).     

What does it mean to check-all-that-apply? Four case studies with beverages

The use of check-all-that-apply (CATA) questions in product-focused consumer research on foods and beverages is now common, and the method is known to provide valid sensory product characterisations. Extensive methodological research has been conducted and has supported uptake, but understanding of how consumers complete CATA questions is incomplete, particularly with regard to their decision to select or not a term to describe the sensory properties of products. The present research was situated within this gap, and using open-ended questions participants (n = 636) were asked to describe how they perceived a pair of samples with regard to an attribute and link this to CATA term selection. The results, obtained for taste (‘sweet’ and ‘sour/acidic’) and flavour (‘cinnamon’ and ‘smoky’) confirmed consumers’ ability to accurately perform sensory characterisation tasks. In particular, it was found that: i) the great majority of the consumers accurately used the CATA terms for describing the sensory characteristics they perceived in a sample, ii) when a term was not selected for describing samples, the majority of the consumers indicated that the corresponding sensory attribute was not perceived, iii) when a term was selected for describing only one of the samples in a pair, consumers reported to have perceived a difference in attribute intensity between the samples. Thus, CATA questions remain a desirable option for sensory product characterisation tasks with consumers, but should be selected with thought as they may not always be able to achieve desired sample discrimination due to the binary nature of the responses.     

RATA questions are not likely to bias hedonic scores

Concurrent elicitation of hedonic and sensory information from consumers helps to identify the sensory attributes that drive consumer preferences. However, there is a risk that asking consumers about specific sensory characteristics can bias their hedonic responses. This research extends previous work by investigating if concurrent use of rate-all-that-apply (RATA) questions, a variant of CATA questions is associated with bias of the hedonic scores. RATA questions extend CATA questions by requiring consumers to also rate the intensity of the terms that are applicable for describing samples (‘high’, ‘medium’ and ‘low’). In eight consumer studies (n = 560) across five product categories there was no evidence of concurrent use of RATA questions influencing hedonic scores. When RATA questions were used concurrently with a hedonic question there was a strong tendency for greater hedonic discrimination between samples (5 of 8 studies). This suggested that the discriminative capacity of the hedonic question was increased by inclusion of the RATA question, a result that could be attributable to the RATA question increasing consumers’ engagement and attention to the task. Further research is required to confirm this result and tentative explanation.     

Lack of evidence that concurrent sensory product characterisation using CATA questions bias hedonic scores

In response to the growing use of consumers for sensory product characterisation, methodological research contributing to development of best practise guidelines is ongoing. We focus here on concurrent elicitation of hedonic and sensory product characterisation by check-all-that-apply (CATA) questions. Jaeger et al. (2013b) reported that CATA questions only caused weak and transient bias of co-elicited hedonic scores. In the current research six studies were conducted, in which more than 700 consumers took part. Five product categories were tested (rice crackers, lite bread, cheese, kiwifruit, black currant drinks) with 4–7 samples per study. In none of these studies was evidence obtained suggesting bias of hedonic scores and it is now possible to conclude with greater certainty that co-elicitation of hedonic scores and product attribute information using CATA questions is unlikely to bias hedonic scores. A second result of the current research was that the use of designs that rotate presentation order of CATA terms was not associated with hedonic bias, and neither was the use of the forced Yes–No CATA question format. In future research, in light of a strong dominance of positive CATA terms used in these studies, we recommend studying more thoroughly the influence of positive/negative/neutral words in CATA lists as a possible source of hedonic bias. An exploratory component to this research suggested that consumers perceive the concurrent elicitation of hedonic and CATA responses as easy, but that too many samples may make the task tedious.     

Further investigations into the reproducibility of check-all-that-apply (CATA) questions for sensory product characterization elicited by consumers

The purpose of the present research was further investigate the reproducibility of check-all-that-apply (CATA) questions for sensory product characterization. Evaluations obtained when such questions are used by consumers are rarely replicated and therefore reproducibility of the data may be at risk. Results from the present work, which included five studies, each with 100–200 consumers across a range of product categories, revealed that sensory product characterizations obtained using CATA questions with consumers are highly reproducible. Hence, the research confirms previous research by Jaeger, Chheang, et al. (2013) and extends it to the use of CATA terms in randomised presentation order as has been recommended to avoid satisficing response behavior. In the future, if CATA studies are conducted without replication and researches seek to examine the reliability of CATA data, the use of a posterior bootstrapping re-sampling approach is suggested.     

Check-all-that-apply (CATA) questions: Sensory term citation frequency reflects rated term intensity and applicability

CATA questions (check-all-that-apply) are popular for sensory product characterisation tasks with consumers. Lingering uncertainty over the meaning of term citation frequency was addressed in the present research, which directly compared CATA questions with intensity and applicability ratings obtained on 10 cm unstructured line scales. In three studies (n ~ 210 consumers per study), evidence fitting with expectations based on the extant literature was obtained to confirm that citation frequency reflects intensity. The two response types were strongly linearly related, meaning that it is possible to infer significant differences between samples for a given CATA term as representing differences in perceived intensity. This finding emphasises how, despite the simplicity of the CATA task and the fact that individual responses are not measures of intensity, the average citation frequencies do reflect perceived intensity. Therefore, across a group of consumers, there is no information loss relative to rating scales. A systematic difference between citation frequencies and intensity ratings was attenuation of scale use in the latter, which was between 15 and 80 on a 0 to 100 scale. Attenuation was less pronounced in CATA responses which spanned approx. 5 to 85%. In the second part of the research, which included six studies (n ~ 135 to 205 consumers per study) CATA questions were compared with applicability ratings obtained on 10 cm unstructured line scales. The results were highly similar to the comparison between CATA questions and intensity scales and further confirmed that average CATA term citation frequency can be interpreted as representing (but not directly measuring) consumers’ intensity perceptions. The attenuation in scale range was again observed. Collectively, the results contribute further support for the popularity of CATA questions for sensory characterization by consumers in product testing.     

Visual attention to rate-all-that-apply (RATA) questions: A case study with apple images as food stimuli

The check-all-that-apply (CATA) question format and variants hereof have become very popular in sensory and consumer research. In the present study, focus is directed to rate-all-that-apply (RATA) questions where participants in addition to selecting terms that are applicable for describing a focal stimulus must also indicate if this term has an intensity corresponding to ‘low’, ‘medium’ or ‘high’. Knowledge of how assessors approach the RATA task is lacking, and this study was dedicated to understanding consumers’ visual attention to RATA questions. This aim was achieved by tracking participants’ eye movements when using a 15-term question to evaluate images of apples with different degrees of internal defect (flesh browning). As expected, participants first looked at the terms, then checked the box corresponding to “applies” and finally selected intensity level, i.e. participants tended to rate attribute intensity immediately after indicating that a term was applicable to describing a focal sample. Commensurate with the two parts to the task, preliminary evidence of more visual attention was suggestive of larger cognitive effort in RATA than CATA questions. Potentially this could be associated with a more “analytical” frame of mind among participants. More visual attention to the RATA question was associated with higher ability to discriminate among samples, in agreement with previous results for CATA questions. Overall, the research supported consumers’ ability to use RATA questions as intended, and indirectly to continued methodological uptake.