What does it mean to check-all-that-apply? Four case studies with beverages

The use of check-all-that-apply (CATA) questions in product-focused consumer research on foods and beverages is now common, and the method is known to provide valid sensory product characterisations. Extensive methodological research has been conducted and has supported uptake, but understanding of how consumers complete CATA questions is incomplete, particularly with regard to their decision to select or not a term to describe the sensory properties of products. The present research was situated within this gap, and using open-ended questions participants (n = 636) were asked to describe how they perceived a pair of samples with regard to an attribute and link this to CATA term selection. The results, obtained for taste (‘sweet’ and ‘sour/acidic’) and flavour (‘cinnamon’ and ‘smoky’) confirmed consumers’ ability to accurately perform sensory characterisation tasks. In particular, it was found that: i) the great majority of the consumers accurately used the CATA terms for describing the sensory characteristics they perceived in a sample, ii) when a term was not selected for describing samples, the majority of the consumers indicated that the corresponding sensory attribute was not perceived, iii) when a term was selected for describing only one of the samples in a pair, consumers reported to have perceived a difference in attribute intensity between the samples. Thus, CATA questions remain a desirable option for sensory product characterisation tasks with consumers, but should be selected with thought as they may not always be able to achieve desired sample discrimination due to the binary nature of the responses.     

Comparison of check-all-that-apply and forced-choice Yes/No question formats for sensory characterisation

The application of check-all-that-apply (CATA) questions for sensory product characterisation is gaining acceptance and popularity. This question format has been reported to be a quick and reliable means of gathering sensory profiles from consumers, concurrently with hedonic assessment. However, a limitation of CATA questions is that they do not encourage deep processing by respondents. Forced-choice questions, where respondents answer “yes” or “no” for each term, may encourage systematic processing and be useful when consumers undertake sensory profiling tasks. This research compared sensory profiles elicited by consumers using CATA questions or forced-choice Yes/No questions and contribute to ongoing investigations of CATA questions and related question formats with a view to developing guidelines for best practise. Across seven consumer studies with 600+ consumers and multiple product categories, consistent evidence was obtained that forced-choice Yes/No questions are associated with higher term citation frequencies. However, this did not consistently translate into greater product discrimination. Conclusions regarding similarities and differences amongst samples and the stability of sample and term configurations were generally independent of question format (i.e., whether the sensory data were elicited by CATA or forced-choice Yes/No questions). Overall, the comparison of CATA and forced-choice Yes/No questions for sensory characterisation suggested parity of the two question formats. This extended to consumers’ perceived difficulty and tediousness for completing the test. Regardless of question format, consumers, on average, perceived the tests as easy and not tedious.     

Comparison of two sensory profiling techniques based on consumer perception

Eight chocolate milk desserts with different formulation were evaluated by two groups of consumers. Fifty consumers evaluated the samples and indicated their overall liking and answered a CATA question. Meanwhile, 40 consumers elicited up to four words to describe the desserts and completed a projective mapping task. Projective mapping and the check-all-that-apply question provided very similar sensory profiles for the evaluated milk desserts. Differences in the sensory characteristics of the samples were explained by differences in their formulations, which suggest the validity of the sensory profiles given by consumers. Projective mapping and the CATA question consisted on valuable tools to understand their perception of the sensory and hedonic characteristics of the desserts. These methodologies could consist on useful and interesting complimentary techniques to trained assessors’ data, being CATA question easier to understand and less time consuming for consumers.     

List length has little impact on consumers’ visual attention to CATA questions

Check-all-that-apply (CATA) questions continue to gain popularity and attract interest in developing guidelines regarding their implementation and analysis. The current research continues on this path and considers visual attention by consumers to CATA questions. This is a pre-requisite for elicitation of valid data. Extending previous research, we consider whether CATA question list length influences consumers’ visual attention to the task when it moderately increases from 12 to 20 terms. In a study with six wheat crackers consumers (n ∼ 120) used a CATA questions with either a “short” list (12 terms) or a “long” list (20 terms). The main difference in visual processing was less attention to individual CATA terms when using “long” lists, but greater sustained visual attention to the task. The sensory characterisations for wheat crackers elicited by “long” and “short” lists differed only in minor ways, pointing to little practical impact of the observed differences in visual processing. Upon replication of the current results, consideration of how these findings generalise to test situations with longer CATA questions (30+ terms), tests with fewer/more samples and/or samples with more complex sensory characteristics is warranted.     

Check-all-that-apply (CATA) questions for sensory product characterization by consumers: Investigations into the number of terms used in CATA questions

In recent years methodological research into application of CATA questions has gained momentum. Yet, key questions for this approach remain unaddressed – how to generate the sensory terms that populate CATA questions and how many terms should be used. The second of these questions was addressed in seven consumer studies, involving a total of 735 consumers and five product categories (crackers, cheese, fruit-flavored drinks, chocolate, milk desserts). Sensory product characterizations elicited with “short” and “long” CATA questions (10–17 terms vs. 20–28 terms) were compared on a number of criteria such as frequency of CATA term use, product differences, spatial configurations (samples and terms) and task perceptions. Two strategies for generating “long” lists of CATA terms were examined: adding synonym terms to those already featuring on the “short” list (e.g., ‘hard’ and ‘firm’), and adding antonym terms to those already featuring on the “short” list (e.g., ‘hard’ and ‘not hard’ or ‘natural’ and ‘artificial’). Between-subjects experimental designs were used to compare product characterizations from “short” and “long” CATA questions. Results revealed that “short” and “long” lists of CATA terms generated largely similar results. In general, sample configurations were very similar, as were task perceptions. However, there were, at times, differences in frequency of CATA term use and term configurations, as well as instances where conclusions about sample differences depended on whether “short” or “long” CATA lists were used. Additionally, here was some evidence that CATA questions with “long” lists of synonym or antonym terms may cause a “dilution” effect of the responses. This fits expectations of idiosyncrasy in consumer perception/expression of sensory stimuli, but may be associated with reduced discriminatory ability of the CATA question. How to best balance these opposing considerations is deserving of further investigation.     

Check-all-that-apply (CATA) questions: Sensory term citation frequency reflects rated term intensity and applicability

CATA questions (check-all-that-apply) are popular for sensory product characterisation tasks with consumers. Lingering uncertainty over the meaning of term citation frequency was addressed in the present research, which directly compared CATA questions with intensity and applicability ratings obtained on 10 cm unstructured line scales. In three studies (n ~ 210 consumers per study), evidence fitting with expectations based on the extant literature was obtained to confirm that citation frequency reflects intensity. The two response types were strongly linearly related, meaning that it is possible to infer significant differences between samples for a given CATA term as representing differences in perceived intensity. This finding emphasises how, despite the simplicity of the CATA task and the fact that individual responses are not measures of intensity, the average citation frequencies do reflect perceived intensity. Therefore, across a group of consumers, there is no information loss relative to rating scales. A systematic difference between citation frequencies and intensity ratings was attenuation of scale use in the latter, which was between 15 and 80 on a 0 to 100 scale. Attenuation was less pronounced in CATA responses which spanned approx. 5 to 85%. In the second part of the research, which included six studies (n ~ 135 to 205 consumers per study) CATA questions were compared with applicability ratings obtained on 10 cm unstructured line scales. The results were highly similar to the comparison between CATA questions and intensity scales and further confirmed that average CATA term citation frequency can be interpreted as representing (but not directly measuring) consumers’ intensity perceptions. The attenuation in scale range was again observed. Collectively, the results contribute further support for the popularity of CATA questions for sensory characterization by consumers in product testing.     

Visual attention to rate-all-that-apply (RATA) questions: A case study with apple images as food stimuli

The check-all-that-apply (CATA) question format and variants hereof have become very popular in sensory and consumer research. In the present study, focus is directed to rate-all-that-apply (RATA) questions where participants in addition to selecting terms that are applicable for describing a focal stimulus must also indicate if this term has an intensity corresponding to ‘low’, ‘medium’ or ‘high’. Knowledge of how assessors approach the RATA task is lacking, and this study was dedicated to understanding consumers’ visual attention to RATA questions. This aim was achieved by tracking participants’ eye movements when using a 15-term question to evaluate images of apples with different degrees of internal defect (flesh browning). As expected, participants first looked at the terms, then checked the box corresponding to “applies” and finally selected intensity level, i.e. participants tended to rate attribute intensity immediately after indicating that a term was applicable to describing a focal sample. Commensurate with the two parts to the task, preliminary evidence of more visual attention was suggestive of larger cognitive effort in RATA than CATA questions. Potentially this could be associated with a more “analytical” frame of mind among participants. More visual attention to the RATA question was associated with higher ability to discriminate among samples, in agreement with previous results for CATA questions. Overall, the research supported consumers’ ability to use RATA questions as intended, and indirectly to continued methodological uptake.     

Visual attention by consumers to check-all-that-apply questions: Insights to support methodological development

The use by consumers of simplifying response strategies when answering check-all-that-apply (CATA) questions for sensory characterization cannot be eliminated. However, ways to motivate consumers to cognitively engage more in the processing of CATA questions and provide their full attention to the task may be welcome. In this context, the aim of the present work was to evaluate patterns of visual attention to CATA questions by consumers and use the insights gained to advance knowledge about CATA question design. A consumer study was carried out in which participants were asked to evaluate two product sets with five samples each (potato chips or plain crackers) and to answer a check-all-that-apply question composed of 20 sensory terms, which was presented on a computer screen. Half of the consumers received the CATA question with the terms in a fixed order, whereas for the other half of participants the order of terms was different for each sample (i.e., simulating within participants balancing of term presentation order). While consumers completed the CATA tasks their eye movements were recorded using a remote eye-tracker. Results showed that the first time that consumers read the CATA question (i.e., for sample 1), they processed the terms from left to right and from top to bottom. In general, they considered the whole list of terms and then selected those that applied to the sample. As the task progressed consumers changed how they answered the CATA question, making fewer and shorter eye fixations to complete the task. Presenting the terms in different order for each sample significantly increased the total number and duration of fixations, which indicates higher cognitive effort. Implications for the design of CATA questions and suggestions for further research are discussed.     

Further investigations into the reproducibility of check-all-that-apply (CATA) questions for sensory product characterization elicited by consumers

The purpose of the present research was further investigate the reproducibility of check-all-that-apply (CATA) questions for sensory product characterization. Evaluations obtained when such questions are used by consumers are rarely replicated and therefore reproducibility of the data may be at risk. Results from the present work, which included five studies, each with 100–200 consumers across a range of product categories, revealed that sensory product characterizations obtained using CATA questions with consumers are highly reproducible. Hence, the research confirms previous research by Jaeger, Chheang, et al. (2013) and extends it to the use of CATA terms in randomised presentation order as has been recommended to avoid satisficing response behavior. In the future, if CATA studies are conducted without replication and researches seek to examine the reliability of CATA data, the use of a posterior bootstrapping re-sampling approach is suggested.     

Lack of evidence that concurrent sensory product characterisation using CATA questions bias hedonic scores

In response to the growing use of consumers for sensory product characterisation, methodological research contributing to development of best practise guidelines is ongoing. We focus here on concurrent elicitation of hedonic and sensory product characterisation by check-all-that-apply (CATA) questions. Jaeger et al. (2013b) reported that CATA questions only caused weak and transient bias of co-elicited hedonic scores. In the current research six studies were conducted, in which more than 700 consumers took part. Five product categories were tested (rice crackers, lite bread, cheese, kiwifruit, black currant drinks) with 4–7 samples per study. In none of these studies was evidence obtained suggesting bias of hedonic scores and it is now possible to conclude with greater certainty that co-elicitation of hedonic scores and product attribute information using CATA questions is unlikely to bias hedonic scores. A second result of the current research was that the use of designs that rotate presentation order of CATA terms was not associated with hedonic bias, and neither was the use of the forced Yes–No CATA question format. In future research, in light of a strong dominance of positive CATA terms used in these studies, we recommend studying more thoroughly the influence of positive/negative/neutral words in CATA lists as a possible source of hedonic bias. An exploratory component to this research suggested that consumers perceive the concurrent elicitation of hedonic and CATA responses as easy, but that too many samples may make the task tedious.     

The concurrent use of JAR and CATA questions in hedonic scaling is unlikely to cause hedonic bias,but may increase product discrimination

The concurrent use of CATA questions (check-all-that-apply) with hedonic scaling in product research with consumers has gained acceptance in recent years. However, CATA questions do not enable product developers to identify consumers’ preferred intensity of a sensory attribute directly, and for this reason just-about-right (JAR) questions are sometimes used together with questions that quantify attribute presence and/or intensity. Using JAR questions, consumers have to describe each of the samples in terms of each specific attribute, typically: 1 = ‘Not enough…’, 3 = ‘Just about right’, and 5 = ‘Much too…’. Past research has suggested that co-elicitation of hedonic and JAR responses bias the hedonic data. This undesired effect could extend to the case where JAR questions are used concurrently with hedonic and CATA questions. In seven studies (N = 762) across seven product categories no such evidence was found. Additionally, there was some evidence pointing to an increase in product discrimination based on hedonic and CATA responses. Preliminary data indicated that the inclusion of JAR questions in product tests also eliciting hedonic and CATA responses did not negatively affect consumers’ perception of the task, which, on average, was still regarded as easy and not tedious. Future research is required to validate the findings across more varied experimental settings.