How to measure consumer's inconsistency in sensory testing?

Consumer sensory testing is the basis for determining directions of product development in the food industry. However, while compliance assessment by trained and expert assessors is well developed, few information is available on testing consumer consistency. Therefore, we provide a statistical framework to rank assessors and attributes according to the level of inconsistency, as well as to identify inconsistent assessors, based on Kendall rank correlation coefficients. The detection of (in)consistency requires evaluations on two connected scales. The suggested approach is illustrated by data from sensory tests of biscuits enriched with three pollens at different levels. 100 consumers evaluated the samples on two different scales (nine category monotonic ascending hedonic response scale, five-category just about right (JAR) intensity scale). The 88 consistent assessors are found using a wider range of both the liking scale and JAR scale than the 12 inconsistent assessors whose evaluations do not have a significantly negative rank correlation. Future consumer tests are recommended to include multiple scales. The proposed framework aims to identify and even filter out the potentially biasing inconsistent evaluations. Questions on attributes leading to highly inconsistent responses should be reconsidered in future sensory tests on the same food product.