Paired Comparison Tests: A System for Informed Decision Making
Have you ever been in a meeting where teams members completely disagree on what to do next? Do you think this happens because one person is super smart and totally knows the future while the other person is a complete idiot and wants the company to fail?
Or is more likely that both people have opinions based on their perspective or best interest?
Well in my experience, the answer is usually that both sides are wrong. How do I know? Because when there is a debate, we stop and go learn more. Instead of making a subjective decision, let's make a data grounded decision. By what method? Paired comparison tests are a survey tool where you can compare two options head to head and isolate the difference that you want to learn more about.
Let's use packaging design as an example. It is a perfect example of something that absolutely matters and can drive purchasing decisions, but can also feel very subjective like picking your favorite color.
Here are two labels for Brain Brew Whiskeys Riverboat series. I created this test back in 2019. Which one do you think more customers preferred?
Left
Right
The answer is the labels on the left, but not significantly. It was 57.6%. Typically you want one option to win 2:1 or 66% to be significant and drive confident decision making.
Research informs decision making, it does not make the decision.
I've seen teams outsource decisions to 100 respondents on the internet. The key is to understand why the respondents picked something and let that support why you'd pick that option too if it aligns with the business goals.
To do this, you need to design the survey with more questions to learn about the respondent and why they like one option over another. The mistake is that people use qualitative questions to ask why they picked what they did. Instead capture these qualitatively and look for strong correlations with purchase intent. Use the open ended questions to ask about what the difference is. I've had respondents admit to taking a few minutes to discover the difference between 2 concepts, because they were so similar. For example with these:
Did you notice the purple versus the blue, or that the words were above or below the image, or that the label was flat versus rounded, or that fonts were easier to read?
Demographics questions:
With all Brain Brew Whiskey surveys, knowing how often they drink bourbon is important and most companies are good about capturing category usage and basic demographics like age, country, and household size. But there other questions we include like trend setter, self esteem, early adopter, and even thinking style that tell use more about their personality.
In this case none of that mattered. This was important as we were curious if there would be any difference between the US, UK and Canada that would drive label design in different markets and cause us to do another test with a bigger sample size.
Sensory Transfer Dimensions:
My favorite questions to add to a paired comparison test and the ones that are frequently under utilized. These give us data for why someone prefers an option and are more helpful then demographic profiles. Examples.
These questions don't take respondents long to answer, so adding more doesn't bog down the survey so when in doubt add the question.
In this survey, "feels more for people like me" informed not only our mindset on bourbon being a personal choice but that if everyone had their own personal bourbon that it would correlate with purchase intent. In these surveys, we expect to see correlations. In package surveys with other clients in various industries and categories, the eye catching question typically has the highest correlation with purchase, supporting the importance of grabbing customers attention. Often quality is the next highest factor along with making the product's benefit stand out more. I've seen this done well in a few ways: using bigger fonts to call out the benefit, like "3x more protein," showing the product already in use alongside other items, like whiskey poured into a cocktail or meat plated with sides, and removing packaging altogether so more of the actual product shows through instead of hiding it inside a box.
The biggest impacts often come down to matching the meaningful uniqueness of the concept, but sometimes you have to make a decision where there is no right or wrong, it is just what people like and grabs their attention.
Research as a system.
At the start I mentioned using research surveys to bring data into a decision and move forward. That is exactly what Brain Brew did back in 2019, and guess what, the current river boat series doesn't look like any of these. Why? Because research is part of the system. When it is fast and cheap to test something new, we learn more and have more confidence to continuously improve. Also the mindset of the people is progress, not perfection. Neither design was perfect and if we wanted to be perfect, we would never ship. Instead we move forward with the best option for now, knowing that next week we will be smarter.
One of the best parts of paired comparison surveys is that they work well with small sample sizes. Often, 50 respondents are enough to tell you whether a difference meaningfully affects purchase intent, or whether you're just debating insignificant details. At about $10 per respondent, that means you can make a far more informed decision for roughly $500 by the next day, instead of losing hours to debate and indecision.
I used Brain Brew as the example here because I'm able to share their data publicly, but this approach works for any organization with an easy way to survey its customers or stakeholders. If you can't reach decision-makers directly, rethink your feedback system: lean on an advisory board, an internal team, or an expert panel instead. We also offer concept feedback tools on Jump Start Your Brain that keep your ideas confidential while using AI to evaluate ideas.