Why Your Product Tests Are Only Telling Half the Story
Survey data told the product team everything looked good. Performance ratings were strong, preference scores beat the competition, and purchase intent hit the target.
Six months after launch, the repurchase numbers told a different story.
Consumers tried the product once and walked away. Reviews mentioned issues the survey never caught. The team went back through the research, searching for answers that weren’t there.
Part of the challenge is timing. Most product tests run for relatively short periods with limited usage. Small irritations get overlooked when people are excited about trying something new, but those same irritations become deal-breakers over time. Longer studies can catch these issues, though they require more time, budget, and product supply than many teams have available.
This discrepancy between what consumers report and how they behave is often referred to in research as the say-do gap, the difference between what people say they do and what they actually do in real-world usage contexts. That’s where independent sensory research becomes particularly valuable. It helps identify potential friction points before they show up in real-world repurchase behavior.
The Limits of Survey-Based Consumer Research Methodology
Surveys have been the backbone of product testing for decades, and for good reason. Closed-ended questions deliver efficient, reliable data for benchmarking and prioritization. They tell teams what consumers think, and that foundation remains essential.
However, the challenge emerges when competition intensifies and margins tighten.
Brands that get closer to their consumers beyond ratings and scores gain an advantage. A product might receive strong overall ratings in a survey, yet sensory testing could reveal that a subtle scent, an unexpected texture, or the way a product feels in-hand creates hesitation during actual use.
These insights are difficult for respondents to articulate because simple surveys with primarily closed-ended questions do not create space for unanticipated issues to surface. Because humans struggle with accurate recall, a key contributor to the say-do gap, behavioral observation and hands-on methods help reveal actions that surveys alone cannot capture. Consumers experience the friction without always being able to communicate it clearly. Well-designed surveys can flag that an issue exists, though sensory testing is often needed to reveal what’s causing it.
Consumers experience the friction without always being able to communicate what they’re experiencing in a way researchers can understand. Well-designed surveys can flag that an issue exists, though sensory testing is often needed to reveal what’s actually causing it.
It’s a challenge that can have significant consequences in product development, according to Sandy Clear, Insights Consultant at L&E Research. Sensory data plays an essential role in creating a more complete picture during product development. It reveals insights that inform better survey design for confirmation testing later in the process.
“Surveys have traditionally been the primary vehicle used to gather consumers’ opinions during product development,” says Clear. “Without complementary approaches like sensory testing, brands often get the ‘what’ from survey ratings but lack confidence in the ‘why.’ That gap can lead to over-investing in features that are merely nice to have or missing critical design flaws until late in development.”
New technologies support this work. Advances in text analytics and AI-driven probing have made open-ended feedback more timely and scalable. The ability to collect and analyze photos and videos reveals behaviors and usage moments consumers may not consciously report. Follow-up conversations with panelists deepen understanding. These tools accelerate analysis and add clarity as complements to hands-on observation, not replacements.
When the “why” is clear, teams can distinguish what’s essential to product success from what can be adjusted, deprioritized, or removed.
Surveys remain a vital starting point, most powerful when paired with sensory, behavioral, and qualitative inputs. Together, these approaches reduce risk, improve decision-making, and lead to products that resonate with consumers in real-world use.
Real Examples Where Hands-On Testing Revealed What Surveys Couldn’t
In one recent study, a product development team tested cleaning performance. The survey focused heavily on efficacy-related measures. Most products performed similarly on cleaning power, though one test product received significantly lower convenience ratings.
The reason wasn’t immediately clear from the data.
Follow-up hands-on research highlighted the issue: a seemingly minor formula change had altered how the product interacted with the packaging, making it difficult to dispense. Consumers struggled to get the product out of the bottle, friction that only became obvious when they physically handled and used it. While the survey’s open-ended questions flagged that a convenience issue existed, observing participants use the product in real time revealed exactly what was causing it.
Another case involved a ready-to-make meal. Consumers evaluated the product and provided mixed feedback that was difficult to interpret through survey responses. The ratings varied widely without a clear pattern.
When participants prepared the meal in person, the issue became clear immediately.
The instructions were being misunderstood. Consumers skipped steps or combined ingredients in the wrong order, leading to inconsistent preparation and a final product that was far less appealing than intended. Observing the process showed friction points that consumers themselves struggled to articulate in a questionnaire.
A third example involved a product that outperformed competitors on traditional, objective measures of absorbency. The lab data was strong, yet consumers didn’t prefer it on that very attribute because the competitive product featured a visual pattern that looked more absorbent. That design signal shaped expectations and perceptions, even though real-world performance differences were minimal and not meaningful to consumers.
Across these examples, surveys accurately flagged where issues existed. Hands-on sensory and observational research, along with skilled conversations with consumers, explained why.
These patterns show when methodology needs to expand beyond surveys alone.
Designing Consumer Research Methodology That Matches Your Needs
Knowing when to combine surveys with sensory testing starts with understanding which phase of product development the brand is in: investigative, development, or confirmation.
“I always start by understanding where the brand is in the product development process, what they already know, what they don’t know, and most importantly, what decisions the research needs to support,” explains Sandy. Once the decision context is clear, we design the research to match those needs rather than defaulting to any single methodology.”
When a product or category has been extensively studied, a well-designed survey may be sufficient, particularly if results come back largely as anticipated. The same holds true for cyclical or tracking work, such as ongoing satisfaction studies, where standardized measures are appropriate and efficient, but even in these cases, relying on closed-ended questions alone leaves insight on the table.
Open-ended responses allow consumers to surface issues or benefits in their own words, often highlighting factors that weren’t originally on the development team’s radar. Photos and videos capture how consumers use a product and what they believe signals effectiveness or quality, adding important behavioral context.
Signals that a survey alone won’t be enough typically emerge when there’s uncertainty, inconsistency, or surprise in the results. Scores don’t align with expectations. Performance and preference don’t move together. Consumers struggle to clearly explain their ratings.
These situations are common in early-stage or innovative product work, where the team is still learning what matters most to the target audience.
The second touchpoint might take different forms depending on the question being answered and the budget. Follow-up conversations with consumers are often the fastest way to add depth and clarity. If questions center on real-world usage or friction, in-home or observational research may be more appropriate.
If consumers like how a product performs without using it as often as expected, targeted sensory testing such as fragrance or tactile evaluation can pinpoint the barrier.
By pairing traditional surveys with behavioral and sensory methods, research can close the say-do gap, ensuring what people say aligns more closely with how they use products in real situations. This combination gives brands the confidence they need to make smarter, lower-risk decisions throughout the development process.
The Role of Controlled Environments and Precise Recruiting
Turning product research into reliable, decision-ready insight requires having the right people, in the right environment, doing the right things the right way.
“If the wrong participants are in the study, even the most sophisticated methodology will produce misleading results,” Sandy describes. “Our extensive database allows us to recruit highly specific audiences that are truly representative of the target consumer, protecting clients from costly missteps.”
Equally important, though often overlooked, is ensuring participants clearly understand what’s expected of them.
Instructions aren’t one-size-fits-all but custom-designed for each study, product, and methodology. When expectations aren’t clear, consumers may misuse a product, skip steps, or disengage, introducing noise and reducing data quality.
For that reason, L&E has dedicated project managers, standardized processes, and real-time support to ensure participants are set up for success and remain compliant throughout the study. These processes are backed by ISO 20252 and ISO 27001 certifications that ensure both research quality and data security meet international standards.
L&E’s facilities make it possible to observe what surveys and remote methods simply can’t capture.
Controlled, in-person environments allow teams to evaluate visual cues, scent, touch, taste, and sound in ways that reflect real usage while maintaining consistency across participants.
For food and beverage research, the test kitchen is a standout asset. Multiple products can be prepared on-site at the same time under identical conditions. This dramatically improves efficiency while eliminating variables that can cloud interpretation. Instead of relying on participants to prepare products correctly at home, the team controls preparation and watches how consumers respond to the finished result.
Teams can watch behaviors unfold, identify friction points in real time, and understand how perceptions are formed rather than just what consumers say after the fact.
L&E’s facilities allow teams to flexibly combine methods, control variables, and gain insights that remote panels and surveys alone simply cannot deliver.
From Data Collection to Actionable Insight
Research gathers the information, but analysis is what turns it into insight.
When a study is thoughtfully designed with a clear purpose for each activity and question, the analysis becomes both efficient and rewarding. The process starts with foundational survey data, reviewing topline results and flagging any outcomes that are unexpected or misaligned with initial hypotheses.
Key objectives are typically addressed through multiple elements of the research design, which allows findings to be validated across different data sources.
New questions commonly emerge during analysis as a natural part of the scientific process. Some questions can be answered through targeted follow-up conversations with consumers, while others may require additional exploratory or sensory-focused research. Rather than seeing this as a limitation, each study sharpens focus and informs smarter next steps.
In many cases, studies are intentionally designed with both a broad survey component and a smaller set of follow-up discussions, knowing that deeper clarification will be needed. A study often serves as a foundational phase, identifying where additional work such as sensory testing, product refinement, or concept optimization will deliver the greatest value.
L&E supports this process from start to finish. Designing the research. Preparing products and creating usage instructions. Programming surveys and tools needed for data collection. Analyzing the information and providing a summary that connects findings to decisions.
The combined insight from surveys and sensory research gives brands far more than direction by providing confidence.
Instead of knowing only what consumers think, brands understand why they think it, how they experience the product, and what actions will most meaningfully improve performance in the market. That level of clarity simply isn’t possible with survey data alone.
What Comes Next
Product testing shouldn’t leave teams guessing. When consumer research methodology combines surveys with sensory testing, brands get complete feedback before launch, understanding what consumers prefer, why they prefer it, and how they actually use the product in real-world conditions.
L&E supports brands across the entire product lifecycle with customized research strategies, precise recruiting, and facilities designed to capture what surveys alone miss. From the test kitchen to controlled sensory environments, the infrastructure is built to deliver reliable, decision-ready insight.
Start a conversation with L&E Research today to learn more about product testing designed to reflect real-world consumer behavior.
