Is Your Idea Any Good? Trusting Research in the Age of AI
You have an idea. Maybe it's a new product, a new service, or a new way of doing something at your company. The next question is always the same: is it any good?
Most people answer that question by trusting their own gut. That's the first mistake.
Don't Trust Just Your Own Judgment
If you came up with the idea, or you're leading the team behind it, you're the worst person to judge it because you're emotionally attached. You've spent hours thinking about it, you believe in it, and you're looking at it from your own perspective instead of the customer's. Also you know more about it than what you communicated and are imagining what it could be versus what it really is.
Listen to Someone Outside Your Own Head
The fix isn't complicated: get an outside perspective before you commit real time or money. That outside perspective can come from actual customers, an expert panel, or, increasingly, an AI system. Each has strengths and weaknesses, and we use them all.
One practical note: confidentiality matters, so using an internal team or a tool like Truth Teller is our go to when we can't show customers yet.
The early version of Truth Teller used 3 expert raters and a computer model to evaluate a concept. It was wrong 10% of the time, but also at this time leaders at companies were wrong about 70% of the time. A system built specifically to evaluate concepts with no emotional attachment and only based on what was written is 7 times smarter than the people inventing the ideas.
Thanks to education, people are smarter than before but also AI has advanced and proven to be fast and useful too.
What Popular AI Tools Are Actually Good For
Since general AI tools like ChatGPT, Claude, and Gemini became part of everyday work, it's natural to ask if you can just paste your idea in and get an answer. You can, and it's genuinely useful. These tools are fast, and they'll give you a helpful first reaction: what's confusing, what's been done before, how you might sharpen the pitch.
What they won't give you is odds of success. A general AI model wasn't trained on a dataset of real products that launched and either survived or quietly disappeared. It's reasoning from general knowledge, not from outcomes. That makes it a great tool for fast stimulus, but not one built for predicting whether something will actually sell or how much.
Worth knowing: some companies don't trust these AI tools to be confidential despite the settings. Without debating where data goes and who has access, just know your organization might have guidelines on usage. Also the tools I've made on Jump Start Your Brian that use LLMs can have that feature easily disabled to comply with your policies
How a Model Can Predict Success, Not Just React to It
The model behind the accuracy numbers above works differently than a general chatbot. It was built on a large set of real products that actually launched, most of which didn't stay on shelves for long. The ones that did succeed shared a set of identifiable traits, 50 factors that impact if an idea will work. The model checks a new idea against the presence or absence of those factors.
What is important is that the model can explain why an idea scored the way it did, instead of just handing back a number. Understanding the math behind the model is what turns a score into an insight you can actually act on.
A great idea can still score poorly if it's explained badly. A weak idea can score well if it's written to oversell itself. Neither a person nor a model is reading your mind, they're only reading what's on the page in front of them.
What if your survey respondents are using AI to answer your questions?
This is a real and growing concern, and it needs to be addressed in how you design a survey and collect responses. Also whatever I say here may be outdated in a few months. In additional to doing the standard quality control to watch for answers that feel too polished, or oddly similar across respondents; ask quantitative questions when possible and use qualitative questions to get information you really need from respondents that you can't get another way. One of my favorite qualitative questions is "if you has a magic wand, what would you change other than price." The survey I did yesterday had 55 responses and only one felt suspicious, so at least with my sampling platform this isn't a major concern yet.
Can an AI model actually evaluate ideas?
Yes, with the right training data. A model trained on real outcomes, not just general knowledge, can predict success with meaningful accuracy, as shown above. The key word is trained: a general-purpose AI chatting about your idea is a different tool than a model built specifically on which ideas actually won or lost in the real world. We've advanced the truth-teller tool to give instant analysis without any need for raters. This upgrade was possible due to having rating data on many public concepts, but it is just the first step to give useful feedback before paying for raters to get a validated score.
Can AI take on a persona?
You can ask a general AI to answer as if it were a certain type of customer, and it will. But that's simulation, not a survey. It's useful for a fast hypothesis or a gut check, but it's still one model's best guess at how a person like that might respond, not an actual response from one. I've built protoypes for these types of simulations, but in real client work we recuit real people for an interact sesssion.
Do business leaders trust AI evaluations more or less than consumer research?
Honestly, I don't know. I think it is good to be a little skeptical of both. When I use my ai tools, I see opportunities for it to get smarter and when I look at survey results I see some responses that make me question if it realy represents a customer. But both are useful and the key is that you are quality skeptical when you agree and when you disagree with the results. Always stop and think.
Build a System so You Can Test Lots of Ideas, Not Just One
You can use humans or AI, and I recommend both. Balance confidentiality, speed and cost. Build your system to allow for more testing of ideas with actionable results and it will not just weed out bad ideas, but also help spark new ones.
Run it in rounds. An idea almost never nails it on the first pass, and that's fine. Each round of testing, whether it comes from customers, an expert panel, or a model built to predict outcomes, makes the next version smarter. We can this cycles of learning.