A diagnostic question makes someone check something. A quiz question just makes them nod.
A roofing contractor sent me his new scorecard last month, proud of it. Question one: "Do you love your home?"
Everyone answers yes. Every homeowner on earth answers yes. That's the problem. He'd built a scorecard and opened it with a question that produces zero information about the person answering it. It feels warm. It feels like it's building rapport. It's actually a quiz question wearing a scorecard's clothes, and it tells you nothing about whether that homeowner has storm damage, an aging roof, or a warranty that expired two years ago.
Here's the tell. A diagnostic question makes the respondent stop and check something, or admit they don't know. "When did you last have your roof inspected for storm damage?" Some people know. Some people have to go find out. Some people realize, mid-scorecard, that they don't know and probably should. That gap, the moment someone realizes they can't answer confidently, is the whole point of the tool. A quiz question skips that moment entirely. It just asks how someone feels, and everyone feels the same way about the thing they already own.
If every respondent would answer a question the same way regardless of their actual situation, you didn't write a diagnostic question. You wrote a quiz question and called it a scorecard.
I saw the same mistake from a completely different angle a few months back. Glenn was building a scorecard for reverse mortgages and wasn't sure whether the tool even fit his niche, whether he needed something custom instead of the standard build. The real issue wasn't the tool. It was that nobody had shown him the difference between a question that scores real variation and one that scores nothing. That gap shows up in every vertical. Diagnosing instead of pitching only works if the questions you're diagnosing with actually diagnose something.
The 3-question test
Before you ship a scorecard question, run it through this.
Would a bored homeowner and a genuinely worried homeowner give the same answer? If yes, it's a quiz question. "Do you love your home?" Yes and yes. "Do you know your roof's remaining warranty coverage?" One knows, one doesn't, and that split is the score.
Does answering it require the respondent to look something up, check a date, or admit uncertainty? A diagnostic question creates friction on purpose. Not annoying friction, useful friction. The kind that makes someone think "I actually don't know that" thirty seconds before your results page tells them why that matters.
Does the answer change what you'd say to that person on the call? If a yes and a no lead to the exact same follow-up conversation, the question isn't pulling weight. Cut it.
A roofing scorecard built on real diagnostic questions, storm damage history, warranty status, attic ventilation, gutter condition, produces a score that moves with actual risk. Not with mood. Not with how good someone's day is going. That's what makes the personalized result on the other end feel like a real assessment instead of a personality test with a lead form attached.
The curse of knowledge makes this worse, not better, the longer you've been in a niche. You know so much about roofs that "do you love your home" feels like a friendly warm-up instead of dead weight. It isn't warm-up. It's the first thing a prospect reads, and if it doesn't diagnose anything, you've spent your best real estate on a question that closes nothing before the call even starts.
The roofing contractor rewrote question one. "When did you last have your roof inspected for storm damage?" Same scorecard, same 8 pillars, one different opening question. That's the whole fix. Not a new tool. Not a different platform. Just a question that actually asks something.
Put it to work
How do I know if a scorecard question is diagnostic or a quiz question in disguise?
Ask whether a bored respondent and a worried respondent would answer it the same way. If the answer is always the same regardless of the person's real situation, it's a quiz question. Rewrite it to require a specific fact, a date, or an honest "I don't know."
Should every question in my scorecard be this specific?
Not every single one, but your opening question sets the tone for the whole tool. If question one is a quiz question, the respondent decides in the first ten seconds that this is entertainment, not a real assessment, and answers everything after it the same shallow way.
What if a diagnostic question feels too hard for a first-time visitor to answer?
That's fine. "I don't know" is a valid, useful answer on a scorecard. It's the moment the prospect realizes they have a gap, which is exactly what should happen right before they see their score.
