The Research
A meta-analysis of self-assessment studies found that the correlation between how people rate their own abilities and their actual performance is... 0.29.
For reference, a perfect correlation would be 1.0. Zero would mean self-assessment is essentially random. A correlation of 0.29 means "there's something there, but it's not great."
It gets worse in specific domains. For managerial competence, the self-performance correlation drops to 0.04. That's essentially no correlation at all. Managers' self-assessments of their management ability are barely better than random.
And physicians? Self-assessment of medical skills often shows non-significant or even negative correlations with objective performance. The doctors who think they're best are sometimes the worst.
This isn't about modesty versus arrogance. It's about a systematic human inability to accurately evaluate our own capabilities. And in the AI age, this inability has consequences.
Why Calibration Matters for AI
Here's where it gets specific. Researchers Caplin, Dean, and colleagues at NBER published a study with the revealing title "The ABCs of Who Benefits from Working with AI." Their core finding: about 20% of the benefit people get from AI collaboration comes from accurate self-assessment.
Think about what this means.
When you work with AI, you're constantly making judgment calls: "Is this AI output good enough?" "Do I know better than the AI on this point?" "Should I trust the AI's recommendation or go with my gut?"
If you're well-calibrated — accurately aware of what you know and don't know — you make these calls well. You trust AI output in areas where you're actually less capable, and you override AI in areas where your expertise genuinely exceeds the model's. Result: high-quality output that combines the best of both.
If you're poorly calibrated, you make these calls badly. You might override good AI output because of unfounded confidence in your own (incorrect) judgment. Or you might accept bad AI output because you underestimate your own ability to evaluate it. Either way, the collaboration suffers.
The Two Flavors of Miscalibration
Miscalibration comes in two flavors, and each one creates a different AI problem:
Overconfidence is the more common and more dangerous flavor. Overconfident people reject AI suggestions too readily ("I know better") even when the AI is actually right. They don't notice errors in their own work because they assume their work is solid. They underuse AI assistance in areas where it would genuinely help.
The classic overconfidence profile: a mid-career professional with strong domain expertise who assumes their judgment is correct across all aspects of their work — including aspects where they're actually mediocre. They're great at the core of their job but mediocre at, say, written communication or data analysis. An AI could dramatically improve those weak areas, but overconfidence prevents them from accepting the help.
Underconfidence is less common overall but disproportionately affects certain groups — notably women (68% of women in tech report imposter syndrome vs. 61% of men) and early-career professionals. Underconfident people accept AI output too readily because they don't trust their own judgment. They defer to the machine even when their human intuition is actually correct.
The classic underconfidence profile: a junior professional who's actually quite capable but doesn't yet know it. They use AI for everything — which sounds productive — but they're not developing their own judgment because they outsource every evaluation to the machine. They become dependent rather than augmented.
Can Calibration Be Improved?
Yes. The meta-analysis shows moderate effect sizes for calibration training — meaning it's genuinely trainable, not a fixed trait.
Here's what works:
Consider-the-opposite prompts. Before making a judgment call, deliberately argue the other side. "I think this AI output is wrong because..." Force yourself to generate reasons it might actually be right. This simple exercise reduces overconfidence by 20-30% in experimental settings.
Pre-mortem analysis. Before starting a project with AI, ask yourself: "If this goes badly, what's the most likely reason?" This surfaces potential blind spots before they become problems. It's like debugging your own thinking before you ship it.
Prediction tracking. Start keeping a record of your predictions and evaluations. "I think this AI analysis is 80% correct." Then check. Over time, you'll notice patterns: areas where you're consistently right, areas where you're consistently wrong, and areas where your confidence doesn't match your accuracy.
Anchoring vignettes. Describe a hypothetical person at a specific skill level in your domain — someone at the 25th percentile, the 50th, the 75th. Then honestly compare yourself. This gives your self-assessment a reference point instead of floating in the void of "I think I'm pretty good."
Immediate, specific feedback. Work with colleagues or mentors who will give you honest, specific feedback on your judgment calls. Not "good job" or "needs improvement" — but "your instinct about the market trend was right, but your estimate of the timeline was off by 6 months."
What doesn't work? Simply knowing about biases. Awareness alone doesn't fix calibration. Neither do monetary incentives for accuracy or generic warnings about overconfidence. You need structured practice, not just information.
The Meta-Point
In a world where AI handles more of the "doing" and humans handle more of the "evaluating" and "deciding," calibration becomes a core professional skill. Maybe the core professional skill.
Because here's the arc of where work is heading: AI produces. Humans evaluate. AI refines. Humans decide. AI executes. Humans verify.
At every step, the human contribution depends on accurate self-knowledge. Can I evaluate this output well? Do I know enough about this domain to make this call? Is my gut feeling here evidence-based or ego-based?
The professional who can honestly answer these questions — who knows what they know and what they don't know — will extract far more value from AI than the professional who can't.
Socrates was right two and a half thousand years ago: knowing yourself is the beginning of wisdom.
Turns out it's also the beginning of AI fluency.