The Vocabulary Problem: Why Two People Can Score the Same and Mean Different Things

Two people take the same Big Five inventory. Both land at the 78th percentile for extraversion. On a matching dashboard, they are interchangeable — same bar, same color, same label.

One is 22, lives in Berlin, and goes out three nights a week. By her own standards she's "about average, maybe a bit social" — but she answered I like parties with a 4 because, well, she does.

The other is 41, lives in Almaty, has two kids, and sees friends roughly twice a month. He also answered I like parties with a 4 — because compared to the people in his life, twice a month is a lot, and because when he does go, he's the one who stays late.

Same number. Wildly different behavior. The score didn't lie, exactly. It just compressed two different sentences into one digit.

This is the vocabulary problem, and it's the quiet tax on every personality test you've ever taken.

Every item is a comparison, not a measurement

A thermometer doesn't ask the air how hot it feels. A personality item does exactly that. When you read I am organized and pick a number from 1 to 5, you are not reporting a physical quantity. You are running a fast, mostly unconscious comparison: organized compared to whom?

Psychologists have a name for what happens next. The reference-group effect describes "the confounding role of context in comparisons of mean questionnaire responses across different groups"[1] — in plainer terms, people judge themselves against the people around them, and different people are surrounded by different people.

The classic demonstration is uncomfortable. Cultural experts broadly agreed that East Asians were more collectivistic than North Americans, yet straightforward cross-cultural comparisons of trait and attitude measures failed to show that pattern; when researchers explicitly manipulated the reference group respondents used, the expected differences appeared.[2] The scale wasn't detecting the thing everyone could see, because each group was silently grading itself on its own curve.

It gets worse at the national level. One study looked at whether cross-national rankings of conscientiousness — built from self-reports, peer reports, and perceptions of national character — lined up with behavioral and demographic indicators of conscientiousness in those countries. The self-report-based rankings held up poorly, and the authors argued that the reference-group effect undermines the validity of national averages of personality scores.[3]

Read that again, because it's the whole article in one sentence: a questionnaire can be reliable, internally consistent, factor-analytically beautiful — and still produce averages that don't track reality, because people are answering different questions.

Four ways meaning drifts under a fixed number

The reference-group effect is the famous one, but it's not alone. In practice, a single Likert item leaks meaning in at least four directions.

1. The comparison class. "Do you like parties?" is answered relative to an implicit peer group: your friends, your city, your profession, your decade of life. Change the peer group and the number moves without the person moving.

2. The quantifier. Items are full of words like often, rarely, a lot, sometimes. These are not units. "I often lose my temper" might mean weekly to one person and twice a year to another who is simply ashamed of it. There's no conversion table printed on the form.

3. The scope. Which version of you is answering? Work-you is organized because deadlines are real. Home-you has four unopened boxes in the hallway. Most inventories ask you to describe yourself "in general," which quietly asks you to average across contexts you've never averaged before — and people weight that average differently depending on which part of their life currently feels most like them.

4. The trait word itself. Organized can mean a tidy desk, a maintained calendar, a clean inbox, or simply "I know where things are even if you don't." These correlate, but not perfectly, and two people who split on which sense they picked will produce the same score from different habits.

None of this is exotic. It's measured routinely: analysts test whether an item behaves the same way across groups using differential item functioning and measurement-invariance methods, checking whether people with the same underlying trait level answer a given item the same way.[4] When items fail these tests, group comparisons on that scale are, strictly speaking, comparisons of apples and slightly different apples.

The honest counterpoint: sometimes the items behave

It would be convenient — and wrong — to conclude that personality scores are noise.

Invariance is a testable property, and it often passes. One analysis applied item response theory to the five-factor scales of the IPIP and the NEO-FFI across groups with high and low cognitive ability, and found the scales measurement invariant, meaning score differences between those groups could be read as genuine trait differences rather than artifacts of item interpretation.[5] Other work finds the opposite in specific places: in a study of affect items among Ecuadorian young adults, researchers reported only partial invariance across gender, with item-level differences concentrated in fear- and hostility-related words.[6]

So the real picture isn't "tests are fake." It's item-dependent. Some items are sturdy. Emotionally loaded, culturally saturated, or vaguely quantified items drift. And the drift doesn't announce itself — it shows up as a perfectly ordinary number.

Survey science already built the patch. It's expensive.

Methodologists noticed this long before dating apps did. When respondents use ordinal response categories in different ways, analyses built on those responses can be biased — and anchoring vignettes were introduced as a design technique to correct for exactly that problem.[7]

The mechanism is elegant. Instead of only asking "how healthy are you?", you also ask the respondent to rate several fixed, described characters — people whose actual level is held constant by design. Any differences in how respondents rate those identical characters reveal how each person is using the scale. Then you rescale the self-rating against that person's own ruler.

It works. It's also brutal in a consumer context: every vignette is extra items, extra minutes, extra abandonment. Nobody finishes a 30-minute dating questionnaire that includes six short stories about strangers named Alison and Jane.

Which leaves the industry with a shortcut that everyone pretends is fine: five boxes, no follow-up, and a dashboard that reports the result to one decimal place.

What a conversation recovers that a checkbox can't

Here's the thing a Likert scale structurally cannot do: ask what you meant.

A conversational assessment can. It can follow "I like parties" with when was the last one, and how long did you stay? It can follow "I'm pretty organized" with walk me through how you handled the last trip you planned. It can notice that someone rated themselves 2 on sociability and then described seeing friends four times last week, and flag the gap instead of averaging it away.

Functionally, this is anchoring-vignette logic delivered as dialogue. The goal is the same — establish each person's private ruler before trusting their self-rating — but instead of rating fictional strangers, you're describing your own last three Fridays, which is faster and less boring.

Three things a conversational pass recovers:

• Base rates instead of adverbs. "Often" becomes "roughly twice a month, usually Fridays." That number means the same thing in Berlin and Almaty.
• Context, not averages. Instead of one blended organization score, you get: ruthless at work, chaotic at home, and a clear sense of which one a partner will actually live with.
• The comparison class, made explicit. When someone says "I'm not very social," the useful follow-up is compared to whom? The answer — my old flatmates, my colleagues, my brother — tells you where the ruler came from.

This is why an avatar that converses is a measurement instrument, not just a chat toy. It can hold the slow, probing interview that psychometrics has always wanted and consumer products have never been able to afford. You can see the raw instruments it starts from when you take the tests, and the conversational layer is what sits on top of them when you create your avatar.

How to read your own results more skeptically

Next time you get a profile back, run it through five questions:

1. Compared to whom did I answer? Picture the specific people you were implicitly measuring against. If that group is unrepresentative — a very extroverted friend circle, a very tidy family — your score is shifted relative to the general population, not wrong, just translated.
2. Can I convert each adverb into a number? Take your three most extreme items and restate them as frequencies and last occurrences. If you can't, the item measured your self-image more than your behavior.
3. Which version of me showed up? Note whether you answered as work-you, home-you, or aspirational-you. Aspirational-you is the most common respondent and the least useful one.
4. Would someone who lives with me produce this profile? You don't need a formal informant report; just imagine their eyebrows at each facet.
5. Does the percentile change my prediction? If moving from the 60th to the 78th percentile doesn't change anything you'd expect to observe in your life, the precision is decorative. (We've written before about why compatibility scores deserve error bars — this is the item-level reason they do.)

Identical scores aren't identical people

The practical upshot for matching is blunt. If two profiles differ by three points, that difference may be entirely linguistic — one person's ruler against another's. Treating it as a meaningful ranking is how platforms generate confident nonsense.

What survives translation is behavior: how often, with whom, how long, what happened last time, what you did when it went badly. Those sentences mean roughly the same thing in any city and at any age. Numbers are a useful index into them — a fast, cheap first pass that tells you where to dig. They are not the finding.

The 22-year-old in Berlin and the 41-year-old in Almaty both scored 78. One of them wants a partner who'll stay out until three. The other wants a partner who'll make the twice-monthly dinner feel worth the babysitter. No Likert scale in the world distinguishes those two wishes.

A good question does. Ask it.

Sources

1. Reference Group Effect (Encyclopedia of Personality and Individual Differences) — Springer
2. What's wrong with cross-cultural comparisons of subjective Likert scales?: The reference-group effect — Semantic Scholar / Journal of Personality and Social Psychology
3. What Do Cross-National Comparisons of Personality Traits Tell Us? The Case of Conscientiousness — University of British Columbia / Psychological Science
4. Measurement invariance and differential item functioning of the positive and negative affect schedule — Frontiers in Psychology
5. Do Personality Scale Items Function Differently in People With High and Low IQ? — American Psychological Association / Psychological Assessment
6. Measurement invariance and differential item functioning of the positive and negative affect schedule: a psychometric study in Ecuadorian young adults — Frontiers in Psychology
7. Comparing Incomparable Survey Responses: New Tools for Anchoring Vignettes — Harvard University (Gary King)