How the data works

The data

Every curve comes from the Google Books Ngram datasets, version 20200217, which count how often each word appears in books scanned by Google, year by year, up to 2019. We use the English 1-gram files: 24 files, 13.4 GB compressed, plus the file of yearly totals. The data is licensed CC BY 3.0 and is described on Google’s datasets page. Words Ago is an independent game and is not affiliated with Google.

From raw counts to a curve

1. Untagged words only. The files also list words with grammar tags (a noun use, a verb use). We keep the plain entries.
2. Capitals merged. “Railway” and “railway” count as one word, so sentence starts and names like the Great Western Railway count too.
3. Dictionary words only. We keep 70,838 words from two public-domain lists by Alan Beale (12dicts 2of12 and 3of6game). 68,414 of them appear often enough to draw a curve and can be guessed.
4. Divided by the year’s total. Far more books were printed in 2000 than in 1850, so each year’s count is divided by the total number of words printed that year.
5. Smoothed. Each year is averaged with the two years either side, which keeps the shape and removes year-to-year noise.
6. Scaled to its own peak. The highest point of every curve is 100%, so you compare shapes, not raw frequency.

Why the chart starts in 1820

Books from around 1800 are few, and their old typefaces used a long s (ſ) that scanning software often read as an f. In this data, “fuch” (a misread of “such”) runs at about 4% of “such” in 1800, about 0.3% by 1815 and under 0.02% by 1820. So the game shows and scores 1820 to 2019; 1818 and 1819 are used only for the smoothing at the left edge.

The match score

A guess is compared with the mystery word using 40 points, one for each five-year span. The score blends two measures:

• Correlation (Pearson’s r, negative values counted as zero): do the curves rise and fall at the same times? This is good at telling eras apart.
• Shape distance: one minus the average gap between the two peak-scaled curves. This is high when two curves simply look alike.

Correlation is unreliable when a curve is nearly flat, so its weight shrinks for flat words. In full: match = 100 × (w·t·r + (1 − w·t)·shape)^0.85, with w = 0.7 and t = the smaller of the two curves’ coefficients of variation divided by 0.8, capped at 1.

We tuned these numbers on 226 word pairs from nine eras, after checking each word’s real peak. Same-era pairs outscore far-apart and flat-against-peaked pairs 97% of the time. Every pair from eras at least 60 years apart scores under 50%, and two flat everyday words always score at least 50%.

What the curves can’t tell you

• Books, not speech. A curve shows printed English. Slang and spoken words show up late or not at all.
• The mix of books changes. The library behind the data isn’t a fixed sample. Around the late 1970s, for example, everyday office words like “program”, “federal” and “percent” all bulge at once, which more likely reflects a change in which books were scanned than a change in how people wrote. We avoid answers whose curves mainly show effects like that.
• Scanning errors remain, especially in old typefaces, even after 1820.
• Hyphens and accents. Hyphenated words such as “well-known” are split into pieces in this data, and accented spellings (café) are counted separately from unaccented ones (cafe). Where that affects an answer, its story says so.
• It stops in 2019. The last year in the dataset is 2019.

The stories

Each daily word has a short history written for it. Every date and fact in a story is checked against the source linked under it, and any statement about the curve (such as the year it peaked) comes from our own data. If you spot a mistake, write to hello@wordsago.com.