The Ukrainian lemmatizer

Why it exists

Every word-list join depends on a lemmatizer that agrees with the list about what a headword is. The frequency list, the validated CEFR overlay and the profiler all use the VESUM convention (the infinitive for verbs, the nominative singular for nouns, the apostrophe written one way). A lemmatizer in another convention turns every token into an unknown. This endpoint is that convention, hosted, so the profiler and any third-party tool can call one thing and get the list’s own lemmas back.

Under the hood it is LanguageTool’s Ukrainian analysis pipeline, the core that nlp_uk’s TagText wraps: the tokenizer, the VESUM tagger and the rule-based disambiguator, over a pinned dictionary (ua.net.nlp:morfologik-ukrainian-lt:6.8.0). The dictionary is CC BY-NC-SA and never leaves the server; the endpoint returns analyses of your own text and nothing else.

Try it

Send a list of tokens (each analyzed on its own, what a word-list join wants):

curl -s https://www.jakelawrence.xyz/api/lemmatize \
  -H 'Content-Type: application/json' \
  -d '{"tokens": ["ім’я", "кав’ярня", "під’їзд", "м’ясо", "п’ять"]}'

Or a text (tokenized and disambiguated in context, with character offsets into the text exactly as you sent it):

curl -s https://www.jakelawrence.xyz/api/lemmatize \
  -H 'Content-Type: application/json' \
  -d '{"text": "Вона повідомила про це вчора."}'

A response looks like this (one token shown):

{
  "tokens": [
    {
      "token": "п'ять",
      "lemma": "п’ять",
      "pos": "numr",
      "ud": "NUM",
      "tags": "numr:p:v_naz",
      "known": true,
      "ambiguous": false,
      "alternatives": [
        {
          "lemma": "п’ять",
          "pos": "numr",
          "ud": "NUM",
          "tags": "numr:p:v_zna"
        }
      ]
    }
  ],
  "meta": {
    "service": "lemmatizer 0.1.0",
    "tagger": "LanguageTool Ukrainian 6.8",
    "vesum": "6.8.0",
    "mode": "tokens",
    "chars": 5,
    "tokens": 1,
    "ms": 3,
    "cache": "miss",
    "hash": "7fb62d674239a1c0"
  }
}

Per token: lemma (apostrophe written U+2019, the list’s glyph), the coarse VESUM pos, a best-effort ud tag, the full tags string, known, ambiguous (more than one distinct lemma survived disambiguation) and every other surviving reading under alternatives. The meta block carries the tagger and VESUM versions on every answer, so a result is never separated from the dictionary that produced it. The full contract is the OpenAPI document.

Limits, and what happens at them

Does it agree with the list?

The check that matters. A deterministic sample of the list (ranks 1 to 200 in full, then every 50th rank: 436 lemmas) was sent to the service, and the answer compared with the list’s own lemma at that rank, apostrophe and stress folded. A match requires the dictionary to know the word; an unknown token echoed back is counted as a miss, not a hit.

The UD column is informational: ud is a best-effort map from VESUM tags, and the list’s own tags came from a different tagger over running text, so particles, prepositions and adverbs disagree where the two conventions draw their lines. The lemma column is the contract.

The 4 misses: корреспондент (rank 1,500, NOUN; unknown to the dictionary), загиблі (rank 1,750, NOUN; returned загиблий), кадри (rank 4,850, NOUN; returned кадр), Ру (rank 8,250, PROPN; unknown to the dictionary).

The five apostrophe lemmas the work order named, which are the words a glyph mismatch breaks first: ім'я → ім’я (rank 311), кав'ярня → кав’ярня (rank 9,053), під'їзд → під’їзд (rank 5,405), м'ясо → м’ясо (rank 2,850), п'ять → п’ять (rank 483).

Timing: 5,000 characters of Wikipedia sentences (710 tokens) in 125 ms wall time, 114 ms inside the service, median of 5 runs against a local instance; the budget was 2,000 ms. Checked 2026-09-28. The artifact behind this section is src/data/ukrainian-frequency/lemmatizer-roundtrip.json; CI recomputes the headline from its rows and fails when a pin changes without a re-run.

What it does not do

It does not ship the dictionary or any wordform table to the browser: VESUM is CC BY-NC-SA and stays server-side. It does not apply nlp_uk’s statistical disambiguation or semantic tags; the rule-based disambiguator is what the round-trip needed, and meta.pipeline says exactly what ran. It does not keep your text. And it does not guess: an unknown word comes back marked known: false with the token as its lemma, and a token that splits into several pieces is reported as such rather than collapsed.