We replaced regex with an LLM in our talent platform. The model was the easy part.

🔁 We replaced regex with an LLM in our talent platform. The model was the easy part.

GPT-4 arrived this week to enormous fanfare — bar exams passed, benchmarks broken. Useful timing, because for the past months my team has been living the less glamorous version of this story: migrating an internal talent platform — matching people to roles and internal opportunities — from the classical stack (regex, keyword rules, hand-tuned NLP) to an LLM-powered one. Some learnings, while they are fresh:

1. The biggest quality jump wasn't intelligence. It was vocabulary. Our rules could only match strings. So the engineer who wrote "led our Kubernetes migration" never matched the role asking for "container orchestration experience" — right person, wrong words, silent miss. Years of tuning synonym lists never fixed this; it was whack-a-mole. A language model simply does not have this problem. That one property — meaning over strings — did more for match quality than everything else combined.

2. The hard work moved; it did not shrink. With rules, effort went into writing rules. With LLMs, effort goes into data: profile quality, skill definitions, what "good match" even means. Our bottleneck was never really the matching logic — it was that profiles were sparse and inconsistent. The LLM exposed that instantly, because it can only reason over what it is given.

3. We got more accurate and less trusted — at the same time. A regex is auditable by definition: here is the rule, here is why it fired. An LLM's judgment arrives without a paper trail, and in HR, "the system says so" is not an acceptable answer to "why was I not shortlisted?" We had to build the explanation layer as a first-class feature, not an afterthought. Accuracy without explainability, in people decisions, is a downgrade.

4. Testing had to be reinvented. Rule-based systems pass or fail deterministically. For the LLM we ended up building a fixed set of known-good matches — real historical cases with agreed right answers — and re-running it after every change. Boring, manual to assemble, and now the single most valuable asset in the project.

5. The failure modes are politer and more dangerous. Regex fails loudly — no match, empty result. An LLM fails fluently: a plausible-sounding match with confident reasoning that is subtly wrong. The old system's errors announced themselves; the new one's have to be hunted.

The meta-lesson, watching GPT-4's launch through this experience: model capability is arriving faster than most organisations' ability to absorb it. The absorption — data, explainability, evaluation, trust — is the actual work, and no launch event ships it for you.

One question worth taking back to your own organisation: somewhere in it, a rule-based system is quietly filtering people, tickets or opportunities on exact words — and silently missing the right answers phrased the wrong way. Do you know which system that is? Because that is where this technology pays for itself first.

Reading: https://techcrunch.com/2023/03/14/openai-releases-gpt-4-ai-that-it-claims-is-state-of-the-art/