RoleRadar
Finds engineering jobs on company boards and scores fit with a calibrated two-step LLM pipeline.
21 of 21 good jobs caught, on 59 postings the pipeline was never tuned on.
Target: ≥ 95% recall. With 21 good jobs the 95% interval is 85–100%, so the target is met on this sample, not proven.
What it does
- Polls 312 company job boards (Greenhouse, Lever, Ashby) every hour from a Raspberry Pi, on systemd timers. In one run, 0 boards failed.
- Extracts each posting's requirements without seeing my profile, then matches them and applies eligibility rules in code.
- Scores new jobs for about $2.31 a month (p95 5–12 s per job), with a spend cap on every run.
- Live as a phone-first web app I use daily: a brief, triage, a shortlist and application tracking, with an alert if the worker stops.
- Tracks about 8,000 open engineering jobs at 149+ companies.
The hard part
Keeping the model from flattering me. Extraction runs without seeing my profile, eligibility is decided by code rules, and the pipeline, targets and threshold were fixed before the test run.
Read the details
The reference labels were written by an agent and corrected by me, so the eval measures agreement with a reviewed reference, not with independent human labels. The limits are listed above: correlation 0.85 against a 0.90 target, and about 17% of jobs crossing the threshold between runs. The known misses come from "X or Y" requirements; a prompt fix is planned.
What was tested
RoleRadar runs on a Raspberry Pi. It reads job postings from company boards and scores how well each one fits. I designed the pipeline, the calibration and this eval. Agents wrote most of the code.
How it was measured
- The reference labels are requirement-by-requirement analyses written by an agent (Claude). I corrected the facts about me in two review rounds.
- I settled 47 disputed requirements by hand: the ones where at least 3 of 4 models disagreed with the reference.
- The protocol was frozen before the run: pipeline, targets and threshold were written down before any model saw the 59 test jobs. Nothing changed after the results.
Where it falls short
- Correlation with the reference is 0.85. The target was 0.90. Not met.
- About 17% of jobs cross the threshold between runs (0.73 points run to run; target 0.5). Planned fix: score borderline jobs (5.5 to 7.5) twice more and average.
- 21 good jobs is too few to prove 95% recall. The lower bound is 85%.
How it's built
What it costs to run
Monthly scoring cost at 100 new jobs a day. The cheap model met the target; the mid-tier models weren't more accurate.
The cost trap: hidden reasoning
Same model, same calibration jobs, reasoning off against its default hidden reasoning.
A 4-call smoke test against a 120-call run. One side-by-side call showed 7× the cost and 16× slower, so the size of the trap varies by job; reasoning stays off.
Show as a table
| Setup | Cost | Latency |
|---|---|---|
| Chosen pipeline (DeepSeek V4.1 Flash, blind extraction then matching) | about $2.31 a month | p95 about 5–12 s per job |
| Target | ≤ $3 a month | p95 ≤ 30 s |
| Mid-tier extractors (GPT-6.1 Terra, GPT-6.1 Sol, Claude Sonnet 5.5) | $14–70 a month | no better accuracy |
| DeepSeek V4.1 Flash, reasoning off (120 calls) | $0.00058 per call | 3.7 s per call |
| DeepSeek V4.1 Flash, default reasoning (4 calls) | $0.0245 per call | 107 s per call |
What the calibration showed
A model that sees my profile while it extracts requirements leaves out the ones I lack, which inflated scores by 0.6 to 0.9 points. Splitting extraction (blind) from matching removed most of that. Holdout correlation rose from 0.88–0.90 to 0.89–0.95.
The mid-tier models were not better extractors and cost 5 to 30 times more. Hidden reasoning was a cost trap: on the same jobs, the same model cost about 42 times more per call and ran about 29 times slower (measured in the chart above).
One model looked best on the dev and holdout sets but did not hold up on the untouched sets. The rule written down beforehand prefers the simpler setup, so a single-model pipeline was chosen.
Calibration cost $8.48 of a $13 budget, across all rounds.
The protocol, as frozen before the test
Written down on 6 November 2026, before any model saw the 59 test jobs. An excerpt:
Pipelines: - P1 (primary): DeepSeek V4.1 Flash blind extraction (split.py SYS_X) -> Qwen 3.8 Flash matching (SYS_M). Reasoning off. - P2: same extraction -> DeepSeek matching. - P3: mean of P1 and P2 (same extraction). - Each with/without the implied-language code rule (posting mostly in French/German/Portuguese/Spanish/Italian/Dutch and no requirement mentions that language -> add a core must-have "Professional <language>" at level N). - Safety net "near-miss": show a job if its uncapped score is >= threshold. Targets (pooled fresh + v4): T1 recall of good jobs (ref >= 7.5) >= 95% at the chosen review threshold; T2 0 false-eligible, binary eligibility >= 95%; T3 auto-CV precision >= 90%; T4 corr >= 0.90, |bias| <= 0.3; T5 consistency <= 0.5, flips <= 5%; T6 <= $3/month, p95 <= 30 s. No changes after seeing the results; any change becomes the next round, tested on new jobs.
From FINAL_PROTOCOL.md, lines 6–12 and 19–22, unedited.
A second eval: who writes the CV
RoleRadar also drafts a tailored CV for each shortlisted job. Five writer setups and four reviewer models were compared on three real shortlisted jobs, and three agents with no context ranked the results blind, as recruiters. The cheapest pair (DeepSeek writing, Gemini reviewing) ranked first on average, and GPT 6.1 Sol's cover letters won 9 of 9. The blind review also found four pipeline bugs that no score had shown. The whole bake-off cost $1.12.
A rule change, checked on the reference set
An earlier change to the eligibility rules was checked across all 129 reference jobs. Three-way agreement went from 106 to 116, with no job wrongly marked eligible.
This set includes the jobs used for tuning, so it is a regression check, not a test. The test is the 59-job result above.
Re-applying the rules to all 768 stored scores needed no new model calls: it reuses the extractor's stored verdicts.
Screens
