I build AI systems that answer questions about company data. They check their own answers, cite their sources, and are tested before anyone relies on them. This page follows one question through the loop that does it.
Xander Bennett · AI engineer · Salt Lake City · Remote or local · Six years in data before this
Retrieval runs over the dbt manifest and catalog rather than the raw schema, so the agent sees model descriptions, column docs, tests, accepted values, and lineage. The five best matches are all it is allowed to use.
In plain terms: the assistant looks up which tables are relevant and is forbidden from using any others. Most wrong answers from AI over data start with a guessed table or column; this removes the guess.
Solid chips are retrieved and allowed. Dashed chips exist in the warehouse but were not retrieved for this question; a query that names one is rejected before it runs.
The model gets the question and the five retrieved model docs, and returns one SELECT as JSON. The first attempt below is what it actually wrote: it reached for a dim_level table that exists in its head but was never retrieved.
In plain terms: the assistant writes the query, and its first try contains a plausible mistake. The next stage is where that mistake gets caught, which is the whole point.
-- attempt 1 · constrained to the five retrieved modelsselect e.job_family, count(*) as below_midpointfrom fct_band_position as bjoin dim_employee as e on e.employee_id = b.employee_idjoin dim_level as l on l.level_id = e.level_idwhere e.is_active and l.level_name = 'L4' and b.compa_ratio < 1.0group by 1order by below_midpoint desc;
Parse as DuckDB SQL, confirm a single read-only SELECT, confirm every referenced table is in the retrieved set, dry-run with EXPLAIN. A failure goes back to the model with the exact error, up to three times. Attempt 1 failed the grounding check; attempt 2 passed all four.
In plain terms: before the query touches any data, it has to pass four tests. The mistake from the previous step is caught here, the assistant is told exactly what was wrong, and it fixes it. Nothing unverified ever runs.
# the model's second attempt, as a diff against the first - join dim_level as l on l.level_id = e.level_id- and l.level_name = 'L4'+ and b.level_id = 'L4'
# agent.py · after validate(), before execute(): catch SQL that is valid but answers a different question v = validate(draft.sql, allowed, con) + back = self.llm.back_translate(v.sql) # "what question does this SQL answer?"+ align = self.judge.alignment(question, back)+ if align < settings.min_alignment:+ return self.repair(draft, reason=back)- return self.execute(v)+ return self.execute(v, confidence=align)
The validated query runs on a read-only DuckDB connection with a 200-row cap. Latency and token cost are logged on every run, so a change that improves accuracy by adding two model calls shows its price.
In plain terms: the query runs against a copy of the data it cannot change, with a limit on how much it can pull back, and the cost and speed of every answer are recorded.
| strjob_family | intbelow_midpoint |
|---|---|
| Software Engineering | 61 |
| Data | 27 |
| Product | 14 |
The rows become a sentence a person can read, with the models that produced it attached. The same loop runs behind three surfaces: a command line, a small FastAPI app, and an MCP server so Claude Desktop or Cursor can call it as a tool.
In plain terms: the person gets a plain answer with its sources and a receipt for what it cost. The same engine can be used from a terminal, a web page, or inside an AI assistant.
A golden set of questions with reference queries, an eval that compares result sets by value, and a CI gate that fails a pull request when accuracy or retrieval recall drops more than three points below the committed baseline. The targets are written down first so the numbers can miss them in public.
In plain terms: the assistant is re-tested every time its code changes, against questions with known correct answers. If it gets worse, the change cannot ship. This is the part most AI features skip.
# one of 12 starter questions · target 50- id: g004 question: Which job families have the most active L4 employees paid below the band midpoint? models: [fct_band_position] sql: > select job_family, count(*) as below_mid from fct_band_position where is_active and level_id = 'L4' and compa_ratio < 1.0 group by 1 compare: set tags: [compa, medium]
| strmetric | strhow it is measured | numv1 target | nummeasured |
|---|---|---|---|
| execution_accuracy | The agent's result set matches the reference query's, by value. Order-insensitive with a float tolerance; scalar and ordered modes where the question demands them. | ≥ 0.80 | — |
| retrieval_recall@5 | The models a golden question needs appear in the top five retrieved. | ≥ 0.90 | — |
| repair_rate | Share of questions that needed at least one repair before a valid query. | reported | — |
| failure_rate | Share of questions with no valid SQL after three repairs. | ≤ 0.05 | — |
| guardrails | Adversarial slice: destructive requests must be blocked, unanswerable questions must be refused rather than answered. | 1.00 blocked | — |
| latency_p50 · cost | Seconds and dollars per question, logged on every run so an accuracy gain shows its price. | ≤ 5 s · ≤ $0.01 | — |
| metric | run 13 | run 14 | Δ |
|---|---|---|---|
| execution_accuracy | 0.78 | 0.84 | +0.06 |
| retrieval_recall@5 | 0.92 | 0.92 | 0.00 |
| repair_rate | 0.31 | 0.22 | −0.09 |
| latency_p50 | 2.4 s | 2.1 s | −0.3 s |
| cost / question | $0.007 | $0.006 | −$0.001 |
Each one ships with a test that says whether it is working. One is the loop above, in progress. Two are queued behind it. Each goes public once it has a committed baseline.
An assistant that turns a plain-English question like “which teams are paid below market?” into a query against the company's data, checks its own work, and shows where the numbers came from.
For a company, this means fewer wrong numbers reaching a CFO, and analysts spending their time on judgment rather than lookups. grounded-sql-agent · in progress
Customer files rarely match the format the software expects. This maps the incoming columns onto a standard set, flags uncertain matches for a person to check, and learns from the corrections.
For a company, this means onboarding a new customer in days rather than weeks, with less manual cleanup each time. schema-mapper · queued
A safety net that re-tests an AI feature every time it changes, compares the results with the previous version, and blocks the release if quality drops.
For a company, this means AI changes ship with evidence behind them, and problems are found in a test run before a customer sees them. regression-gate · queued
I came to AI engineering from the data side. In that work, a wrong answer is a number a finance leader has already repeated in a meeting, and that is the standard I hold AI systems to. It is why the loop above spends more of its time checking than generating.
I spent six years at Addepar, a wealth-management data platform, going from operations analyst to owning the reporting and data-modeling layer. I then led an analytics rebuild at Dorsia, including a cloud migration, and ran a small team.
Today I'm a Customer Success Engineer at Pave, a compensation platform. I run technical implementations, clean up the HR and pay data that makes customers' software work, and build the AI workflows my team runs every week: a comp-cycle forecasting agent (Cycle Radar), scheduled CRM and reporting updates, and a customer-feedback sweep that drafts product requests from calls and email.
Outside work: two kids, tabletop games, and a side project teaching game development.
| ability | score | evidence |
|---|---|---|
| INT | 17 | Six years of data modeling, now building the retrieval, validation, and eval layers of an AI system from scratch. |
| CON | 16 | Keeps five scheduled AI workflows running at work while building the flagship at night. |
| WIS | 15 | Grades an AI feature on its result set, not on whether the SQL looked right. |
| DEX | 14 | dbt, LookML, SQL, Python, DuckDB, FastAPI, MCP. Comfortable moving between the data layer and the application layer. |
| CHA | 13 | Runs implementations with customer HR and finance teams and translates between them and engineers. |
| STR | 12 | Carried an analytics stack through a cloud migration at Dorsia. |
I'm open to full-time AI engineering roles, in Salt Lake City or remote. Send the problem you're stuck on and I'll reply with how I'd approach it and how I'd measure it.