- 1 Projects
- 2 Details
- 3 Source: private client work
Computer science | AI Data Engineer at Handshake AI and Solumn AI | Jul 2026 to present
Coding-agent evaluation
AI Data Engineer on coding-agent evaluation at Handshake AI, as a Platinum Fellow, and part-time at Solumn AI. Each task reconstructs an open-source project at a historical commit, reproduces a reported defect, verifies the maintainers' fix against the project's own tests, and ships as a reproducible benchmark item with a reference solution and an automated grader. Reviewer acceptance has stayed at 100% across three programmes.
- Docker
- pytest
- Python
- Integer linear algebra
- Modular arithmetic
- Git and pull requests
- Evaluation protocols
- 100%
- reviewer acceptance across three programmes
- 2
- organisations: Handshake AI and Solumn AI
- byte-exact
- grading of the agent's real output
- 1 Dockerfile
- builds the environment for agent and verifier
Workstream 1
Adversarial evaluation tasks
A task is only useful if a capable model can fail it for a real reason. Each of mine has to meet three conditions at the same time. The naive reading of the data looks correct and is wrong. The cases that decide the answer are hidden from the sample the agent sees, yet fully determined by it. And the result can still be graded byte-exact, so passing or failing is never a judgement call.
Each task ships self-contained. This is the path one task takes from idea to accepted submission.
Design the trap
Choose a problem where the obvious interpretation of the data is plausible and wrong, and where the deciding cases never appear in the sample but follow from it.
Generate the data
Build a synthetic corpus for the task instead of borrowing real data, which keeps every case that matters under control.
Prove there is exactly one answer
Show computationally that the corpus admits exactly one consistent interpretation. Where a task hides structure, I recover it with integer linear algebra and modular arithmetic.
Write the reference solution
Solve the task by genuine computation, the way an agent would have to, with nothing hard-coded.
Pin the environment
A digest-pinned Dockerfile builds the environment for both the agent and the verifier, so both work against the same pinned environment.
Grade the real output
A pytest suite checks what the agent actually produced, byte-exact, instead of matching strings in its answer.
Pass the review gates
The task goes in as a pull request and iterates against automated review gates until every check is green.
Workstream 2
Repositories rebuilt at a historical commit
Most tasks start from real open-source code. The job is to reconstruct a project at an exact historical commit in a digest-pinned Docker image, reproduce a reported defect, and verify that the project's own tests fail before the maintainers' fix and pass after it, then attach a reference solution and an automated grader so the task is reproducible and comparable across agents.
Check out the commit
Start from the exact historical commit the task is based on.
Write a build recipe
Capture how that repository builds in its own recipe, so the environment comes out the same every time it is rebuilt.
Pin until the suite flips
Pin dependencies until the test suite that failed before the fix passes after it.
Verify against the reference
Confirm that fail-to-pass flip against the reference solution before the environment ships.
Workstream 3
Submission pipeline tooling
Around the core sits the tooling I built around the platform's own checks, so quality held while volume grew. Part of the pipeline runs through model agents in separate lanes; I write the protocols they follow, set their budgets and check what they produce.
Repository registry
A registry of candidate repositories, so the next task starts from a known pool.
Static screen against the bar
Pre-checks against the review bar before anything is built, so work that cannot pass is never started.
Duplicate detection and build caching
Work that has already been done is recognised and reused.
Independent audit and rework
An independent audit before submission, and a review-and-rework loop after it: returned tasks are taken through rework rather than dropped.
The bar
What the tasks are held to
What a benchmark counts as a task decides what it measures. I keep a written definition that every task is screened against, a list of the ways a task goes wrong, and a log of reviewer returns.
What counts as a valid task
A written definition that every task is screened against, so the benchmark measures the same thing from one task to the next.
A taxonomy of failures
The ways a task can go wrong, named and sorted, so a failure is classified rather than described from scratch each time.
Reviewer returns as data
Every return from review is recorded and read as evidence about the bar, and the bar changes when the evidence says it should.
Comparable across agents
Reproducible environments, verified fixes and automated graders mean the same task gives every agent the same test.
Step 3 of 3
Source code
This work belongs to the client, so the tasks, environments and tools are not public. Ask me about the engineering, or read the code of my open-source projects.