1. 1 Projects
  2. 2 Details
  3. 3 Source: private client work

Computer science | AI Data Engineer at Handshake AI and Solumn AI | Jul 2026 to present

Coding-agent evaluation

AI Data Engineer on coding-agent evaluation at Handshake AI, as a Platinum Fellow, and part-time at Solumn AI. Each task reconstructs an open-source project at a historical commit, reproduces a reported defect, verifies the maintainers' fix against the project's own tests, and ships as a reproducible benchmark item with a reference solution and an automated grader. Reviewer acceptance has stayed at 100% across three programmes.

  • Docker
  • pytest
  • Python
  • Integer linear algebra
  • Modular arithmetic
  • Git and pull requests
  • Evaluation protocols
100%
reviewer acceptance across three programmes
2
organisations: Handshake AI and Solumn AI
byte-exact
grading of the agent's real output
1 Dockerfile
builds the environment for agent and verifier

Workstream 1

Adversarial evaluation tasks

A task is only useful if a capable model can fail it for a real reason. Each of mine has to meet three conditions at the same time. The naive reading of the data looks correct and is wrong. The cases that decide the answer are hidden from the sample the agent sees, yet fully determined by it. And the result can still be graded byte-exact, so passing or failing is never a judgement call.

Each task ships self-contained. This is the path one task takes from idea to accepted submission.

  1. Design the trap

    Choose a problem where the obvious interpretation of the data is plausible and wrong, and where the deciding cases never appear in the sample but follow from it.

  2. Generate the data

    Build a synthetic corpus for the task instead of borrowing real data, which keeps every case that matters under control.

  3. Prove there is exactly one answer

    Show computationally that the corpus admits exactly one consistent interpretation. Where a task hides structure, I recover it with integer linear algebra and modular arithmetic.

  4. Write the reference solution

    Solve the task by genuine computation, the way an agent would have to, with nothing hard-coded.

  5. Pin the environment

    A digest-pinned Dockerfile builds the environment for both the agent and the verifier, so both work against the same pinned environment.

  6. Grade the real output

    A pytest suite checks what the agent actually produced, byte-exact, instead of matching strings in its answer.

  7. Pass the review gates

    The task goes in as a pull request and iterates against automated review gates until every check is green.

Workstream 2

Repositories rebuilt at a historical commit

Most tasks start from real open-source code. The job is to reconstruct a project at an exact historical commit in a digest-pinned Docker image, reproduce a reported defect, and verify that the project's own tests fail before the maintainers' fix and pass after it, then attach a reference solution and an automated grader so the task is reproducible and comparable across agents.

  1. Check out the commit

    Start from the exact historical commit the task is based on.

  2. Write a build recipe

    Capture how that repository builds in its own recipe, so the environment comes out the same every time it is rebuilt.

  3. Pin until the suite flips

    Pin dependencies until the test suite that failed before the fix passes after it.

  4. Verify against the reference

    Confirm that fail-to-pass flip against the reference solution before the environment ships.

Workstream 3

Submission pipeline tooling

Around the core sits the tooling I built around the platform's own checks, so quality held while volume grew. Part of the pipeline runs through model agents in separate lanes; I write the protocols they follow, set their budgets and check what they produce.

Repository registry

A registry of candidate repositories, so the next task starts from a known pool.

Static screen against the bar

Pre-checks against the review bar before anything is built, so work that cannot pass is never started.

Duplicate detection and build caching

Work that has already been done is recognised and reused.

Independent audit and rework

An independent audit before submission, and a review-and-rework loop after it: returned tasks are taken through rework rather than dropped.

The bar

What the tasks are held to

What a benchmark counts as a task decides what it measures. I keep a written definition that every task is screened against, a list of the ways a task goes wrong, and a log of reviewer returns.

What counts as a valid task

A written definition that every task is screened against, so the benchmark measures the same thing from one task to the next.

A taxonomy of failures

The ways a task can go wrong, named and sorted, so a failure is classified rather than described from scratch each time.

Reviewer returns as data

Every return from review is recorded and read as evidence about the bar, and the bar changes when the evidence says it should.

Comparable across agents

Reproducible environments, verified fixes and automated graders mean the same task gives every agent the same test.

Step 3 of 3

Source code

This work belongs to the client, so the tasks, environments and tools are not public. Ask me about the engineering, or read the code of my open-source projects.