Hi, I'm

Vipul Raj Jha

AI Data Engineer at Handshake AI. Evaluating AI coding agents, research in design cognition, led a team of 35.

I build and evaluate the data that trains AI coding agents, at Handshake AI and Solumn AI. Further down: the research, the mathematics and the teams behind it. Based in Delhi, India.

Vipul Raj Jha

Who I am

About

Computer science is my base. I build end-to-end systems, from coding tasks to multi-step pipelines, and the infrastructure that proves they work: digest-pinned Docker environments, reference solutions that compute the real answer, and automated tests that check output rather than claims. That work now trains and evaluates AI coding agents, with a 100% acceptance rate across three programmes.

Research came next. At Ariel University, Israel, I studied design cognition with linkography, built the Java tooling for the analysis, ran 150+ structured evaluations and co-authored a peer-reviewed publication.

Management is the third kind of work. I led a distributed team of 35 on an LLM coding-data programme: task allocation, rubrics and quality bars, mentoring, 11,000+ hours delivered at 3.9/5 and a personal 4.3/5 across 51 reviews.

Mathematics runs through all of it. Graphs and entropy in the research, statistics in the quality work, algorithms and number theory in competitive programming, and the linear algebra, probability and discrete mathematics from my B.Tech at IIIT Delhi.

100%
of my submissions accepted across three coding-agent programmes
150+
structured evaluations in the linkography study, and one peer-reviewed paper
35
people in the squad I led at Scale AI
11,000+
hours shipped with that squad, at 3.9/5 quality
1485
Codeforces rating, Specialist, 1000+ problems solved
4.3/5
across 51 internal quality reviews

2023 to now

Where I have worked

  1. Solumn AI

    Jul 2026 to present, part-time

    AI Data Engineer

    • Coding-agent evaluation data, part-time. Each task rebuilds an open-source project at a historical commit in a digest-pinned Docker image, checks that the project's own tests fail before the fix and pass after it, and ships with a reference solution and an automated grader.
    • I keep the tooling around it running: a registry of candidate repositories, a pre-check against the acceptance bar, duplicate detection, build caching and an audit step before anything is submitted.
    • Part of the pipeline runs through model agents in separate lanes; I write the protocols they follow, set their budgets and check what they produce.
    • Each reviewer return is logged and read back into the pre-check, so the same miss is not made twice. Submissions so far have all been accepted.
  2. Handshake AI

    Jul 2026 to present

    AI Data Engineering Fellow (Platinum)

    • Platinum Fellow on coding-agent evaluation. Each task reconstructs an open-source project at a historical commit, reproduces a reported defect, verifies the maintainers' fix against the project's own tests, and ships as a reproducible benchmark item with a reference solution, grader and load-bearing artifact.
    • I built my own tooling around the platform's checks (repository registry, pre-checks against the review bar, duplicate detection, build caching, independent audit) so quality held while volume grew.
    • I take returned tasks through rework.
    • Every submission so far has been accepted, and my throughput has been the highest on my programmes.
  3. Scale AI

    Jul 2025 to Jul 2026

    AI Engineer and squad lead

    • Led a distributed technical squad of 35 on an LLM coding-data programme. I scoped the task sets, wrote the rubrics and technical assessments, built the review workflow and ran delivery.
    • 11,000+ hours shipped with my squad of 35, at a 3.9/5 average quality rating.
    • Lead reviewer on more than half my assigned projects, including multi-model coding and reasoning evaluation. 4.3/5 across 51 internal quality reviews.
    • Promoted to the platform's senior reviewer tier and held its top contributor designation.
    • Python validation pipelines, RLHF and SFT feedback loops, fine-tuning with LoRA and PEFT, and RAG pipelines with Model Context Protocol.
  4. Zinnia Digital Services

    Apr 2025 to Jun 2025

    Automation and AI Intern

    • End-to-end QA automation in Python and Selenium for insurance client applications.
    • Delivered two phases of the CI/CD pipeline. Regression testing time fell about 65%, and each release cycle lost more than three hours of manual estimation.
    • REST API contract validation across several releases.
  5. ETiDM Lab, IIIT Delhi

    Jan 2024 to Aug 2024

    VR Developer

    • A Blender-inspired 3D modelling application in Unity and C#, with 3-axis manipulation, validated through user testing.
  6. Ariel University, Israel

    Jul 2023 to Apr 2024

    Research Intern, with the Human-Centred Design Lab, IIIT Delhi

    • A study of how design methods shape ideation and cost in collaborative architectural design, built on linkographs (Kan & Gero, 2008) of recorded design sessions for climate-acclimatised buildings.
    • Wrote the evaluation modules in Java that encode moves, record forelinks and backlinks, and compute link index, critical moves and entropy per session, and ran more than 150 structured evaluations through them.
    • Framed the question with my advisor, designed the protocol, built the tooling, ran the analysis and wrote up the findings, which became a co-authored, peer-reviewed publication.

What I have studied

Research

A published study of design cognition, the evaluation data behind a coding-agent benchmark, and a literature tool built along the way.

Linkography at Ariel University

At Ariel University, Israel, I studied how design methods shape ideation and cost in collaborative architectural design. We took recorded design sessions for climate-acclimatised buildings, transcribed them move by move and built linkographs (Kan & Gero, 2008) that map every design move to the moves it draws on; the coding scheme went through two full passes.

Hand-coding at that density is not reproducible, so I wrote the evaluation modules in Java that encode moves, record forelinks and backlinks, and compute link index, critical moves and entropy per session, and ran more than 150 structured evaluations through them. The findings became a co-authored, peer-reviewed publication.

How the study was done · Read the paper

Evaluation data at Solumn AI

The question at Solumn AI is what makes a benchmark task valid. I keep a written definition that every task is screened against, a list of the ways a task goes wrong, and a log of reviewer returns; each return changes the pre-check.

The pipeline under it rebuilds open-source projects at exact historical commits, checks that each project's own tests fail before the fix and pass after it, and attaches a reference solution and an automated grader, so every task is reproducible and comparable across agents.

How the work is built

research-harvest, applied research tooling

A command-line tool and Python library for literature screening. It searches PubMed and OpenAlex, merges and deduplicates the results, and reads the methods, cohort size and best score out of each abstract. The extraction is deliberately narrow, so a p-value is never reported as a cohort.

115 tests at 95% coverage, all running offline against recorded fixtures.

Details and code

In the work

Mathematics in the work

Competitive programming, the graph measures in the research, the statistics in the quality work, and the coursework underneath.

1485

Competitive programming

Codeforces Specialist, world rank 979 in Round 994 out of 35,000+ entrants, and 1000+ problems solved: algorithms, complexity, combinatorics and number theory in practice.

Codeforces profile
Graphs

Inside the research

A linkograph is a graph of design moves. For every session the Java tooling computed the link index and an information-theoretic entropy, so sessions could be compared on numbers rather than impressions.

The linkography study
Statistics

Inside the quality work

Ratings aggregated over 11,000+ hours of delivered work and 51 reviews of my own, with rubrics designed so that scores from different reviewers are comparable.

How the squad ran
B.Tech

The coursework base

Linear algebra, probability and statistics, and discrete mathematics at IIIT Delhi, 2021 to 2025, with a Dean's List year in 2023-24.

Squad lead at Scale AI

The team I have led

A distributed squad of 35 people on an LLM coding-data programme.

On an LLM coding-data programme at Scale AI I led a distributed technical team of 35: task allocation, rubrics and quality bars, mentoring, and delivery. The work ran from what the squad would build to whether it was ready to ship.

  1. Scope

    Scoped the task sets the squad worked from.

  2. Standards

    Wrote the evaluation rubrics and the technical assessments.

  3. Review

    Built the review workflow every piece of work went through.

  4. Delivery

    Ran delivery for the whole distributed squad.

11,000+
hours shipped with my squad of 35
3.9/5
average quality rating
4.3/5
my own score across 51 quality reviews

Read how I ran it

11 projects

What I have built

Six from work, described without client details, one research study, and four open-source projects. Each card is labelled by the kind of work. Open any card for the full write-up.

  1. 1 Pick a project
  2. 2 Read how it works
  3. 3 Open the code on GitHub
Filter by:

Showing all 11 projects

Computer sciencegrade --byte-exact

Adversarial coding-agent tasks

Handshake AI

Tasks where the naive reading of the data looks correct and is wrong, and the deciding cases are hidden from the sample but fully determined by it. I generate synthetic corpora, prove computationally that each admits exactly one consistent interpretation, and recover hidden structure with integer linear algebra and modular arithmetic.

  • Docker
  • pytest
  • Python
  • Linear algebra
Computer sciencefail -> pass, verified

Repositories rebuilt at a historical commit

Handshake AI and Solumn AI

Each task reconstructs an open-source project at a historical commit in a digest-pinned Docker image, reproduces a reported defect, and verifies that the project's own tests fail before the fix and pass after it, then ships with a reference solution and an automated grader.

  • Docker
  • Dependency pinning
  • pytest
Computer sciencebuild-lock acquired

Submission pipeline tooling

Solumn AI and Handshake AI

A registry of candidate repositories, a static screen against the acceptance bar, duplicate detection, build caching, independent audit and a review-and-rework loop, built around the platform's own checks so quality held while volume grew. Every task goes in as a pull request and iterates against automated review gates until each check is green.

  • Repository registry
  • Pre-checks and audit
  • Pull request gates
Managementsquad=35 hours=11000+

LLM coding-data programme

Scale AI

Led a distributed squad of 35 from scope to delivery. I scoped the task sets, wrote the rubrics and technical assessments, and built the review workflow. Together we shipped 11,000+ hours at a 3.9/5 average quality rating.

  • Leadership
  • Rubric design
  • Review workflow
Managementreviews=51 score=4.3

Multi-model coding and reasoning evaluation

Scale AI

Lead reviewer on multi-model coding and reasoning evaluation, and on more than half the projects I was assigned. Scored 4.3/5 across 51 internal quality reviews and was promoted into the platform's senior reviewer tier.

  • LLM evaluation
  • Code review
  • Reasoning
Computer sciencerlhf sft lora peft rag

Validation and fine-tuning pipelines

Scale AI

Python validation pipelines for data quality assurance and model output monitoring. RLHF and SFT feedback loops, fine-tuning with LoRA and PEFT, and RAG pipelines using Model Context Protocol for context-aware evaluation.

  • Python
  • LoRA and PEFT
  • RLHF
  • RAG
  • MCP
Researchlinkograph --entropy --per-session

Linkography study of collaborative design

Ariel University, Israel, with the Human-Centred Design Lab, IIIT Delhi

How design methods shape ideation and cost in collaborative architectural design. Recorded sessions were transcribed move by move and built into linkographs (Kan & Gero, 2008); I wrote the Java modules that encode moves, record forelinks and backlinks, and compute link index, critical moves and entropy per session.

More than 150 structured evaluations, two full coding passes, and a co-authored, peer-reviewed publication.

  • Java
  • Linkography
  • Entropy
  • Scientific writing
Open source115 passed, 95% coverage

research-harvest

Searches PubMed and OpenAlex, merges the results, removes papers found in both, and reads the methods, cohort size and best score out of each abstract. The extraction is the hard part: anything clever enough to catch every phrasing also catches a p-value and calls it a cohort.

Built on the official REST APIs and deduplicated on DOI, then source id, then normalised title. 115 tests at 95% coverage, all running offline against recorded fixtures.

  • Python
  • REST APIs
  • pytest
Open source135 tests, 2 backends

SilverPass

Passport services for someone who is 78 and cannot read 16px type. 20px base text with three size steps, high contrast mode, an English and Hindi interface, and the first two hours at every centre held for people aged 60 and over.

Express API and React client with a status machine that refuses illegal transitions. A JSON store and MongoDB are held to one contract test, and CI runs the server tests once per backend. 135 tests in all.

  • Express
  • React
  • MongoDB
  • Accessibility
Open source108 tests, deployed

CourseBot360

Answers questions like "is there a lab in OS?" about 31 university courses. It is a deterministic intent parser on purpose: a static site cannot hide an API key, and a parser is fast, free, needs no server and cannot invent a prerequisite.

The intent engine never touches the DOM and returns answers as data, so a whole conversation runs in a test. 108 tests, deployed by GitHub Actions.

  • JavaScript
  • Testing
  • GitHub Pages
Open sourcephp -l 8.0 8.2 8.3 ok

Cab Booking System

Fleet and booking management in PHP and MySQL. Customers book a cab and follow its status, drivers move each job from confirmed to dropped off, and staff run the fleet. It runs with docker compose up.

Passwords use bcrypt at cost 12, and old MD5 hashes are upgraded in place at sign-in. CI lints on PHP 8.0, 8.2 and 8.3, then boots the stack in Docker, signs the admin in and checks that a wrong password leaks no SQL.

  • PHP
  • MySQL
  • Docker
  • GitHub Actions

Stack and methods

Skills

I write production code in

  • Python
  • C++
  • JavaScript
  • Java
  • SQL
  • PHP
  • C#
  • Bash

I ship and test with

  • Docker
  • GitHub Actions
  • pytest
  • vitest
  • node:test

On the ML side I use

  • PyTorch
  • Hugging Face
  • scikit-learn
  • LoRA and PEFT
  • RAG
  • Embeddings and vector search

For the web I build with

  • Node
  • React
  • Express
  • Vite
  • MySQL
  • MongoDB
  • Selenium

In research I work with

  • experimental design
  • evaluation protocols
  • linkography
  • statistical analysis
  • scientific writing

The mathematics I use is

  • linear algebra
  • probability and statistics
  • discrete mathematics
  • algorithms and complexity

Beyond work

Achievements

1485

Codeforces Specialist

World rank 979 in Round 994, out of more than 35,000 entrants.

Codeforces profile
1000+

DSA problems solved

Data structures and algorithms across Codeforces, LeetCode and practice sets.

LeetCode profile
2023-24

Dean's List

IIIT Delhi, B.Tech in Computer Science and Social Sciences, 2021 to 2025.

2023

Research paper

"To study the influence of design methods on design ideation and cost", a linkograph study of collaborative design sessions. With Pratham Kumar, advised by Sonal Keshwani, IIIT Delhi.

Read the paper

Get in touch

Let's connect

Hiring for evaluation, research, data or infrastructure work, or want to talk about a hard problem? Every message here reaches me at vipul21435@iiitd.ac.in.

Your message goes straight to my inbox at vipul21435@iiitd.ac.in, and I reply from there.