- 1 Projects
- 2 Details
- 3 Source: private client work
Management | Scale AI | Jul 2025 to Jul 2026
LLM coding-data programme
At Scale AI I led a distributed team of 35 on an LLM coding-data programme: task allocation, rubrics and quality bars, mentoring, and delivery. Together we shipped 11,000+ hours at a 3.9/5 average quality rating, and my own work scored 4.3/5 across 51 reviews.
- Leadership
- Rubric design
- Code review
- LLM evaluation
- Python
- LoRA and PEFT
- RLHF and SFT
- RAG
- 35
- people in the squad I led
- 11,000+
- hours shipped with my squad
- 3.9/5
- average quality rating
- 4.3/5
- my own score across 51 reviews
Squad lead
Running a squad of 35
The squad was distributed, and my job covered the whole loop: what we would build, what good looked like, how every piece of work was checked, and getting it delivered. Rubrics were designed so that scores from different reviewers are comparable. These are the four stages I owned.
Scope
Scoped the task sets the squad worked from.
Standards
Wrote the evaluation rubrics and the technical assessments.
Review
Built the review workflow that every piece of work went through before delivery.
Delivery
Ran delivery for the distributed squad. Together we shipped 11,000+ hours of task work at a 3.9/5 average quality rating.
Lead reviewer
Multi-model coding and reasoning evaluation
Alongside the squad, I was lead reviewer on more than half of the projects I was assigned to, multi-model coding and reasoning evaluation among them. My own work was reviewed as well, and 51 internal quality reviews averaged 4.3/5.
I was promoted into the platform's senior reviewer tier and held its top contributor designation.
- 51
- internal quality reviews of my work
- 4.3/5
- average across those reviews
- Senior
- reviewer tier, promoted into it
- Top
- contributor designation, held
Engineering
Validation and fine-tuning pipelines
Validation pipelines
Python pipelines for data quality assurance and for monitoring model output.
RLHF and SFT feedback loops
Feedback loops for reinforcement learning from human feedback and for supervised fine-tuning.
Fine-tuning with LoRA and PEFT
Parameter-efficient fine-tuning of models with LoRA and the PEFT library.
RAG with Model Context Protocol
Retrieval-augmented generation pipelines that use Model Context Protocol to give evaluations the right context.
Step 3 of 3
Source code
This was client work, so the task sets, rubrics and pipelines are not public. Ask me how the squad and the review workflow ran, or read the code of my open-source projects.