Analyst, Applied AI

HeyMilo is building AI interviewers that automate and improve hiring through conversational AI. We work closely with companies to bring AI into real hiring workflows. We also run an applied AI team that studies where today's models succeed and fail across verticals.

The Role

We're hiring an Analyst to join our Applied AI team in Colombo. You'll design the tasks, rubrics, and datasets we use to evaluate AI systems on real-world workflows, and define the ground truth that decides whether a model's attempt counts as correct.

The quality of everything we build downstream depends on the quality of the tasks and ground truth you produce.

What you'll do

Design realistic, multi-step tasks that test AI systems on real-world workflows, and author the grading rubrics that score them

Build and curate ground-truth datasets from real operational data and domain research

Review model attempts against your tasks: analyze failure modes, catch scoring loopholes, and identify where models pass for the wrong reasons

Calibrate task difficulty so results separate models in a meaningful way

Analyze score distributions across models and turn results into clear, defensible findings

Contribute to research write-ups and published evaluations alongside our engineers and academic reviewers

What we're looking for

Master's or PhD in AI, Machine Learning, Computer Science, Statistics, or a closely related quantitative field

Strong understanding of how large language models are trained and evaluated

Hands-on experience assessing LLM outputs: evals, annotation, red-teaming, or benchmark work

Comfortable with Python and data tooling for building datasets and analyzing results

Exceptional attention to detail and intellectual honesty; you flag a broken task even when the scores look good

Clear written communication; you can explain a subtle failure mode to both engineers and non-technical readers

Bonus

Familiarity with reinforcement learning concepts (reward design, verifiable rewards)

Experience contributing to published research or open evaluation suites

Domain knowledge in recruiting, staffing, or other operations-heavy industries

Why join

Ground-floor role on a new applied AI team with direct ownership of what we measure and publish

Work reviewed by and published alongside experienced researchers

High visibility, fast-paced, execution-driven environment

Competitive pay and benefits