Research Engineer, Benchmarks

About the Role

Join a small technical research and engineering team building benchmarks for evaluating AI agents on realistic, domain-specific workflows. You will own the design and implementation of rigorous evaluations that help technical teams understand how agents perform in real-world tasks.

What You'll Do

Design, implement, and maintain benchmarks for evaluating AI agents on domain-specific tasks.

Work with subject-matter experts to translate real workflows into benchmark tasks and evaluation criteria.

Build and operate reliable infrastructure for running models and agents against tasks at scale.

Develop metrics and analyses to assess benchmark difficulty, reliability, and failure modes.

Validate whether benchmark results align with real-world performance and technical user needs.

Write clear documentation and reports for research and engineering audiences.

What We're Looking For

Two to four years of experience in research engineering, machine learning engineering, or a related technical role, including at least two years building AI benchmarks, evaluation infrastructure, or agent environments.

Strong Python skills and practical experience with Docker and Linux.

Experience designing, implementing, and running benchmarks or evaluation environments for AI agents or large language models.

Experience collaborating with subject-matter experts and analyzing workflows across technical or business domains.

Experience developing metrics, statistical analyses, or validation studies for evaluations.

Strong technical writing, attention to detail, and ability to work independently in an early-stage environment.

A technical educational background; experience with reinforcement learning pipelines, published evaluation work, or widely used benchmarks is a plus.

Compensation & Benefits

Salary range: USD 150,000 to 250,000 annually. Visa sponsorship is available.

Location

On-site in Singapore, Singapore.