hirly

Nolla Health

Research Engineer, Evals & RL Environments

New York City

hirly Pro

Get in front of the people hiring at Nolla Health

  • Reach the hiring manager at Nolla Health on LinkedIn, from your own account
  • hirly applies to this job for you, with unlimited applications
  • A resume and cover letter written for this job, unlimited
  • The hiring manager and recruiter at Nolla Health, with contact details
Get Pro for this job →

$49 a month · ₹1,999 for 30 days in India · cancel any time

See how you match this job — and similar ones. Free.

Upload your resume and hirly scores it against this role at Nolla Health first, then against similar open jobs, and shows where you fit and why.

PDF or DOCX, up to 12MB. No sign-up to see your matches.

Get past the screening software and onto a recruiter's desk

hirly rewrites your resume for this job — matching the keywords and skills in the posting, moving your most relevant experience to the top, and writing a cover letter to fit. About 30 seconds.

  • Keywords matched to this posting
  • Fit score before you apply
  • Cover letter included

Matched against 2.9M live jobs from 220,000+ employers in 200+ countries.

Tailor my resume for this job →

Apply from your AI assistant

Connect hirly to Claude or ChatGPT and ask it to apply to this job. hirly tailors your resume, fills the employer’s form and asks before sending.

Some employer sites stop an application at a CAPTCHA or sign-in and hand it back with a link. Applying needs a paid plan. Works with any assistant that supports MCP.

hirly's read of this role

Seniority
Mid level
Stated salary
$180,000 – $240,000 per year
Country
US
Work mode
On-site / unstated
First seen by hirly
27 Sept 2026

Derived automatically from the posting. Upload your resume above to see how the role scores against it.

the posting

New York City · Full-time

Build the environments that teach us what healthcare AI can do—and where it needs to improve.

At Nolla Health, we’re building AI-powered healthcare. To improve the systems involved in delivering care, we need realistic ways to test them: cases that unfold over time, tools an agent can use, and clear criteria for judging what happens.

We’re looking for a research engineer to build evaluation harnesses, simulated environments, and data pipelines that turn clinical knowledge and appropriately governed data into useful evaluation and training assets.

You’ll work with engineers and clinicians to define tasks, build the systems that run them, and make the results trustworthy. The work will support our own agents and models, with the opportunity to develop benchmarks and environments for external AI teams.

What you'll do

Build harnesses that run agents through reproducible tasks, capture their actions, and make failures easy to investigate.

Develop multi-step and multi-visit environments with patient state, clinical tools, and realistic consequences for an agent’s decisions.

Work with clinicians to turn clinical workflows into well-defined tasks, rubrics, and success criteria.

Build data pipelines for preparing, validating, versioning, and tracing evaluation and training datasets.

Develop scoring and reward mechanisms, combining programmatic checks, model-based grading, and clinician review where appropriate.

Test graders and environments for ambiguity, shortcuts, data leakage, and reward hacking.

Run experiments, analyze trajectories, and identify whether changes improve performance across meaningful slices of cases.

Work with our applied AI engineer to turn evaluation findings into product improvements.

Build with privacy, access controls, and permitted data use in mind.

What you bring

Strong programming skills and experience building reliable software for experiments, data processing, or agent systems.

Hands-on experience building an agent evaluation harness, an RL environment, or closely related infrastructure.

The ability to translate an open-ended problem into a runnable task with measurable outcomes.

Careful judgment about data quality, reproducibility, grading reliability, and what a benchmark result actually supports.

Comfort debugging across datasets, environment state, agent behavior, and scoring code.

The ability to work with clinicians and turn domain expertise into precise specifications and tests.

Evidence of independent execution through shipped systems, research implementations, open-source work, or projects you’ve owned.

We expect many candidates to have roughly two to six years of relevant experience, but we’re flexible for someone with exceptional work. Research experience is valuable; you should also enjoy building and maintaining the systems behind the experiments.

Bonus points

RL environments, simulated tool use, or long-horizon agent benchmarks.

Synthetic data generation, human review workflows, or dataset quality systems.

Reward design, model-based graders, and validation against expert judgments.

Building evaluation or training data products for AI labs.

Healthcare data, clinical reasoning evaluations, or longitudinal patient workflows.

Healthcare experience is welcome, but it isn’t required. Curiosity about the domain and a willingness to work closely with clinicians matter.

You’ll help define how we measure progress toward AI that can support care over time. Your work will make it possible to investigate failures, compare approaches, and build training signals grounded in realistic tasks.

You’ll own both the experimental design and the engineering needed to make it useful, with room to shape a new area of work at Nolla.

We’d ideally like you to be based in New York City.

Role logistics, compensation & benefits

Role Type: Engineering

Salary: $180,000–$240,000, based on experience

Job Type: Full-time

Work Setup: New York City preferred

Equity: Meaningful equity, commensurate with experience

Health Insurance: Medical, dental, and vision

HSA/FSA: Eligible

Time Off: Flexible, unlimited vacation

Additional Perks: Meal stipends, team retreats

Work Hours: Flexible but demanding. We're building something that matters

Growth: Founding team members step into expanded roles as we scale

Original posting on Nolla Health's site ↗

Listed on hirly, a job board. hirly is not the employer: Nolla Health is hiring for this role.

Browse similar roles

Want this one?

Upload your resume and hirly rewrites it for this job and writes the cover letter — in about thirty seconds, before you sign up.

Tailor my resume for this job