hirly

Rhoda AI

Research Member of Technical Staff- Training Systems

Mountain View

hirly Pro

Get in front of the people hiring at Rhoda AI

  • Reach the hiring manager at Rhoda AI on LinkedIn, from your own account
  • hirly applies to this job for you, with unlimited applications
  • A resume and cover letter written for this job, unlimited
  • The hiring manager and recruiter at Rhoda AI, with contact details
Get Pro for this job →

$49 a month · ₹1,999 for 30 days in India · cancel any time

See how you match this job — and similar ones. Free.

Upload your resume and hirly scores it against this role at Rhoda AI first, then against similar open jobs, and shows where you fit and why.

PDF or DOCX, up to 12MB. No sign-up to see your matches.

Get past the screening software and onto a recruiter's desk

hirly rewrites your resume for this job — matching the keywords and skills in the posting, moving your most relevant experience to the top, and writing a cover letter to fit. About 30 seconds.

  • Keywords matched to this posting
  • Fit score before you apply
  • Cover letter included

Matched against 2.9M live jobs from 220,000+ employers in 200+ countries.

Tailor my resume for this job →

Apply from your AI assistant

Connect hirly to Claude or ChatGPT and ask it to apply to this job. hirly tailors your resume, fills the employer’s form and asks before sending.

Some employer sites stop an application at a CAPTCHA or sign-in and hand it back with a link. Applying needs a paid plan. Works with any assistant that supports MCP.

hirly's read of this role

Seniority
Lead / management
Stated salary
$200,000 – $300,000 per year
Country
US
Work mode
Remote-friendly
First seen by hirly
1 Sept 2026

Derived automatically from the posting. Upload your resume above to see how the role scores against it.

the posting

At Rhoda AI, we’re building the next generation of generalist intelligent robots. We own the full robotics stack from high-performance hardware and robot systems to the infrastructure and state-of-the-art foundation world models that control our robots. Our robots are designed to be generalists capable of operating in complex, real-world environments and handling long-tail edge cases, made possible by our cutting edge research and end-to-end system design. We've raised over $450M and are investing aggressively in model research, infrastructure, hardware development, and manufacturing scale-up to make generalist robotics a reality.

We're looking for a Staff / Principal ML Training Systems Engineer to own training systems performance end-to-end. You will define how our models train at scale — driving efficiency, scalability, and correctness across large-scale multimodal training. This is a core systems role, not infrastructure support. Your work directly determines how efficiently we use compute, how well models scale across thousands of GPUs, and how quickly research can iterate.

What You'll Do

Own training performance end-to-end

Diagnose and improve performance of large-scale multimodal training (vision, video, proprioception, actions, language)

Build systematic performance attribution: step-time decomposition (compute vs communication vs input pipeline), scaling curves across cluster sizes, and bottleneck identification and prioritization

Drive measurable gains in:

Distributed efficiency (comm/compute overlap, bucketization, topology-aware mapping, parallelism strategies)

Compute efficiency (kernel hotspots, operator fusion, attention optimization, framework/runtime overhead)

Memory efficiency (activation checkpointing, sequence packing/bucketing, fragmentation reduction)

Design training systems (not just tune them)

Define and evolve parallelism strategies: data / tensor / pipeline / sharding / hybrid approaches

Improve execution efficiency through communication scheduling and overlap, graph capture and execution optimization, and runtime-level improvements

Contribute to and extend training frameworks where needed

Make performance observable and measurable

Establish source-of-truth performance metrics: step-time breakdowns, MFU / throughput / scaling efficiency

Build tools to identify bottlenecks quickly, track performance across model families, and compare scaling behavior across configurations

Develop regression detection: microbenchmarks, performance baselines, and automated detection of efficiency regressions

Partner deeply with researchers

Work side-by-side with research scientists and research engineers — no silos

Translate model innovations into scalable, efficient implementations

Advise on training tradeoffs for robotics world models: long-horizon sequences, rollout/evaluation cadence, multimodal and variable-length data

Collaborate on cluster-level efficiency

Work with infrastructure/SRE teams to improve utilization across large distributed jobs, impact of network and collective performance on training, and topology-aware job placement and scaling behavior

What We're Looking For

Proven track record improving large-scale distributed training performance

Deep hands-on experience with modern ML stacks (PyTorch required; JAX a plus)

Strong understanding of data / tensor / pipeline parallelism, sharded training (FSDP / ZeRO-style), communication patterns and overlap strategies, and scaling behavior across large GPU clusters

Strong systems intuition — ability to reason across compute, communication, and memory bottlenecks

Exceptional debugging and measurement ability: turn "training is slow" into clear bottlenecks, experiments, and validated improvements

High ownership mindset and comfort in a fast-moving environment

Nice to Have (But Not Required)

GPU kernel or compiler-level experience (CUDA, Triton, graph capture, operator fusion)

Experience with multimodal or video training (variable-length sequences, packing/bucketing)

Experience working on large-scale training frameworks or distributed runtimes

Familiarity with cluster topology, networking, and large-scale scheduling effects

Why This Role

Direct leverage on research velocity — every efficiency gain you make accelerates model iteration across the entire research team

Own the scalability and performance of large-scale multimodal training for real-world embodied intelligence, not static benchmarks

Improvements you make compound across every training run the company executes — high ownership, high impact, small elite team

Original posting on Rhoda AI's site ↗

Listed on hirly, a job board. hirly is not the employer: Rhoda AI is hiring for this role.

Browse similar roles

Want this one?

Upload your resume and hirly rewrites it for this job and writes the cover letter — in about thirty seconds, before you sign up.

Tailor my resume for this job