hirly

Sunset

Data Scientist

New York

hirly Pro

Get in front of the people hiring at Sunset

  • Reach the hiring manager at Sunset on LinkedIn, from your own account
  • hirly applies to this job for you, with unlimited applications
  • A resume and cover letter written for this job, unlimited
  • The hiring manager and recruiter at Sunset, with contact details
Get Pro for this job →

$49 a month · ₹1,999 for 30 days in India · cancel any time

See how you match this job — and similar ones. Free.

Upload your resume and hirly scores it against this role at Sunset first, then against similar open jobs, and shows where you fit and why.

PDF or DOCX, up to 12MB. No sign-up to see your matches.

Get past the screening software and onto a recruiter's desk

hirly rewrites your resume for this job — matching the keywords and skills in the posting, moving your most relevant experience to the top, and writing a cover letter to fit. About 30 seconds.

  • Keywords matched to this posting
  • Fit score before you apply
  • Cover letter included

Matched against 2.2M live jobs from 220,000+ employers in 200+ countries.

Tailor my resume for this job →

Apply from your AI assistant

Connect hirly to Claude or ChatGPT and ask it to apply to this job. hirly tailors your resume, fills the employer’s form and asks before sending.

Some employer sites stop an application at a CAPTCHA or sign-in and hand it back with a link. Applying needs a paid plan. Works with any assistant that supports MCP.

hirly's read of this role

Role family
Data & ML
Seniority
Mid level
Country
US
Work mode
On-site / unstated
First seen by hirly
2 Sept 2026

Derived automatically from the posting. Upload your resume above to see how the role scores against it.

the posting

About Replay

We are building the first-ever bank for enterprise data. There are over 35 million businesses in the US alone, many of which are sitting on an asset they don’t know they have. At Replay, we help businesses monetize their historical operating data by licensing it to the frontier AI and data labs for model training. Replay sits at the intersection of advancing the frontier of AI, while helping companies generate millions in growth capital without having to give up equity, take on debt, or lay off employees.

Why Join Replay Now

We have scaled from $0 to a multi-eight-figure run rate in a matter of months

We have raised from top-tier investors, including Floodgate, Afore, Ludlow, and Hustle Fund

We are small enough that you will carry outsized responsibility and grow as quickly as the company does

You will partner with and build for some of the fastest and most important companies in the world

You will help build a massive, category-defining business from the ground floor

The Role

Replay turns sensitive internal enterprise data into de-identified datasets without destroying the structure and meaning that make the data valuable. That creates a difficult measurement problem. A system can improve aggregate F1 while missing a high-risk slice, remove more sensitive information while also destroying useful context, or pass one stage while defects escape somewhere else in the pipeline.

As Replay's first Data Scientist focused on evaluation, you will establish how we know whether that data is actually getting better. You will build the datasets, experiments, quality measures, and feedback loops that expose hidden failures, accelerate model and pipeline improvement, and give the team confidence in what it delivers.

This is a hands-on, zero-to-one role at the intersection of data science, AI, and a real production system. You will write Python and SQL, construct evaluation corpora, study failure patterns, design comparisons, calibrate human and model-based judgments, and turn the result into a clear decision. The questions are scientifically difficult, but the output must be practical enough to change what the team builds and ships.

You will work closely with Machine Learning, Product Engineering, Data Engineering, Security, Quality, domain experts, and the team making delivery decisions. Machine Learning Engineers own changing model behavior. You own the credibility of the evidence used to decide whether a model, pipeline, or delivery change actually made the data safer or more useful.

Questions You Might Answer

Did a higher NER or entity-resolution score actually reduce sensitive misses across the messages, documents, tables, and providers that matter?

Is a new model finding more sensitive information, or simply removing more of the useful structure our customers need?

Can we trust a golden dataset, a human review process, or an LLM judge enough to use it for a release decision?

Which customer, modality, entity, language, or format slices are hidden by a strong aggregate result?

Where did a quality loss enter between source data, processing, de-identification, review, and delivery?

What is the smallest credible experiment that would tell us whether to ship, revise, or stop a change?

What You'll Do

Define what high-quality and safe-to-deliver data mean across de-identification, structure preservation, semantic coherence, and customer utility

Design representative samples and build golden, adversarial, replay, and production-like corpora with explicit provenance, labeling policy, agreement, adjudication, and versioning

Turn ambiguous concepts such as “useful,” “clean,” or “safe” into measurable claims with known uncertainty and clear decision consequences

Evaluate detectors, models, prompts, judges, thresholds, review workflows, and pipeline changes using comparisons that can support a real decision

Break aggregate results into the modalities, providers, entity classes, customer contexts, languages, formats, and risk tiers that reveal consequential failures

Connect local measures to escaped sensitive information, avoidable over-redaction, preserved data utility, review burden, rework, and delivery acceptance

Build reproducible analysis, evaluation pipelines, and high-fidelity environments using Python, SQL, synthetic data, historical replay, seeded failures, and programmatic verifiers

Establish holdout and evaluation practices that keep the evidence trustworthy while model and product teams iterate quickly

Use modern AI tools deeply for analysis, corpus development, coding, review, and hypothesis generation while independently verifying their output

What Success Looks Like

The team has a decision-grade baseline for a priority Clean Data quality claim and trusts it enough to use in model, pipeline, release, and delivery decisions

Improvements are judged by the slices and failure costs that matter, not only by an aggregate benchmark

The company can distinguish a true gain from label noise, sample bias, leakage, evaluator error, or a shifted workload

Changes that improve one stage cannot hide escaped defects, over-redaction, utility loss, or review burden somewhere else

At least one consequential decision changes because the evidence reveals a risk, tradeoff, or opportunity that was previously unclear

Evaluation becomes faster and more repeatable without sacrificing independence or rigor

Quality claims communicate uncertainty honestly and remain understandable to engineers, customers, and risk owners

You Might Thrive Here If

You have at least three years of professional experience in applied science, data science, machine learning, quantitative research, or a closely related role

You have designed evaluations or experiments that changed a product, model, release, or operational decision

You understand sampling, uncertainty, precision, recall, F1, calibration, agreement, class imbalance, distribution shift, and imperfect labels

You can investigate messy, multi-stage data systems and determine where an apparent gain or loss actually came from

You are comfortable writing Python and SQL and building reproducible technical artifacts rather than handing requirements to an engineering team

You can protect the independence of an evaluation while collaborating closely with the people whose work it evaluates

You have startup experience and enjoy broad ownership, changing context, and building the measurement foundation while decisions are already moving quickly

You use AI tools fluently but do not confuse an articulate model output with valid evidence

You communicate uncertainty and difficult findings directly, without hiding behind false precision

This Role May Not Be for You If

You want to optimize models as your primary job rather than determine whether changes actually improve delivered data

You prefer descriptive dashboards that stop short of changing a decision

You treat labels, benchmarks, or model-based judges as ground truth without investigating how they fail

You need a perfectly defined dataset and research plan before you can make progress

You are uncomfortable disagreeing with a technically strong team when the evidence does not support its conclusion

You do not want AI tools to be part of your daily scientific and technical workflow

Bonus

Experience evaluating NER, entity resolution, information extraction, document understanding, multimodal, retrieval, or LLM systems

Experience with privacy, de-identification, data quality, model risk, safety, or other high-trust decision systems

Experience designing human-review, adjudication, weak-supervision, or active-learning systems

Experience building adversarial corpora, replay systems, simulation environments, programmatic verifiers, or model-judge evaluations

Experience connecting offline measures to escaped defects, customer outcomes, review effort, or

Original posting on Sunset's site ↗

Listed on hirly, a job board. hirly is not the employer: Sunset is hiring for this role.

Browse similar roles

Want this one?

Upload your resume and hirly rewrites it for this job and writes the cover letter — in about thirty seconds, before you sign up.

Tailor my resume for this job