Mpathic2
AI/ML Evaluation Reviewer- Healthcare AI (Contract)
Remote
hirly Pro
Get in front of the people hiring at Mpathic2
- Reach the hiring manager at Mpathic2 on LinkedIn, from your own account
- hirly applies to this job for you, with unlimited applications
- A resume and cover letter written for this job, unlimited
- The hiring manager and recruiter at Mpathic2, with contact details
$49 a month · ₹1,999 for 30 days in India · cancel any time
Get past the screening software and onto a recruiter's desk
hirly rewrites your resume for this job — matching the keywords and skills in the posting, moving your most relevant experience to the top, and writing a cover letter to fit. About 30 seconds.
- Keywords matched to this posting
- Fit score before you apply
- Cover letter included
Matched against 2.1M live jobs from 220,000+ employers in 190+ countries.
Tailor my resume for this job →Apply from your AI assistant
Connect hirly to Claude or ChatGPT and ask it to apply to this job. hirly tailors your resume, fills the employer’s form and asks before sending.
Some employer sites stop an application at a CAPTCHA or sign-in and hand it back with a link. Applying needs a paid plan. Works with any assistant that supports MCP.
hirly's read of this role
- Seniority
- Mid level
- Work mode
- Remote-friendly
- First seen by hirly
- 20 Oct 2026
Derived automatically from the posting. Upload your resume above to see how the role scores against it.
the posting
About the Role
mpathic provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in real-world clinical settings. As part of our work supporting Utah's Office of Artificial Intelligence Policy (OAIP) regulatory sandbox, we are recruiting experienced machine learning (ML) and data science professionals to serve as technical reviewers on expert evaluation panels.
In this role, you will independently assess the technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, and proposed monitoring and safeguards. Your assessments will help inform recommendations regarding whether and under what conditions an AI technology may be appropriate for pilot deployment.
You will work alongside clinical and other subject-matter experts to evaluate whether applicants' technical claims are supported by credible evidence and whether their proposed evaluation and monitoring plans are appropriate for the intended healthcare setting and patient population.
This is an evaluation and scientific review role, not a model development position. We are looking for experts who can critically assess existing evidence, identify limitations, and communicate actionable recommendations to technical and non-technical stakeholders.
What You'll Be Doing
Responsibilities may include:
Evaluate model performance and validation: Assess the appropriateness of validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty.
Assess data quality and representativeness: Review whether training, testing, and validation datasets are appropriate for the intended clinical application, including potential data leakage, subgroup performance differences, and generalizability across patient populations and healthcare settings.
Evaluate technical reliability: Assess evidence of model robustness, reproducibility, known failure modes, and performance under atypical or out-of-distribution inputs.
Review proposed evaluation plans: Determine whether applicants have established scientifically defensible methods, performance benchmarks, and success criteria for evaluating AI technologies during pilot deployment, particularly when pre-deployment evidence is limited.
Assess technical safeguards and human oversight: Evaluate whether proposed technical safeguards, escalation mechanisms, and human-in-the-loop processes are operationally feasible and supported by appropriate testing.
Evaluate monitoring and change management: Assess applicants' plans for detecting performance degradation, monitoring model behavior, managing updates, and establishing criteria for intervention or rollback.
Collaborate with multidisciplinary reviewers: Work alongside clinicians and other technical experts to identify concerns that require cross-disciplinary judgment, particularly where model performance may affect clinical outcomes.
Produce structured evaluation findings: Document strengths, limitations, missing evidence, and technical concerns using a standardized review framework. Provide clear, evidence-based recommendations and identify what additional evidence or changes would resolve identified concerns.
Participate in reviewer calibration and quality assurance: Contribute to discussions intended to improve consistency, transparency, and defensibility across evaluations.
Maintain independence and confidentiality: Follow conflict-of-interest, confidentiality, and data-handling requirements when reviewing proprietary or unpublished materials.
What We're Looking For
We are seeking technically rigorous, independent reviewers who can distinguish between a promising technology and evidence that adequately supports its proposed use.
Strong candidates are comfortable questioning unsupported performance claims, identifying limitations in study design, and recognizing when additional evidence is needed before meaningful conclusions can be drawn. They understand that healthcare AI evaluation is context-dependent: technical performance must be considered alongside the intended population, clinical application, degree of autonomy, and potential consequences of model error.
Candidates should also be able to translate complex technical findings into clear, actionable recommendations for multidisciplinary audiences, including clinicians and state regulatory stakeholders.
Basic Qualifications
At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems.
Demonstrated understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error.
Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset.
Hands-on experience evaluating at least one relevant class of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models.
Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions.
Strong analytical writing skills, including the ability to communicate complex findings clearly to non-technical audiences.
Ability to work independently while contributing to multidisciplinary expert review and calibration processes.
Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training requirements.
Preferred Qualifications
Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments.
Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards.
Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards.
Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis.
Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices.
Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments.
Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device.
Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations.
Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment.
Engagement Structure
This is a contract, per-review opportunity. Reviewers will be assigned to applications based on their technical expertise and the specific AI systems being evaluated.
Assignments will involve reviewing applicant materials, documenting findings using mpathic's standardized evaluation framework, and participating in review discussions or calibration activities as needed.
Reviewers will receive orientation to the evaluation framework and project-specific requirements prior to their first assignment.
Listed on hirly, a job board. hirly is not the employer: Mpathic2 is hiring for this role.
Want this one?
Upload your resume and hirly rewrites it for this job and writes the cover letter — in about thirty seconds, before you sign up.
Tailor my resume for this job