Senior/Staff AI Evaluation Engineer

Job Description

Role: Senior/Staff AI Evaluation Engineer

Experience: 6+ Years

Location: Remote (Pan India)

Mode: Hyparz Payroll

About the Role

We are looking for a Senior/Staff AI Evaluation Engineer to lead the benchmarking, validation, reliability, and safety evaluation of next-generation Agentic AI platforms .

This role sits at the intersection of Quality Engineering, Software Engineering, and AI/ML . You will build automated evaluation frameworks, benchmark AI behavior, validate LLM/RAG/agentic workflows, and establish measurable quality standards for production AI systems.

What You'll Do

• •Design and implement automated evaluation frameworks for LLM, RAG, and agent-based applications .
• •Build and maintain golden datasets, benchmark datasets, test datasets, and regression suites for AI evaluation.
• •Develop measurable evaluation criteria for accuracy, relevance, consistency, safety, reliability, and agent behavior.
• •Perform adversarial testing to identify hallucinations, prompt vulnerabilities, unsafe behavior, and edge cases.
• •Evaluate LangGraph-based and multi-agent systems at the node, state-transition, routing, and workflow levels.
• •Use LangSmith for tracing, debugging, experiments, datasets, and evaluation workflows.
• •Work extensively with LangChain and LangGraph , including sub-graphs and conditional routing.
• •Build Python-based automation for functional, regression, integration, and end-to-end AI testing.
• •Evaluate prompts, embeddings, vector search, RAG pipelines, tool calling, and agentic workflows.
• •Validate AI systems against safety, security, privacy, and Responsible AI expectations.
• •Integrate evaluation and regression testing into GitHub-based CI/CD workflows .
• •Work with AWS and Amazon Bedrock to validate AI-powered application architectures.
• •Perform API and backend validation using REST APIs, JSON, SQL, and modern application architectures.
• •Collaborate with Engineering, Product, Data, and Quality teams to investigate failures and drive improvements.
• •Take ownership of ambiguous AI quality problems and convert them into repeatable, measurable evaluation approaches.

What We're Looking For

• •6+ years of experience in software engineering, quality engineering, test automation, AI engineering, ML engineering, or a related technical discipline.
• •Demonstrable hands-on experience testing or evaluating LLM, NLP, ML, RAG, or agentic AI applications .
• •Strong Python development and automation experience.
• •Deep hands-on knowledge of LangChain, LangGraph, and LangSmith .
• •Experience evaluating LangGraph node execution, state transitions, conditional routing, sub-graphs, or multi-agent workflows .
• •Strong understanding of LLMs, prompt engineering, embeddings, vector databases/search, RAG, and AI agents.
• •Experience building golden datasets, benchmark datasets, or structured AI test datasets.
• •Experience with adversarial testing and AI safety evaluation.
• •Practical understanding of security, privacy, and Responsible AI evaluation .
• •Strong GitHub experience covering repositories, branching, pull requests, code reviews, and CI/CD.
• •Experience with Claude Code or comparable AI-assisted development tools .
• •Experience with AWS and Amazon Bedrock .
• •Strong REST API, JSON, and SQL knowledge.
• •Strong experience in automated, regression, integration, and end-to-end testing.
• •Excellent analytical, troubleshooting, communication, and problem-solving abilities

Skills:- Large Language Models (LLM), Retrieval Augmented Generation (RAG), AI Evaluation, Agentic AI and AWS Bedrock