Associate Lead - Testing (QA + MLOps)

While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.

If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!

Role: Lead/Associate Lead – QA + MLOps & Generative AI

Experience: 10+ years

Location: Mumbai/Bangalore (Hybrid)

Key Responsibilities:

AI/ML & GenAI Testing Strategy (AWS Ecosystem)

Define testing approaches for AI systems built on AWS services such as:

Amazon SageMaker

Amazon Bedrock

AWS Lambda

Amazon API Gateway

Amazon Kinesis

AWS Glue

Amazon S3

Amazon CloudWatch

Design validation frameworks covering:

Model accuracy & performance validation

Data drift & concept drift detection

Hallucination detection for LLMs

Prompt robustness testing

RAG validation (retrieval accuracy + grounding)

Bias & fairness validation

Safety & toxicity testing

MLOps Quality Engineering (AWS-Centric)

Validate the end-to-end ML lifecycle including:

Data ingestion & feature pipelines

Model training & hyperparameter tuning

Model versioning & registry

Deployment validation

Canary & blue/green release validation

Work with AWS-native services such as:

SageMaker Pipelines

SageMaker Model Monitor

SageMaker Feature Store

Bedrock model evaluation workflows

CloudWatch-based observability

Implement CI/CD quality gates for ML pipelines integrated with AWS DevOps tools.

GenAI & Agentic AI Testing

Define quality engineering approaches for:

LLM-based applications using Amazon Bedrock

Prompt engineering validation

Multi-agent orchestration testing

Chatbot & Voice bot conversational testing

Intent classification validation

Conversation drift & fallback validation

API contract validation for LLM integrations

Build reusable evaluation harnesses for:

BLEU / ROUGE scoring

Embedding similarity scoring

Response consistency

Safety scoring frameworks

Framework & Capability Development

Design reusable AI testing accelerators

Create AWS-aligned AI test automation frameworks (Python-first)

Develop synthetic data generation strategies

Establish AI quality scorecards

Build an internal AI QA Center of Excellence

Client Engagement & Leadership

Lead AI/ML quality strategy workshops

Perform AI risk & readiness assessments

Present quality architecture to CXOs

Drive QA transformation programs

Mentor QA teams on AWS-based AI testing

Own delivery for AI testing engagements end-to-end

Must have skills:

Testing Expertise

8–12+ years in Quality Engineering

Strong test strategy, automation & governance experience

Experience leading QA transformation initiatives

Experience building frameworks from scratch AI/ML & GenAI Expertise

Deep understanding of ML lifecycle

Experience testing ML models (NLP preferred)

Hands-on experience validating LLM applications

Strong understanding of:

Prompt engineering

RAG architecture

Embeddings

Bias & explainability AWS AI/ML Expertise

Hands-on experience with:

Amazon SageMaker (training, deployment, monitoring)

Amazon Bedrock (LLM integration & evaluation)

S3-based data pipelines

AWS IAM (security validation)

CloudWatch monitoring

Lambda & API Gateway integrations

AWS CI/CD (CodePipeline / CodeBuild preferred)

Understanding of:

Infrastructure as Code (Terraform / CloudFormation)

Observability in AI systems

Cost monitoring for ML workloads

Technical Skills

Python (mandatory)

Experience with ML libraries (Scikit-learn, TensorFlow, PyTorch)

Experience with LLM frameworks (LangChain, etc.)

API & automation testing frameworks

Git-based workflows

Leadership & Communication

Strong client-facing communication

Experience leading QA teams

Ability to create strategy decks & solution proposals

Strong stakeholder management

If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us !