Staff Engineer – AI Data Engineering

Prudential’s purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people’s career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.

We are looking for an experienced AI Data Engineering Lead to define and lead the engineering of enterprise-scale data platforms and data products that power Artificial Intelligence, Machine Learning, Generative AI and advanced analytics.

This individual will build the organisation’s AI-ready data foundation by bringing together modern data engineering, cloud platforms, batch and real-time processing, unstructured data, vector and semantic technologies, data governance, and AI/ML engineering practices.

The role requires a senior technology leader with 15+ years of experience, strong hands-on technical depth, architecture capability, and a proven ability to lead engineering teams and deliver complex data platforms at enterprise scale.

Key Responsibilities

AI Data Engineering Strategy & Architecture

· Define the enterprise AI Data Engineering strategy, architecture and engineering standards.

· Design scalable data architectures supporting AI/ML, GenAI, LLM, analytics and enterprise data products.

· Establish reusable architectural patterns for structured, semi-structured and unstructured data.

· Define patterns for data lakes, lakehouse architectures, data warehouses, streaming platforms, vector databases and AI knowledge layers.

· Drive architectural decisions covering performance, scalability, resiliency, security and cost.

AI-Ready Data Platforms

· Lead engineering of data platforms providing trusted and reusable data for ML models, LLMs, AI agents and GenAI applications.

· Build ingestion and transformation frameworks for data from databases, APIs, documents, events, logs and enterprise applications.

· Establish pipelines for batch, near-real-time and streaming data processing.

· Drive automated data validation, quality controls, metadata management and lineage.

· Establish engineering approaches for preparing text, documents, images and other unstructured data for AI consumption.

GenAI & LLM Data Engineering

· Provide technical leadership for Retrieval-Augmented Generation (RAG), document ingestion, parsing, chunking and enrichment.

· Lead embedding generation and management, vector and hybrid search, semantic retrieval and reranking.

· Design knowledge repositories, metadata and access-control filtering, and enterprise search integration.

· Create data pipelines for LLM evaluation, AI grounding, authorised retrieval, fine-tuning where applicable, and AI agents.

· Understand how poor retrieval, incomplete or stale data, weak metadata and inappropriate chunking affect enterprise AI reliability.

Modern Data Engineering

· Provide leadership across ETL/ELT, distributed processing, lakehouse and data warehouse architectures, streaming, event-driven design, data APIs and data services.

· Drive data modelling, data contracts, observability, quality engineering, lineage and metadata management.

· Embed Infrastructure-as-Code, CI/CD, DevOps and DataOps practices in the engineering lifecycle.

· Improve reusability, automation, reliability and developer productivity through common frameworks and standards.

Cloud & Technology Platforms

· Demonstrate significant expertise across modern cloud and data platforms such as Microsoft Azure, Azure Databricks, Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Snowflake and Databricks.

· Bring strong capability in Python, SQL, Apache Spark/PySpark, Kafka, Delta Lake, Airflow and dbt.

· Apply AI and GenAI data technologies including Azure AI Search, vector databases, embedding models, semantic search, RAG frameworks, MLflow and feature stores.

· Experience with Azure OpenAI, enterprise LLM platforms and agent architectures is highly valuable. Deep expertise is expected across a relevant enterprise-scale stack, rather than every technology listed.

Data Governance, Security & Responsible AI

· Embed data governance, privacy, security and regulatory controls into data engineering architecture.

· Implement appropriate access controls for sensitive and confidential information.

· Ensure lineage, traceability and auditability across AI data pipelines.

· Ensure AI solutions retrieve and process only appropriately authorised data.

· Partner with Cybersecurity, Risk, Compliance, Data Governance and AI Governance teams, while promoting Responsible AI engineering practices.

· Experience handling PII, sensitive customer information, data residency and regulatory requirements is highly valuable.

Data Quality & Observability

· Define enterprise standards for AI data quality and automated validation across ingestion, transformation and retrieval pipelines.

· Monitor data freshness, completeness, accuracy, schema changes and pipeline failures.

· Drive observability across data pipelines and AI knowledge systems.

· Establish measurable engineering SLAs and SLOs for critical data services.

Engineering Leadership

· Lead and mentor senior data engineers, AI engineers and platform engineers.

· Provide technical direction across multiple engineering squads and programmes.

· Establish engineering standards, reference architectures and reusable frameworks.

· Conduct architecture and engineering design reviews, and resolve complex technical issues.

· Build capability through coaching, communities of practice and technical mentoring.

· Partner with Enterprise Architecture, Cloud, Cybersecurity, Data, AI/ML and application engineering teams while remaining technically credible and sufficiently hands-on.

Required Experience

· 15+ years of overall technology and engineering experience, with substantial depth in data engineering, data platforms or related disciplines.

· Significant experience leading enterprise-scale data engineering programmes and teams.

· Strong experience designing and implementing cloud-native data platforms.

· Deep understanding of distributed data processing and modern data architectures.

· Demonstrated experience supporting AI/ML or GenAI workloads through data engineering.

· Strong understanding of structured and unstructured data processing.

· Experience with enterprise-grade security, governance and data-quality practices.

· Strong coding capability, particularly in Python, SQL and PySpark or equivalent technologies.

· Experience with CI/CD, DevOps/DataOps and automated engineering practices.

· Proven ability to operate across architecture, engineering and senior stakeholder environments.

Preferred AI-Specific Experience

· GenAI and LLM solutions

· RAG architectures, vector databases and embedding pipelines

· Semantic and hybrid search, knowledge graphs and AI agents

· Feature engineering and feature stores

· MLOps and LLMOps

· Model evaluation datasets and data preparation for fine-tuning

· Unstructured-document processing

· Responsible AI and AI governance

Leadership Competencies

· Technology Vision: Converts AI strategy into practical data-engineering architecture.

· Engineering Excellence: Drives scalable, resilient and maintainable engineering practices.

· AI Mindset: Understands how data architecture changes for GenAI, LLMs and AI agents.

· Enterprise Thinking: Designs reusable capabilities rather than point solutions.

· Technical Leadership: Guides senior engineers and architects through complex technology decisions.

· Stakeholder Influence: Communicates complex concepts effectively to senior business and technology stakeholders.

· Execution: Converts architecture into production-grade platforms and measurable outcomes.

· Talent Devel