Data Engineering Lead

• You’ll I nfluence T he Lives O f M illions G lobally
• Do you want to be part of one of the global leading innovators in the biopharmaceutical business in shaping the technology that reinforces everything we do that helps AstraZeneca push the boundaries and turns ideas into life changing medicines ? The Drug Development Data Platform team here at AZ plays a key role introducing process and technology improvements such as Data Mesh , Agile, DevOps and the very latest engineering tools to maximise velocity and business value.

• Apply Y our E xpertise I n A D ynamic T eam
• You will join the Drug Discovery Data Platform leadership team, delivering data and digital capabilities across Target Identification and discovery through design, make, test and analyse.

This is a hands-on technical leadership role. You will lead multidisciplinary teams delivering trusted data products, analytical products and AI-enabled solutions across the full lifecycle—from discovery and architecture through engineering, deployment, adoption, monitoring and continuous improvement.

Your work will include:

Building discoverable, interoperable and trusted data products for scientific and business users.

Developing AI solutions such as predictive analytics, intelligent search, knowledge assistants, RAG, agentic workflows and automation.

Setting standards for data engineering, APIs, analytics, machine learning and generative AI.

Providing hands-on technical leadership through architecture, design reviews, prototypes, code contributions, performance optimisation and production support.

Improving delivery through Agile, DevOps, DataOps , MLOps and LLMOps practices, automation and self-service capabilities.

Working with product managers, architects, data owners, scientists and stakeholders to prioritise opportunities and deliver measurable outcomes.

Coaching senior engineers and technical leads and promoting responsible, well-governed use of AI.

You will primarily work with our AWS data and analytics ecosystem, while collaborating with teams using Azure. Strong engineering judgement and experience selecting appropriate technologies are essential.

Key Responsibilities

Technical Leadership and Architecture

Define technical direction for data products, analytics platforms and AI-enabled capabilities.

Design secure, scalable, resilient and cost-effective architectures for batch, streaming, event-driven, API and AI workloads.

Establish reusable patterns for data contracts, domain-owned data products, semantic models, metadata, lineage and self-service.

Lead decisions across cloud infrastructure, data storage, processing, orchestration, integration, AI services and application development.

Manage technical debt, platform risks, performance, dependencies and cloud costs.

Ensure solutions meet relevant standards for security, privacy, compliance, validation, resilience, auditability and responsible AI.

Hands-on Engineering

Contribute to production-quality code, prototypes and technical investigations, primarily using Python and SQL.

Build and review data pipelines, transformation frameworks, APIs, analytical products and AI application components.

Develop cloud-native solutions using containers, serverless services, managed platforms, event-driven architectures and infrastructure as code.

Establish practices for Git-based development, automated testing, CI/CD, release management, observability and operational support.

Apply data quality checks, monitoring, performance testing and secure software supply-chain practices.

Support incident investigation, root-cause analysis and continuous improvement.

AI and Advanced Analytics

Identify valuable and feasible applications of AI and advanced analytics in drug discovery.

Engineer production AI solutions covering data preparation, model or foundation-model integration, retrieval, tool use, evaluation, deployment, monitoring and cost management.

Move suitable concepts from experimentation into reliable services.

Product and Delivery Leadership

Translate user and business needs into product outcomes, technical options, delivery plans and success measures.

Balance speed with quality, maintainability, security, compliance, resilience and total cost of ownership.

Lead delivery across globally distributed teams, including the UK, Chennai, Barcelona and Guadalajara.

Maintain clear technical designs, data definitions, data contracts, AI documentation, runbooks, support models and ownership.

Promote reuse, interoperability and continuous feedback across products and domains.

What You’ll Bring

Essential Experience

Experience as a Data Engineering Lead, Software Engineering Lead, Platform Engineering Lead, AI Engineering Lead or equivalent.

A strong hands-on background building and operating production data, software or AI products in the cloud.

Strong programming and engineering capability in Python and SQL , including testing, code review, debugging, optimisation and production support.

Experience with data modelling, batch and streaming pipelines, orchestration, data quality, metadata, lineage, data contracts and observability.

Experience with AWS, Azure or comparable cloud platforms, including CI/CD, infrastructure as code, containers and production operations.

Experience partnering with product, architecture, security, governance, data science and business teams to deliver measurable outcomes.

The ability to make pragmatic architectural decisions and communicate complex technical topics clearly.

Experience coaching engineers and raising engineering standards.

Relevant Technical Experience

Our environment includes AWS and Azure services, S3, Redshift, Athena, Aurora, PostgreSQL, Snowflake, Starburst, Glue, Lambda, EMR, Spark, dbt , Power BI, GitHub Enterprise and container platforms. Equivalent technologies are equally relevant.

We are particularly interested in experience with:

Data platforms and processing: Lakehouse or warehouse architectures, Spark, SQL transformation, streaming and workflow orchestration.

Software engineering: Python, APIs, microservices, containers, serverless architectures, automated testing and secure development.

Cloud and platform engineering: Terraform or equivalent, CI/CD, secrets management, observability, resilience, automation and cloud cost optimisation.

Data product engineering: Data contracts, semantic layers, metadata, lineage, catalogues, discoverability, interoperability and self-service.

AI engineering: LLM APIs, vector or hybrid search, RAG, prompt and model management, tool integration, evaluation and monitoring.

Analytics and visualisation: Power BI or equivalent tools and scalable, user-centred insight products.

Engineering operations: Service ownership, incident management, reliability engineering and operational readiness.

Experience with third normal form, dimensional modelling and star schemas is useful, alongside the judgement to select the right modelling approach for each use case.

Desirable Experience

Experience in pharmaceutical, biotechnology, healthcare, clinical research or another regulated industry.

Understanding of drug discovery or scientific data, including Target Identification, design, make, test and analyse.

Experience establishing engineering standards, reference architectures, reusable platforms or internal developer platforms.

Experience with FinOps, platform scalability and sustainable technology delivery.

Experience taking AI or machine learning solutions from experimentation into supported production services.

Experience with data mesh or domain-oriented data products, event-driven architectures, real-time data products, knowledge graphs or graph-based search.

Experience with human-in-the-loop decision support or workflow automation.

A degree or equivalent experience in computer science, engineering, data science or a related discipl