Staff Data Engineer

Entity:

• Technology
• Job Family Group:

• IT&S Group
• Job Description:

Staff Data Engineer

Individual Contributor, Grade G

About the role

The Staff Data Engineer plays a critical role in shaping, designing and delivering data and AI solutions that translate complex business needs into scalable, secure and reusable capabilities. Operating at staff level, this role provides senior technical leadership across data products, data platforms and AI-enabled solutions, with clear accountability for end-to-end system design, from high-level architecture through to low-level implementation design.

This is a hands-on engineering leadership role. The postholder independently identifies problems to solve, leads the technical design of solutions and acts as technical lead across one large-scale or multiple medium-scale delivery workstreams. A strong command of modern engineering patterns and practices is essential, as this role helps set the engineering standard for others to follow.

The role carries clear technical leadership accountability without line management responsibility. It guides delivery teams, influences internal and third-party engineering outcomes, and ensures solutions are well designed, maintainable, secure and fit for enterprise use. The postholder is also expected to mentor others and actively contribute to engineering excellence across the wider team.

You will work with

This role sits within the Finance Technology team and works closely with data engineers, data analysts, AI and software engineers, product managers, architects, cyber, platform and business stakeholder teams. It also partners with central technology and architecture teams to ensure solutions are designed for reuse, interoperability and future scalability, rather than being developed in isolation.

You will deliver

End-to-end system design: Translate high-level architectural direction into detailed, buildable solution designs. This includes decomposing architectural patterns, such as event-driven ingestion, medallion lakehouse and RAG pipelines, into concrete engineering specifications including table schemas, transformation logic, interface contracts, retry and error-handling strategies, SLA thresholds and observability requirements. Produce HLD artefacts that communicate solution intent and platform choices, alongside LLD artefacts that delivery engineers can implement without ambiguity. Design outputs should be reviewable through architecture governance and usable as the baseline for technical assurance.

Strong engineering practices and patterns: Set and reinforce standards for software design, code quality, testing, CI/CD, observability and documentation. Advocate for and apply established engineering patterns, including event-driven architecture, microservices, pipeline orchestration, RAG and layered data architecture, using sound judgement on when each pattern is appropriate. Apply these patterns effectively across data-intensive pipelines, products and applications.

Scalable data infrastructure and product delivery: Design, build and maintain reliable data pipelines, data products and integration layers that move, process and serve data across the enterprise at scale.

AI-enabled solution engineering: Integrate AI and machine learning capabilities into production-grade applications, including LLM-based solutions, retrieval-augmented generation, AI agents and unstructured data processing. Evaluate AI tooling pragmatically, prioritising measurable business value and sustainable delivery over experimentation for its own sake.

Technology selection, reuse and scalable solution design: Exercise strong technical judgement in determining when to build custom capability, configure or extend existing platforms, adopt SaaS solutions, or use low-code/no-code approaches. Ensure decisions consider both the immediate requirement and broader enterprise value, balancing delivery speed, maintainability, cost and scalability. Identify opportunities to reuse existing capabilities, create common patterns and shared services, adopt platform-based solutions and actively reduce unnecessary duplication and technical debt.

Technical governance and design assurance: Work with architecture, cyber, data governance and platform teams to ensure solutions meet enterprise standards for security, resilience, compliance, performance and operational readiness.

Engineering leadership across internal and third-party teams: Provide technical direction and engineering leadership across internal teams, delivery partners and third-party suppliers. Set clear engineering expectations, challenge technical decisions where appropriate, and ensure delivered solutions meet required standards for quality, architecture, security, maintainability and long-term ownership.

Operational excellence and lifecycle ownership: Design for supportability, observability, reliability and cost effectiveness throughout the solution lifecycle. Define service reliability expectations, SLA requirements and appropriate site-reliability engineering practices for systems owned or technically led by the role.

Hands-on delivery and prototyping: Translate ambiguous problems into practical technology options, solution designs and working prototypes. Use modern engineering and AI-assisted development tools to accelerate design and

build activities, test assumptions and validate ideas rapidly with users before committing to larger-scale implementation.

Technical mentorship and engineering excellence: Mentor engineers across the team, contribute to data engineering learning and development, and actively participate in the evolution of engineering communities, standards, reusable patterns and practices.

What you will need to be successful

Essential

Bachelor’s degree in Computer Science or a related field.

8+ years of hands-on experience designing, building, productionising and maintaining reliable, scalable data infrastructure and data products in complex environments.

Demonstrable experience producing HLD and LLD artefacts for data or technology solutions, including solution architecture diagrams, data-flow designs, component specifications, API contracts, data models and operational runbooks.

Ability to take architectural decisions and decompose them into engineering specifications that developers can build from without further clarification.

Strong command of software engineering best practices, including technical design review, unit testing, CI/CD pipeline development, monitoring and alerting, code review and documentation.

Development experience in one or more object-oriented programming languages, with Python preferred. Scala, Java or C# are also applicable.

Expert SQL knowledge and experience working with large-scale distributed data systems.

Experience integrating AI and machine learning capabilities into production systems, including LLMs through APIs, RAG patterns, vector search, embeddings or AI agents.

Deep knowledge and hands-on experience across the full data lifecycle.

Strong stakeholder management skills and the ability to lead through technical influence across internal teams and external partners.

A continuous learning and improvement mindset.

Desired

Experience with AI/ML frameworks such as LangChain, LlamaIndex, Hugging Face, Weaviate or similar ecosystems.

Familiarity with agentic frameworks such as CrewAI, AutoGen or similar LLM orchestration toolkits.

Experience with big data technologies such as Spark, Hadoop or Hive.

Familiarity with cloud data platforms such as AWS, Azure or GCP, and infrastructure-as-code practices.

Experience with data governance tools such as Alation or Collibra, and enterprise data management standards.

No prior experience in the energy industry is required.

Why join bp

Our purpose is to deliver energy to the world, today and tomorrow. For over 100 years, bp has focused on discovering, developing, and producing oil and gas in the nations where we operate. We are o