Data Architect - Data Engineering 4D

Data Architect

• Ready to turn bold ideas into real-world impact?
• At Genpact, we don’t just adapt to change, we lead it. AI and digital innovation are transforming the way businesses work, and we’re at the forefront of it. Genpact’s AI Gigafactory , our industry-first accelerator, exemplifies how we scale advanced technology solutions to help global enterprises work smarter, grow faster, and transform at scale. Whether tackling complex challenges through large-scale models or agentic AI , our breakthrough solutions tackle companies’ most complex challenges.

If you thrive in a fast-moving, innovation-driven environment, love building and deploying cutting-edge AI solutions, and want to push the boundaries of what’s possible, this is your moment.

• Genpact (NYSE: G) is an agentic and advanced technology solutions company. We leverage process intelligence and artificial intelligence to deliver measurable outcomes. With a strong partner ecosystem and decades of client trust, we provide innovative solutions that transform how businesses run. Powered by a team with an active learning mindset and client centricity at its core, we deliver lasting value for the world’s leading enterprises.
• Get to know us at genpact.com and on LinkedIn , YouTube , X , and Facebook .

Job Description

Senior DataOps Engineer

Team: Global Digital Operations — Data Platform

About the Role

Data Platform Services is building a governed, metadata-driven data platform on Azure Databricks / Unity Catalog — spanning source ingestion, a source-faithful central repository, long-term archival, and controlled distribution to multiple downstream systems. We're looking for a DataOps Engineer who treats pipeline and access-control logic as configuration, not one-off code — someone who has built or maintained systems where a metadata table drives what gets generated (jobs, views, permissions), rather than hand-coding each new dataset or consumer.

You'll work across ingestion (landing → central → archive), retention/partitioning strategy, and the routing-rules engine that determines how data is split, filtered, or masked across four major downstream systems.

What You'll Do

Build and operate Auto Loader–based ingestion pipelines (batch and incremental) from Unity Catalog volumes into managed Delta tables, using MERGE patterns for upsert/dedup logic

Design and maintain metadata-driven pipeline generation — where dataset onboarding (new source, new schema, new retention rule) is a config change against a rules table, not new pipeline code per dataset

Implement and maintain partition-based retention (year partitioning; rolling window of current + N prior years) and the archive extraction process for datasets requiring longer history

Build and maintain a dataset routing-rules engine: a governed metadata table (splittable, target_systems[], row_filter_expr, column_mask_expr) that programmatically generates consumption-layer views per downstream system

Own Unity Catalog governance: catalog/schema/storage-location design, access grants, lineage, and audit trail across landing, central, archive, and consumption layers

Manage Databricks Asset Bundles (DAB) or equivalent IaC for repeatable environment and job deployment across dev/test/prod

Monitor pipeline health, checkpoint integrity, schema drift/evolution, and job SLAs; build alerting for ingestion failures and data-freshness breaches

Collaborate with upstream source-system owners to define data contracts (e.g. ensuring _source_changed_at semantics are honored) and with downstream teams to translate their access/filter requirements into routing rules

Document pipeline architecture, retention policy, and routing logic for audit and onboarding purposes

Required Skills & Experience

Core platform

Azure Databricks and Unity Catalog (catalogs, schemas, managed/external volumes, managed/external tables, storage-location hierarchy, grants)

Strong PySpark and SQL, including Structured Streaming / Auto Loader (cloudFiles, foreachBatch, trigger(availableNow=True)), MERGE INTO, and Delta Lake internals (transaction log, OPTIMIZE, VACUUM, ZORDER)

Practical experience with partitioning strategy and retention management (partition-drop vs. rolling delete tradeoffs) at production data volumes

Metadata-driven engineering (critical)

Demonstrable background designing or maintaining metadata/config-driven code generation — e.g. a rules or registry table that drives dynamically generated SQL, views, jobs, or DDL, rather than hardcoded per-object pipelines

Comfort with templated/parameterized pipeline patterns (e.g. DAB with variable substitution, dbt-style codegen, or custom Python/SQL generation from a config table)

Understanding of how to keep generated artifacts (views, jobs) in sync with a source-of-truth ruleset, including versioning and change auditability

Platform engineering / DataOps

Azure fundamentals: ADLS Gen2, Azure Data Factory or equivalent orchestration, networking/private endpoints, cost management basics

CI/CD for data pipelines (Azure DevOps or GitHub Actions), ideally with Databricks Asset Bundles or Terraform for Databricks

Git-based workflow discipline; comfortable owning pipeline code as a shared, reviewed codebase rather than notebook-only development

Monitoring/observability for data pipelines (job alerting, data quality checks, schema drift detection)

Governance & collaboration

Working knowledge of data governance concepts: access control models, data lineage, retention/compliance-driven deletion, PII masking

Experience working with multiple downstream consumers with differing data needs — building filtered/masked views rather than granting broad table access

Clear written documentation habits; able to explain pipeline and governance logic to both technical and non-technical stakeholders

Nice to Have

Experience with Delta Live Tables or Databricks Streaming Tables (declarative SQL ingestion)

SnowFlake, Iceberg

Exposure to multi-tenant data platforms serving several internal business-unit consumers with differing retention/access needs

What Success Looks Like in the First 6 Months

New source onboarding (landing → central) reduced to a config-driven task, not custom pipeline development

Partition-based retention live and operating automatically across central repository schemas

Routing-rules table operational, with at least the initial set of downstream consumption views generated from it rather than hand-written

Clear, current documentation of the ingestion → central → archive → consumption architecture

Qualifications

Bachelors - Business Analytics, Bachelors - Computer Science, Bachelors - Statistics, Masters - Data Science

Certifications

Certified Data Management Professional - RiversandRiversand, Databricks Certified Associate Developer for Apache Spark 3.0 - Databricks AcademyDatabricks Academy, Databricks Certified Associate Developer for Apache Spark - Databricks AcademyDatabricks Academy, Microsoft Certified: Azure Data Engineer Associate - MicrosoftMicrosoft, Oracle Database 12c Certified Implementation Specialist - OracleOracle

Required Skills

Agile Methodology, Change Management, Consumer Packaged Goods (CPG), Databricks Platform, Data Engineering, Snowflake GenAI

Language

English

Language Proficiency -

Proficient - C2

Additional Job Location -

Job Type

Regular

Master Skill List -

Data Engineering

Remote Type -

Office

Work Shift -

Flex Time (India)

• Why join Genpact?
• •Lead AI-powered transformation – Drive innovation and solve real-world business challenges that matter
• •Make an impact – Help global enterprises solve business challenges that matter
• •Accelerate your career – Gain hands-on experience, mentorship, and world-class learning opportunities to stay ahead
• •Work with the best – Join 140,000+ bold thinkers and problem-solvers who push boundaries every day
• •Thrive in a values-driven culture – Our courage, cu