Databricks Technical Lead

Databricks Technical Lead

Req number:

R8395

Employment type:

Full time

Worksite flexibility:

Hybrid

Who we are

CAI is a global services firm with over 9,000 associates worldwide and a yearly revenue of $1.3 billion+. We have over 40 years of excellence in uniting talent and technology to power the possible for our clients, colleagues, and communities. As a privately held company, we have the freedom and focus to do what is right—whatever it takes. Our tailor-made solutions create lasting results across the public and commercial sectors, and we are trailblazers in bringing neurodiversity to the enterprise.

Job Summary

We are looking for a motivated Databricks Technical Lead ready to take us to the next level! If you have experience with AWS, Databricks, and AI data product management, and are looking for your next career move, apply now.

Job Description

We are looking for a Databricks Technical Lead to lead the strategy, governance, productization, and operational excellence of AI-ready data products that enable scalable and reliable AI solutions across the enterprise. This position will be full-time and hybrid in Bengaluru.

What You’ll Do

Own the AI data product roadmap and translate business priorities into a sequenced backlog of reusable data products with defined ownership, service levels, and lifecycle controls

Partner with dashboard, analytics, and AI delivery teams to identify reporting assets that should evolve into governed AI-ready data products

Establish data product charters defining purpose, consumers, source ownership, data contracts, access models, and retention requirements

Implement table-, column-, and row-level access standards, classification, audit trails, and data handling controls using Databricks Unity Catalog and AWS services

Lead the design and operation of incremental, observable, and cost-efficient data ingestion and curation pipelines across enterprise data sources

Create and maintain versioned, reproducible training datasets, evaluation datasets, inference inputs, feature tables, and monitoring datasets with clear metadata and documentation

Define and implement AI-specific data quality standards including completeness, consistency, timeliness, uniqueness, referential integrity, distribution stability, and drift sensitivity

Establish automated quality controls and enforce readiness gates that prevent model training, deployment, or production promotion when critical data quality requirements are not met

Develop operational runbooks for recovery, backfills, reprocessing, incident response, and escalation management

Participate in operational reviews and drive root-cause analysis and corrective actions to improve platform reliability

Optimize storage, compute utilization, partitioning, scheduling, and overall platform cost efficiency

Approve data readiness gates for model training, evaluation, deployment, and significant production inference input changes

Escalate data ownership, quality, access, and cost issues that introduce risk or impede business outcomes

Drive dashboard-to-data-product transformations and establish governed, reusable data products that support critical AI use cases

Deliver quality and cost scorecards that provide transparency into data health, reliability, usage, and platform cost drivers

What You’ll Need

Required:

Bachelor’s degree or higher in Computer Science, Data Science, AI/ML, Applied Mathematics, Engineering, or a related field

8+ years of experience in data engineering, data management, AI data foundations, or related technology leadership roles

Demonstrated ownership of data products or mission-critical data pipelines delivering measurable business impact

Hands-on experience with AWS, Databricks, Delta Lake, Unity Catalog, data lineage, observability, and production support

Experience defining and managing data contracts, quality gates, service levels, and data access models

Strong stakeholder management skills with experience partnering across business leaders, product owners, data and AI engineering teams, security, and platform organizations

Ability to operate at a senior level, challenge incomplete requirements, evaluate trade-offs, and drive decisions in ambiguous environments

Strong communication skills with the ability to simplify complex technical concepts for business stakeholders

Proven ownership mindset with accountability for business and technical outcomes

Strong judgment balancing speed, quality, security, cost, and maintainability

Excellent problem-solving skills and ability to adapt in fast-paced, evolving environments

Commitment to data quality, operational excellence, documentation, and continuous improvement

Preferred:

Databricks Data Engineer Professional certification

Databricks Machine Learning Professional certification

AWS Certified Data Analytics certification

AWS Solutions Architect Associate certification

AWS Solutions Architect Professional certification

Data governance, data management, product ownership, or agile delivery certifications

Physical Demands

Ability to safely and successfully perform the essential job functions

Sedentary work that involves sitting or remaining stationary most of the time with occasional need to move around the office to attend meetings, etc.

Ability to conduct repetitive tasks on a computer, utilizing a mouse, keyboard, and monitor

Reasonable accommodation statement

If you require a reasonable accommodation in completing this application, interviewing, completing any pre-employment testing, or otherwise participating in the employment selection process, please direct your inquiries to application.accommodations@cai.io or (888) 824 – 8111.