ADF (Azure Data Factory)Databricks+Pyspark

You’ll lead the design and delivery of modern data engineering solutions that turn raw data into trusted, analytics-ready assets. Working at the intersection of orchestration, scalable processing, and cloud-native platforms, you’ll partner closely with product owners, analysts, and engineering teams to build reliable pipelines that power business decisions. This role is ideal for someone who enjoys owning end-to-end delivery—shaping architecture, guiding implementation, and mentoring engineers—while continuously improving performance, quality, and operational excellence. If you’re excited by solving complex data challenges, enabling self-serve analytics, and building a collaborative culture that values craftsmanship and learning, this is the place to make a meaningful impact.

Responsibilities

Key Responsibilities:

Data Engineering & Delivery

Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing

Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability

Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing

Build reusable frameworks, templates, and standards for pipeline development and deployment

Architecture & Performance

Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices

Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing)

Establish monitoring, alerting, and operational runbooks for production pipelines

Leadership & Collaboration

Provide technical leadership, code reviews, and mentoring to ensure high engineering standards

Collaborate with stakeholders to translate business requirements into scalable data solutions

Drive delivery planning, estimation, and risk management for data engineering initiatives

Minimum Qualifications:

BTECH, MTECH, MCA, MSC (or equivalent) in Computer Science, Engineering, or related field

7–9 years of experience in data engineering with strong hands-on delivery ownership

Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns

Strong expertise in Databricks for building scalable data processing solutions

Hands-on proficiency with PySpark for building and optimizing distributed data transformations

Experience building production-grade pipelines with logging, error handling, and operational support readiness

Technical requirements

Technology->Big Data - Data Processing->PySpark

Technology->Cloud Integration->Azure Data Factory (ADF)

Technology->Data Engineering->Databricks

Additional responsibilities

Preferred Qualifications:

Experience designing medallion/layered data architectures and implementing reusable transformation patterns in Databricks

Strong understanding of data modeling concepts and building curated datasets for analytics consumption

Experience implementing CI/CD practices for data pipelines and notebooks, including automated testing and deployment

Proven ability to lead technical discussions, mentor team members, and drive engineering best practices

Experience improving observability (metrics, alerts, dashboards) and reducing pipeline failures through proactive monitoring

Education

MCA,MSc,MTech,Bachelor of Engineering,BTech