Sr Data Engineer (AWS | Multi-Cloud | Databricks)

Job Description – Data Engineer (AWS | Multi-Cloud | Databricks)

Experience: 5+ Years

Role: Senior Data Engineer

Primary Cloud Platform: AWS

Secondary Cloud Platforms: GCP and Microsoft Azure

Core Technology: Databricks, Apache Spark, PySpark, SQL, Data Engineering

Job Overview

We are looking for an experienced Senior Data Engineer with 5+ years of hands-on experience in designing, developing, and implementing scalable data engineering solutions across cloud platforms.

The ideal candidate should have strong expertise in AWS as the primary cloud platform , along with working knowledge and exposure to data engineering services across Google Cloud Platform (GCP) and Microsoft Azure.

The candidate must possess excellent data engineering fundamentals, strong programming and problem-solving skills, and very good hands-on experience with Databricks and Apache Spark .

The role involves building modern data pipelines, implementing data lakes and lakehouse architectures, optimizing large-scale data processing workloads, and developing reliable, scalable, and cost-efficient data platforms.

Key Responsibilities

• •Design, develop, and maintain scalable end-to-end data pipelines using AWS and Databricks.
• •Build and optimize batch and real-time data processing solutions using Apache Spark, PySpark, and SQL.
• •Design and implement modern data lake and lakehouse architectures using S3, Delta Lake, Apache Iceberg, and Databricks.
• •Work with AWS data engineering services for data ingestion, transformation, orchestration, storage, cataloging, and analytics.
• •Develop ETL/ELT workflows integrating structured, semi-structured, and unstructured data from multiple sources.
• •Implement data transformation, data quality, validation, and reconciliation frameworks.
• •Optimize Spark jobs, SQL queries, cluster configurations, and distributed processing workloads.
• •Implement workflow orchestration using Airflow, AWS Step Functions, Databricks Workflows, or equivalent services.
• •Work with streaming technologies and services for real-time data ingestion and processing.
• •Develop reusable frameworks, automation scripts, and standardized data engineering components.
• •Implement security, access control, encryption, monitoring, logging, and governance across data platforms.
• •Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable technical solutions.
• •Participate in architecture discussions, technical design, code reviews, and performance optimization initiatives.

Required Experience & Qualifications

• •5+ years of experience in Data Engineering.
• •Strong hands-on experience with AWS data engineering services.
• •Very good practical experience with Databricks and Apache Spark.
• •Strong programming skills in Python, PySpark, and SQL.
• •Practical exposure to at least one of GCP or Azure, with familiarity with the data engineering ecosystem of both.
• •Experience designing and implementing end-to-end data pipelines.
• •Understanding of modern data lakehouse architecture and distributed data processing.
• •Experience working with large-scale datasets and performance optimization.

Good to Have

• •Experience working on multi-cloud data engineering architectures.
• •Exposure to AWS, Azure, and GCP migrations.
• •Experience with Apache Kafka, Amazon MSK, or other streaming platforms.
• •Knowledge of Apache Iceberg, Delta Lake, and open table formats.
• •Exposure to data governance, lineage, and metadata management.
• •Knowledge of DevOps practices and CI/CD automation.
• •Experience with Infrastructure as Code (Terraform / CloudFormation).
• •Exposure to Generative AI data pipelines and vector databases.

Ideal Candidate Profile

The ideal candidate should be a hands-on Data Engineer who can independently design, develop, troubleshoot, and optimize complex data engineering pipelines.

We are specifically looking for someone who:

• •Has strong practical expertise in AWS data engineering services.
• •Has very good hands-on experience with Databricks and PySpark.
• •Understands data engineering ecosystems across AWS, GCP, and Azure.
• •Has strong fundamentals in distributed data processing, SQL, Python, and data architecture.
• •Can work independently on end-to-end data engineering implementations.
• •Can contribute to technical architecture and engineering best practices.