Data Engineer

Job Description – Data Engineer (Databricks on AWS)

Experience: 1–5 Years

Technology Stack: Databricks, Apache Spark, PySpark, SQL, AWS, Delta Lake

Employment Type: Full-time

Job Summary

We are looking for a skilled Data Engineer with 1–5 years of experience in data engineering to design, develop, and maintain scalable data pipelines using Databricks on AWS. The ideal candidate should have hands-on experience with PySpark, SQL, data processing, and ETL/ELT development, along with a good understanding of cloud-based data engineering concepts.

The candidate will work closely with data architects, analysts, and other engineering teams to build reliable, high-performance data solutions that support business and analytical requirements.

Key Responsibilities

• •Design, develop, test, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
• •Develop data transformation workflows using Apache Spark and Delta Lake.
• •Work with structured and semi-structured data from multiple sources, including databases, APIs, and files such as CSV, JSON, and Parquet.
• •Build and manage Databricks Jobs and Workflows for automated data processing and scheduling.
• •Implement incremental data processing, data validation, error handling, and data quality checks.
• •Work with AWS services such as Amazon S3, IAM, and other relevant cloud services to build and support data pipelines.
• •Optimize Spark jobs and SQL queries for performance, scalability, and cost efficiency.
• •Implement appropriate data partitioning, file management, and Delta Lake optimization techniques.
• •Work with Unity Catalog and implement data access controls and governance practices.
• •Troubleshoot pipeline failures, investigate data discrepancies, and resolve production issues.
• •Follow coding standards, version control practices, and development, testing, and deployment processes.
• •Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.
• •Maintain technical documentation for data pipelines, transformations, and operational procedures.

Required Technical Skills

1. Databricks and Apache Spark

• •Hands-on experience with Databricks on AWS.
• •Good understanding of Apache Spark architecture and distributed data processing.
• •Experience developing data pipelines using PySpark and Spark SQL.
• •Familiarity with Databricks Jobs, Workflows, notebooks, and job clusters.
• •Understanding of Delta Lake, including MERGE, schema evolution, and incremental data processing.

2. SQL and Data Engineering

• •Strong SQL skills, including joins, aggregations, CTEs, window functions, and complex transformations.
• •Experience with relational databases and data integration.
• •Understanding of ETL/ELT concepts, data modeling, and data warehouse fundamentals.
• •Ability to troubleshoot data quality issues and validate pipeline outputs.

3. AWS Cloud

• •Working knowledge of Amazon S3 for data storage.
• •Understanding of AWS IAM roles, policies, and permissions.
• •Familiarity with AWS cloud architecture and secure access to cloud resources.
• •Exposure to AWS Glue, Lambda, or other AWS data services is an advantage.

4. Delta Lake and Data Governance

• •Understanding of Delta tables, ACID transactions, and schema enforcement.
• •Familiarity with Unity Catalog, catalogs, schemas, tables, and access permissions.
• •Awareness of data lineage, governance, and secure data access practices.

5. Development and Deployment

• •Experience with Git and version control.
• •Familiarity with Python programming, debugging, and exception handling.
• •Exposure to CI/CD pipelines and automated deployments is an advantage.
• •Ability to write reusable, maintainable, and well-documented code.

Good to Have

• •Experience with Change Data Feed (CDF), incremental data loads, and Slowly Changing Dimensions (SCD).
• •Exposure to Databricks Auto Loader, Lakeflow, streaming data pipelines, and Databricks Genie.
• •Familiarity with Databricks AI/BI dashboards, Genie spaces, and natural-language analytics.
• •Experience with business intelligence and data visualization tools such as Power BI, Tableau, or similar analytical tools.
• •Exposure to Azure Databricks, Azure Synapse Analytics, or other Azure data services.
• •Familiarity with Google Cloud Platform (GCP) and its data engineering services, such as BigQuery, Dataflow, or Cloud Storage.
• •Familiarity with data integration tools such as Fivetran.
• •Experience with orchestration tools such as Apache Airflow or AWS Step Functions.
• •Familiarity with Terraform or other infrastructure-as-code tools.
• •Understanding of performance optimization techniques, including partitioning, caching, shuffle optimization, and Adaptive Query Execution (AQE).
• •Databricks Data Engineer Associate or Professional certification.

Qualifications

• •1–5 years of relevant experience in data engineering, ETL development, or big data technologies.
• •Strong analytical and problem-solving skills.
• •Good communication skills and the ability to work effectively in a collaborative environment.
• •Willingness to learn new technologies and contribute to continuous improvement.

Expected Candidate Profile

The candidate should be able to independently develop and troubleshoot data pipelines appropriate to their experience level, write efficient PySpark and SQL code, understand distributed data processing fundamentals, and contribute to building scalable and reliable data engineering solutions on Databricks running on AWS.