Data Engineer

Responsibilities

Role demands a highly skilled Data Engineer to design, build, and optimize scalable data pipelines and data platforms. The ideal candidate will have strong expertise in data modeling, cloud-based data architectures, and modern data engineering tools across Azure, Snowflake, and Databricks environments.

Key Responsibilities

Data Engineering & Pipeline Development

Design, develop, and maintain robust ETL/ELT pipelines using Databricks, PySpark, and Azure Data Factory (ADF).

Build scalable and efficient data ingestion frameworks for structured and unstructured data.

Optimize pipeline performance through performance tuning and orchestration best practices.

Data Modeling & Management

Develop and maintain data models using modern tools (DBT preferred).

Implement Master Data Management (MDM) solutions to ensure data consistency and integrity.

Design scalable and efficient Snowflake schemas (star/snowflake schema, dimensional modeling).

Database & Query Optimization

Write and optimize advanced SQL queries across Snowflake, Azure SQL, and Synapse.

Develop and manage stored procedures and database objects.

Ensure efficient data retrieval through indexing, partitioning, and query optimization.

Cloud & Platform Integration

Work with Azure data services including:

o Azure Data Factory (ADF)

o Azure Data Lake Storage (ADLS)

o Azure Synapse Analytics

o Azure SQL Database

Integrate and maintain Snowflake with Azure ecosystem.

Python Development

Develop data transformation and automation scripts using Python libraries:

o pandas

o pyodbc

o SQLAlchemy

Build reusable components for data processing and validation.

Data Quality, Validation & Monitoring

Implement data validation rules, quality checks, and anomaly detection frameworks.

Perform root cause analysis for data inconsistencies.

Develop dashboards or tools for data quality monitoring.

Collaboration & DevOps

Use GitHub for version control, branching strategies, and code reviews.

Manage workload scheduling and dependency management for pipelines.

Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders.

Required Skills & Qualifications

Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field.

Strong experience in data engineering and data platform development.

Technical requirements

Technical Skills

Expertise in DBT (preferred) for data modeling.

Strong SQL skills with hands-on experience in:

o Snowflake

o Azure SQL

o Stored procedures

Proficiency in Python for data engineering workflows.

Hands-on experience with:

o Databricks & PySpark

o Azure Data Services (ADF, ADLS, Synapse)

Strong knowledge of Snowflake architecture and schema design.

Experience with data validation, quality frameworks, and analysis tools.

Familiarity with GitHub and CI/CD practices.

Additional responsibilities

Preferred Qualifications

Experience implementing data governance and MDM solutions.

Knowledge of performance tuning in distributed processing systems.

Familiarity with workflow orchestration tools.

Experience in agile environments (SCRUM/Kanban).

Education

Master Of Engineering,Master Of Technology,Bachelor of Engineering,Bachelor Of Technology