Senior Data Engineer

We are looking for a highly skilled Senior Data Engineer to contribute to the engineering, modernization, and continuous evolution of data processing platform on Databricks on AWS. While supporting the transition from the legacy Cloudera Hadoop platform to Databricks on AWS, this role will continue to play a key part in enhancing performance, simplifying pipelines, and delivering new capabilities on the Databricks platform over the long term.

The ideal candidate is a strong hands‑on Spark engineer with solid design experience, capable of contributing to architectural decisions while leading complex implementation and optimization efforts.

Responsibilities:

1. Platform Engineering & Modernization

Refactor and modernize existing Spark pipelines to Databricks native architectures

Eliminate legacy Hadoop dependencies and adopt cloud native AWS patterns

Enhance and extend existing processing logic using optimized Spark (ScalaSpark) on Databricks

2. Databricks Native Development

Build and optimize solutions using Databricks features, including Delta Lake, Databricks Workflows for orchestration and Auto scaling and job clusters

3. Design & Solution Engineering

Contribute to low and mid level architecture and design

Translate high level architecture into detailed technical designs

Define data models, pipeline patterns, and reusable components

Ensure solutions are scalable, maintainable, and production ready

4. Performance Optimization & Simplification

Analyze, improve Spark job performance and simplify complex or over engineered pipelines into standardized, efficient patterns

5. Engineering Standards & Best Practices

Follow and contribute to Databricks and Spark engineering standards

Write clean, modular, and testable code

Contribute to shared frameworks, reusable libraries, and quality standards

6. Collaboration & Stakeholder Engagement

Work closely with senior architects, platform teams, and DevOps engineers

Provide technical inputs, troubleshooting support, and implementation guidance

Participate in design discussions and technical decision making

7. Testing & Quality Assurance

Develop unit, integration, and data validation tests

Support production releases and post deployment validation

Qualifications:

Core Technical Skills:

8+ years in data engineering or distributed systems

Strong expertise in Apache Spark (ScalaSpark/ JavaSpark / PySpark), Databricks on AWS, and Delta Lake

Experience on SQL

Experience with AWS services and large‑scale distributed data processing

Modernization & Optimization Experience

Experience modernizing or refactoring legacy data platforms into cloud‑based architectures

Strong background in Spark performance tuning and large‑scale batch optimization

Design Capability

Ability to translate architecture into implementable designs

Understanding of data modeling and pipeline orchestration patterns

Behavioral Competencies

Strong problem‑solving mindset for complex distributed systems

Comfortable working in time‑bound, high‑impact environments

Proactive, accountable, and collaborative

Clear communication skills across global teams

Education:

Bachelor’s degree/University degree and equivalent experience

------------------------------------------------------

Job Family Group:

Technology ------------------------------------------------------

Job Family:

Applications Development ------------------------------------------------------

Time Type:

Full time ------------------------------------------------------

Most Relevant Skills

Please see the requirements listed above. ------------------------------------------------------

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter. ------------------------------------------------------

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi .

View Citi’s EEO Policy Statement and the Know Your Rights poster.