[Job- 29791] Senior Data Engineer [Databricks required] (Hybrid 3x/wk)

Senior Data Engineer

Job Purpose

As a Senior Data Engineer , you will serve as a technical leader and mentor within cross-functional project teams, taking ownership of complex data solutions and architectural decisions within your area of expertise. You will be responsible for designing and implementing high-performance data systems, including data pipelines, storage solutions, and processing frameworks, while mentoring junior and middle-level colleagues and ensuring technical excellence through comprehensive code reviews and testing practices.

In this role, you will contribute to data strategy discussions, lead the implementation of critical data workflows, and bridge the gap between technical execution and business objectives. You will also maintain strong client relationships and support pre-sales activities when needed.

Required Technical Stack

Core Tech Stack: SQL | Python | PySpark | Databricks | AWS

SQL – Strong proficiency in writing complex queries, data transformation, optimization, and working with large datasets

Python – Strong experience in Python for data engineering, automation, data processing, and pipeline development

PySpark – Hands-on experience developing and optimizing distributed data processing workflows using PySpark

Databricks – Strong hands-on experience with Databricks for data engineering, data processing, pipeline development, and analytics

AWS – Hands-on experience designing, implementing, and supporting cloud-based data solutions using AWS services

Key Accountabilities

Technical Leadership & Engineering Excellence

Lead the design and implementation of features, including complex data processing workflows and pipelines, with high attention to detail and quality standards

Design, develop, and optimize scalable data solutions using SQL, Python, PySpark, Databricks, and AWS

Contribute to architecture decisions within project scope and provide technical input for broader data strategy discussions

Establish and maintain engineering standards, best practices, and comprehensive testing strategies within development teams

Conduct thorough code reviews and drive adoption of a strong peer review culture for continuous improvement

Lead troubleshooting of complex technical issues and provide innovative solutions to challenging data engineering problems

Drive performance optimization initiatives for data systems and ensure scalability and reliability in technical implementations

Evaluate data processing frameworks, cloud technologies, and emerging tools for potential adoption within projects

Lead proof-of-concept development and technical risk assessments for new data initiatives

Ensure data solutions are secure, maintainable, reliable, and aligned with business and technical requirements

Team Development & Mentorship

Mentor and develop junior and middle-level colleagues across different technical areas and specializations

Provide technical guidance, knowledge sharing, and support for the career progression of team members

Support technical hiring processes through candidate evaluation, interviewing, and technical assessments

Contribute performance evaluation input and provide constructive feedback for team members

Develop and deliver technical training sessions to elevate team capabilities and foster a culture of continuous learning

Lead by example in adopting engineering best practices, including test-driven development and automated testing approaches

Support team collaboration and knowledge transfer across different technical domains and projects

Share expertise in SQL, Python, PySpark, Databricks, AWS , and other relevant data engineering technologies

Project Execution & Delivery

Take ownership of complex technical tasks and ensure timely, high-quality delivery within project timelines

Design and implement reliable and scalable data pipelines and processing workflows

Provide accurate technical estimations and planning input for development tasks and project milestones

Coordinate technical dependencies and collaborate effectively across different organizational units

Contribute to Agile development practices and ensure technical considerations are represented in sprint planning

Support release management activities and participate in deployment processes with comprehensive testing and validation

Balance technical debt management with feature delivery to maintain sustainable development practices

Monitor and optimize data workflows to ensure performance, scalability, reliability, and maintainability

Client & Stakeholder Engagement

Participate in client interactions and technical discussions to understand requirements and provide appropriate data engineering solutions

Contribute to technical documentation, solution design, and clear communication of complex technical concepts to stakeholders

Support pre-sales activities through technical expertise, solution demonstrations, and client consultations when needed

Assist in translating business requirements into technical specifications and implementation approaches

Provide technical input on project feasibility, resource requirements, and timeline estimations for stakeholder planning

Explain technical solutions and recommendations clearly to both technical and non-technical stakeholders

Maintain professional relationships with clients and contribute to long-term client satisfaction through technical excellence

Business Adaptability & Professional Growth

Demonstrate Technical Leadership: Lead technical initiatives with confidence, make informed decisions, and take ownership of complex data engineering challenges while mentoring others

Drive Adaptability & Continuous Growth: Execute seamless transitions between different projects, technologies, and client requirements while continuously upskilling in emerging data engineering technologies and methodologies

Execute Quality-Focused Development: Apply analytical thinking with attention to detail, prioritize security and maintainability, and ensure comprehensive testing coverage in all deliverables

Practice Effective Communication: Communicate complex technical concepts clearly to various stakeholders, collaborate effectively across teams, and maintain high ethical standards with transparency

Stay Current with Technology: Continuously evaluate advancements in data engineering, cloud computing, distributed processing, and analytics technologies to identify opportunities for improvement

Promote Engineering Excellence: Advocate for scalable, reliable, and maintainable data solutions while continuously improving development and delivery practices

Core Technology Requirements

SQL | Python | PySpark | Databricks | AWS

Strong hands-on experience across the core technology stack is expected, particularly in building and optimizing scalable data pipelines, distributed data processing solutions, and cloud-based data platforms.