Senior Data Engineer - Databricks

Mobile Wave Solutions is a professional services company specializing in software development as a service. With a team of over 120 engineers, we deliver scalable, high-quality software that empowers our global clients to innovate and grow. We value collaboration, technical excellence, and a pragmatic approach to solving complex problems.

About the Role

We are seeking a seasoned, hands-on Senior Data Engineer to serve as the subject matter expert and technical anchor for our next-generation Databricks Lakehouse platform. In this role, you will design and build scalable ingestion pipelines, establish production best practices, set up CI/CD standards, and elevate the team’s technical capabilities.

Our current environment is hosted on AWS , but we welcome strong Databricks experts with background in Azure or other major cloud providers, as your core Lakehouse mastery is what matters most. We need a confident, self-directed engineer with deep production experience across Databricks and Apache Spark who can tackle unstructured data sources, optimize distributed workloads, and drive decisive architectural standards in a fast-paced rebuild environment.

Key Responsibilities

Platform Standards & Best Practices: Lead the architecture, design, and implementation of robust Lakehouse ingestion and transformation pipelines using Databricks and PySpark. Establish standards that level up the broader team's engineering maturity.

CI/CD & Operational Excellence: Set up automated delivery pipelines using Databricks Asset Bundles (DABs) and modern orchestration (Airflow or similar) to ensure seamless, reliable deployments.

Complex Data Integration: Architect pipelines to ingest, process, and structure both structured (OLTP) and unstructured/semi-structured data (APIs, logs, cloud storage streams) into high-quality analytical datasets.

Delta Lake & Governance: Enforce Delta Lake principles (CDC, schema evolution, optimization) and integrate data quality frameworks (e.g., Great Expectations) and Unity Catalog governance into daily development.

AI Agent Direction: Direct and evaluate AI agents for rapid code generation, ensuring generated code meets strict quality, security, and performance standards.

Technical Mentorship & Execution: Clearly articulate complex technical trade-offs to stakeholders, mentor peers, and confidently drive pragmatic solutions without needing step-by-step guidance.

Qualifications

Deep Databricks & Spark Expertise: 5+ years of dedicated data engineering experience with proven, large-scale production exposure to Databricks, PySpark performance tuning, and distributed processing architectures.

Cloud Ecosystem Experience: Strong proficiency in major cloud environments (AWS primary, but candidate expertise in Azure or GCP with Databricks is fully welcomed and transferable).

Advanced CI/CD Knowledge: Hands-on experience implementing enterprise CI/CD for Databricks workflows, specifically utilizing Databricks Asset Bundles (DABs) or equivalent infrastructure-as-code patterns.

Unstructured & Diverse Data Processing: Demonstrated history of turning raw, unstructured, or semi-structured data sources into structured, production-ready Lakehouse layers.

Production Discipline: Expert SQL optimization, deep Python proficiency, and practical experience with workflow orchestrators (Airflow) and quality/governance tooling (Unity Catalog, Great Expectations).

Senior Communication: Concise, direct communicator capable of explaining technical concepts clearly and mentoring team members on best practices.

Pragmatic & Autonomous: Comfortable tackling ambiguous, legacy datasets in a fast-paced rebuild environment; excels at guiding AI-assisted coding tools while enforcing high stability standards.

Our Benefits

Remote Office – Flexible hybrid form of working

Parking Space – We provide free parking spots

Fun Office Space – We offer a game zone and a relaxation area

Health Insurance – Additional private health insurance, including a dental care plan

Personal Development – Company-sponsored training budget to further develop your skills

Employee Referral Program – Receive a bonus for referring a friend

Holidays – Enjoy an extra 5 days after your 1st and 5th year

Social Events – We love to celebrate our success together

Family Insurance – Add insurance to a family member

Offering sports cards – 100% sponsored by the company