Senior Data Engineer (Fully In-Office)

Description

JOB DESCRIPTION

JOB TITLE: Senior Data Engineer

DEPARTMENT: 500 – IT

REPORTS TO: Chief of Information Officer

JOB SUMMARY:

The Senior Data Engineer – Databricks is responsible for building and evolving PharmaLink’s enterprise data, analytics, and AI platform. This hands-on role spans the modern Databricks ecosystem, including data engineering, lakehouse architecture, analytics and dashboards, AI, Databricks Apps, and Lakebase. The role requires strong engineering fundamentals, independent problem-solving, technical ownership, and the ability to design, build, and operate reliable, scalable production solutions.

A hands-on builder approach is essential, with a focus on developing solutions rather than simply administering platforms. The ideal candidate can work across data engineering, platform architecture, analytics, AI, and data applications; independently investigate complex problems; and take solutions from initial design through production.

KEY RESPONSIBILITIES:

• •Design, build, and operate enterprise data solutions on Databricks.
• •Develop scalable ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake.
• •Design curated data models and reusable data products for analytics, applications, and AI.
• •Implement Unity Catalog, security, lineage, access controls, and data-governance capabilities.
• •Build and optimize Databricks SQL Warehouses and AI/BI Dashboards.
• •Develop data and AI applications using Databricks Apps.
• •Design and implement appropriate solutions using Databricks Lakebase.
• •Enable AI/ML and Generative AI use cases using Databricks capabilities.
• •Build automated data-quality, reconciliation, anomaly-detection, and monitoring processes.
• •Integrate data from databases, SaaS platforms, APIs, files, and other enterprise systems.
• •Optimize Spark workloads, SQL queries, pipelines, performance, and platform cost.
• •Establish engineering standards for Git, CI/CD, testing, monitoring, and production deployment.
• •Troubleshoot complex production and data issues and perform root-cause analysis.
• •Partner with Data Governance, BI, application, and business teams to deliver trusted data products.

Requirements

QUALIFICATIONS:

• •8+ years of experience in data engineering, data platforms, or related engineering roles.
• •4+ years of hands-on Databricks experience in production environments.
• •Advanced SQL, Python, PySpark, and Apache Spark skills.
• •Strong experience with Delta Lake and Unity Catalog.
• •Strong ETL/ELT, data modeling, and enterprise data architecture experience.
• •Experience with Databricks SQL and AI/BI Dashboards.
• •Familiarity with Databricks capabilities across AI/ML, Generative AI, Databricks Apps, and Lakebase.
• •Strong relational database and data-engineering fundamentals.
• •Experience with Git, CI/CD, automated testing, monitoring, and production support.
• •Experience working with cloud-based enterprise data platforms.
• •Must maintain a positive attitude in a high-performance team environment.
• •Must pass background check and random drug screenings.
• •Must be adaptable and flexible in dealing with a diverse work force

PREFERRED QUALIFICATIONS:

• •Databricks on Microsoft Azure.
• •Azure Data Factory.
• •Microsoft SQL Server.
• •Fivetran or similar CDC/ELT technologies.
• •Power BI.
• •Databricks Workflows.
• •MLflow and AI/model lifecycle capabilities.
• •API and application development experience.
• •Azure DevOps or GitHub-based CI/CD.
• •Databricks professional certification.
• •Experience in healthcare, life sciences, or another regulated industry.

PHYSICAL REQUIREMENTS:

Requires light physical activity performing non-strenuous daily activities of an administrative nature (i.e., sustained phone, headset, and keyboard use) as well as periods of prolonged sitting.