Senior Data Platform Engineer

OVERVIEW

Reporting to the VP, Technical Operations, the Senior Data Platform Engineer is a senior, hands-on individual contributor on our Engineering, Data Platform team. This is a full-time, remote role open to candidates based in the United States or Canada, with no direct reports.

StoreForce helps specialty retailers run their stores, from workforce scheduling to performance analytics, across tens of thousands of locations worldwide. We are building a new data platform on Azure Databricks that turns the operational data behind those stores into governed metrics, embedded analytics, and AI-driven insight for our customers.

As a Senior Data Platform Engineer, you will build, run, and scale that platform: Spark and Delta Lake pipelines from our multi-tenant SQL Server estate, Unity Catalog governance with a shared semantic layer, and the analytics agent on top. You will work in a small platform team, alongside an implementation partner during the initial build, and own the platform’s growth from the first customers to the full base.

WHAT YOU'LL DO

1. Lakehouse Engineering & Data Pipelines

Build and run the lakehouse on Azure Databricks, including change tracking and change data capture (CDC) ingestion from SQL Server through Lakeflow Connect.

Develop Spark and Delta Lake transformations in Lakeflow Spark Declarative Pipelines and jobs, along with orchestration and serving layers.

Implement and maintain the medallion and dimensional data model in Delta Lake.

Tune Spark jobs, Photon, liquid clustering, and storage for performance and cost.

Measure page and report latency against Lakebase and Databricks SQL, adding a columnar serving store only if the numbers require it.

2. Semantic Layer & Data Governance

Maintain the governed semantic layer in Unity Catalog (Metric Views) so that dashboards, embedded analytics, applications, and AI agents consume the same certified metrics.

Enforce per-tenant row and column security at query time.

Implement StoreForce’s retail KPI logic in Metric Views, including fiscal calendars, 53-week comparisons, comp days, and targets.

Reconcile Metric View results to the existing application’s numbers.

3. Platform Scale & Tenant Onboarding

Scale the platform by configuration: onboard tenants and add KPIs by repeating the established pattern.

Map each tenant’s configurable KPI columns to one governed schema.

Extend and fix the pattern when it needs a new code path.

4. Data Quality, Reliability & Cost

Enforce data quality in the pipeline: expectations, data contracts, reconciliation to source, and freshness and volume checks.

Maintain lineage and Lakehouse Monitoring, treating data incidents like production incidents.

Keep the platform reliable in production, favoring serverless and managed services over undifferentiated infrastructure.

Deliver everything as code through CI/CD, with close attention to compute sizing and DBU spend.

5. AI-Powered Analytics

Build and operate the LLM-powered analytics agent that answers natural-language questions against the semantic layer.

Use Databricks AI tooling alongside Anthropic Claude and OpenAI models.

Ensure tenant isolation, runtime guardrails, and an automated evaluation suite that acts as a quality gate.

Use AI-assisted development to ship quickly and effectively.

6. Collaboration & Platform Ownership

Work alongside the implementation partner during the initial build and take ownership of the platform at handover.

Serve as the go-to engineer for data at StoreForce, contributing design decisions backed by evidence.

Keep engineering standards and runbooks current.

Support the teams consuming the platform and triage data incidents.

Other duties as assigned by your manager and senior leadership.

• QUALIFICATIONS
• Competencies Required:

Technical Depth - Hands-on expert across the Databricks and Spark core who writes production code every day

Ownership - Takes end-to-end accountability for the platform, from build through production support

Data Quality Mindset - Treats data correctness and data incidents with the rigor of production software

Evidence-Based Decisions - Measures before deciding and backs design choices with data

Learning Agility - Picks up new tools quickly and embraces AI-assisted development

Collaboration & Communication - Works effectively with partners and consuming teams; explains trade-offs clearly to technical and non-technical audiences

EXPERIENCE / KNOWLEDGE

5+ years building data pipelines or backend systems, including at least 2 years running a modern data stack in production

Hands-on production experience with Databricks: Unity Catalog, Delta Lake, Lakeflow Jobs and Spark Declarative Pipelines (or their predecessors, Workflows and Delta Live Tables), Databricks SQL, and deploying through Databricks Asset Bundles or an equivalent as-code practice

Strong Apache Spark skills: PySpark and Spark SQL, Structured Streaming, reading query plans, diagnosing shuffle and skew, and tuning jobs and compute for performance and cost

Strong SQL (window functions, complex joins, query optimization) and strong Python for data work

Solid data modeling across medallion, dimensional, and incremental patterns, with experience tuning Delta tables (liquid clustering, change data feed, OPTIMIZE and VACUUM strategy)

Experience implementing data quality and observability (expectations, contracts, reconciliation, lineage, monitoring, alerting)

Hands-on experience with modern LLM providers (Anthropic Claude and OpenAI), building AI features against their APIs

Fluent in modern software engineering practices (version control, automated testing, CI/CD) and able to explain everyday trade-offs such as serverless vs. classic compute or streaming vs. batch; infrastructure or DevOps specialization is not required.

PREFERRED QUALIFICATIONS

Databricks Certified Data Engineer Associate or Professional

Experience deploying LLMs against structured data in production on Databricks (Genie, the agent framework, Model Serving for external models, MLflow evaluation), particularly evaluation, guardrails, and CI gating

Experience with multi-tenant SaaS data on Databricks, including per-tenant isolation with Unity Catalog row filters and column masks, and catalog design for many tenants

Experience ingesting from SQL Server and Azure SQL Managed Instance with Change Tracking or CDC over private networking

Experience serving curated data to always-on applications from Lakebase or PostgreSQL

Experience delivering embedded analytics inside a product, not only internal business intelligence

Experience taking over a platform from a consultancy or another team and running it well afterward.

TECHNICAL ENVIRONMENT

Ingestion - Lakeflow Connect with SQL Server change tracking

Storage - Delta Lake on Azure Data Lake Storage Gen2 in StoreForce’s own tenancy, in open formats so the data outlives any engine

Transformation - PySpark, Spark SQL, and Lakeflow Spark Declarative Pipelines

Governance - Unity Catalog for governance, lineage, and the semantic layer (Metric Views)

Orchestration & Compute - Lakeflow Jobs, with serverless compute where it fits

Serving - Databricks SQL for BI and embedded analytics; Lakebase or PostgreSQL as the always-on serving store

Infrastructure & Security - Everything as code with Databricks Asset Bundles and Terraform; private networking with Entra ID throughout

You don’t need experience with every technology: bring hands-on depth across the Databricks and Spark core and be ready to learn the rest. This is a data engineering role, not infrastructure.

KEY PERFORMANCE INDICATORS (KPIs)

Pipeline success rates, data freshness against agreed SLAs, and availability of serving layers

Reconciliation of governed KPIs to the existing application’s numbers, data quality check pass rates, and number and severity of data incidents

Time