Senior Data Engineer, Platform

Senior Data Engineer, Platform

Salary range: $180,000 - $220,000 per year

About the Role

Two kinds of data decide whether we can do our job. What a health plan tells us happened to a member, in eligibility files, claims, pharmacy and hospital event feeds. And what our own software records about what we did for them. When a file is late, malformed, or quietly incomplete, care teams work from a stale list. When the two do not line up, nobody can say whether the work is helping.

You will own the infrastructure our data runs on: where and how our jobs run and whether they run efficiently, including compute choices, serverless versus clusters, cost and performance; how work is orchestrated and deployed; how we connect to partners and systems, over SFTP, S3, private networking and partner feeds; and whether all of it is reliable and observable. You will set the technical direction for that platform, make the architecture calls, and build it as software, with tests, version control and infrastructure as code. This is a senior individual contributor role, with a clear path to leading data engineering as the team grows.

This role is complementary to analytics engineering. Analytics owns the questions and the answers, analytics engineering builds the gold tier (curated tables) they work from, and you own the platform underneath it, so those tables arrive complete, on time and at a sensible cost.

We are a small, AI-forward team. The whole team builds with AI coding agents, and part of this platform's job is to make our data usable by AI. We build and evaluate AI systems on real data, where a wrong answer has consequences, so evaluation is part of the work rather than something bolted on at the end.

We want the person who looks at a 384 GB Spark driver and asks why. We are adding markets and data quickly, and we want a platform that grows by adding machines rather than by buying bigger ones. You will know what belongs on a single machine and what does not, and you will keep our pipelines fast, predictable and economical to run as they grow.

Our partners send and expect files on fixed monthly schedules. Eligibility, claims and attribution files come in; engagement, roster and service files go out. Part of the job is making those exchanges boring: replacing recurring manual work with automation, so routine exchanges run themselves and people are pulled in only when something needs judgment. And you will not become the person everything depends on. We treat that as a design flaw, not a badge.

In your first 90 days, you might take on one or two of these:

Take over the performance work already underway on our heaviest batch jobs, carry it from single-machine processing onto distributed compute, and prove the output did not change

Move our legacy Databricks jobs onto our newer platform, with the orchestration, deploys and runbooks to operate them

Connect a new payer data feed end to end, from the first file over SFTP or S3 to members live in our care platform, monitored from day one

Review where each of our scheduled jobs runs, serverless or cluster, and right-size the ones that cost more than they should

You will also be close to the people whose work this data describes: coaches, clinicians and researchers who spend their days with our members. When you cannot tell what a field means, or whether a number is plausible, you can go and ask the person who generated it. Most data engineers never get to do that. Our members are among the most underserved people in American healthcare, and these data problems are not abstract.

About Vanna Health

At Vanna Health, we believe that a sense of belonging and engagement in the community is the foundation for improving overall health and wellbeing for individuals with serious mental illness. We help people living with serious mental illness thrive by strengthening the community ecosystem around them. Through innovative payment models and technology, Vanna Health promotes collaboration across existing behavioral health and primary care providers, hospitals, payers, clubhouses and other community based organizations. Additionally, Vanna Health can deploy our own top quality providers and case managers, as needed, in areas where there may be a shortage of resources.

Vanna Health was founded in 2021 by a team of mission-driven entrepreneurs, working in close partnership with payers, providers, and community based organizations to promote collaboration and improve outcomes while driving down costs. We are committed to the goal of supporting all individuals with culturally competent care regardless of their race, ethnicity, religion, sexual orientation, or gender identity. With modern technology, extensive development opportunities, diverse careers paths, and a culture that values diversity, equity, inclusion, and belonging, we are redefining what it means to work in community health.

Responsibilities

In your first year, most of your time goes to compute, ingestion and orchestration; the AI and governance work grows with the team.

Shape the data platform roadmap with data leadership, own delivery of the work you lead, and keep stakeholders informed of progress, risks and trade-offs

Set technical direction for data engineering: design and code-review standards, mentoring, and unblocking the engineers and contractors who work on the platform

Own the platform architecture on Databricks and AWS, including ingestion, storage, compute, access control and environments. Write the design docs and make the build-versus-buy calls

Own connectivity to partners and systems, built as tested, versioned software: payer claims, eligibility and attribution files over SFTP and S3, private networking, clinical and event feeds (FHIR, HL7), and data exchanged with our care platform

Decide where and how jobs run, and make them fast, memory-predictable and economical: compute choices, serverless versus clusters, right-sizing, and moving single-machine Python onto distributed compute

Own orchestration and deploys, and treat infrastructure and pipelines as code: Terraform, GitLab CI/CD, automated tests and release practices

Make the platform reliable and observable, and turn failures into fixes that hold. Know that a job failed, quietly skipped, or ran twice as long as usual before anyone downstream notices, then ask what would have caught it earlier and what could have recovered it without you

Make our data usable by AI: governed metric definitions, Genie spaces and MCP servers, so colleagues and AI agents work from trusted data rather than guesses

Validate what arrived against what was promised: row counts, date spans, expected fields, and whether history reaches as far back as it should

Replace recurring manual work with automation and AI agents, so routine data operations run themselves and people are pulled in only when something needs judgment

Take part in a shared data operations rotation, which we keep light through automation

Protect PHI by design: access controls, auditing, de-identification and secure data exchange, in partnership with our security leadership

Give analytics engineering a platform to build on: they build the gold tier the analytics team uses, and you make sure it builds reliably, on time and at a sensible cost

Represent data engineering with partners, vendors and colleagues, and share what you learn

About You

Typically seven or more years building production software or data systems, including several years owning data infrastructure or pipelines end to end, from design through operation

Hands-on production experience with at least one distributed data processing engine at scale, such as Spark or Databricks, Snowflake, BigQuery, Flink, Dask or Ray, rather than single-machine pandas: you know what belongs on one machine and what does not, have made large jobs fast and memory-predictable, and can describe one with before and after numbers

Strong Python, and S