Principal Backend / Performance / Database Engineer

Context

iClosed.io is a SaaS platform for sales and marketing teams, and we're growing quickly. As usage scales to millions, performance becomes a core part of the product experience, so we're investing in it early and deliberately. We've formed a dedicated Performance Team to lead that work.

We run a polyglot data layer spanning relational, document, key-value and in-memory stores, and a backend built across multiple modern languages and frameworks. We believe the biggest gains come from design rather than tuning alone: matching each store to its access pattern, moving work off the critical path, and drawing service boundaries that scale cleanly. That's why this role is weighted toward system design. We want someone who solves the problem in front of them, then shapes the architecture so entire classes of problems never come up.

The role

• Title: Principal Backend / Performance / Database Engineer
• Reports to: CTO / Product Manager
• Type: Full-time, permanent
• Level: Senior individual contributor. No direct reports, high autonomy, significant technical influence.

You own the performance and scalability of our backend systems and the data layer underneath them. The work splits roughly into three parts:

Design: how a request flows, what's synchronous, where state lives, and what happens when a dependency is slow.

Diagnosis: reading traces, plans and metrics to find the real cause rather than the obvious one.

Prevention: instrumentation, review standards and regression gates, so problems surface before customers notice.

We want genuine depth in both halves: someone who can design a system that scales, and who can also open an execution plan or a key-value access pattern and see what's wrong with it. Candidates strong in only one half will find this role frustrating.

What you'll own

System and service design

You'll recognise which of these a problem calls for, and what each costs:

Scaling reads: caching, read replicas and index strategy, with invalidation and staleness decided deliberately.

Scaling writes: sharding, partitioning, batching, and queue-buffered writes where needed.

Read/write separation: replica routing, and CQRS where it earns its complexity.

Background processing: queues, workers, idempotency, retries, dead letters and backpressure.

Real-time updates: WebSockets, SSE or polling, and their fan-out and cost implications.

Traffic spikes and high availability: autoscaling on leading indicators, headroom by choice, failover, and a clear answer to what degrades when a component is lost.

Reliability: timeouts, backoff, circuit breakers and graceful degradation.

Search, data growth and safe change: dedicated search indexes, archival and retention, feature flags, and online migrations that don't lock production.

API design: pagination, rate limits, payload budgets, and batch rather than chatty access.

You'll also review feature teams' architecture proposals, and write the design docs and decision records other engineers build from.

Data layer across engines

Deciding which workload belongs where, and correcting cases where that was decided by habit:

Relational: execution plans, index write cost, lock contention, connection pooling, replica lag and partitioning.

Document: modelling, index and shard key design, aggregation performance and working set sizing.

Key-value: access-pattern-first modelling, hot partition avoidance, secondary index strategy and capacity cost.

In-memory cache: caching patterns, key design, eviction, memory pressure and cold-cache behaviour.

Across engines: consistency boundaries and dual-write problems.

Latency and throughput

Diagnose latency across the full request path with evidence.

Eliminate N+1 and chatty access, including ORM-generated traffic.

Reduce sync and ETL latency, including third-party integration paths.

Own capacity planning for the data layer, and cost per request alongside latency.

Standards and prevention

Introduce design and query review for PRs touching hot paths or the data layer.

Write standards for migrations, index justification, and what must never run synchronously.

Instrument data access so issues appear in application traces.

Contribute CI regression gates and load testing at production-realistic volumes.

Investigate per-tenant performance in our largest accounts.

What success looks like

Diagnosis: Latency is attributed to real causes with evidence. You tell us things about our systems we didn't know. Investigations into large-account slowness end with an answer, not a theory.

Design: Critical paths have documented designs others can reason about. Store selection is deliberate and written down. Work that doesn't need to be synchronous isn't, and queues behave predictably under load. Feature teams bring you proposals early because you make them better, not slower.

Measurable improvement: Meaningful latency reduction on the highest-cost paths, with before-and-after numbers. No production operation over 1s without a known reason and owner. Data layer utilisation within target at peak, with articulated headroom. Cost per request trending down.

Prevention: Design and query review is standard practice for hot paths. Regressions are caught before merge. Migration standards are followed by teams you don't sit with.

Ultimately: Backend and data layer performance stops being a meaningful source of customer-visible latency, and we can see the next constraint coming instead of discovering it under load.

What we need from you

Experience

5+ years building and operating production backend systems, with 3+ where performance and scale were a named part of your job.

Owned performance for a multi-tenant SaaS product under real load.

Systems you designed, not only implemented: the alternatives you rejected, the tradeoffs you accepted, and what you'd do differently.

A specific track record: a problem you diagnosed, its cause, your fix, and what the numbers did.

System design (equal weight with data depth)

What we expect you to reason about:

Scaling reads — Caching, replicas, indexes, invalidation, staleness

Scaling writes — Sharding, partitioning, batching, queue buffering

Read/write separation — Replica routing, CQRS, and when it isn't justified

Background work — Queues, workers, orchestration, idempotency, dead letters

Real-time updates — WebSockets, SSE, long polling, connection cost, fan-out

Traffic spikes — Load balancing, autoscaling, headroom, load shedding

High availability — Replication, failover, degraded operation

Reliability — Timeouts, backoff, circuit breakers

Search — Dedicated indexes instead of the primary store

Data growth — Partitioning, archival, retention

Safer deployment — Canary, feature flags, online migrations

Observability — Logs, metrics, traces and alerts designed in, not added later

We care less about whether you can name these than whether you know when not to use one. Explaining why CQRS was the wrong answer somewhere, and what it cost to find out, tells us more than listing its benefits. You should also be able to write a design doc a team can build from and an executive can follow.

Data systems depth

Real depth in at least two of relational, document, key-value and in-memory stores, with working competence across the rest. Nobody is deep in all four, and we'd rather you say so.

Relational: execution plans as routine, composite and partial indexes and their write cost, locking, isolation levels, pooling and replica lag.

Document: embedding vs referencing, shard key selection, aggregation performance and working set.

Key-value: access-pattern modelling, partition keys, hot partitions, secondary indexes and capacity cost.

Cache: patterns, eviction and TTL, and system behaviour when the cache is cold or gone.

Across all: consistency boundaries, and recognising when a query problem is really an application design problem.

Back