Databricks Data Engineer / Business Analyst – Data Contracts & OCDS

Position Summary

Scicom Infrastructure Services is seeking a Data Engineer / Business Analyst to support a large-scale Databricks implementation. This hybrid role will bridge business, contracting, data-governance, and engineering teams to define business requirements, construct data contracts, map source data, and develop reliable Databricks data pipelines.

The successful candidate must have hands-on experience with the Open Contracting Data Standard (OCDS) and understand how contracting and procurement information is structured across planning, tender, award, contract, and implementation stages. OCDS provides a standardized model for publishing and analyzing data throughout the public-contracting process and uses defined schemas, codelists, releases, records, and packages.

This individual will work closely with business stakeholders, procurement subject-matter experts, data architects, Databricks engineers, governance teams, and program leadership to translate complex business and contracting requirements into enforceable technical specifications and production-ready data products.

Key Responsibilities

Data Contracts and Business Analysis

Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders.

Define, construct, document, and maintain data contracts between data producers and consumers.

Establish data-contract requirements covering:

Dataset purpose and ownership

Source and target systems

Schemas, fields, and data types

Required and optional attributes

Business definitions and transformation rules

Data-quality expectations

Validation and reconciliation rules

Refresh frequency and delivery schedules

Versioning and schema-evolution requirements

Security classifications and access controls

Service-level expectations

Issue ownership and change-management procedures

Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements.

Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping.

Identify differences between existing procurement data and required OCDS structures.

Facilitate agreement among data owners, producers, consumers, architects, and governance teams.

Maintain traceability from business requirements through data models, engineering implementation, testing, and acceptance.

Evaluate requested changes for downstream effects on data products, reports, integrations, and analytical use cases.

Support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance.

OCDS Responsibilities

Apply the Open Contracting Data Standard to procurement and public-contracting datasets.

Map source-system data to appropriate OCDS fields and structures.

Work with data across the contracting lifecycle, including:

Planning

Tender

Award

Contract

Implementation

Develop and maintain mappings for OCDS releases, records, release packages, record packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related contracting elements.

Interpret and apply OCDS schemas, codelists, validation rules, and implementation guidance.

Determine whether standard OCDS fields meet project requirements or whether documented extensions are necessary.

Support the construction of complete contracting records from multiple transactional releases.

Establish rules for handling amendments, updates, cancellations, corrections, and historical changes.

Validate transformed data against applicable OCDS JSON schemas.

Identify missing, invalid, inconsistent, or nonconforming procurement data and work with stakeholders to resolve deficiencies.

Document assumptions, business rules, mappings, extensions, and exceptions.

Support the publication, exchange, analysis, or internal use of standardized contracting data.

Databricks Data Engineering

Design, develop, test, and maintain data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL.

Build ingestion and transformation pipelines for structured and semi-structured procurement data.

Process JSON, CSV, XML, Parquet, relational database, API, and file-based data sources.

Implement Bronze, Silver, and Gold data layers using medallion architecture.

Build normalized, dimensional, analytical, and OCDS-aligned data models.

Use Delta Lake capabilities for schema enforcement, schema evolution, versioning, auditability, and reliable processing.

Develop reusable frameworks for mapping source procurement data into OCDS-compatible outputs.

Implement batch and incremental ingestion patterns.

Use Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables or Lakeflow capabilities, as appropriate.

Develop REST API integrations for source ingestion and downstream data delivery.

Implement automated data-quality, reconciliation, completeness, and conformity checks.

Support Unity Catalog implementation for metadata, ownership, lineage, access control, and data discovery.

Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts.

Participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support.

Investigate pipeline failures, data discrepancies, and performance issues.

Data Quality and Governance

Define measurable quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness.

Develop validation controls for required OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships.

Create dashboards or reports that show data-contract compliance and data-quality results.

Establish processes for detecting and managing schema drift.

Document data lineage from original procurement systems through Databricks transformations and downstream products.

Work with governance teams to assign data owners, stewards, classifications, retention requirements, and access policies.

Ensure sensitive procurement and supplier information is handled according to security and privacy requirements.

Support auditability through documented rules, version-controlled mappings, validation results, and change histories.

Required Qualifications

Bachelor’s degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related field.

At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience.

Hands-on experience implementing or working with the Open Contracting Data Standard .

Demonstrated experience constructing, documenting, negotiating, or maintaining data contracts.

Experience mapping procurement or contracting data to OCDS schemas.

Strong knowledge of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation practices.

At least three years of hands-on experience with Databricks.

Strong experience with:

Apache Spark

PySpark

Python

SQL

Delta Lake

ETL and ELT pipelines

Data modeling

JSON and JSON Schema

REST APIs

Data-quality validation

Source-to-target mapping

Experience gathering requirements and translating them into implementable engineering specifications.

Ability to communicate effectively with technical and nontechnical stakeholders.

Experience writing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation.

Strong analytical, troubleshooting, facilitation, and documentation skills.

Experience working within Agile delivery teams.

Preferred Qualifications

Experience with government procurement, public-sector contracting, grants, acquisition, supplier, or financ