hirly

Black Canyon Consulting

Lead Data Engineer

Bethesda, MD

hirly Pro

Get in front of the people hiring at Black Canyon Consulting

  • Reach the hiring manager at Black Canyon Consulting on LinkedIn, from your own account
  • hirly applies to this job for you, with unlimited applications
  • A resume and cover letter written for this job, unlimited
  • The hiring manager and recruiter at Black Canyon Consulting, with contact details
Get Pro for this job →

$49 a month · ₹1,999 for 30 days in India · cancel any time

See how you match this job — and similar ones. Free.

Upload your resume and hirly scores it against this role at Black Canyon Consulting first, then against similar open jobs, and shows where you fit and why.

PDF or DOCX, up to 12MB. No sign-up to see your matches.

Get past the screening software and onto a recruiter's desk

hirly rewrites your resume for this job — matching the keywords and skills in the posting, moving your most relevant experience to the top, and writing a cover letter to fit. About 30 seconds.

  • Keywords matched to this posting
  • Fit score before you apply
  • Cover letter included

Matched against 2.8M live jobs from 220,000+ employers in 200+ countries.

Tailor my resume for this job →

Apply from your AI assistant

Connect hirly to Claude or ChatGPT and ask it to apply to this job. hirly tailors your resume, fills the employer’s form and asks before sending.

Some employer sites stop an application at a CAPTCHA or sign-in and hand it back with a link. Applying needs a paid plan. Works with any assistant that supports MCP.

hirly's read of this role

Role family
Data & ML
Seniority
Lead / management
Country
US
Work mode
Remote-friendly
First seen by hirly
25 Sept 2026

Derived automatically from the posting. Upload your resume above to see how the role scores against it.

the posting

Overview

We are seeking a Lead Data Engineer to design and implement the core analytical data platform for a federal permitting modernization initiative.

The Lead Data Engineer will establish a modern lakehouse environment using DuckDB, Apache Iceberg, Amazon S3, and AWS Glue Catalog and will lead the engineering patterns used to ingest, harmonize, and serve data across multiple source systems. The role will also help implement metadata-driven and agentic approaches to data engineering, including source analysis, mapping, pipeline generation, testing, reconciliation, and quality assurance.

This is a hands-on technical leadership role with responsibility for guiding and reviewing the work of other Data Engineers.

This position is remote.

Duties & Responsibilities

Design and implement the core DuckDB / Apache Iceberg data platform on AWS.

Define physical data organization, schema, partitioning, file layout, snapshot, compaction, and optimization strategies for Iceberg tables.

Integrate DuckDB with Amazon S3, Apache Iceberg, and AWS Glue Catalog.

Establish bronze, silver, and gold data-layer conventions and reusable engineering patterns.

Lead and mentor Data Engineers, providing technical direction, code reviews, and engineering guidance.

Build and maintain analytical datasets supporting application services, reporting, analytics, and data science.

Develop reusable patterns for incremental loads, CDC, schema evolution, late-arriving data, backfills, and historical reconstruction.

Work with the Data Architect to implement the common canonical data model and source-to-canonical mappings.

Lead migration and harmonization of datasets from multiple source systems.

Implement reconciliation processes that demonstrate completeness and accuracy between source and target systems.

Establish automated data-quality monitoring, operational observability, and testing.

Implement agentic and AI-enabled data engineering patterns supporting activities such as source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance.

Support integration of structured data with associated document and geospatial datasets.

Lead development of certified data products, analytical datasets, and business metrics.

Optimize DuckDB queries and Iceberg datasets for performance and efficiency.

Develop engineering documentation, operational procedures, and technical handoff materials.

Educational Requirements

Bachelor's degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related field.

Requirements

6+ years of relevant data engineering or data platform experience, including experience leading or mentoring other engineers.

Strong hands-on experience with DuckDB and Apache Iceberg .

Advanced SQL and strong Python development experience.

Experience building cloud-based analytical platforms using object storage and columnar formats such as Parquet.

Strong understanding of partitioning, schema evolution, snapshots, compaction, incremental processing, and analytical query optimization.

Experience implementing bronze/silver/gold or comparable layered data architectures and transforming heterogeneous enterprise datasets.

Experience with AWS data services, particularly Amazon S3 and AWS Glue.

Experience designing reusable, metadata-driven ingestion and transformation frameworks.

Experience implementing automated data quality, reconciliation, monitoring, and operational controls.

Experience with agentic or AI-enabled data engineering solutions , including the use of agents or AI-assisted workflows for activities such as data mapping, pipeline generation, validation, testing, or troubleshooting.

Hands-on experience with agentic frameworks such as LangChain, LangGraph, or similar frameworks for building tool-using or multi-step AI workflows.

Strong troubleshooting, performance tuning, and technical leadership skills.

Bonus Skills

Experience with Kafka or Amazon MSK and CDC architectures.

Experience with GeoParquet, PostGIS, or geospatial datasets.

Experience with OpenSearch, document stores, or vector search technologies.

Experience integrating enterprise SaaS platforms, APIs, databases, and file-based sources.

Experience working in the U.S. Federal Government space .

Experience with permitting, regulatory, land management, environmental, energy, or natural-resource datasets.

Benefits and Salary

We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement.

If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!

Original posting on Black Canyon Consulting's site ↗

Listed on hirly, a job board. hirly is not the employer: Black Canyon Consulting is hiring for this role.

Browse similar roles

Want this one?

Upload your resume and hirly rewrites it for this job and writes the cover letter — in about thirty seconds, before you sign up.

Tailor my resume for this job