Databricks SME

Job Description

CSCI Consulting is looking for a Databricks Subject Matter Expert (SME) to support our Department of War client. In this role, the consultant will serve as the technical lead for Databricks-based data modernization initiatives, providing architecture, engineering, governance, analytics, and cloud data platform expertise to support mission-critical data, reporting, AI/ML, and decision-support capabilities.

Responsibilities

Serve as the primary Databricks SME supporting enterprise data and analytics initiatives

Design, develop, and optimize scalable data platforms using the Databricks Lakehouse architecture

Lead the implementation of data engineering, data integration, and data transformation solutions using Databricks and Apache Spark

Design and implement Medallion Architecture frameworks, including bronze, silver, and gold data layers

Develop and maintain ETL/ELT pipelines for structured and unstructured data sources

Configure and optimize Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables

Support cloud-native data solutions within Azure, AWS, or multi-cloud environments

Collaborate with business stakeholders, architects, developers, and data teams to translate requirements into scalable solutions

Support data governance, metadata management, data quality, and master data management initiatives

Optimize platform performance, scalability, availability, and cost efficiency

Develop technical documentation, architecture diagrams, data-flow diagrams, and operational procedures

Support enterprise analytics, business intelligence, artificial intelligence, and machine learning initiatives

Implement and support DevSecOps, CI/CD, and Infrastructure-as-Code practices

Integrate Databricks solutions with enterprise applications, APIs, databases, and cloud services

Support cybersecurity, compliance, audit, and data protection requirements

Provide technical mentorship and guidance to engineers, analysts, and project teams

Participate in Agile ceremonies, technical reviews, solution demonstrations, and program planning activities

Minimum Requirements

7+ years of experience supporting enterprise data engineering, analytics, or data platform initiatives

5+ years of hands-on experience with Databricks and Apache Spark

Experience designing and implementing enterprise data lake, data warehouse, or Lakehouse architecture

Strong expertise with Databricks Lakehouse Platform and Delta Lake

Advanced proficiency with SQL, Python, PySpark, and distributed data processing

Experience building and maintaining ETL/ELT pipelines and data integration solutions

Experience supporting Azure, AWS, or other cloud-based data platforms

Knowledge of data governance, data quality, metadata management, and data security best practices

Experience troubleshooting performance, scalability, and reliability issues in large-scale data environments

Experience developing technical documentation and communicating complex technical concepts to both technical and non-technical audiences

Ability to obtain and maintain the security clearance required by the program

U.S. Citizenship required where mandated by the applicable government contract

Able to obtain security clearance

Creativity and adaptability in problem-solving

Ability to work with clients to understand their needs

Strong organizational and time-management skills

Excellent written and verbal communication skills

Professional presence

Preferred Skills

Databricks Certified Data Engineer Associate, Professional Data Engineer, or related Databricks certification

Experience supporting Department of War, DoD, Intelligence Community, or Federal data modernization programs

Experience with Unity Catalog, Delta Live Tables, MLflow, Databricks Workflows, and Databricks Asset Bundles

Experience implementing modern data architecture including Data Mesh, Data Fabric, and Lakehouse strategies

Experience supporting Artificial Intelligence (AI), Machine Learning (ML), and Generative AI initiatives

Experience integrating Databricks with SAP, Salesforce, ServiceNow, Oracle, or other enterprise platforms

Experience with Power BI, Tableau, Qlik, or other enterprise reporting and visualization platforms

Experience implementing DevSecOps, CI/CD pipelines, and Infrastructure-as-Code frameworks

Experience using Azure DevOps, GitHub, GitLab, Jenkins, or other enterprise development platforms

Knowledge of Kubernetes, containerization technologies, and cloud-native architecture

Experience with Terraform, Bicep, or other infrastructure automation tools

Experience supporting data governance and compliance requirements within regulated environments

Experience working within Agile or SAFe delivery frameworks

Ability to work in a team environment, as well as independently

Strong customer and vendor relationship skills

Demonstrated ability to comply with data standards and policies

Motivation to learn new technologies and methodologies that demonstrate value

Past experience working with a Federal government agency

Department of War, etc. experience is a plus!

About CSCI

CSCI is an award-winning information technology and financial management consulting firm founded on one simple philosophy: “Do what is right, always.” We apply this philosophy across all elements of our growing business, from delivering world-class s ervices for customers to providing an environment where associates thrive both personally and professionally. At CSCI, work and fun aren’t diametrically opposed!

At CSCI, our goal is to hire people with proven track records and retain them with an energizing, diverse company culture. We value each associate’s natural drive to excel, and we provide them with the freedom to do things their way. CSCI seeks the best and brightest in the industry—those who are ready to move their lives and career forward. Join us today and get excited about Mondays again!

Benefits of Working at CSCI

Competitive salaries

Generous Paid Time Off (PTO) package

Paid holidays aligned to the Federal calendar

Full health benefits including medical, dental, vision, and life insurance

401(k) retirement plan

Team building events

Professional development support

Legal

CSCI complies with all applicable Federal, state, and local employment regulations. Please reach out to [email protected] with any questions.

Equal Employment Opportunity

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability or veteran status, or any other applicable state or federal protected class. CSCI provides affirmative action in employment for qualified Individuals with a Disability and Protected Veterans in compliance with Section 503 of the Rehabilitation Act and the Vietnam Era Veterans’ Readjustment Assistance Act.

If you require an accommodation, please contact CSCI Human Resources for further assistance. For questions regarding the qualifications listed above, please contact the CSCI Recruiting Team.

E-Verify

CSCI participates in E-Verify to confirm the employment eligibility of all newly hired employees. For more information, please visit www.dhs.gov/E-Verify .

California Consumer Privacy Act (CCPA) Notice

As part of the application and recruitment process, CSCI Consulting, Inc. may collect personal information as described under the California Consumer Privacy Act (CCPA).

This information is collected solely for employment-related purposes, including evaluating your qualifications, processing your application, and complying with legal obligations.

By applying to this position, you acknowledge that you have read and understand the following notice regarding the collection of your personal information as a job applicant:

Categories of Information Collecte