Operations Support Engineer - Lead

Overview

Black Canyon Consulting (BCC) is looking for a Lead Operations Support Engineer to join our DevOps and infrastructure team to support application developers and operations at the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM), an institute of the National Institutes of Health. This opportunity is full time at the NCBI in Bethesda, MD and/or remote.

NCBI is part of the National Library of Medicine (NLM) at the National Institutes of Health (NIH). NCBI advances science and public health by providing free access over the Internet to biomedical literature and genomic data, making it one of the 400 top most-visited sites in the world. NCBI has a diverse staff of smart, talented, and deeply technical people who collaborate to build critically valuable services for researchers, physicians, educators, students, and the general public. For example, NCBI develops and delivers PubMed, an index of over 29 million biomedical research abstracts; and maintains ClinicalTrials.gov, a database of over 400,000 clinical studies around the world.

The Systems & DevOps team is responsible for the efficient operation of infrastructure to run NCBI's many applications. This includes providing convenient, scalable solutions for development, deployment, and operations across teams, languages, and cloud and on-prem environments.

This is a great opportunity to work on challenging problems in a technical, scientific, and goal-oriented environment. NCBI offers flexible working hours, remote options, on- and off-site training courses, and conference attendance and tuition reimbursement.

We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement. If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!

Duties & Responsibilities

NCBI has built a modern DevOps platform based on GitLab and Kubernetes, and is looking to create a team of support engineers to assist internal developers with transitioning legacy software development and deployment to the new DevOps platform. You would be the leader of our platform operations team.

Identify and resolve operational problems in a micro-service environment

Work with developers to resolve deployment and runtime problems

Perform analysis and debugging work across multiple technologies

Prioritize issues to keep applications within error budgets and meeting their SLOs

Provide technical solutions to a wide range of problems and user requests

Document process, procedures and SOPs by soliciting feedback and suggestions from team members

Compile postmortems and action items to minimize future outages

Interview other people for team member roles, and decide which ones to recommend for hire.

Train new team members, and assist them with issues.

Provide on-call support to NCBI's internal developers and other staff.

Requirements

BS degree in STEM or equivalent experience

Customer-focused, team-oriented disposition

Good systems debugging skills

Comfortable with the Linux environment or UNIX CLI

Experience with some programming or scripting language

Have experience creating processes, procedures and SOP documentation

General understanding of TCP/IP, HTTP, and related protocols

Initiative to take ownership of tasks and drive them to completion

Comfortable dealing with users with varying levels of IT knowledge

Eager to learn new technologies

Strong communication and soft skills to interface with customers, peers and management

Good judgement, sense of integrity, and responsibility

Preferred Experience / Skillsets

Kubernetes, OpenShift, Cloud or Linux experience

Experience with:

Service Reliability Engineering in any capacity

Linux systems administration

Automated CI servers, especially TeamCity and/or GitLab

Automation programming/scripting in any of: bash, Ruby , Python, Go, Java , Scala, Rust , C++, Perl

Automated configuration management , such as Puppet, Ansible, Chef, bcfg2, cfengine, etc. Puppet is preferred.

Version control systems, especially git

Service Mesh technologies (e.g., linkerd, Istio)

Configuring or using monitoring and alerting technologies (TIGK stack, Grafana, Prometheus, OpsGenie)

Confluence, Jira, and Microsoft Office suite

GitOps tools, especially ArgoCD

Google Anthos

Understanding of:

Linux internals (system calls, file systems, processes, etc.)

Linux network configuration

Linux application containerization, especially Docker

Attached network storage technologies

Cloud computing environment such as AWS, GCP or Azure

Automated CI/CD pipelines

Distributed systems design principles