Observability Engineer

Position Overview

The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.

The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.

Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilitie s

· Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.

· Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.

· Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.

· Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).

· Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.

· Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.

· Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.

· Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.

· Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.

· Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.

· Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.

· Maintain service-based error budgets which our development teams use to dictate feature work.

Qualifications

· Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.

· Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.

· Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.

· Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.

· Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.

· Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.

· Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.

· Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.

· Strong troubleshooting, analytical, communication, and cross-team collaboration skills.

· Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.

Preferred Qualifications

· Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.

· Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.

· Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.

· Experience establishing SLIs/SLOs or service-health measures for production applications.

· Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.

· Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.

· Familiarity with ITIL Incident Management and Problem Management practices.