Principal Software Engineering - Reliability and FinOps Engineer

This is a full-time onsite role based in the location(s) posted and is performed from the designated work location five days per week. No relocation is offered.

AT&T will not hire any applicants for this position who require employer sponsorship now or in the future.

Join AT&T and reimagine the communications and technologies that connect the world. The Technology Services organization is responsible for advancing information technology performance and delivering solutions with a focus on maximizing ROI, increasing efficiency and enhancing the experience of end users. Guided by experienced leaders, Corporate Systems seamlessly integrate with advanced Technology and Operations to drive our enterprise forward. Our Systems Reliability and Software Delivery teams are unwavering in their commitment to excellence, ensuring every solution is robust and efficient. When you step into a career with AT&T, you won't just imagine the future, you'll create it.

About The Tea m

AT&T Technology Services is modernizing the technology ecosystem through secure, scalable, cloud-enabled, and AI-powered platforms. Within Reliability & FinOps Engineering, our purpose is to elevate the customer experience by strengthening the reliability and resiliency of critical systems, maximizing the value of technology investments, and accelerating responsible automation.

Our organization brings together reliability engineering, FinOps and infrastructure optimization, AI and automation enablement, and customer-centric business intelligence. We build reusable engineering patterns, policy-driven controls, self-service capabilities, and intelligent optimization services that improve operational outcomes at enterprise scale.

What You'll Do

As a Principal System Engineer, you will serve as a senior technical leader across FinOps, cloud optimization, system reliability, and AI-driven automation. You will shape and execute enterprise strategies that improve the reliability of critical services, reduce cloud waste, strengthen financial accountability, and automate operational workflows across Azure, AWS, Kubernetes, and hybrid-cloud environments.

In this role, you will translate business priorities into durable technical solutions, influence cross-organizational roadmaps, and move optimization opportunities from insight through governed execution and measurable value realization.

Key responsibilities include:

Provide principal-level technical leadership for resilient, secure, scalable, and cost-efficient platforms across Azure, AWS, Kubernetes, and hybrid-cloud environments.

Lead FinOps and infrastructure optimization capabilities, including cost allocation, showback, anomaly detection, forecasting, rightsizing, workload scheduling, commitment optimization, waste reduction, and validated savings realization.

Develop policy-driven, closed-loop optimization workflows that move recommendations through identification, application-owner engagement, approval, ServiceNow change control, execution, validation, evidence capture, and reporting.

Apply an automation-by-default approach using Python, PowerShell, Bash, REST APIs, Azure Functions, Power Automate, and event-driven services to reduce manual handoffs and operational toil.

Design practical generative AI, AI-fueled coding, and agentic automation solutions for engineering operations, cost analysis, anomaly investigation, remediation guidance, executive reporting, knowledge retrieval, and decision support.

Establish cloud-governance guardrails, policy-as-code controls, exception processes, auditability, and risk-based automation patterns aligned with security, architecture, compliance, and change-management requirements.

Engineer secure identity and authorization models using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, and time-bound permissions.

Integrate optimization and automation capabilities with Terraform, GitHub, Azure DevOps, CI/CD pipelines, ServiceNow, monitoring platforms, cost-management tools, and enterprise data sources while preventing configuration drift.

Advance reliability engineering practices, including observability, service-level objectives, capacity management, performance analysis, incident prevention, operational readiness, post-change validation, rollback planning, and continuous improvement.

Build executive dashboards and operational scorecards that communicate cloud spend, reliability, automation coverage, optimization pipeline health, risk, realized savings, and customer or business outcomes.

Lead proofs of concept, production readiness, phased adoption, and complex technical issue resolution in partnership with internal teams and strategic vendors.

Mentor engineers, create reusable reference architectures and standards, and influence priorities across Reliability, FinOps, Security, Architecture, Finance, Operations, and application teams.

Demonstrate AT&T leadership behaviors by serving customers first, moving faster, acting boldly, winning as one, and developing talent.

What You'll Need

• •7+ years of progressive experience in systems engineering, cloud engineering, site reliability engineering, platform engineering, DevOps, FinOps, or enterprise automation.
• •Experience creating, managing and support ephemeral runner model GitHub supported ARC controllers using helm charts in Kubernetes and manage securely via github apps
• •Experience creating and managing finops agents in the Kubernetes (AKS, EKS)

Demonstrated ability to lead complex, cross-functional engineering initiatives and influence technical direction without relying solely on direct authority.

Deep hands-on experience with Azure and/or AWS architecture, networking, compute, storage, identity, monitoring, governance, and cost-management services.

Strong knowledge of FinOps practices, including allocation, tagging, showback or chargeback, budgeting, forecasting, unit economics, commitment management, anomaly detection, optimization, and savings validation.

Experience implementing cloud optimization for virtual machines, storage, databases, platform services, Kubernetes, and idle or underutilized resources.

Advanced scripting and automation experience using Python, PowerShell, Bash, REST APIs, JSON, and event-driven integration patterns.

Strong Infrastructure as Code and CI/CD experience using Terraform, GitHub Actions, Azure DevOps, or comparable enterprise tooling.

Hands-on experience with Kubernetes, AKS, EKS, or OpenShift, including workload rightsizing, Helm, observability, node lifecycle management, and container cost allocation.

Experience applying generative AI, prompt engineering, AI assistants, AI-fueled coding, or agentic automation to engineering and operational use cases.

Strong understanding of reliability engineering, observability, incident management, capacity planning, performance optimization, recovery, change control, and operational readiness.

Experience designing secure automation using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, secrets management, and auditable approval controls.

Ability to analyze complex cloud cost, utilization, telemetry, and operational data and convert findings into prioritized actions and clear executive insights.

Excellent written and verbal communication skills, including the ability to present complex technical and financial information to engineers, application owners, finance partners, vendors, and senior leaders.

What You'll Bring

Bachelor's or master's degree in Computer Science, Computer Engineering, Information Systems, Data Science, Finance preferred.

FinOps Certified Practitioner or certifications in Azure, AWS, Kubernetes, Terraform, ITIL, site reliability engineering, security, data, or artificial intelligence.

Experience