DevOps Engineer - Data Platform Department (DPD)

Job Description:

Department Overview

The Data Platform Department (DPD) at Rakuten Group develops and maintains a comprehensive data platform, empowering over 70 Rakuten services with solutions for data ingestion, discovery, governance, analytics, and querying. We support data-driven decision-making across one of Japan's largest data ecosystems, providing the tools and infrastructure to support key domains such as Data Lakes, Data Warehouses, and Business Intelligence.

Position:

Position Details

We are seeking a DevOps Engineer to manage the end-to-end operations of our enterprise Databricks platform hosted on Azure and GCP. The role will focus on platform administration, automation, CI/CD, monitoring, security, incident management, and ensuring a secure, reliable, and highly available platform.

Key Responsibilities

• •Manage day-to-day operations of Azure and GCP Databricks environments
• •Automate infrastructure provisioning and platform configuration using Terraform
• •Build and maintain CI/CD pipelines for Databricks deployments
• •Monitor platform health, performance, availability, and costs
• •Troubleshoot production issues, perform root cause analysis, and drive operational improvements
• •Manage platform security, access controls, secrets, and audit logging
• •Collaborate with engineering, cloud, networking, and security teams to support platform operations

Mandatory Qualifications:

• •3+ years of hands-on experience administering Azure Databricks in production environments
• •Strong knowledge of Unity Catalog, workspace administration, cluster policies, compute management, Jobs & Workflows, permissions, secret scopes, and audit logs
• •Experience managing multiple Databricks workspaces across development, test, and production environments

• •Strong hands-on experience with Microsoft Azure and working knowledge of Google Cloud Platform (GCP)
• •Experience with cloud networking concepts including VNet/VPC, Private Link/Private Endpoints, firewall rules, DNS, VPN/Peering, and secure connectivity

• •Hands-on experience with Terraform for provisioning and managing cloud infrastructure
• •Experience automating infrastructure deployment, configuration management, and environment standardization

• •Experience building and maintaining CI/CD pipelines using Azure DevOps, GitHub Actions, GitLab CI, or Jenkins
• •Strong knowledge of Git, branching strategies, release management, and deployment automation

• •Experience implementing monitoring, logging, and alerting using Azure Monitor, Log Analytics, GCP Cloud Monitoring, or similar tools
• •Strong experience in production support, incident management, root cause analysis (RCA), and operational runbook creation

• •Experience implementing RBAC, Microsoft Entra ID (Azure AD), IAM, service principals, managed identities, secret management, and audit logging
• •Good understanding of cloud security best practices and compliance requirements

• •Proficiency in Python and Bash (PowerShell is an advantage)
• •Experience developing automation scripts for operational and deployment tasks

Strong understanding of DevOps principles, Infrastructure as Code, CI/CD, GitOps, change management, and platform reliability engineering

• •Experience managing multi-cloud Databricks platforms (Azure & GCP)
• •Knowledge of Delta Sharing, Delta Live Tables, Databricks Asset Bundles, and Lakehouse architecture
• •Experience with Docker, Kubernetes, Grafana, Prometheus, Splunk, or Datadog.
• •Experience with cloud cost optimization (FinOps)
• •Azure, GCP, Databricks, or Terraform certifications

Preferred Qualifications:

• •Strong troubleshooting and analytical skills
• •Experience supporting mission-critical production platforms
• •Excellent communication and documentation skills
• •Automation-first mindset with a focus on continuous improvement

#engineer #infrastructureenginee #technologyplatformdiv