GCP Data Engineer

Strong hands-on GCP Data Engineering experience across scalable cloud data platforms.

Expertise in Google Cloud Storage, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Functions, Cloud Monitoring, IAM and Secret Manager.

Strong experience in batch and real-time streaming using Pub/Sub, Dataflow / Apache Beam, Spark Structured Streaming or Kafka.

Hands-on coding proficiency in Python, PySpark and SQL, including debugging and performance optimization.

Experience building high-volume, low-latency ingestion and transformation pipelines with retry, idempotency, deduplication and dead-letter handling.

Strong experience in Databricks, Delta Lake, Structured Streaming, Auto Loader and Spark performance tuning.

Experience handling TB/PB-scale datasets, partitioning, clustering, file compaction, schema evolution and data lifecycle management.

Strong understanding of Data Lake, Lakehouse and modern cloud data-platform architectures.

Practical data-modeling experience for analytics, reporting and downstream consumption.

Experience implementing CI/CD using Cloud Build, GitHub Actions, Jenkins or equivalent, with Terraform for Infrastructure as Code.

Strong troubleshooting, observability, production support and cloud cost-optimization skills.

Excellent client communication, stakeholder management and problem-solving skills.

Responsibilities

Responsibilities and JD in brief along with additional criteria to be considered:

Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms.

Build batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark and Databricks.

Develop and optimize Databricks, Delta Lake, Spark and Structured Streaming workloads for large-scale processing.

Implement high-throughput streaming solutions using Pub/Sub and Dataflow, including windowing, watermarking, late-arriving data, duplicate handling and dead-letter queues.

Handle high-volume TB/PB-scale data with focus on performance, scalability, reliability and cost optimization.

Develop complex SQL transformations, data models and analytics-ready BigQuery datasets using partitioning and clustering.

Orchestrate pipelines using Cloud Composer / Airflow and implement CI/CD and Terraform-based Infrastructure as Code.

Ensure data quality, governance, security, monitoring, alerting and production support.

Collaborate with architects, stakeholders and cross-functional teams in an Agile delivery model.

Troubleshoot and optimize end-to-end GCP, Databricks and streaming data pipelines.

Technical requirements

1. GCP BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, batch and real-time streaming, Databricks , Pyspark

2. Python, strong SQL

Databricks with Delta Lake and Structured Streaming

Responsibilities and JD in brief along with additional criteria to be considered:

Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms.

Build batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark and Databricks.

Develop and optimize Databricks, Delta Lake, Spark and Structured Streaming workloads for large-scale processing.

Implement high-throughput streaming solutions using Pub/Sub and Dataflow, including windowing, watermarking, late-arriving data, duplicate handling and dead-letter queues.

Handle high-volume TB/PB-scale data with focus on performance, scalability, reliability and cost optimization.

Develop complex SQL transformations, data models and analytics-ready BigQuery datasets using partitioning and clustering.

Orchestrate pipelines using Cloud Composer / Airflow and implement CI/CD and Terraform-based Infrastructure as Code.

Ensure data quality, governance, security, monitoring, alerting and production support.

Collaborate with architects, stakeholders and cross-functional teams in an Agile delivery model.

Troubleshoot and optimize end-to-end GCP, Databricks and streaming data pipelines.

Additional responsibilities

Infosys is a global leader in next-generation digital services and consulting. Within the Data & Analytics unit, you will work on cutting-edge data engineering and analytics initiatives for global clients across industries. The role offers opportunities to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and large-scale enterprise data modernization programs. Employees benefit from continuous learning programs, certifications, and career growth opportunities within Infosys' data and analytics practice

Education

Master Of Comp. Applications,Master Of Technology,Bachelor Of Science,Bachelor of Engineering,Bachelor Of Technology