Applications Support Analyst

• The Senior Application Support Analyst (C11) is a critical role responsible for ensuring the stability, reliability, and optimal performance of Citi's core business applications.
• This position requires a highly analytical and proactive professional who excels in Incident and Problem Management within an ITIL framework, combined with strong, hands-on technical expertise in Unix/Linux environments, SQL database querying, enterprise monitoring via ITRS Geneos, and batch workload scheduling via Autosys. Crucially, the successful candidate must have working experience supporting Big Data and EAP (Enterprise Application/Analytics Platform) environments, with a proven track record of driving automation and process enhancements to eliminate manual toil and optimize support workflows.

Key Responsibilities

1. Big Data & EAP (Enterprise Application/Analytics Platform) Support

• •Support Distributed Environments: Provide Level 2 (L2) and Level 3 (L3) support for applications hosted on Big Data platforms and Citi's Enterprise Application/Analytics Platform (EAP).
• •Troubleshoot Data Pipelines: Diagnose and resolve failures in complex data ingestion and processing pipelines, including distributed processing frameworks (e.g., Apache Spark, Hadoop MapReduce).
• •Cluster & Resource Monitoring: Monitor cluster resource utilization (using YARN, Cloudera Manager, or similar tools) to identify and resolve memory bottlenecks, queue congestion, and job failures (e.g., Spark Out-Of-Memory errors).
• •Data Querying & Validation: Query and validate large-scale datasets stored in distributed data warehouses and file systems (e.g., HDFS, Hive, Impala, or HBase).
• •Message Queue Management: Monitor and troubleshoot real-time streaming and messaging platforms (e.g., Apache Kafka), managing consumer groups, offsets, and partition lags.

2. Batch Management & Job Scheduling (Autosys)

• •Monitor and manage batch execution: Oversee the execution of critical daily, weekly, and monthly batch processing cycles scheduled via Autosys.
• •Troubleshoot batch failures: Rapidly diagnose and resolve Autosys job failures, analyzing log files, identifying dependency issues, and performing necessary job overrides, force-starts, or hold/release actions to minimize business impact.
• •Optimize job flows: Collaborate with development and engineering teams to define, configure, and optimize Autosys job definitions using JIL (Job Information Language).

3. Automation & Process Enhancement (Toil Reduction)

• •Identify and eliminate manual bottlenecks: Actively analyze daily support activities to identify repetitive, manual tasks ("toil") and design automated solutions to eliminate them.
• •Develop automation scripts: Write, test, and deploy robust scripts (using Python, Bash, or PowerShell) to automate routine operations, such as daily health checks, application restarts, log archiving, and data reconciliation.
• •Drive process improvements: Evaluate existing support workflows, runbooks, and escalation paths, implementing enhancements to streamline operations and reduce Mean Time to Repair (MTTR).

4. Incident Management & Production Recovery

• •Own and drive the end-to-end resolution of L2/L3 production incidents, ensuring strict adherence to corporate Service Level Agreements (SLAs) and Service Level Objectives (SLOs).
• •Lead technical triage during Major Incidents (MIM) and high-severity outages. Coordinate effectively with cross-functional global teams (Infrastructure, Database, Networks, Development, and Business Operations) to restore services rapidly.
• •Act as the primary technical escalation point during incidents, translating complex technical issues into clear, concise, and business-friendly updates for senior leadership and stakeholders.
• •Ensure accurate and timely logging, categorization, and tracking of incidents within ServiceNow.

5. Problem Management & Root Cause Analysis (RCA)

• •Lead proactive Problem Management initiatives by analyzing incident trends, identifying systemic patterns, and pinpointing recurring failure points.
• •Conduct deep-dive technical investigations—including log analysis, database queries, and infrastructure health checks—to perform comprehensive Root Cause Analysis (RCA).
• •Author high-quality Post-Incident Reviews (PIRs) and RCA documents, detailing the timeline, root cause, impact, and preventative actions.
• •Collaborate closely with Development and Engineering teams to prioritize, track, and implement permanent bug fixes, structural workarounds, and long-term remediations.

6. Hands-on Unix/Linux & Application Troubleshooting

• •Perform deep-dive technical troubleshooting directly within Unix/Linux production environments (analyzing system resources, CPU/memory bottlenecks, process states, and network connectivity).
• •Conduct advanced log analysis using Unix command-line utilities (e.g., grep, awk, sed, find, tail) to rapidly isolate application errors and system anomalies.
• •Maintain and debug shell scripts (Bash/Korn) used for application startup, shutdown, health checks, and automated maintenance.

7. Database Support & SQL Querying

• •Troubleshoot database-related application issues by writing and executing complex SQL queries (including multi-table joins, subqueries, and aggregations) on databases such as Oracle, MS SQL Server, or Sybase.
• •Analyze database performance, identify slow-running queries, and collaborate with DBAs to resolve locks, blocks, and indexing issues affecting production.

8. Proactive Monitoring & Alerting with ITRS Geneos

• •Actively monitor application health and infrastructure performance using ITRS Geneos (Active Console).
• •Configure, customize, and maintain ITRS Geneos samplers, rules, alerts, and Netprobes to ensure comprehensive coverage of critical system components.

Technical Skills & Experience

Core Requirements (Mandatory & Hands-on)

Competency / Technology

Required Hands-on Experience

Big Data & EAP Environments

Working experience supporting Big Data ecosystems and Enterprise Application/Analytics Platforms (EAP). Hands-on familiarity with Hadoop, HDFS, Hive, Spark, YARN, and Kafka. Ability to troubleshoot distributed job failures and monitor cluster health.

Autosys

Strong hands-on experience with Autosys (or similar enterprise job schedulers). Proficient in monitoring batch cycles, troubleshooting job failures, managing dependencies, performing run-time overrides, and writing/modifying JIL (Job Information Language) configurations.

Unix / Linux

Advanced hands-on experience navigating Unix/Linux file systems, managing processes, analyzing system performance (CPU, memory, disk I/O), and writing/debugging Shell scripts (Bash/Ksh).

SQL & Databases

Proficient in writing complex SQL queries to extract, analyze, and troubleshoot data issues across relational databases (Oracle, Sybase, or MS SQL Server). Understanding of database locks, indexing, and basic performance tuning.

ITRS Geneos

Hands-on experience using ITRS Geneos for real-time monitoring. Ability to navigate the Active Console, interpret alerts, and configure basic samplers, rules, and alerts.

Automation & Process Enhancement

Proven experience in automating manual support tasks and optimizing operational processes. Strong ability to identify automation opportunities and implement scripting solutions to reduce operational toil.

Incident & Problem Management

Strong expertise in ITIL processes, specifically Incident and Problem Management. Proven track record of leading major incident triage, managing SLAs, conducting Root Cause Analysis (RCA), and managing the ticket lifecycle in ServiceNow.

Secondary Technical Skills (Highly Desirable)

• •Scripting Languages: Strong proficiency in Python or Perl for building automation tools and utility scripts.
• •Middleware: Familiarity with middleware technologies (TIBCO EMS, IBM MQ, WebSphere, or Tomcat).
• •Cloud & Containers: Basic exposure to containerized environments (OpenShift, Kubernetes) or Cloud