Engineering - DXR Engineering - Senior Scheduling Platform Production Engineer - Dallas - Vice President

The Runtime Platforms team delivers a highly available and scalable ecosystem for Goldman Sachs engineers worldwide, enabling them to run workloads across various compute platforms. As part of the firm's strategic move to increase the usage of containerization to package, deliver and run software, the runtime team plays a critical role in ensuring that we offer container platforms that are secure, reliable and operationally inexpensive so that business teams can focus on delivering value to shareholders. The platforms offered by the runtime team support some of the firm's most critical businesses and services both on-premises and in public cloud.

At Goldman Sachs, our culture is one of teamwork, innovation and meritocracy. We often say our people are our greatest asset and we take pride in supporting each colleague both professionally and personally. You will be joining a talented globally distributed team passionate about delivering the best possible experience to our users. 
This is an experienced-hire role for a self-directed engineer who will act as the regional subject matter expert (SME) for the Procmon platform.

Procmon Platform delivers a highly scalable and reliable ecosystem for scheduling business critical jobs across Goldman Sachs.

Our platform is responsible for scheduling tens of millions of daily jobs for Global Banking & Markets, Asset & Wealth Management, Risk and other business and engineering functions.

The ecosystem includes a number of high availability, very large scale systems including

• Job scheduling
• Event streaming
• Log shipping
• Data warehouses
• Security infrastructure
   

RESPONSIBILITIES

• Proactively identify operational risks, capacity concerns and reliability gaps; propose and implement remediation without requiring close supervision.
• Continuously refine runbooks, tooling and automation to reduce manual toil and improve platform reliability.
• Build observability for new deployments to ensure robustness. Work closely with development team to ensure observability is built in from the start.
• Lead real-time outage investigations, write and present postmortems to senior management.
• Define SLIs and SLOs and partner with development teams to ensure system are well designed and live up to our SLAs.
• Work closely with users to facilitate planned migration from the previous to the newest Procmon platform.
• Communicate well and work efficiently with remote colleagues in other regions.
• Provide regional coverage for the Procmon platform, including participation in an on-call rotation and acting as the escalation point for user queries and L3 escalations.

REQUIREMENTS

• Extensive experience (8+ years) in DevOps or Production Engineering, with expertise in architecting, deploying, and scaling distributed computing systems,
• Excellent problem-solving and automation skills, leveraging real-time monitoring and alerting tools like Prometheus for proactive incident detection and automated response, and utilizing data analysis and Grafana for historical data visualization, deep-dive root cause analysis, and automated operational reporting.
• Deep knowledge of Linux operating systems and system administration
• Good networking fundamentals (familiarity with TCP/IP, IP routing, firewalls, secure tunneling protocols).
• Experience working with distributed computing systems and Cloud computing within enterprise environments.
• Proficiency in at least one programming language; the team uses a mix of Go, Shell, Python and Erlang
• Able to operate effectively in a mission critical, highly regulated financial services environment.
• Strong communication skills, able to explain complex technical issues clearly to both technical and non-technical users.
• Previous experience with supporting and mentoring junior team members and interns is a plus.