Software Engineer, Hardware Health

Responsibilities

• Define and maintain health signals across GPUs, CPUs, networking, and platform infrastructure
• Build and evolve health checks that detect, remediate, and verify failures at scale
• Ensure critical health checks execute with minimal latency to maximize workload uptime
• Investigate hardware failures and system-level issues across large-scale compute environments
• Own node lifecycle workflows including drain, quarantine, repair, RMA, and return-to-service processes
• Build automation and tooling that enables global cluster management with minimal manual intervention
• Partner with workload, reliability, and provider teams to integrate health signals into training and inference systems

Requirements

• 7+ years of industry experience in software or infrastructure engineering
• Strong proficiency with Python and shell scripting
• Experience building large-scale distributed systems or infrastructure platforms
• Comfort digging into noisy operational data using SQL, PromQL, or similar tooling
• Experience building reproducible analyses and operational tooling
• Strong systems debugging and operational instincts with an ownership mindset

Preferred Qualifications

• Experience with low-level hardware systems and Linux tooling (PCIe, InfiniBand, RoCE, networking, power management, kernel performance tuning, FW/SW debugging)
• Experience operating or debugging large-scale GPU or accelerator clusters
• Expertise in network operations, observability, or systems telemetry
• Experience with automated remediation systems or fleet lifecycle management
• Experience improving reliability, utilization, or workload uptime in distributed compute environments

About the Company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity.