Software Engineer, Model Routing & Inference

About the Company

Our mission is to automate coding. The first step in our journey is to build the best tool for professional programmers, using a combination of inventive research, design, and engineering. Our organization is very flat, and our team is small and talent dense.

Responsibilities

• Build the inference platform that powers every AI interaction in the product
• Own the full inference path to make AI faster, more reliable, and more cost-effective
• Build and evolve the inference gateway to abstract provider API semantics
• Design intelligent cross-provider failover to prevent user-visible degradation
• Design routing backpressure and admission control to manage traffic spikes

Requirements

• Deep experience building high-throughput, low-latency distributed systems
• Experience in inference serving, traffic routing, or real-time data pipelines
• Ability to reason about cost/performance tradeoffs including GPU utilization and provider economics
• Strong software engineering fundamentals with experience shipping production systems at scale