Software Engineer, ML Serving

About the Company

Rime is a foundation modeling company that builds voice AI for enterprises running customer experiences at scale. Our models are purpose-built for high-volume conversational deployments, engineered for the accuracy, performance, and deployment flexibility that production environments actually demand.

Responsibilities

• Architecture and implementation of Rime's TTS serving infrastructure, from GPU-backed inference engines to the API surface
• Model optimization from a single-node to disaggregated fleet serving
• Ensuring compatibility with different NVIDIA hardwares from Hopper to Blackwell and beyond
• Managing continuous integration and deployment workflows for the model serving pipeline
• Maintaining site reliability through on-call rotation, monitoring, alerting, and observability
• Resource provision and cost management across the GPU fleet

Requirements

• Hands-on experience with real-time multinode ML serving infrastructure (NVIDIA Triton, vLLM, SGLang, or equivalent)
• Experience with distributed or disaggregated model serving (Tensor Parallel, Pipeline Parallel, etc.)
• Strong cloud infrastructure fundamentals including Linux internals, networking, and containerization (Docker, Kubernetes)
• Experience with Infrastructure as Code (Terraform, Packer, or comparable)
• Willingness to participate in an on-call rotation

Preferred Qualifications

• Experience with multinode training (DDP, FSDP, etc.)
• Experience with gRPC or other bidirectional binary streaming protocols
• Experience with audio streaming technologies (WebRTC, WebSockets, etc.)
• Experience with multi-cloud infrastructures (AWS, GCP, OCI, etc.)
• Background in SRE, DevOps, or platform engineering at a startup