Software Engineer, Ray Serve

Responsibilities

• Build and scale Ray Serve, a production-grade serving framework for high-performance machine learning applications.
• Design and implement intelligent request routing systems and sub-millisecond model routing to balance load across thousands of replicas.
• Develop sophisticated traffic management systems for zero-downtime model updates and seamless transitions between versions.
• Architect frameworks for multi-model orchestration and complex ML pipelines to maintain end-to-end latency guarantees.
• Solve fundamental computer science problems involving asynchronous inference, state management at scale, and distributed observability.
• Write performance-critical code using Python and C++ to handle millions of requests per second.

Requirements

• Strong systems fundamentals, including deep knowledge of operating systems, networking, and concurrency.
• Proven experience with distributed systems and managing production-scale systems that serve real users.
• High standards for code quality, simplicity, testing coverage, and generality.
• An ownership mindset with experience managing the full lifecycle from design to deployment and incident response.

Preferred Qualifications

• Experience with distributed systems frameworks such as gRPC or Ray.
• Background in ML/AI systems or specialized serving infrastructure.
• Contributions to major open-source projects.
• Experience with performance optimization, profiling, and cloud-native technologies like Kubernetes or Istio.

About the Company

Anyscale is on a mission to democratize distributed computing. We are commercializing Ray, a popular open-source project that creates an ecosystem of libraries for scalable machine learning. Our technology is used by industry leaders like OpenAI, Uber, and Spotify to accelerate the deployment of AI applications at scale.