Member of Technical Staff - ML Systems & Inference

About the Company

Gimlet is building the next generation of AI infrastructure: large-scale AI datacenters and the orchestration platform that coordinates them. We work with foundation labs, hyperscalers, and AI-native companies to power production workloads at massive scale.

Responsibilities

• Build and optimize inference systems that improve latency, throughput, and efficiency for production AI workloads
• Design execution strategies that intelligently balance batching, scheduling, concurrency, and resource utilization
• Improve KV cache management, memory efficiency, and execution behavior across large-scale serving environments
• Enable new model architectures and inference techniques to run efficiently in production
• Partner with compiler, kernel, networking, and distributed systems engineers to drive end-to-end performance improvements

Requirements

• Strong software engineering fundamentals
• Experience building or operating ML inference or model serving systems
• Comfort reasoning about performance, memory usage, and system behavior under load

Preferred Qualifications

• Experience with inference runtimes such as TensorRT-LLM, vLLM, or custom serving systems
• Deep understanding of modern model architectures and attention mechanisms
• Experience with batching, scheduling, and concurrency control in inference systems
• Familiarity with KV cache management and memory placement strategies
• Experience profiling and tuning latency- and throughput-critical systems
• Software development experience in Python and C++