LLM Inference Engineer

About the Company

Hippocratic AI is the leading generative AI company in healthcare, building the world’s first healthcare-only, safety-focused LLM to transform patient outcomes at a global scale.

Responsibilities

• Design and implement multi-node serving architectures for distributed LLM inference
• Optimize multi-LoRA serving systems and apply advanced quantization techniques
• Implement speculative decoding and other latency optimization strategies

Requirements

• Experience optimizing LLM inference systems at scale
• Expertise with distributed serving architectures for large language models
• Proficiency in Python, C++, and CUDA programming

Preferred Qualifications

• Contributions to open-source frameworks like vLLM, SGLang, or TensorRT-LLM
• Experience with custom CUDA kernels
• Track record of deploying inference systems in production environments