Software Engineer - GenAI inference

Responsibilities

• Design and implement the inference engine and model-serving stack optimized for large-scale LLMs
• Collaborate with researchers to integrate new model architectures like sparsity and mixture-of-experts
• Optimize for latency, throughput, memory efficiency, and hardware utilization across GPUs and accelerators
• Build instrumentation, profiling, and tracing tooling to identify performance bottlenecks
• Develop scalable routing, batching, scheduling, and memory management mechanisms
• Support reliability and fault tolerance in inference pipelines, including A/B launches and model versioning
• Integrate with distributed inference infrastructure to orchestrate across nodes and balance load

Requirements

• BS/MS/PhD in Computer Science or a related field
• 3+ years of software engineering experience in performance-critical systems
• Solid understanding of ML inference internals including attention, MLPs, and quantization
• Hands-on experience with CUDA, GPU programming, and libraries like cuBLAS, cuDNN, or NCCL
• Experience designing distributed systems, RPC frameworks, and memory partitioning
• Ability to solve performance bottlenecks across kernel, memory, networking, and scheduler layers
• Experience building instrumentation and profiling tools for ML models

Preferred Qualifications

• Published research or open-source contributions in ML systems, inference optimization, or model serving

About the Company

Databricks is the data and AI company. More than 10,000 organizations worldwide rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics and AI. Databricks was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake and MLflow.