Software Engineer, Distributed Data Systems

About the Company

Exa is an applied AI lab building a search engine unlike the world has ever seen. We build massive-scale infra to crawl the entire web, train state-of-the-art embedding models to process it, and design super high performant vector databases to retrieve over it.

Responsibilities

• Design a lakehouse architecture that handles 100+ PB of web crawl data
• Build streaming pipelines that process billions of documents per day for real-time indexing
• Architect the data layer for our embedding training infrastructure on Ray
• Scale our ClickHouse deployment to handle analytical queries across petabytes of search logs

Requirements

• Deep understanding of lakehouse architectures (Delta Lake, Iceberg, Hudi)
• Experience building and operating large-scale distributed data processing pipelines
• Hands-on experience with streaming data systems (Kafka, Flink, or similar)
• Familiarity with Ray, Spark, or ClickHouse at production scale
• Obsessive focus on reliability and building systems that don't page you at 3am

Preferred Qualifications

• Experience with Lance or other vector-native storage formats
• Background in GPU-accelerated data processing (RAPIDS, cuDF)

Benefits

• Premium healthcare benefits (medical, dental, vision)
• Fertility benefits
• 16 weeks of fully paid parental leave
• Monthly wellness stipend