Research, Evals Engineer

About the Company

Exa is an applied AI lab building a search engine unlike the world has ever seen. We build massive-scale infra to crawl the entire web, train state-of-the-art embedding models to process it, and design super high performant vector databases to retrieve over it. We power search for Cursor, Cognition, HubSpot, and over 400,000 developers.

Responsibilities

• Design and implement evaluation frameworks that probe the limits of search
• Build scalable, reliable eval pipelines that track regressions, drift, and quality signals across billions of documents
• Create golden datasets, synthetic benchmarks, agentic tasks, and real-world test suites
• Partner closely with ML researchers, data engineers, infra engineers, and product to shape feedback loops
• Investigate how to evaluate search engines in an LLM world

Requirements

• Hands-on ML experience in training, finetuning, or evaluating models
• Experience with embeddings or LLMs is a bonus
• Strong engineering fundamentals in Python or Rust
• Experience with distributed pipelines, GPU/cluster jobs, or reliable system building
• Ability to design creative measurement strategies and inspect edge cases