Software Engineer, Web Crawling

About the Company

Exa is an applied AI lab building a search engine unlike the world has ever seen. We build massive-scale infra to crawl the entire web, train state-of-the-art embedding models to process it, and design super high performant vector databases to retrieve over it.

Responsibilities

• Build a distributed crawler that can handle 100M+ pages per day
• Optimize crawl politeness and rate limiting across thousands of domains
• Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
• Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

Requirements

• Extensive experience building and scaling web crawlers
• Experience with high performance languages such as C++ or Rust
• Familiarity with TypeScript, Playwright, modern web design, and CDP (Chrome DevTools Protocol)
• Ability to optimize systems to an exceptional degree

Benefits

• Premium healthcare benefits (medical, dental, vision)
• Fertility benefits
• 16 weeks of fully paid parental leave
• Monthly wellness stipend