Member of Technical Staff (Backend Software Engineer, Search Crawler)

Responsibilities

• Take ownership of and lead projects focused on developing large-scale web crawlers, ingestion pipelines, and data processing systems
• Build, maintain, and optimize core backend and frontend components for crawler services, including storage, retrieval, and UI dashboards
• Collaborate closely with Search and Infrastructure teams to ensure reliable ingestion of billions of web pages
• Architect and implement fullstack features and scalable distributed systems for high-load and real-time data operations
• Rapidly iterate and experiment with novel approaches to enhance system performance and reliability

Requirements

• Minimum of 5 years of software development experience
• Strong knowledge of data structures and algorithms
• Proficiency in Python, C++, Rust, or Go
• Experience building, deploying, and optimizing high-load, distributed services
• Deep understanding of cloud infrastructure with hands-on experience in Kubernetes (K8s) and AWS

Preferred Qualifications

• Experience with large-scale web crawlers