Member of Technical Staff (Data Scientist/Engineer, Online Metrics)

Responsibilities

• Discover and validate online signals from user interactions that serve as reliable proxies for true answer quality
• Design and implement novel online metrics to be tracked both in A/B testing and on product health dashboards
• Analyze experimental results to validate metrics and ensure they accurately predict user satisfaction
• Build and maintain data pipelines that calculate metrics at scale for Search, Product, and model training teams
• Communicate findings through close collaboration with Product and Search teams

Requirements

• MS in a technical field or equivalent experience
• 4+ years of experience as a Data Scientist, Analytics Engineer, or related role
• Experience with search, recommendation, or LLM-based products
• Expertise in designing online metrics and analyzing A/B experiments
• Strong proficiency in Python and SQL with production-grade coding skills
• Deep knowledge of statistical analysis
• Experience with Business Intelligence (BI) tools for visualization
• Comfort with agentic coding workflows and AI-assisted development tools

Preferred Qualifications

• Proficiency with Apache Spark and Databricks
• Experience with the development or validation of LLM-as-a-judge systems
• Prior work supporting customer-facing products at scale