Research Engineer, Evals

Responsibilities

• Build proprietary benchmarks and datasets to evaluate models on fraud, identity, and risk workflows
• Design and run offline and online evals measuring model performance on real customer tasks
• Define quality metrics for judgment systems, including precision, calibration, and consistency
• Analyze model and agent failure modes to improve datasets and training loops
• Build reusable evaluation tools and quality building blocks for various product surfaces
• Partner with research, engineering, and product teams to drive system quality through experimentation

Requirements

• Strong engineering fundamentals and programming skills
• Experience with research-heavy codebases
• Ability to design clean experiments and draw conclusions from noisy results
• Interest in fraud, risk, trust and safety, or regulated domains
• Experience in building benchmarks, datasets, or evaluation pipelines

Preferred Qualifications

• Experience training, evaluating, or improving modern ML systems
• Familiarity with LLMs, agent systems, retrieval, or post-training
• Strong engineering judgment and a bias toward building

Benefits

• Competitive salary and meaningful equity
• Platinum-level medical, dental, and vision insurance
• Unlimited PTO, sick leave, and parental leave
• Up to $100 per month for personal health and wellness expenses
• 401(k) plan

About the Company

Variance is building AI agents for high-stakes risk investigations, fraud, and identity reviews. We are a small, talent-dense team in San Francisco focused on making AI systems reliable in messy, adversarial, real-world environments.