Director of Evaluations

Responsibilities

• Define the evaluation strategy and roadmap, prioritizing measurement needs in coordination with research and product teams
• Build the Evaluations team (8-10 specialists) within the first 3 to 6 months
• Lead the team independently from core research to avoid conflicts of interest
• Oversee the design and creation of new datasets, tasks, and interactive environments to measure model performance, safety, and robustness
• Direct the evaluation of Scientist AI as a guardrail for frontier models, focusing on harm detection and resistance to adversarial attacks
• Establish and lead automated and manual red-teaming programs
• Drive the creation of internal tools and infrastructure for large-scale automated evaluations
• Manage public communication of evaluation results, including model cards and technical reports
• Represent LawZero externally to AI safety institutes, research collaborators, and funders

Requirements

• Graduate degree (Master, PhD, or equivalent) in Machine Learning, Computer Science, or a related field
• Over 10 years of experience in Machine Learning
• At least 5 years in a leadership role building or advancing technical ML teams
• Practical expertise in large-scale evaluation of LLMs or frontier ML systems
• Proven experience creating evaluation datasets, benchmarks, or interactive environments
• Excellent written and verbal communication skills for both technical and non-technical audiences
• Ability to thrive in a fast-paced, research-oriented, and ambiguous environment

Preferred Qualifications

• Experience leading automated or manual red-teaming exercises
• Experience collaborating with third-party partners for benchmark and dataset creation
• Experience publishing open-source datasets, benchmarks, or evaluation tools
• Familiarity with AI safety policies and standards (e.g., NIST, EU AI Act)
• Experience coordinating with external safety institutes or government agencies

Benefits

• Opportunity to contribute to a high-impact mission
• Comprehensive health benefits
• Minimum of 20 days of vacation from start
• Minimum 4% employer retirement savings contribution
• Generous and flexible benefits for well-being
• Collaborative and inclusive work environment in Montreal

About the Company

LawZero is a non-profit organization dedicated to advancing research and developing technical solutions for safe-by-design AI systems. Based in Montreal, LawZero develops "Scientist AI," a cutting-edge system designed to be highly performant and secure, utilizing research methods proposed by Professor Yoshua Bengio.