Director of Evaluations

Responsibilities

• Define LawZero’s evaluations strategy and roadmap in coordination with research and product teams
• Build and scale the Evaluations Team to approximately 8–10 people within the first 3–6 months
• Operate the team independently to ensure unbiased evaluation of the Scientist AI and frontier LLMs
• Oversee the design of new datasets, tasks, and environments to measure capabilities, safety, and adversarial robustness
• Lead evaluation of the Scientist AI when deployed as a guardrail around frontier models
• Establish and lead automated and manual red-teaming programs
• Lead the construction of internal tooling and infrastructure for large-scale automated evaluations
• Own public communication of evaluation results, including technical reports and peer-reviewed publications
• Represent LawZero externally with AI safety institutes, research collaborators, and funders

Requirements

• Advanced degree (MSc or higher) in machine learning, computer science, or a related field
• 10+ years of experience in machine learning
• 5+ years in a leadership role building or scaling technical teams
• Hands-on expertise in large-scale evaluations of LLMs or frontier ML systems
• Proven track record of building evaluation datasets and benchmarks from scratch
• Strong written and verbal communication skills for technical and non-technical audiences
• Ability to operate in a fast-moving, research-driven environment with high ambiguity

Preferred Qualifications

• Experience leading automated or manual red-teaming exercises
• Experience working with third-party partners for benchmark and dataset creation
• Experience releasing open-source datasets, benchmarks, or evaluation tooling
• Familiarity with AI safety policy and standards (UK AISI, CAISI, NIST, EU AI Act)
• Experience coordinating with external safety institutes or government bodies

Benefits

• Opportunity to contribute to a unique mission with major impact
• Comprehensive health benefits
• Minimum of 20 days vacation per year
• Minimum 4% employer retirement savings contribution
• Generous flexible benefits
• Collaborative and inclusive work environment in Montreal

About the Company

LawZero is a non-profit organization committed to advancing research and creating technical solutions that enable safe-by-design AI systems. Based in Montreal, its scientific direction is inspired by the research of Professor Yoshua Bengio, aiming to build AI that prioritizes truthful, transparent, and probabilistic reasoning.