Member of Technical Staff, Evals

About the Company

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone.

Responsibilities

• Build and maintain the internal evals platform used across Magic
• Design, implement, and validate eval tasks for pre-training, post-training, reinforcement learning, inference, and product systems
• Develop infrastructure for running large-scale evaluations
• Build systems to measure dataset quality and identify opportunities to improve training data
• Improve evaluation correctness, reproducibility, and reliability
• Audit and improve upon public benchmarks, evaluation methodologies, and open-source implementations
• Partner with research, data, inference, and product teams to define metrics that accurately reflect model quality
• Build tooling and frameworks that enable teams across Magic to make decisions based on trustworthy measurements

Requirements

• Experience building production systems, internal platforms, or developer infrastructure
• Experience working with machine learning systems, evaluation frameworks, data infrastructure, or research tooling
• Track record of owning technical projects end-to-end
• Skepticism toward results that cannot be reproduced, validated, or explained
• Ability to reason critically about benchmarks, metrics, and experimental methodology
• Experience designing, implementing, or operating systems that run at scale
• Comfortable navigating ambiguity and determining whether a measurement is actually capturing the behavior it claims to measure

Benefits

• Annual salary range between $200K - $550K depending on experience
• Equity is a significant part of total compensation
• 401(k) plan with 6% salary matching
• Generous health, dental, and vision insurance
• Unlimited paid time off
• Visa sponsorship and relocation support