Machine Learning Scientist - Open Source Lead

About the Company

Arena is the platform for evaluating how AI models perform in the real world. Founded by researchers from UC Berkeley's SkyLab, we're on a mission to measure and advance the frontier of AI for real-world use, and to build the foundation for everyone to understand, shape, and benefit from it.

Responsibilities

• Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions
• Develop new metrics, methodologies, and evaluation protocols that go beyond traditional benchmarks
• Analyze large-scale human voting and interaction data to uncover insights into model performance and user preferences
• Communicate results with the broader research community via academic papers, educational content, and conference talks
• Collaborate with engineers to implement and scale research findings into production systems
• Prototype and test research ideas rapidly, balancing rigor with iteration speed
• Partner with model providers to shape evaluation questions and support responsible model testing
• Contribute to the scientific integrity and transparency of the LMArena leaderboard and tools

Requirements

• PhD or equivalent research experience in Machine Learning, Natural Language Processing, Statistics, or a related field
• Hands-on experience training large-scale models (reward models, preference models, fine-tuning LLMs)
• Proficiency in Python and ML research libraries such as PyTorch, JAX, or TensorFlow
• Strong understanding of LLMs and modern deep learning architectures (Transformers, RLHF, DPO)
• Demonstrated ability to design and analyze experiments with statistical rigor
• Experience publishing research or working on open-source projects in ML, NLP, or AI evaluation
• Ability to translate research questions into practical systems

Preferred Qualifications

• Skilled at public speaking, writing, and presenting research work to diverse audiences
• Active participation in conferences, panels, and online forums
• Experience building trust through transparent communication and community engagement

Benefits

• Competitive compensation and equity
• Comprehensive health and wellness benefits, including medical, dental, and vision