Research Engineer

About the Company

Mercor's mission is to organize human intelligence to power the AI economy. We partner with leading AI labs and enterprises to provide the human intelligence essential to AI development. Mercor is a profitable Series C company valued at $10 billion.

Responsibilities

• Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning
• Build and operate end-to-end LLM evaluation systems including runs, scoring, dashboards, and reporting
• Run systematic failure analysis on model outputs to categorize failure modes and quantify prevalence
• Create and refine rubrics, automated evaluators, and scoring frameworks
• Quantify data usability and impact on key benchmarks to guide data generation and curation
• Collaborate with AI researchers and data producers to align evals with training objectives

Requirements

• Strong applied research background in model evaluation, benchmarking, or failure analysis
• Strong coding skills with hands-on experience in ML models and evaluation code
• Solid grasp of data structures, algorithms, and backend systems
• Proficiency with APIs, SQL/NoSQL, and cloud platforms
• Ability to reason about model behavior and experimental results
• Ability to work in person in San Francisco five days a week

Preferred Qualifications

• Industry experience on a post-training or evaluation/benchmarking team
• Publications at top-tier venues such as NeurIPS, ICML, or ACL
• Experience building LLM evaluation pipelines or failure-analysis tooling
• Experience with synthetic data generation or RL-style workflows
• Relevant work samples or code demonstrating evaluation frameworks