Assessment Designer & Learning Analyst

About the Company

Mercor's mission is to organize human intelligence to power the AI economy. We partner with leading AI labs and enterprises to provide the human intelligence essential to AI development. Mercor is a profitable Series C company valued at $10 billion.

Responsibilities

• Design and continuously improve assessments and certification frameworks that validly and reliably measure expert readiness
• Build assessments and measurements of skills that are predictive of on-the-job performance
• Develop item banks, scoring guides, and inter-rater reliability protocols
• Run validity studies to ensure assessments measure intended constructs
• Analyze the relationship between instructional materials, assessments, and expert performance
• Analyze assessment data at the item level including difficulty, discrimination, and reliability
• Investigate the relationship between assessment performance and real-world expert effectiveness
• Build reports and dashboards that surface actionable insights to program and operations teams
• Design and analyze quasi-experimental, quantitative, and qualitative studies
• Track certification and assessment outcomes over time and flag when programs need revision
• Partner with learning designers to translate findings into program improvements

Requirements

• Master's degree in Learning Sciences, Educational Psychology, Educational Measurement, Psychometrics, or a closely related field
• Coursework in quantitative research methods, psychometrics, and educational statistics
• Familiarity with classical test theory (CTT) and ideally item response theory (IRT)
• Proficiency in item-level analysis including difficulty index, discrimination index, and inter-rater reliability
• Ability to perform correlation, regression, and basic predictive modeling
• Fluency in Excel or Google Sheets for data cleaning and summaries
• Basic proficiency in Python, STATA, or R
• 1–2 years of experience in assessment design, educational research, or learning analytics
• Strong writing skills to communicate complex findings to non-technical audiences

Preferred Qualifications

• Experience with item response theory (IRT) or latent variable modeling
• Familiarity with data annotation, labeling, or AI evaluation workflows
• Experience in tech, AI/ML, or data operations environments
• Background in competency-based or mastery learning frameworks