Strategic Project Lead

About the Company

Contra Labs is a human-centered research lab studying how creatives actually use AI, how tools can serve creative work better, and how to protect the originality that gets flattened as everything starts to look the same. We connect a global network of top creative talent with frontier AI labs and product companies.

Responsibilities

• Design annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs
• Manage project scope, evaluator staffing, quality routing, and client relationships
• Ensure data quality and inter-rater reliability through real-time course correction
• Perform qualitative and quantitative analysis to produce clear findings and reports
• Partner with engineering to build and improve internal tooling and processes
• Develop case studies and benchmark publications to sharpen market positioning

Requirements

• 3+ years of experience designing studies, running evaluations, and analyzing data
• Experience with data labeling, annotation, human-in-the-loop systems, or AI/ML evaluation workflows
• Proven track record managing multiple concurrent projects
• Strong understanding of creative quality and ability to build frameworks to capture it
• Must be based in NYC and able to work in the Williamsburg office 5 days a week

Preferred Qualifications

• Master's or PhD in HCI, cognitive science, psychology, or design research
• Proficiency in statistical methods like inter-rater reliability, sampling, and significance testing
• Experience with RLHF, LLM evaluation, or benchmark design
• Background in ethnography, UX research, or design-study methods
• Startup or 0-to-1 experience