Senior Member of Technical Staff, AI Quality

About the Company

Harper is an AI-native commercial insurance company in San Francisco. We're rebuilding the entire business as software, turning expert human judgment into compute to transform a trillion-dollar industry.

Responsibilities

• Build capability and regression evaluation suites for assigned agents (intake, submissions, placements, renewals, CRM, or voice)
• Curate golden datasets from real failure modes, including transcripts and call recordings
• Design graders using deterministic methods, LLM-as-judge, and human calibration
• Ship pre-merge evaluation gates to run in CI for every PR touching an agent, prompt, or tool
• Implement production trajectory monitoring to catch drift within hours
• Convert operational findings into permanent regression tests

Requirements

• 3–6 years of software engineering experience
• Hands-on production LLM/agent evaluation experience (capability/regression suites, LLM-as-judge, golden datasets)
• Experience designing LLM-as-judge rubrics that survive human calibration
• Familiarity with at least one major evaluation framework
• Strong written communication skills for rubric documentation and failure-mode taxonomies
• Ability to work on-site in San Francisco with high standards and a fast pace

Preferred Qualifications

• Open-source evaluation framework contributions
• Red-teaming or adversarial testing experience
• Voice evaluation experience (latency, interruption, transcription accuracy)
• Background in ML evaluation or observability