Software Engineer, Agent Evaluation and Quality

About the Company

Our mission is to automate coding. The first step in our journey is to build the best tool for professional programmers, using a combination of inventive research, design, and engineering. Our organization is very flat, and our team is small and talent dense.

Responsibilities

• Design and build best-in-class AI evaluation systems including curated datasets, offline replay, scorers, and dashboards
• Design feedback loops from real usage by collecting and interpreting user signals
• Develop analysis tooling and workflows for debugging agent behavior and surfacing actionable insights
• Improve reliability and guardrails by defining quality metrics, alerting, and triage primitives
• Partner with research, product, and infrastructure teams to turn insights into improvements

Requirements

• Experience building and operating evaluation or measurement systems (AI evals, experimentation, ranking, or search quality)
• Ability to turn ambiguous quality questions into concrete metrics and pipelines
• Strong data acumen and ability to collaborate with data scientists and researchers
• Strong software engineering fundamentals and experience shipping production systems
• Deep interest in emerging research and industry trends regarding model and agent behaviors