Staff Back End Engineer, Evals

About the Company

Hazel is building the AI engine for wealth management that unlocks 10x growth, efficiency and value for financial advisors and their clients in a regulated industry. Hazel is a part of Altruist’s broader mission to make financial advice better, more affordable, and accessible to all.

Responsibilities

• Design and build Hazel's evals platform end-to-end, including online scoring, offline benchmarks, and LLM-as-judge pipelines
• Build production observability and monitoring for AI quality, such as hallucination rates and factual accuracy
• Architect data curation pipelines to turn advisor interactions into evaluation datasets
• Build and steward golden datasets in partnership with subject matter experts
• Develop LLM verification agents to catch hallucinations and compliance violations
• Integrate evals into the deployment pipeline as a first-class deployment gate
• Partner with the orchestration team to evaluate cross-model performance across Anthropic, OpenAI, and self-hosted models
• Define quality SLOs and build alerting for high-stakes financial planning flows

Requirements

• 8+ years of engineering experience
• 2+ years focused on evaluation infrastructure, model quality, fine-tuning, or ML platform work
• Deep familiarity with RAG evaluation, document processing, and LLM-as-judge frameworks
• Experience designing and curating golden datasets and sampling strategies
• Proficiency in data engineering (SQL, dbt, warehouses) and backend integration (APIs, async pipelines)
• Strong communication skills to translate domain requirements into measurable eval criteria

Preferred Qualifications

• Prior experience at an applied AI company building research infrastructure
• Experience evaluating multi-step agentic workflows or RAG pipelines in production
• Familiarity with frameworks like Braintrust or Langfuse
• Background in regulated industries like financial services, healthcare, or legal
• Experience building human-in-the-loop labeling workflows or red-teaming programs
• Domain knowledge of wealth management or tax planning