Data Domain Architect Senior Associate - Agentic AI Evaluation & Annotation

Responsibilities

• Define and operationalize evaluation metrics for agentic AI workflows, including task success, tool-use quality, and policy adherence.
• Design annotation schemas, rubrics, and taxonomies for evaluating agent trajectories across multi-turn and multi-tool scenarios.
• Build and maintain agent-specific gold sets, challenge sets, and regression suites to assess planning and grounding.
• Train, calibrate, and support annotators to ensure consistent application of labeling guidelines and quality expectations.
• Identify and report agent failure patterns such as incorrect planning, invalid arguments, or unsafe recommendations.
• Partner with ML engineers and product managers to align evaluation criteria with agent architecture and business requirements.
• Contribute to LLM-as-judge and automated evaluation workflows, including evaluator prompt design and rubric-based scoring.

Requirements

• Master’s degree in Computer Science, Data Science, AI/ML, or a related field.
• 3+ years of experience in AI evaluation, annotation programs, LLM applications, or agentic AI systems.
• Hands-on experience evaluating LLM-based systems, specifically regarding multi-step reasoning, planning, and tool use.
• Proven experience designing annotation schemas, evaluation rubrics, and labeling guidelines.
• Working knowledge of agentic AI concepts such as orchestration, tool invocation, and context management.
• Hands-on Python experience for data analysis, validation, and automation.
• Strong analytical and communication skills to translate complex agent behavior into measurable decisions.

Preferred Qualifications

• Experience evaluating autonomous or tool-using agents in production environments.
• Experience building agent evaluation benchmarks, trajectory datasets, or regression suites.
• Experience with agent observability, trace review, and evaluation pipelines.
• Knowledge of emerging agentic AI evaluation methods and best practices.

About the Company

JPMorganChase is a leading global financial services firm with a history spanning over 200 years. We provide innovative financial solutions to millions of consumers, small businesses, and prominent corporate and institutional clients. Our Consumer & Community Banking division leverages cutting-edge data and analytics to build competitive advantages and provide industry-leading digital solutions.