Agentic AI Test Engineer

Responsibilities

• Design and implement automated evaluation frameworks using LLM-as-a-judge patterns to assess AI Agent accuracy and safety
• Perform hands-on validation of AI Agents within complex environments using semantic validation techniques
• Automate the testing of real-time data streams and LLM outputs to prevent regression in agent behavior
• Build and maintain high-performance automated tests for modern web interfaces and RESTful APIs
• Develop custom test utilities, framework extensions, and internal quality dashboards using Python and Streamlit
• Integrate automated tests into CI/CD pipelines such as Jenkins or Concourse to ensure continuous quality feedback
• Participate in technical code reviews and perform deep-dive root cause analysis on automation failures

Requirements

• 5+ years of experience as an SDET or Automation Engineer
• Expert-level hands-on coding proficiency in Python
• Proven experience building and maintaining custom automation frameworks
• Solid understanding of Agent Evaluation techniques and using LLMs as a judge
• Extensive experience in both Web and API automation

Preferred Qualifications

• Experience with TypeScript, Java, or React environments
• Familiarity with DevOps tools and CI/CD pipeline management (Jenkins/Concourse)
• Experience building internal tooling or dashboards using Streamlit

About the Company

Comcast is a Fortune 30 global media and technology company that reaches hundreds of millions of customers, viewers, and guests worldwide. We create space to innovate and invest in cutting-edge products, platforms, and solutions that our customers love.