AI Systems QA Engineer

Responsibilities

• Design, develop, and execute evaluation frameworks (Evals) for Large Language Models (LLMs) and AI systems
• Perform end-to-end system testing, regression testing, and performance testing for AI-driven applications
• Validate model outputs for accuracy, consistency, safety, hallucination detection, and edge cases
• Build automated test pipelines and quality benchmarks for AI systems
• Collaborate with AI/ML engineers, product teams, and platform engineers to improve system reliability
• Analyze failures, identify root causes, and provide actionable feedback to improve model behavior
• Develop datasets, prompts, and testing scenarios to measure model performance
• Monitor production performance and continuously improve evaluation metrics

Requirements

• Bachelor’s degree in Computer Science, Engineering, or related field
• 5–8 years of experience in QA/testing, preferably in AI/ML or data-driven systems
• Strong experience in AI/LLM evaluation frameworks and system testing
• Hands-on experience with automated testing methodologies and QA processes
• Familiarity with prompt engineering, AI benchmarking, and model validation techniques
• Experience working with Python and testing frameworks
• Understanding of LLM behaviors, hallucinations, prompt injection risks, and AI safety concepts
• Experience with API testing tools like Postman and automation frameworks
• Understanding of NLP concepts such as tokenization, embeddings, and text generation
• Familiarity with cloud platforms (AWS, GCP, or Azure)

Preferred Qualifications

• Exposure to tools such as OpenAI Evals, LangSmith, DeepEval, or Promptfoo
• Experience testing AI/ML models or data pipelines
• Knowledge of CI/CD pipelines and automation workflows
• Experience with Generative AI or conversational AI products
• Startup experience or experience in the construction industry

About the Company

MeltPlan is building the “planning engine” for the $14 Tn construction industry, an AI system designed specifically to optimize decisions before construction begins. Founded by experienced operators, MeltPlan aims to reduce rework and change orders by surfacing constraints and tradeoffs early in the planning phase.