Quality Assurance Engineer

Responsibilities

• Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against "Gold Standard" datasets
• Implement automated RAG evaluation metrics such as faithfulness and answer relevance to verify pipeline accuracy
• Design regression suites to monitor "prompt drift" and ensure model updates do not degrade document quality
• Build integration tests to validate data consistency between AI agents and enterprise systems like SAP S/4HANA
• Design performance benchmarks to validate latency and throughput for forecasting models and risk scoring engines
• Automate API testing to verify secure gateways, RBAC, and PII redaction logic
• Map automated test cases to system requirements to provide digital evidence for formal Verification and Validation (V&V) reports
• Prepare "Test Readiness" packages for formal Stage Gate reviews to ensure system stability for production deployment

Requirements

• 5+ years of experience in QA Automation
• 2+ years of experience testing ML models, AI agents, or complex data-driven applications
• Expert proficiency in Python for building custom test harnesses (Pytest) and automation libraries
• Experience with LLM evaluation frameworks (e.g., DeepEval, TruLens, or RAGAS)
• Proficiency in SQL and data validation tools like Great Expectations
• Strong experience with performance testing tools (e.g., Locust, JMeter, or K6)
• Experience integrating automated tests into CI/CD pipelines (e.g., GitLab CI/CD)
• Familiarity with requirements management and test reporting tools (e.g., Jira, TestRail, or Allure)

Preferred Qualifications

• Experience in Defense, Aerospace, or highly regulated industries
• Understanding of IV&V (Integration, Verification, and Validation) processes
• Knowledge of modern Quality Engineering practices such as Shift Left and the Test Pyramid

About the Company

AI71 leads the validation and verification strategies for major AI transformations, focusing on ensuring that Large Language Models and predictive engines meet the rigorous reliability standards required for the defense sector.