Quality Assurance Engineer

Responsibilities

• Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy.
• Implement automated RAG evaluation metrics (e.g., RAGAS) to verify retrieval-augmented generation pipelines.
• Design regression suites to monitor prompt drift and ensure model stability.
• Build robust tests to validate data consistency between AI agents and enterprise systems like SAP S/4HANA.
• Design performance benchmarking tests for latency and throughput using tools like Locust or K6.
• Automate testing of secure API gateways, verifying RBAC and PII redaction logic.
• Map automated test cases to system requirements to provide digital evidence for formal V&V reports.

Requirements

• 5+ years of experience in QA Automation.
• 2+ years of experience testing ML models, AI agents, or complex data-driven applications.
• Expert proficiency in Python and Pytest.
• Hands-on experience with LLM evaluation frameworks (e.g., DeepEval, TruLens).
• Proficiency in performance engineering with tools such as Locust or K6.
• Strong expertise in SQL and data quality tools (e.g., Great Expectations).
• Experience integrating quality gates into GitLab CI/CD pipelines.

Preferred Qualifications

• Background in Defense, Aerospace, or highly regulated industries.
• Familiarity with Independent Verification and Validation (IV&V) processes.
• Experience with Selenium, Playwright, or Requests for web/API automation.

Benefits

• Market-leading, 100% tax-free salary and benefits package.
• Full relocation support, including flights, housing assistance, and visa sponsorship for you and your family.
• Opportunity to work on pioneering AI technology in a safety-critical environment.

About the Company

AI71 is a leading AI company focused on driving AI transformation across defense and enterprise sectors. We build intelligent systems that meet the highest reliability and security standards for mission-critical environments.