Senior QA Engineer (AI Solutions)

Responsibilities

• Own the end-to-end QA strategy for Elixirr’s agent platform and AI-enabled accelerators, covering functional, non-functional, and safety testing.
• Build and maintain automated test suites across UI, API, contract, and integration layers within CI/CD pipelines.
• Design LLM and agent evaluation harnesses using automated checks, LLM-as-judge scoring, and rubric-based evaluation.
• Use AI tooling and prompt engineering to accelerate test generation, synthetic data creation, and automated triage.
• Conduct performance, load, and chaos testing against agent workflows and backend services.
• Partner with engineering teams to implement "shift-left" testing during design and "shift-right" observability in production.

Requirements

• 5+ years of experience in QA, SDET, or Test Engineering with hands-on ownership of automation frameworks.
• Proficiency in at least one automation stack such as Python (pytest), TypeScript (Playwright), or Java (JUnit/RestAssured).
• Experience testing cloud-native applications and microservices in AWS and/or Azure.
• Strong understanding of API testing, contract testing, and CI/CD pipelines (e.g., GitHub Actions, GitLab CI).
• Understanding of LLM evaluation concepts, including hallucination, prompt injection, and model drift.
• Solid understanding of observability tooling including logs, metrics, and traces.

Preferred Qualifications

• Experience with performance testing tools like k6, Locust, or JMeter.
• Familiarity with AI evaluation frameworks such as Ragas, DeepEval, Promptfoo, or Langfuse.
• Exposure to security testing (SAST/DAST), accessibility testing, and compliance-driven environments.

Benefits

• Competitive compensation including performance bonuses and employee stock options.
• Employee Share Purchase Plan (ESPP).
• Health benefits plan and pension plan.
• Flexible working hours.
• Access to modern equipment and cutting-edge technologies.
• Internal learning management systems and knowledge hubs for professional growth.

About the Company

Elixirr Digital is a dynamic global consulting firm recognized for delivering transformative solutions. As part of our growing AI practice, we develop enterprise-grade products, accelerators, and agentic solutions that help leading organizations harness advanced AI technologies like RAG and multi-agent orchestration.