Senior AI QA Automation Engineer

Responsibilities

• Own test strategy and execution for AI features across unit, API, UI, and end-to-end layers
• Create and maintain automated AI evaluation suites, including golden datasets and regression baselines
• Validate prompt, retrieval, and orchestration changes using repeatable test harnesses
• Build and extend automation frameworks and reusable utilities to improve test stability and speed
• Integrate automated tests and AI evaluations into CI/CD pipelines and triage failures
• Partner with engineers to improve testability, logging, and observability through shift-left practices
• Perform root cause analysis on defects and AI output issues to implement regression coverage
• Provide guidance to junior SDETs on automation, test design, and AI validation patterns

Requirements

• Bachelor's degree in Computer Science, Engineering, or equivalent practical experience
• 4+ years of experience in SDET, QA automation, or quality engineering
• Proficiency in Python, C#, or Java, and strong SQL skills
• Experience with API, UI, and ETL automation
• Experience working with CI/CD pipelines
• Experience testing AI-enabled systems or demonstrated applied AI fluency

Preferred Qualifications

• Experience with LLM evaluation techniques such as rubric-based grading and automated scoring
• Familiarity with AI safety and guardrail testing (grounding, PII handling, injection risks)
• Experience with cloud tooling, containers, observability, and release automation

Benefits

• Medical Insurance (coverage includes spouse and parents up to age 80)
• Provident Fund (PF) and Life Insurance
• Performance-based increments
• 32 paid annual leaves
• Easy loan facility and company-sponsored trips
• Gym Passport membership providing access to 250+ gyms

About the Company

CodeNinja is a global technology and AI firm with over 10 years of industry experience. We help enterprises modernize systems and embed AI into their workflows across sectors including financial services, defense, health, and logistics. We partner with organizations worldwide to deliver AI infrastructure and dedicated engineering teams.