Senior Software Development Test Engineer - AI

Responsibilities

• Own end-to-end quality for ML models, AI agents, and AI-powered features.
• Design and implement automated testing frameworks for model inference, prompt pipelines, agentic workflows, and RAG/retrieval systems.
• Define and execute evaluation strategies, including creating ground-truth datasets and building LLM-as-judge scoring pipelines.
• Validate model correctness, accuracy, latency, and behavioral consistency across versions and prompt changes.
• Build regression suites to detect quality drift in models, prompts, embeddings, or training data.
• Use AI and LLMs to accelerate quality engineering by generating synthetic test data and automation scaffolding.
• Drive root cause analysis for production model issues and partner with ML Engineers on preventive improvements.

Requirements

• 5–8 years of experience in SDET, quality engineering, or software engineering with a focus on test automation.
• Strong programming skills in Python (preferred) and/or Java.
• Hands-on experience testing data-intensive or ML/AI systems.
• Solid understanding of ML concepts, including model training, inference, and evaluation metrics (precision, recall, F1).
• Familiarity with LLM/agent concepts such as prompting, RAG, embeddings, and vector search.
• Experience with CI/CD, test orchestration, and API/integration testing.
• Strong analytical and debugging skills for performing root cause analysis across model, data, and code layers.

Preferred Qualifications

• Experience with ML/eval tooling such as MLflow, Weights & Biases, LangSmith, Ragas, or DeepEval.
• Experience with cloud platforms (AWS/GCP/Azure) and containerized environments like Docker or Kubernetes.
• Exposure to model observability, drift detection, and production ML monitoring.
• Experience testing agentic systems, tool use, or multi-step LLM workflows.

About the Company

Tekion is disrupting the automotive industry with the first cloud-native automotive platform. Our platform connects the entire automotive retail ecosystem through cutting-edge technology, big data, machine learning, and AI, enabling seamless experiences for OEMs, retailers, and consumers.