Staff Software Development Test Engineer - AI Evaluation

Responsibilities

• Design, enhance, and own Tekion’s AI evaluation infrastructure as a shared capability used across ML teams.
• Create, curate, and maintain evaluation datasets and golden/ground-truth sets across various use cases.
• Define quality metrics for AI outputs, including accuracy, relevance, faithfulness, consistency, and safety.
• Build automated scoring pipelines using LLM-as-judge, rubric-based, and reference-based evaluation methods.
• Identify hallucinations, unsafe or biased outputs, and edge cases through targeted evaluation suites.
• Implement both offline benchmarking and online production quality monitoring, including drift detection.
• Establish evaluation gates within CI/CD pipelines to ensure quality before model or prompt releases.
• Develop dashboards to make AI quality signals visible and actionable for ML and product teams.

Requirements

• 5–8 years of experience in SDET, quality engineering, ML engineering, or data science.
• Strong programming skills in Python with the ability to build robust, reusable evaluation pipelines.
• Deep understanding of ML/LLM evaluation, benchmark design, and non-deterministic system testing.
• Hands-on experience with LLM-as-judge, rubric-based scoring, or human-in-the-loop evaluation.
• Solid grasp of LLM/agent concepts including prompting, RAG, embeddings, and tool use.
• Understanding of generative failure modes such as hallucination, drift, and prompt sensitivity.
• Strong statistical intuition for interpreting evaluation results and significance.

Preferred Qualifications

• Experience with evaluation frameworks such as Ragas, DeepEval, LangSmith, TruLens, or Promptfoo.
• Experience building online evaluation, guardrails, or production model monitoring systems.
• Familiarity with responsible AI, safety evaluation, and red-teaming.
• Experience with experiment tracking tools like MLflow or Weights & Biases.
• Experience standing up evaluation capabilities as a platform service for multiple teams.

About the Company

Tekion is positively disrupting the automotive industry with the first cloud-native automotive platform. Our ecosystem connects OEMs, retailers, and consumers through a seamless platform powered by cutting-edge technology, big data, and AI. With nearly 3,000 employees across North America, Asia, and Europe, Tekion is enabling the best automotive retail experiences ever through continuous innovation.