Senior Quality Engineer 2 - AI - SOC Automation

Responsibilities

• Create and extend software frameworks and tools to run, monitor, and report on automated tests
• Build and maintain automation frameworks and CI/CD pipelines
• Architect, design, and build software to automate testing methods including API validation, performance/load testing, and fuzzing
• Develop and maintain test frameworks for GenAI and AI/ML applications, focusing on model validation, LLM prompt-response consistency, and hallucination detection
• Implement evaluation pipelines to test LLM accuracy, latency, bias, toxicity, and reliability
• Review testing strategies, approaches, and test plans to provide actionable insights to development teams

Requirements

• Experience building automation frameworks and CI/CD pipelines
• Proven ability to architect software for API validation and performance testing
• Experience developing test frameworks specifically for GenAI and AI/ML applications
• Expertise in implementing evaluation pipelines for LLM accuracy and reliability
• Experience deploying services on cloud computing platforms such as AWS, Azure, or GCP
• Expertise in using common tools for functional, load, and fuzz testing (e.g., Locust, CATS)

Preferred Qualifications

• Experience working with distributed systems and microservices architecture
• Proficiency with our technology stack: Pytest, Robot, Python, Playwright, Docker, Kubernetes, and LangChain

About the Company

Arctic Wolf is a leader in security operations, providing an AI-driven Security Operations Cloud that processes hundreds of billions of security signals daily. We combine advanced AI and automation with expert human analysis to help organizations detect and respond to threats faster and more effectively.