Software Developer (Agentic Evaluation)

Responsibilities

• Develop and orchestrate multi-agent AI systems for automated test generation and development workflow optimization using frameworks like LangGraph, AutoGen, or Anthropic Agent SDK.
• Design agentic workflows to autonomously drive test automation across UI, API, integration, and system levels.
• Build evaluation frameworks and custom benchmarks for agentic systems, including comparisons against commercial solvers using tools like AgentBench and Langfuse.
• Evaluate MCP server and tool performance across agentic pipelines, measuring latency, accuracy, and task completion rates.

Requirements

• BS/MS in Computer Science, Machine Learning, or a related applied AI field.
• Expertise in Python and ML frameworks such as PyTorch, Transformers, or scikit-learn.
• Experience with Large Language Models applied to software understanding or test generation.
• Knowledge of AI evaluation methodologies and metrics for agentic task completion.
• Strong foundation in statistical analysis and experimental design.
• Experience with developer workflow and productivity measurement frameworks.

Preferred Qualifications

• Hands-on experience with MCP (Model Context Protocol) and optimizing MCP server integrations.
• Experience with agentic AI frameworks including LangGraph, AutoGen, or Claude Code.
• Background in software engineering or QA with familiarity with automation frameworks like Playwright, Selenium, or Pytest.
• Knowledge of vision-language models or multi-modal AI for UI understanding.
• Experience with Azure AI Foundry/ML or AWS cloud ML platforms.

About the Company

Autodesk helps innovators turn their ideas into reality, transforming not only how things are made, but what can be made. From the greenest buildings to the smartest factories, our software empowers creators to shape the world.