Senior AI Engineer

Responsibilities

• Shape AI experiences for APM by designing and shipping LLM/agentic workflows that analyze telemetry to generate diagnoses and guided fixes.
• Own the full development loop, from rapid prototyping and defining success metrics to productionizing for scale and reliability.
• Build robust agent systems, including tools, retrieval/planning strategies, guardrails, and human-in-the-loop paths.
• Integrate AI capabilities with Datadog’s platform, leveraging Trace Explorer, Service Catalog, and other core surfaces.
• Collaborate deeply with Product Management and Design to build cohesive, user-centric experiences.
• Write performant, maintainable backend code and manage services in high-throughput, low-latency production environments.

Requirements

• 4+ years of experience building backend or real-time ML systems.
• Proven track record of delivering LLM or agent-based features to production environments.
• Solid grasp of the ML lifecycle (task definition, modeling, evaluation, deployment) and statistics.
• Proficiency in Go, Java, or Python.
• Experience with microservices, distributed systems, and observability patterns.
• Ability to iterate from prototype to GA while measuring impact with clear product metrics.

Preferred Qualifications

• Hands-on experience with distributed tracing stacks (OpenTelemetry/Datadog APM), profilers, or logs/metrics pipelines.
• Exposure to agent frameworks, tool-use orchestration, RAG, and retrieval/indexing for observability data.
• Familiarity with SLO/SLA practices and incident response.

Benefits

• Competitive salary and equity packages.
• Comprehensive healthcare, dental, parental planning, and mental health benefits.
• 401(k) plan with company match.
• Paid time off and fitness reimbursements.
• Continuous professional development and opportunities for high product influence.

About the Company

Datadog is the leading observability and security platform for the AI era, providing businesses with unified visibility across the technology stack to manage complexity at scale. Trusted globally by Fortune 500 companies, Datadog enables businesses to move faster with clarity and confidence.