Staff Software Engineer - ML Observability

Responsibilities

• Drive the design and implementation of foundational LLM observability features
• Ideate, prototype, and scale new product features to provide insights for generative AI systems
• Work cross-functionally with engineering, product, UX, and applied science teams to find product-market fit
• Develop and extend tools for tracing, evaluating, and debugging Large Language Models
• Influence architecture decisions and mentor engineers to build resilient, high-performance systems
• Stay current with industry trends in machine learning and observability to drive team innovation

Requirements

• BS/MS/PhD in Computer Science, Engineering, or a related scientific field
• Deep understanding of distributed systems and scalable backend architectures
• Hands-on experience building and shipping LLM-powered or GenAI applications
• Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
• Experience with observability tools and platforms
• Ability to thrive in ambiguous, fast-changing environments with a product-oriented mindset

Benefits

• Competitive salary and equity package
• Comprehensive healthcare, dental, parental planning, and mental health benefits
• 401(k) plan and company match
• Paid time off and fitness reimbursements
• Continuous professional development opportunities

About the Company

Datadog is the leading observability and security platform for the AI era, providing businesses with unified visibility across the technology stack to manage complexity at scale. Trusted globally by Fortune 500 companies and high-growth AI leaders, Datadog enables businesses to move faster with clarity and confidence.