LLM Ops Engineer

Responsibilities

• Design, implement, and maintain ingestion pipelines for LLM training and RAG datasets
• Develop and optimise chunking, embedding, enrichment, and indexing processes using LangChain
• Build and operate infrastructure for AI components using Azure AI Foundry and Azure OpenAI
• Define and maintain CI/CD pipelines for LLM artefacts using Azure DevOps
• Establish observability for LLM systems including telemetry, latency tracking, and cost visibility using LangSmith
• Implement automated evaluation pipelines for prompts and RAG relevance metrics
• Apply responsible AI principles and ensure privacy and access control for model interactions
• Collaborate with AI developers, architects, and product owners to integrate LLM components into product features

Requirements

• Bachelor’s degree in Computer Science, Software Engineering, Data Engineering, or related field
• 4+ years of experience in software engineering, data engineering, ML engineering, or DevOps
• Hands-on experience with Python and modern AI frameworks like LangChain or Semantic Kernel
• Experience operating cloud-based AI solutions using Azure AI Foundry, Azure OpenAI, or Azure App Services
• Familiarity with vector databases and retrieval pipelines (Azure AI Search, Pinecone, Chroma, etc.)
• Strong understanding of CI/CD, version control, and Azure DevOps
• Experience with container orchestration using Kubernetes (AKS)
• Experience with observability tooling such as Azure Monitor, logging, and tracing

Preferred Qualifications

• Knowledge of modern front-end or service development technologies like React, TypeScript, or C#